Building Resiliency for Data with Backups

How do we get resiliency for data?

In our introduction to resiliency, we discussed the importance of resiliency in the face of failure. We distinguished between providing resiliency for your processes running on your servers and your critical data itself.

In this guide, we will walk you through a hands-on implementation for resiliency of your data on Equinix Metal via backups. A different guide addresses redundancy.

In order to keep things simple, we will not spend time with advanced enterprise backup systems, or any system that coordinates across multiple servers. Instead, we will focus on a single server using basic freely available tools.

We will provide backups to another server running on Equinix Metal. Rather than simple copy, we will use the S3 protocol. This is pretty much a standard for object storage. The same protocol will allow you to back up to another server, or even a cloud service, such as S3 in Amazon Web Services.

Software and Tools

For this getting started guide, we will show replication at the operating system level, using the popular open source database MariaDB.

For the backup software itself, we will use the very powerful yet basic rsync protocol. The rsync site contains great information on rsync itself and how to use it for backups. Perhaps most importantly, it includes a library, which is included in lots of other software, making it easier to use across protocols. To keep things simple, we will use the open-source and easy-to-use duplicity.

In addition to wrapping the rsync protocol, which is highly efficient, duplicity also provides management of full and incremental backups, can verify the state of backups, and encrypts the backups using GnuPG.

To receive the backups on our Equinix Metal server, we will run minio, which is an open source S3-compatible object storage server.

Prerequisites

You need the following for this guide:

• An Equinix Metal account

Overview

Here are the steps we need to take to provide resiliency for our data:

1. Deploy a server on Equinix Metal
2. Install and configure database software on the server
3. Deploy our backup server on Equinix Metal
4. Install and configure our backup tool on the backup server
5. Install and configure our backup tool on the database server
6. Test our backups

1. Deploy a Server on Equinix Metal

Let's start by setting up our primary server. You can do this via the Equinix Metal Console, the CLI, Terraform, or any tool that interacts with the Equinix Metal API.

We will use the Equinix Metal Console for this example. For a basic getting started guide to deploying your first server, see Deploy Your First Server.

We are deploying to Dallas, using our c3.small.x86, but of course, you can deploy whichever server types to whichever metro suits your purposes.

Next, we select our operating system, Ubuntu 22.04, which is widely accepted in the cloud and easy to use. Pick whichever operating system you prefer. We are deploying just one server, our database server.

And we click "Deploy Now".

Now we wait for the server to be ready. This can take a few minutes.

2. Install and configure database software on the server

Let's navigate back to our main manage servers page. We can see that our server was given a public IP by Equinix Metal. We will use that IP to ssh into the server and set it up.

In the above example, the IP is:

• 147.75.47.207

In your project, the IP address is likely to be different, so be sure to change it as you follow the examples.

On our server, we will do the following.

1. ssh to the server
2. update the package manager
3. install the software we need, specifically mariadb, as well as some other utilities

Let's ssh to the database server.

$ ssh root@147.75.47.207
The authenticity of host '147.75.47.207 (147.75.47.207)' can't be established.
ED25519 key fingerprint is SHA256:OydezYaS2j/5/t4fd06g+34lZjFan8X/GRRVqwdMXAA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '147.75.47.207' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-58-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sun Apr  9 17:49:02 UTC 2023

  System load:            0.00439453125
  Usage of /:             0.7% of 438.04GB
  Memory usage:           1%
  Swap usage:             0%
  Temperature:            50.0 C
  Processes:              241
  Users logged in:        0
  IPv4 address for bond0: 147.75.47.207
  IPv6 address for bond0: 2604:1380:4642:4900::1

 * Introducing Expanded Security Maintenance for Applications.
   Receive updates to over 25,000 software packages with your
   Ubuntu Pro subscription. Free for personal use.

     https://ubuntu.com/pro

6 updates can be applied immediately.
6 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@database:~#

We now are logged into the server. Notice that the first time we log in, it asks us if we want to trust the ssh key of the server, as our local system has not seen it before. We can answer yes to this question for now.

With each login, it also gives us information about the server and operating system.

We end up with our command-prompt, giving us the username with which we are logged in (root), the name of the server (database), and the # prompt, indicating that we are the superuser, or root.

We can now update the package manager and install the software we need.

root@database:~# apt update -y
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [728 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [990 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [147 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [9020 B]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [701 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [109 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [576 B]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [716 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [210 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [13.9 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [744 kB]
Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [118 kB]
Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [14.2 kB]
Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [19.4 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [115 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [576 B]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [899 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [180 kB]
Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [4068 B]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [18.6 kB]
Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [24.1 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [6312 B]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B]
Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [40.6 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [9800 B]
Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [20.3 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [14.4 kB]
Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [480 B]
Fetched 6191 kB in 2s (2744 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
83 packages can be upgraded. Run 'apt list --upgradable' to see them.

That went well enough. We are not going to bother upgrading packages at this point, as we are not interested in general operating system updates, just getting our example to work.

Next, let's install the software we need. The output includes a lot of extraneous data, so out output in this guide is somewhat truncated, to keep it readable.

root@database:~# apt install -y mariadb-server mariadb-client
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  galera-4 libcgi-fast-perl libcgi-pm-perl libclone-perl libconfig-inifiles-perl libdaxctl1 libdbd-mysql-perl libdbi-perl
  libencode-locale-perl libfcgi-bin libfcgi-perl libfcgi0ldbl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
  libhttp-date-perl libhttp-message-perl libio-html-perl liblwp-mediatypes-perl libmariadb3 libmysqlclient21 libndctl6
  libpmem1 libsnappy1v5 libtimedate-perl liburi-perl liburing2 mariadb-client-10.6 mariadb-client-core-10.6 mariadb-common
  mariadb-server-10.6 mariadb-server-core-10.6 mysql-common socat
Suggested packages:
  libmldbm-perl libnet-daemon-perl libsql-statement-perl libdata-dump-perl libipc-sharedcache-perl libbusiness-isbn-perl
  libwww-perl mailx mariadb-test
The following NEW packages will be installed:
  galera-4 libcgi-fast-perl libcgi-pm-perl libclone-perl libconfig-inifiles-perl libdaxctl1 libdbd-mysql-perl libdbi-perl
  libencode-locale-perl libfcgi-bin libfcgi-perl libfcgi0ldbl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
  libhttp-date-perl libhttp-message-perl libio-html-perl liblwp-mediatypes-perl libmariadb3 libmysqlclient21 libndctl6
  libpmem1 libsnappy1v5 libtimedate-perl liburi-perl liburing2 mariadb-client mariadb-client-10.6 mariadb-client-core-10.6
  mariadb-common mariadb-server mariadb-server-10.6 mariadb-server-core-10.6 mysql-common socat
0 upgraded, 36 newly installed, 0 to remove and 83 not upgraded.
Need to get 18.6 MB of archives.
After this operation, 165 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 mysql-common all 5.8+1.0.8 [7212 B]

...
...
Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 mariadb-server all 1:10.6.12-0ubuntu0.22.04.1 [11.8 kB]
Fetched 18.6 MB in 4s (4220 kB/s)
Extracting templates from packages: 100%
Preconfiguring packages ...

...
...
Selecting previously unselected package mariadb-server.
Preparing to unpack .../20-mariadb-server_1%3a10.6.12-0ubuntu0.22.04.1_all.deb ...
Unpacking mariadb-server (1:10.6.12-0ubuntu0.22.04.1) ...
Setting up libconfig-inifiles-perl (3.000003-1) ...
...
...
Setting up mariadb-server (1:10.6.12-0ubuntu0.22.04.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

Notice that mariadb-server and mariadb-client have a large number of dependencies, and that apt installs them all for us.

Now check that it is running:

root@database:~# systemctl status mariadb
● mariadb.service - MariaDB 10.6.12 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-04-09 17:51:10 UTC; 1min 46s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 3311 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 3312 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 3314 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera>
    Process: 3357 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 3359 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
   Main PID: 3343 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 9 (limit: 38073)
     Memory: 61.1M
        CPU: 201ms
     CGroup: /system.slice/mariadb.service
             └─3343 /usr/sbin/mariadbd

Apr 09 17:51:10 database mariadbd[3343]: 2023-04-09 17:51:10 0 [Note] Plugin 'FEEDBACK' is disabled.
Apr 09 17:51:10 database mariadbd[3343]: 2023-04-09 17:51:10 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_b>
Apr 09 17:51:10 database mariadbd[3343]: 2023-04-09 17:51:10 0 [Warning] You need to use --log-bin to make --expire-logs-days>
Apr 09 17:51:10 database mariadbd[3343]: 2023-04-09 17:51:10 0 [Note] Server socket created on IP: '127.0.0.1'.
Apr 09 17:51:10 database mariadbd[3343]: 2023-04-09 17:51:10 0 [Note] InnoDB: Buffer pool(s) load completed at 230409 17:51:10
Apr 09 17:51:10 database mariadbd[3343]: 2023-04-09 17:51:10 0 [Note] /usr/sbin/mariadbd: ready for connections.
Apr 09 17:51:10 database mariadbd[3343]: Version: '10.6.12-MariaDB-0ubuntu0.22.04.1'  socket: '/run/mysqld/mysqld.sock'  port>
Apr 09 17:51:10 database systemd[1]: Started MariaDB 10.6.12 database server.
Apr 09 17:51:10 database /etc/mysql/debian-start[3361]: Upgrading MySQL tables if necessary.
Apr 09 17:51:10 database /etc/mysql/debian-start[3376]: Triggering myisam-recover for all MyISAM tables and aria-recover for >

The status is active (running), which sounds good to us.

3. Deploy our backup server on Equinix Metal

This process is pretty much identical to the process of deploying our database server, so I won't go into too much detail here. The only difference is that we will be naming it backup instead of database. That should make it easier to keep track of which server is which.

One other step you could take, is to deploy the backup server in a different Equinix Metal metro. This is good practice as it provides extra safety in case of failure of a metro. To be fair, the Equinix Metal metros are very reliable, composed of multiple facilities, but it's always good to have a "backup" plan.

For the purposes of this guide and to keep it simple, however, we will deploy in the same Dallas metro as our database server. When deploying to a different metro, the private IP address space is not shared between servers, even in the same project, which means you need to set up connectivity. Backend transfer is great for that. You do have to pay for transfer between regions, but the cost, as always with Metal, is very reasonable.

As we are trying to keep this guide simple, we will not be doing that.

Once again, we wait a few minutes for the server to be deployed, and then we can start to use it. Don't forget to record its IP address. In my case, it is 147.75.53.31.

4. Install and configure our backup tool on the backup server

On our backup server, we will do the following.

1. ssh to the server
2. update the package manager
3. install the software we need, specifically minio
4. Configure minio

First, let's ssh to the server and update the package manager.

$ ssh root@147.75.53.31
The authenticity of host '147.75.53.31 (147.75.53.31)' can't be established.
ED25519 key fingerprint is SHA256:OA1v9dYFvNEkNujDT+RmpdYq8MFgLDHr/3BZUXa6nps.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '147.75.53.31' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-58-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sun Apr  9 18:53:20 UTC 2023

  System load:            0.03515625
  Usage of /:             0.7% of 438.04GB
  Memory usage:           1%
  Swap usage:             0%
  Temperature:            41.0 C
  Processes:              246
  Users logged in:        0
  IPv4 address for bond0: 147.75.53.31
  IPv6 address for bond0: 2604:1380:4642:4900::3

6 updates can be applied immediately.
6 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@backup:~# apt update -y
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [728 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [990 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [147 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [9020 B]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [701 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [109 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [576 B]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [716 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [210 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [13.9 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [744 kB]
Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [118 kB]
Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [14.2 kB]
Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [19.4 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [115 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [576 B]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [899 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [180 kB]
Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [4068 B]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [18.6 kB]
Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [24.1 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [6312 B]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B]
Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [40.6 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [9800 B]
Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [20.3 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [14.4 kB]
Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [480 B]
Fetched 6191 kB in 2s (2755 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
83 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@backup:~#

This should look familiar from the same steps on our database server.

Next, let's download minio and install it.

root@backup:~# curl -L -o /usr/local/bin/minio https://dl.min.io/server/minio/release/linux-amd64/minio
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 94.0M  100 94.0M    0     0  31.7M      0  0:00:02  0:00:02 --:--:-- 31.7M
root@backup:~# chmod +x /usr/local/bin/minio

Let's see that it worked:

root@backup:~# minio --version
minio version RELEASE.2023-04-07T05-28-58Z (commit-id=260a63ca73b09cf029a872554aceed809ae47231)
Runtime: go1.20.3 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2023 MinIO, Inc.

Excellent!

We are going to create a minio user and group, then a special directory to store our data, /var/s3, and change ownership of that directory to our minio user and group:

root@backup:~# adduser --system --no-create-home --shell /bin/bash --group minio
Adding system user `minio' (UID 114) ...
Adding new group `minio' (GID 121) ...
Adding new user `minio' (UID 114) with group `minio' ...
Not creating home directory `/home/minio'.
root@backup:~# mkdir -p /var/s3
root@backup:~# chown minio:minio /var/s3

We are going to do two more things to make this run a little more secure, even if it is just a getting started guide.

First, we will run minio server as our minio user, and not as root. Running software as root when it does not need to be is pretty bad security practice, as it makes any weaknesses in the software that much more powerful, with full system access.

Second, we are going to have minio server listen only on our private IP address, not on all, and especially not our public, interfaces. Since both of our servers are deployed in the same Equinix Metal project, they should have no problem communicating with each other over their private IP addresses.

Let's find our private IP address:

root@backup:~# ip addr show bond0
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b4:96:91:70:26:78 brd ff:ff:ff:ff:ff:ff
    inet 147.75.53.31/31 brd 255.255.255.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet 10.65.7.131/31 brd 255.255.255.255 scope global bond0:0
       valid_lft forever preferred_lft forever
    inet6 2604:1380:4642:4900::3/127 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::b696:91ff:fe70:2678/64 scope link
       valid_lft forever preferred_lft forever

There are two IPv6 addresses marked as inet6, which are not of interest to us (or at least, not for this guide). That leaves two IPv4 addresses, one of which is our public IP address, and the other is our private IP address. We want the private IP address, so that is 10.65.7.131.

Now, let's become the minio user and launch minio:

root@backup:~# su - minio
minio@backup:/root$ cd /var/s3
minio@backup:/var/s3$ minio server --address 10.65.7.131:9000 /var/s3
Formatting 1st pool, 1 set(s), 1 drives per set.
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2023-04-07T05-28-58Z (go1.20.3 linux/amd64)

Status:         1 Online, 0 Offline.
API: http://10.65.7.131:9000
RootUser: minioadmin
RootPass: minioadmin
Console: http://10.65.7.131:40405 http://147.75.53.31:40405 http://127.0.0.1:40405
RootUser: minioadmin
RootPass: minioadmin

Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
   $ mc alias set myminio http://10.65.7.131:9000 minioadmin minioadmin

Documentation: https://min.io/docs/minio/linux/index.html
Warning: The standard parity is set to 0. This can lead to data loss.

There are lots of warnings here, but we can ignore them for now. If you run minio in production, we strongly recommend that you configure all of your servers and software in the correct manner. This is, after all, just a getting started guide!

Our backup server now is ready.

5. Install and configure our backup tool on the database server

Let's switch back to our database server. If you logged out, be sure to ssh back in.

Before we do anything, we are going to be sure we can reach out backup server. First, let's ping the backup server's private IP address:

root@database:~# ping 10.65.7.131
root@database:~# ping 10.65.7.131
PING 10.65.7.131 (10.65.7.131) 56(84) bytes of data.
64 bytes from 10.65.7.131: icmp_seq=1 ttl=61 time=0.967 ms
64 bytes from 10.65.7.131: icmp_seq=2 ttl=61 time=0.679 ms
^C
--- 10.65.7.131 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.679/0.823/0.967/0.144 ms

Looks great! Now, let's install minio client and try to connect to the minio server:

root@database:~# curl -L https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24.9M  100 24.9M    0     0  24.3M      0  0:00:01  0:00:01 --:--:-- 24.3M
root@database:~# chmod +x /usr/local/bin/mc

And let's see that it runs:

root@database:~# mc --version
mc version RELEASE.2023-04-06T16-51-10Z (commit-id=1c6f4f48aba72b4c9770d911e95225f9de6e9488)
Runtime: go1.20.3 linux/amd64
Copyright (c) 2015-2023 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

Now we are going to use it to see if we can connect to our minio server:

root@database:~# mc alias set myminio http://10.65.7.131:9000 minioadmin minioadmin
Added `myminio` successfully.
root@database:~# mc ping myminio
  1: http://10.65.7.131:9000:9000   min=1.95ms     max=1.95ms     average=1.95ms     errors=0   roundtrip=1.95ms
  2: http://10.65.7.131:9000:9000   min=1.32ms     max=1.95ms     average=1.63ms     errors=0   roundtrip=1.32ms
  3: http://10.65.7.131:9000:9000   min=1.32ms     max=1.95ms     average=1.53ms     errors=0   roundtrip=1.32ms

Looks great!

Let's install our backup software. Unfortunately, the version of duplicity packages by Ubuntu is really old, and creates all sorts of issues. So we are going to install the latest version from the duplicity PPA:

root@database:~# add-apt-repository ppa:duplicity-team/duplicity-release-git
Repository: 'deb https://ppa.launchpadcontent.net/duplicity-team/duplicity-release-git/ubuntu/ jammy main'
Description:
Stable release versions of duplicity
More info: https://launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git
Adding repository.
Press [ENTER] to continue or Ctrl-c to cancel.
Adding deb entry to /etc/apt/sources.list.d/duplicity-team-ubuntu-duplicity-release-git-jammy.list
Adding disabled deb-src entry to /etc/apt/sources.list.d/duplicity-team-ubuntu-duplicity-release-git-jammy.list
Adding key to /etc/apt/trusted.gpg.d/duplicity-team-ubuntu-duplicity-release-git.gpg with fingerprint AF953139C1DF9EF3476DE1D58F571BB27A86F4A2
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 https://ppa.launchpadcontent.net/duplicity-team/duplicity-release-git/ubuntu jammy InRelease [17.6 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
Get:6 https://ppa.launchpadcontent.net/duplicity-team/duplicity-release-git/ubuntu jammy/main amd64 Packages [584 B]
Get:7 https://ppa.launchpadcontent.net/duplicity-team/duplicity-release-git/ubuntu jammy/main Translation-en [420 B]
Fetched 355 kB in 2s (189 kB/s)
Reading package lists... Done
root@database:~# apt update
Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:5 https://ppa.launchpadcontent.net/duplicity-team/duplicity-release-git/ubuntu jammy InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
78 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@database:~# apt install duplicity
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  grub-pc-bin python3-lockfile
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  lftp
Suggested packages:
  ncftp python3-boto python3-kerberos
The following NEW packages will be installed:
  duplicity lftp
0 upgraded, 2 newly installed, 0 to remove and 78 not upgraded.
Need to get 1035 kB of archives.
After this operation, 3832 kB of additional disk space will be used.
Do you want to continue? [Y/n] ^C
root@database:~# apt install -y duplicity
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  grub-pc-bin python3-lockfile
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  lftp
Suggested packages:
  ncftp python3-boto python3-kerberos
The following NEW packages will be installed:
  duplicity lftp
0 upgraded, 2 newly installed, 0 to remove and 78 not upgraded.
Need to get 1035 kB of archives.
After this operation, 3832 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 lftp amd64 4.9.2-1build1 [720 kB]
Get:2 https://ppa.launchpadcontent.net/duplicity-team/duplicity-release-git/ubuntu jammy/main amd64 duplicity amd64 1.2.2-ppa202301261750~ubuntu22.04.1 [315 kB]
Fetched 1035 kB in 1s (1064 kB/s)
Selecting previously unselected package lftp.
(Reading database ... 83274 files and directories currently installed.)
Preparing to unpack .../lftp_4.9.2-1build1_amd64.deb ...
Unpacking lftp (4.9.2-1build1) ...
Selecting previously unselected package duplicity.
Preparing to unpack .../duplicity_1.2.2-ppa202301261750~ubuntu22.04.1_amd64.deb ...
Unpacking duplicity (1.2.2-ppa202301261750~ubuntu22.04.1) ...
Setting up lftp (4.9.2-1build1) ...
Setting up duplicity (1.2.2-ppa202301261750~ubuntu22.04.1) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning candidates...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

Restarting services...
Service restarts being deferred:
 systemctl restart networkd-dispatcher.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

Duplicity itself is just a tool to manage a single backup. You can run an incremental or a full, verify a backup, or run a restore. It does not, however, have scheduling services build in.

We need to make a bucket to which we can back up. Earlier, we aliased our server to the name myminio, so the bucket we create, named backups (super-original!) will be named myminio/backups.

In addition, we need to create credentials, and give those credentials access to the bucket. We will use the mc tool to do all of the above, and use the very insecure credentials ACCCESSKEY1 and SECRETKEY1 to do so.

root@database:~# mc mb myminio/backups
Bucket created successfully `myminio/backups`.

root@database:~# mc admin user add myminio ACCESSKEY1 SECRETKEY1
Added user `ACCESSKEY1` successfully.

root@database:~# cat > /tmp/policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
            "s3:ListBucket",
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject"
        ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::backups/*", "arn:aws:s3:::backups"
      ],
      "Sid": "BucketAccessForUser"
    }
  ]
}
EOF

root@database:~# mc admin policy create myminio backups-bucket-policy /tmp/policy.json
Created policy `backups-bucket-policy` successfully.

root@database:~# mc admin policy attach myminio backups-bucket-policy --user ACCESSKEY1
Policy `backups-bucket-policy` successfully attached to user `ACCESSKEY1`

6. Testing backups

With all of that in place, let's do some backups!

We are aware that we are doing this a little bit suboptimally for databases. Although they are built to handle inconsistent data, for example, if you run a backup during a write, you will get a backup that is not consistent. Good database servers are built from the ground up to handle this, but it still is good practice to use database-native tools to take a dump on a regular basis, preferably from a replica, and then back that up. For our purposes, however, we will just back up the data directory.

The default data directory for mariadb is /var/lib/mysql, per this official mariadb knowledge base article.

Looking in that directory, we see all of our database files:

root@database:~# ls -l /var/lib/mysql
total 123328
-rw-rw---- 1 mysql mysql    417792 Apr  9 17:51 aria_log.00000001
-rw-rw---- 1 mysql mysql        52 Apr  9 17:51 aria_log_control
-rw-rw---- 1 mysql mysql         9 Apr  9 17:51 ddl_recovery.log
-rw-r--r-- 1 root  root          0 Apr  9 17:51 debian-10.6.flag
-rw-rw---- 1 mysql mysql       942 Apr  9 17:51 ib_buffer_pool
-rw-rw---- 1 mysql mysql 100663296 Apr  9 17:53 ib_logfile0
-rw-rw---- 1 mysql mysql  12582912 Apr  9 17:51 ibdata1
-rw-rw---- 1 mysql mysql  12582912 Apr  9 17:51 ibtmp1
-rw-rw---- 1 mysql mysql         0 Apr  9 17:51 multi-master.info
drwx------ 2 mysql mysql      4096 Apr  9 17:51 mysql
-rw-r--r-- 1 root  root         15 Apr  9 17:51 mysql_upgrade_info
drwx------ 2 mysql mysql      4096 Apr  9 17:51 performance_schema
drwx------ 2 mysql mysql     12288 Apr  9 17:51 sys

Let's run a single full backup of this directory:

root@database:~# duplicity --s3-endpoint-url http://10.65.7.131:9000 /var/lib/mysql s3://backups/database
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption:
Retype passphrase for decryption to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1681068957.81 (Sun Apr  9 19:35:57 2023)
EndTime 1681068959.58 (Sun Apr  9 19:35:59 2023)
ElapsedTime 1.76 (1.76 seconds)
SourceFiles 207
SourceFileSize 130450962 (124 MB)
NewFiles 207
NewFileSize 130450962 (124 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 207
RawDeltaSize 130426386 (124 MB)
TotalDestinationSizeChange 1139231 (1.09 MB)
Errors 0
-------------------------------------------------

For the GnuPG passphrase, I used the top-secret abcdefg. In real systems, of course, you would configure this to use a proper key.

Now it is time to have real fun. Let's update our database, and run it again.

root@database:~# cat > servers.txt <<EOF
a3.large.x86    2   1024
c3.medium.x86   24    64
m3.large.x86    32 256
EOF
root@c3-small-x86-01:~# mysql
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.001 sec)

MariaDB [(none)]> create database metal;
Query OK, 1 row affected (0.001 sec)

MariaDB [(none)]> use metal;
Database changed
MariaDB [metal]> create table servers (class varchar(20), cores smallint, memgb int);
Query OK, 0 rows affected (0.006 sec)

MariaDB [metal]> describe servers;
+-------+-------------+------+-----+---------+-------+
| Field | Type        | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| class | varchar(20) | YES  |     | NULL    |       |
| cores | smallint(6) | YES  |     | NULL    |       |
| memgb | int(11)     | YES  |     | NULL    |       |
+-------+-------------+------+-----+---------+-------+
3 rows in set (0.002 sec)

MariaDB [metal]> load data local infile 'servers.txt' into table servers;
Query OK, 3 rows affected, 9 warnings (0.001 sec)
Records: 3  Deleted: 0  Skipped: 0  Warnings: 9

MariaDB [metal]> select * from servers;
+----------------------+-------+-------+
| class                | cores | memgb |
+----------------------+-------+-------+
| a3.large.x86    2    |  NULL |  NULL |
| c3.medium.x86   24   |  NULL |  NULL |
| m3.large.x86    32 2 |  NULL |  NULL |
+----------------------+-------+-------+
3 rows in set (0.001 sec)

That looks pretty good.

Now let's run a backup.

root@database:~# duplicity --s3-endpoint-url http://10.65.7.131:9000 /var/lib/mysql s3://backups/database
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Apr  9 19:35:47 2023
GnuPG passphrase for decryption:
Retype passphrase for decryption to confirm:
--------------[ Backup Statistics ]--------------
StartTime 1681069164.21 (Sun Apr  9 19:39:24 2023)
EndTime 1681069164.62 (Sun Apr  9 19:39:24 2023)
ElapsedTime 0.41 (0.41 seconds)
SourceFiles 211
SourceFileSize 130533505 (124 MB)
NewFiles 5
NewFileSize 74360 (72.6 KB)
DeletedFiles 0
ChangedFiles 2
ChangedFileSize 100675584 (96.0 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 7
RawDeltaSize 328237 (321 KB)
TotalDestinationSizeChange 3719 (3.63 KB)
Errors 0
-------------------------------------------------

Notice that there are some changed files and some new files.

For the moment of truth, let's test it. We are going to do the following:

1. Shut down our database.
2. Move our data directory out of the way, and make a clean one. This will replicate the situation where we have a new database server.
3. Restore the database from our backup.
4. Restart our database.
5. See if our data is there.

root@database:~# systemctl stop mariadb
root@database:~# mv /var/lib/mysql /var/lib/mysql.bak
root@database:~# mkdir /var/lib/mysql
root@database:~# duplicity --s3-endpoint-url http://10.65.7.131:9000 s3://backups/database /var/lib/mysql
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Apr  9 19:35:47 2023
GnuPG passphrase for decryption:
root@database:~# systemctl start mariadb
root@database:~# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 31
Server version: 10.6.12-MariaDB-0ubuntu0.22.04.1 Ubuntu 22.04

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use metal;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [metal]> select * from servers;
+----------------------+-------+-------+
| class                | cores | memgb |
+----------------------+-------+-------+
| a3.large.x86    2    |  NULL |  NULL |
| c3.medium.x86   24   |  NULL |  NULL |
| m3.large.x86    32 2 |  NULL |  NULL |
+----------------------+-------+-------+
3 rows in set (0.001 sec)

Victory!