A way to backup your servers using a s3 bucket

Imagine you are hosting an application, a blog on a cloud provider and you don’t want to pay a extra fee for backup storage.

You already have a local infrastructure and but you don’t already have a s3 storage.

Here I will explain how I deployed a backup architecture for all my servers selfhosted or on cloud.

Source: https://github.com/sinnwerkstatt/runrestic

Deploy Minio

As always, I will use my Synology NAS to host the minio service and use a docker-compose.

At first I just create a new directory (or share if you prefer) to host my minio data

1
$ mkdir -p /volume1/docker/minio

Personnaly I use portainer to manage my docker containers as I don’t like the too simple and basic docker app on DSM….

My docker-compose.yml look like this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
version: '3'
services:
  minio:
    image: quay.io/minio/minio:RELEASE.2023-05-27T05-56-19Z
    volumes:
      - /volume1/docker/minio:/data
    ports:
      - 32769:9000
      - 32768:9001
    environment:
      MINIO_ROOT_USER: 'root'
      MINIO_ROOT_PASSWORD: 'password'
      MINIO_ADDRESS: ':9000'
      MINIO_CONSOLE_ADDRESS: ':9001'
      VIRTUAL_HOST: 's3.example.com'
      VIRTUAL_PORT: ':443'
    command: minio server /data

It may be easier to reverse proxy Minio behind a Nginx or traefik but I won’t cover this for now. I assume that you already did it and your appliance is reachable from restic using s3.example.com.

Create restic bucket

Connect to the minio admin console (let’s say that my server hosting minio is named backup.example.com) using port 9001 : http://backup.example.com:9001

Use the value in MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.

From the Administrator section click on Buckets and then Create Bucket Define a name for your bucket and select the features you want. Here I don’t need file versionning as I will let restic manage as well as for Object locking.

I just define a Quota by security to 1Gb (only for the demo)

Create user and policy

Next thing to do is to create a user able to read and write on the bucket.

We will first create a user, under the Administrator section, click on Identity - Users and Create user:

Don’t select any policy for now, we will create a new one !

Under Administrator section, click on Policies then Create Policy

As policies are json based, we will have to paste something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BucketAccessForUser",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::backup",
                "arn:aws:s3:::backup/*"
            ]
        }
    ]
}

⚠️ Note that we allow all method under Action (this mean that the user is admin of the bucket) and we filter under Resource where does this apply. In our case it’s related to the backup bucket.

Next you can edit your previously created user and assign the policy.

Last step is to create a Service account for this user (used by restic). In the Service Accounts tab click on Create Access Key and just click on Create You should be prompted to download a json file containing the access key and secret key.

And you are done with the Minio part !

Configure restic client server

Install the required tools

1
2
$ yum install -y restic python3-pip
$ pip3 install --upgrade runrestic

Create the runrestic configuration

You will need to adapt this config file in order to fit to your needs:

repositories : you need to change the fqdn or just put the ipadress and port if it is on the same network
AWS_ACCESS_KEY_ID : get it from the json file
AWS_SECRET_ACCESS_KEY : get it from the json file
RESTIC_PASSWORD : use whatever password generator

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
name = "blog backup" # optional. if not set, the filename will be used without the extension

repositories = [
    "s3:https://s3.example.com/backup/"
    ]

[execution]
parallel = true
retry_count = 10
retry_backoff = "1:00 exponential"  # 00:00 = min:sec; 00:00:00 = hour:min:sec
# strategies:
#  - static (same duration every try)
#  - linear (duration * retry number)
#  - exponential

[environment]
AWS_DEFAULT_REGION="eu-west-1"
AWS_ACCESS_KEY_ID="xxxxxxxxxxxxxxx"
AWS_SECRET_ACCESS_KEY="zzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
RESTIC_PASSWORD="qqqqqqqqqqqqqqqqqqqqqq"

# or RESTIC_PASSWORD_FILE
# https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables

[backup]
sources = [
    "/opt/"
    ]

exclude_patterns = []
# exclude_files = []
# exclude_if_present = []

pre_hooks = ["systemctl stop myapp"]
post_hooks = ["systemctl start myapp"]

[prune]
keep-last =  3
keep-hourly =  5
keep-weekly = 10
keep-monthly = 30
group-by = "host,paths"
# https://restic.readthedocs.io/en/latest/060_forget.html#removing-snapshots-according-to-a-policy

[check]
checks = ["check-unused", "read-data"]

Initialize restic repository

The first time you deploy restic, you will need to initialize the repository on the s3 bucket (this has to be done just once for all your servers)

1
/usr/local/bin/runrestic init 

If everything goes well you should have a some directories and files created Run your first backup

Just simply run without any arguments

1
/usr/local/bin/runrestic

Schedule using systemd timers

Create a systemd file in /etc/systemd/system/runrestic.service

1
2
3
4
5
6
[Unit]
Description=runrestic backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/runrestic

And create a timer service

1
2
3
4
5
6
7
8
9
[Unit]
Description=Run runrestic backup

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable systemd configuration

1
2
systemctl enable runrestic.timer
systemctl start runrestic.timer

And you are done !

Going further

If you want to get a dashboard of your backup jobs, you can deploy a prometheus restic exporter : https://github.com/ngosang/restic-exporter

Of course you will need a prometheus instance and a grafana instance to display the dashboard