A backup solution with restic, systemd, and backblaze

Nov 23, 2021 – Blog

Linux’s beauty lies in being coerced to learn a shitload of stuff to administrate your own system. I must admit I feel I’ve been dumbed down by all these years of blissful macOS usage. Last year I built myself a Linux desktop, and since then, I’ve been taking my digital sovereignty more seriously.

I still use cloud services for sure, but I don’t trust them anymore with my data. My Music library was absolutely crushed by Apple Music. Also, data is leaking everywhere, and what is supposed to be trusted shouldn’t be (I’m looking at you, Apple, and your dubious encryption claims.)

That said. Let’s talk about backups. Due to some rave reviews here and there on the internet, I decided to use restic as my backup solution. I want to back up my important files to a USB-connected disk and regularly upload them to an object-storage service for an easy and cheap off-site solution. I just needed a way to automatize its use.

First, a little bit of background:

  • All my computer’s disks are encrypted and decrypted at boot.
  • My USB disk is also encrypted and decrypted at boot.
  • The USB disk is not auto-mounted by systemd when the disk is first accessed, like this:

/dev/mapper/backup_crypt /backup ext4 defaults,noauto,user,x-systemd.automount,x-systemd.requires=systemd-cryptsetup@backup_crypt.service 0 2

To back up, I initially wrote a script that was run through some cronjobs, but it had the awful tendency to crash. I suspect the mix of systemd automount and restic wasn’t successful, but I did not investigate in that direction.

Instead, I searched for a way to have systemd manage this fun zoo. That way, the unit could depend on the disk being mounted and ensure that the /backup mount would be already available before the backup process. Plus, having a learning opportunity always makes things more fun for me. Instead of going through the analytical route, I went through the learning one.

As always when it comes to system administration, the Arch Wiki delivers. Systemd offers a Timers primitive that can replace crons. It works a bit differently, and it’s interesting. 

#!/bin/bash

set -e -o pipefail

DIR="/etc/backup"

BACKUP=$1

if [ -z $BACKUP ]
then
    >&2 echo "Please define backup name through the BACKUP environment variable."
    exit 1
fi

BACKUP_RETENTION_DAYS=7
BACKUP_RETENTION_WEEKS=4
BACKUP_RETENTION_MONTHS=6
BACKUP_RETENTION_YEARS=2
BACKUP_ENV="${DIR}/${BACKUP}.env"
BACKUP_FILES="${DIR}/${BACKUP}.files"
BACKUP_IEXCLUDE="${DIR}/${BACKUP}.iexclude"
BACKUP_OPTS=""

source /etc/backup/${BACKUP}.env

if [ ! -f "${BACKUP_ENV}" ]
then
    >&2 echo "[$BACKUP] ${BACKUP_ENV} does not exist. Exiting."
    exit 1
fi

if [ ! -f "${BACKUP_FILES}" ]
then
    >&2 echo "[$BACKUP] ${BACKUP_FILES} does not exist. Exiting."
    exit 1
fi

if [ -f $BACKUP_IEXCLUDE ]
then
    BACKUP_OPTS="--iexclude-file $BACKUP_IEXCLUDE ${OPTS}"
fi

if [ ! -d $RESTIC_REPOSITORY ]
then
    restic init &
    wait $!
fi

# Remove locks in case other stale processes kept them in
restic unlock &
wait $!


# Do the actual backup
restic backup \
    --exclude-caches \
    $BACKUP_OPTS \
    --files-from $BACKUP_FILES &
wait $!


# Removing old snapshots
restic forget \
    --verbose \
    --prune \
    --keep-daily $BACKUP_RETENTION_DAYS \
    --keep-weekly $BACKUP_RETENTION_WEEKS \
    --keep-monthly $BACKUP_RETENTION_MONTHS \
    --keep-yearly $BACKUP_RETENTION_YEARS &
wait $!

# Verifying data integrity
restic check --read-data &
wait $!

>&2 echo "[$BACKUP] Backup done"

The above script is executed like that: backup.sh name. It looks for different files allowing to configure restic execution (environment variables, etc.). Once it has found everything, it runs the backup and data verification process.

Now, it is needed to create a template unit file for systemd: 

[Unit]
Description=Backup with Restic (%i)
# Do not run in case another backup is already running
Conflicts=backup@.service
# This service requires the backup disk to be mounted
Requires=backup.automount

[Service]
Type=simple
Nice=10
ExecStart=/etc/backup/backup.sh %i
Environment="HOME=/root"

[Install]
WantedBy=multi-user.target

[Unit]
Description=Backup Nas Configuration

[Timer]
# Run everyday at 2:15am
OnCalendar=*-*-* 2:15:00
# My assumption: This should allow the job to start later if it wasn't started due to a conflict
Persistent=True

[Install]
WantedBy=timers.target

Now we run: 

systemctl daemon-reload
systemctl enable backup@nas-config.timer
systemctl start backup@nas-config.timer

And voila! It’s ready. Now we can repeat the same solution for sending backup to an off-site location. Also, the good thing is, it is possible to have execution logs with journalctl -f -u backup@nas-config.timer

Tags