A backup solution with restic, systemd, and backblaze
Linux’s beauty lies in being coerced to learn a shitload of stuff to administrate your own system. I must admit I feel I’ve been dumbed down by all these years of blissful macOS usage. Last year I built myself a Linux desktop, and since then, I’ve been taking my digital sovereignty more seriously.
I still use cloud services for sure, but I don’t trust them anymore with my data. My Music library was absolutely crushed by Apple Music. Also, data is leaking everywhere, and what is supposed to be trusted shouldn’t be (I’m looking at you, Apple, and your dubious encryption claims.)
That said. Let’s talk about backups. Due to some rave reviews here and there on the internet, I decided to use restic as my backup solution. I want to back up my important files to a USB-connected disk and regularly upload them to an object-storage service for an easy and cheap off-site solution. I just needed a way to automatize its use.
First, a little bit of background:
- All my computer’s disks are encrypted and decrypted at boot.
- My USB disk is also encrypted and decrypted at boot.
- The USB disk is not auto-mounted by systemd when the disk is first accessed, like this:
/dev/mapper/backup_crypt /backup ext4 defaults,noauto,user,x-systemd.automount,x-systemd.requires=systemd-cryptsetup@backup_crypt.service 0 2
To back up, I initially wrote a script that was run through some cronjobs, but it had the awful tendency to crash. I suspect the mix of systemd automount and restic wasn’t successful, but I did not investigate in that direction.
Instead, I searched for a way to have systemd manage this fun zoo. That way, the unit could depend on the disk being mounted and ensure that the /backup
mount would be already available before the backup process. Plus, having a learning opportunity always makes things more fun for me. Instead of going through the analytical route, I went through the learning one.
As always when it comes to system administration, the Arch Wiki delivers. Systemd offers a Timers primitive that can replace crons. It works a bit differently, and it’s interesting.
#!/bin/bash set -e -o pipefail DIR="/etc/backup" BACKUP=$1 if [ -z $BACKUP ] then >&2 echo "Please define backup name through the BACKUP environment variable." exit 1 fi BACKUP_RETENTION_DAYS=7 BACKUP_RETENTION_WEEKS=4 BACKUP_RETENTION_MONTHS=6 BACKUP_RETENTION_YEARS=2 BACKUP_ENV="${DIR}/${BACKUP}.env" BACKUP_FILES="${DIR}/${BACKUP}.files" BACKUP_IEXCLUDE="${DIR}/${BACKUP}.iexclude" BACKUP_OPTS="" source /etc/backup/${BACKUP}.env if [ ! -f "${BACKUP_ENV}" ] then >&2 echo "[$BACKUP] ${BACKUP_ENV} does not exist. Exiting." exit 1 fi if [ ! -f "${BACKUP_FILES}" ] then >&2 echo "[$BACKUP] ${BACKUP_FILES} does not exist. Exiting." exit 1 fi if [ -f $BACKUP_IEXCLUDE ] then BACKUP_OPTS="--iexclude-file $BACKUP_IEXCLUDE ${OPTS}" fi if [ ! -d $RESTIC_REPOSITORY ] then restic init & wait $! fi # Remove locks in case other stale processes kept them in restic unlock & wait $! # Do the actual backup restic backup \ --exclude-caches \ $BACKUP_OPTS \ --files-from $BACKUP_FILES & wait $! # Removing old snapshots restic forget \ --verbose \ --prune \ --keep-daily $BACKUP_RETENTION_DAYS \ --keep-weekly $BACKUP_RETENTION_WEEKS \ --keep-monthly $BACKUP_RETENTION_MONTHS \ --keep-yearly $BACKUP_RETENTION_YEARS & wait $! # Verifying data integrity restic check --read-data & wait $! >&2 echo "[$BACKUP] Backup done"
The above script is executed like that: backup.sh name
. It looks for different files allowing to configure restic execution (environment variables, etc.). Once it has found everything, it runs the backup and data verification process.
Now, it is needed to create a template unit file for systemd:
[Unit] Description=Backup with Restic (%i) # Do not run in case another backup is already running Conflicts=backup@.service # This service requires the backup disk to be mounted Requires=backup.automount [Service] Type=simple Nice=10 ExecStart=/etc/backup/backup.sh %i Environment="HOME=/root" [Install] WantedBy=multi-user.target
[Unit] Description=Backup Nas Configuration [Timer] # Run everyday at 2:15am OnCalendar=*-*-* 2:15:00 # My assumption: This should allow the job to start later if it wasn't started due to a conflict Persistent=True [Install] WantedBy=timers.target
Now we run:
systemctl daemon-reload systemctl enable backup@nas-config.timer systemctl start backup@nas-config.timer
And voila! It’s ready. Now we can repeat the same solution for sending backup to an off-site location. Also, the good thing is, it is possible to have execution logs with journalctl -f -u backup@nas-config.timer