The rsync-system-backup program uses rsync to create full system backups of Linux systems. Supported backup destinations include local disks (possibly encrypted using LUKS) and remote systems that are running an SSH server or rsync daemon. Each backup produces a timestamped snapshot and these snapshots are rotated according to a rotation scheme that you can configure. The package is currently tested on cPython 2.6, 2.7, 3.4, 3.5, 3.6 and PyPy (2.7).
While this project brings together more than ten years of experience in creating (system) backups using rsync, all of the actual Python code was written in the first few months of 2017 and has seen limited real world use. The project does however have an automated test suite with more than 90% test coverage and my intention is to extend the test coverage further.
In May 2018 I changed the status from alpha to beta as part of release 1.0. The bump in major version number was triggered by a backwards incompatible code change, however at that point I had been using rsync-system-backup to make local backups of several of my Linux systems for the majority of a year. Also several colleagues of mine have used the how-to on setting up unattended backups to an encrypted USB disk.
Warning
Please use the --dry-run option when you're getting familiar
with how rsync-system-backup works and don't remove the option
until you're confident that you have the right command line,
because using rsync-system-backup in the wrong way can cause
data loss (for example by accidentally swapping the SOURCE
and DESTINATION arguments).
The rsync-system-backup package is available on PyPI which means installation should be as simple as:
$ pip install rsync-system-backupThere's actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).
There are two ways to use the rsync-system-backup package: As the command
line program rsync-system-backup and as a Python API. For details about the
Python API please refer to the API documentation available on Read the Docs.
The command line interface is described below.
Usage: rsync-system-backup [OPTIONS] [SOURCE] DESTINATION
Use rsync to create full system backups.
The required DESTINATION argument specifies the (possibly remote) location where the backup is stored, in the syntax of rsync's command line interface. The optional SOURCE argument defaults to '/' which means the complete root filesystem will be included in the backup (other filesystems are excluded).
Please use the --dry-run option when getting familiar with this program and
don't remove it until you're confident that you have the right command line,
because using this program in the wrong way can cause data loss (for example
by accidentally swapping the SOURCE and DESTINATION arguments).
Supported locations include:
- Local disks (possibly encrypted using LUKS).
- Remote systems that allow SSH connections.
- Remote systems that are running an rsync daemon.
- Connections to rsync daemons tunneled over SSH.
The backup process consists of several steps:
- First rsync is used to transfer all (relevant) files to a destination directory (whether on the local system or a remote system). Every time a backup is made, this same destination directory is updated.
- After the files have been transferred a 'snapshot' of the destination
directory is taken and stored in a directory with a timestamp in its
name. These snapshots are created using 'cp
--archive--link'. - Finally the existing snapshots are rotated to purge old backups according to a rotation scheme that you can customize.
Supported options:
| Option | Description |
|---|---|
-b, --backup |
Create a backup using rsync but don't create a snapshot and don't rotate
old snapshots unless the --snapshot and/or --rotate options are also given. |
-s, --snapshot |
Create a snapshot of the destination directory but don't create a backup
and don't rotate old snapshots unless the This option can be used to create snapshots of an rsync daemon module using
a 'post-xfer exec' command. If DESTINATION isn't given it defaults to the
value of the environment variable |
-r, --rotate |
Rotate old snapshots but don't create a backup and snapshot unless the
This option can be used to rotate old snapshots of an rsync daemon module
using a 'post-xfer exec' command. If DESTINATION isn't given it defaults to
the value of the environment variable |
-m, --mount=DIRECTORY |
Automatically mount the filesystem to which backups are written. When this option is given and An entry for the mount point needs to be defined in /etc/fstab for this to work. |
-c, --crypto=NAME |
Automatically unlock the encrypted filesystem to which backups are written. When this option is given and the An entry for the encrypted filesystem needs to be defined in /etc/crypttab for this to work. If the device of the encrypted filesystem is missing and rsync-system-backup is being run non-interactively, it will exit gracefully and not show any desktop notifications. If you want the backup process to run fully unattended you can configure a key file in /etc/crypttab, otherwise you will be asked for the password each time the encrypted filesystem is unlocked. |
-t, --tunnel=TUNNEL_SPEC |
Connect to an rsync daemon through an SSH tunnel. This provides encryption
for rsync client to daemon connections that are not otherwise encrypted.
The value of TUNNEL_SPEC is expected to be an SSH alias, host name or IP
address. Optionally a username can be prefixed (followed by '@') and/or a
port number can be suffixed (preceded by ':'). |
-i, --ionice=CLASS |
Use the 'ionice' program to set the I/O scheduling class and priority of
the 'rm' invocations used to remove backups. CLASS is expected to be one of
the values 'idle', 'best-effort' or 'realtime'. Refer to the man page of
the 'ionice' program for details about these values. |
-u, --no-sudo |
By default backup and snapshot creation is performed with superuser
privileges, to ensure that all files are readable and filesystem
metadata is preserved. The -u, --no-sudo option disables
the use of 'sudo' during these operations. |
-n, --dry-run |
Don't make any changes, just report what would be done. This doesn't
create a backup or snapshot but it does run rsync with the --dry-run
option. |
-x, --exclude=PATTERN |
Selectively exclude certain files from being included in the backup.
Refer to the rsync documentation for allowed PATTERN syntax. Note that
rsync-system-backup always uses the 'rsync --one-file-system' option. |
-f, --force |
By default rsync-system-backup refuses to run on non-Linux systems because
it was designed specifically for use on Linux. The use of the -f, --force
option sidesteps this sanity check. Please note that you are on your own if
things break! |
--disable-notifications |
By default a desktop notification is shown (using notify-send) before the system backup starts and after the backup finishes. The use of this option disables the notifications (notify-send will not be called at all). |
-v, --verbose |
Make more noise (increase logging verbosity). Can be repeated. |
-q, --quiet |
Make less noise (decrease logging verbosity). Can be repeated. |
-h, --help |
Show this message and exit. |
I've been finetuning my approach to Linux system backups for years now and during that time rsync has become my swiss army knife of choice :-). I also believe that comprehensive documentation can be half the value of an open source project. The following sections attempt to provide a high level overview of my system backup strategy:
Each backup is a full copy of the filesystem tree, stored in the form of individual files and directories on the destination. This "backup format" makes it really easy to navigate through and recover from backups because you can use whatever method you are comfortable with, whether that is a file browser, terminal, Python script or even chroot :-).
Note
You may want to configure updatedb to exclude the directory containing your system backups, otherwise the locate database will grow enormously.
Every time a backup is made the same destination directory is updated with
additions, updates and deletions since the last backup. After the backup is
done a snapshot of the destination directory is created using the command cp
--archive --link with the current date and time encoded in the name.
Due to the use of hard links each "version" of a file is only stored once. Because rsync by default doesn't modify files inplace it breaks hard links and thereby avoids modifying existing inodes. This ensures that the contents of snapshots don't change when a new backup updates existing files. The combination of hard links and the avoidance of inplace modifications effectively provides a limited form of deduplication. Each snapshot requires a couple of megabytes to store the directory names and hard links but the contents of files aren't duplicated.
The article Easy Automated Snapshot-Style Backups with Linux and Rsync contains more details about this technique.
Snapshots can be rotated according to a flexible rotation scheme, for example I've configured my laptop backup rotation to preserve the most recent 24 hourly backups, 30 daily backups and endless monthly backups.
While developing, maintaining and evolving backup scripts for various Linux laptops and servers I've learned that backups for different systems require different backup destinations and connection methods:
There's a LUKS encrypted USB disk on my desk at work that I use to keep hourly, daily and monthly backups of my work laptop. The disk is connected through the same USB hub that also connects my keyboard and mouse so I can't really forget about it :-).
Before the backup starts, the encrypted disk is automatically unlocked and mounted. The use of a key file enables this process to run unattended in the background. Once the backup is done the disk will be unmounted and locked again, so that it can be unplugged at any time (as long as a backup isn't running of course).
My personal laptop transfers hourly backups to the rsync daemon running on the server in my home network using a direct TCP connection without SSH. Most of the time the laptop has an USB Ethernet adapter connected but the backup runs fine over a wireless connection as well.
My VPS (virtual private server) transfers nightly backups to the rsync daemon running on the server in my home network over an SSH tunnel in order to encrypt the traffic and restrict access. The SSH account is configured to allow tunneling but disallow command execution. This setup enables the rsync client and server to run with root privileges without allowing the client to run arbitrary commands on the server.
Backing up to a local disk limits the effectiveness of backups but using SSH access between systems gives you more than you bargained for, because you're allowing arbitrary command execution. The rsync daemon provides an alternative that does not allow arbitrary command execution. The following sections discuss this option in more detail.
To be able to write files as root and preserve all filesystem metadata, rsync must be running with root privileges. However most of my backups are stored on remote systems and opening up remote root access over SSH just to transfer backups feels like a very blunt way to solve the problem :-).
Fortunately another solution is available: Configure an rsync daemon on the destination and instruct your rsync client to connect to the rsync daemon instead of connecting to the remote system over SSH. The rsync daemon configuration can restrict the access of the rsync client so that it can only write to the directory that contains the backup tree.
In this setup no SSH connections are used and the traffic between the rsync client and server is not encrypted. If this is a problem for you then continue reading the next section.
On Debian and derivatives like Ubuntu you can enable and configure an rsync daemon quite easily:
Make sure that rsync is installed:
$ sudo apt-get install rsync
Enable the rsync daemon by editing
/etc/default/rsyncand changing the lineRSYNC_ENABLE=falsetoRSYNC_ENABLE=true. Here's a one liner that accomplishes the task:$ sudo sed -i 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsyncCreate the configuration file
/etc/rsyncd.confand define at least one module. Here's an example based on my rsync daemon configuration:# Global settings. max connections = 4 log file = /var/log/rsyncd.log # Defaults for modules. read only = no uid = 0 gid = 0 # Daily backups of my VPS. [vps_backups] path = /mnt/backups/vps/latest post-xfer exec = /usr/sbin/process-vps-backups # Hourly backups of my personal laptop. [laptop_backups] path = /mnt/backups/laptop/latest post-xfer exec = /usr/sbin/process-laptop-backups
The
post-xfer execdirectives configure the rsync daemon to create a snapshot once the backup is done and rotate old snapshots afterwards.Once you've created
/etc/rsyncd.confyou can start the rsync daemon:$ sudo service rsync start
If you're using a firewall you should make sure that the rsync daemon port is whitelisted to allow incoming connections. The rsync daemon port number defaults to 873. Here's an iptables command to accomplish this:
$ sudo iptables -A INPUT -p tcp -m tcp --dport 873 -m comment --comment "rsync daemon" -j ACCEPT
When your backups are transferred over the public internet you should definitely use SSH to encrypt the traffic, but if you're at all security conscious then you probably won't like having to open up remote root access over SSH just to transfer backups :-).
The alternative is to use a non privileged SSH account to set up an SSH
tunnel that redirects network traffic to the rsync daemon. The login shell of
the SSH account can be set to /usr/sbin/nologin (or something similar like
/bin/false) to disable command execution, in this case you need to pass
-N to the SSH client.
The latest version of rsync-system-backup is available on PyPI and GitHub. The documentation is hosted on Read the Docs and includes a changelog. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at [email protected].
This software is licensed under the MIT license.
© 2018 Peter Odding.