Borg backup is an open source software project providing deduplication, encryption, and compression. Not all files which are present on a machine should be copied to the backup system (e.g. caches, metadata stores). Files or directories which should be ignored during backup are specified using “exclusion lists” or “patterns” which are then fed into borg. Unfortunately macOS has many peculiarities not present in standard *nix distributions which makes crafting a sensible list difficult, furthermore these can change with each OS update.
The purpose of this repository is to collect an annotated list of exclusion rules tailored to the macOS environment that can be shared and critiqued among the community. Ideally there’s a universally agreed upon “core” set of exclusions which can be used in all situations for macOS, and then suggested exclusions available as addenda.
Clone this repo or copy the exclusion files to the machine which is going to run borg backup.
Run borg create with --patterns-from=/path/to/borg-exclude.lst. To combine multiple lists add them in as an additional argument.
This will exclude any directory with a .nobackup file present, while following the core and programming exclusion lists.
borg create --list --exclude-if-present .nobackup --patterns-from=/Users/exampleuser/sysadmin/borg-exclude-macos-core.lst --patterns-from=/Users/exampleuser/sysadmin/borg-exclude-macos-programming.lst [email protected]:example/directory::'{hostname}-{utcnow} /
Although it might seem like copying all items from a machine would be the ideal case, these can create problems on recovery or unduly slow down the backup process. The wallclock time to perform a backup is strongly correlated with the number of files that need to be traversed, so omitting unnecessary directories can significantly speed up backups.
Excluding these files leads to faster backups, less storage use/churn, and easier recoveries.
These are items that would be present in any macOS system, that would not make sense to restore under any circumstances. This should be safe to use in any backup scenario.
These are items that pertain to applications which are not a part of the default macOS system but should not be backed up. These are things like special databases or caches that can be generated again, or may change too quickly for borg to copy.
For example, adguard creates a database which borg complains about.
! **/com.adguard.mac.adguard/adguard.db
These are not exclusions of the app data itself to save space (e.g. the ~/Applications/adguard.app is untouched).
WARNING: Never use fast and loose in a standard backup
Borg isn’t designed as a continuous backup system (which would make duplicates of files on modification), but it can be run frequently (every 5 minutes). Borg’s speed seems to be
The expectation is that this is run on a home directory (not the root dir), and excluded items from “fast and loose” will be included in a nightly backup, but aren’t (hopefully) relevant for recovering work from the past few hours.
The primary risk vector is that an excluded file changes more frequently than the full backup period (daily), and then isn’t available to be recovered from. But this risk would be present under the standard backup scheme too.
Directories which should never be included in the backup should contain a .nobackup file instead of being excluded through a pattern.
WARNING: Do not copy this blindly
This is included more as an example, you should profile your own frequent backups to see where they hang. Additionally certain directories contain random characters that are different on your system.
Most git repositories are hosted on an external server, and public repos of large projects will typically be available elsewhere. Thus, there’s usually no need to back them up.
Apple recommends backing these files up before attempting to Repair your library in Photos on Mac.
- More programs are built in than standard *nix operating systems. These require their own forms of exclusions
- There are virtual folders which can lead to standard rules like
~/Library/examplebeing ignored because they’re reparsed asSystem/Volumes/Data/Users/yourname/Library/example
I’m starting with the core items that should be excluded in any situation, but there are many additional files which are reasonable to exclude for most situations, and even more which can be overlooked if the goal is speed/frequency rather than completeness. I plan to add these in as optional configurations later.
- Create bug reports if you find something included that should be excluded (or the converse)
- Include commentary about risk or failure
- Add or correct annotations
- Collect and post timing data for traversing certain file paths
- Suggest changes to this README file
- Link to this repository in stack exchange, blogs, and reddit posts
[fn:1] A more effective way of excluding large files would be to search for files over a certain size (say 50mb), and then exclude these specifically for the frequent backups.