Rsync-Incr

What is rsync-incr

rsync-incr is a linux wrapper shell (bash) script around rsync to perform automated, unattended, incremental, disk to disk backups, automatically removing old backups to make room for new ones. It produces standard mirror copies browsable and restorable without specific tools.I have been using it in production daily at work and at home since 2004.

This page is at http://colas.nahaboo.net/Software/RsyncIncr

Goals

I wanted to have a backup system with the following properties:

• standard based on standard tools (rsync), and restorable with only standard tools.
• simple as possible.
• automatable to be run daily (or more) by crontab, managing error conditions reliably so we can mail on errors, and making automatically room for new backups.

User Manual

rsync-incr [options] N sourcedir destdir

rsync-incr will create destdir as a perfect mirror, and save in destdir.past a directory per run with copies of changed files old versions. These N (at most) directories of old versions of changed files are named by their dates in the form destdir.past/YYYY-MM-DD.HHhMN-SIZEm

• e.g: for rsync-incr 10 /home /backups/home
/backups/home.past/2005-01-24.04h23-122m

Date of last backup is in the contents of file destdir.past/LAST_DATE. No need to append a trailing / to source and destination.
Dest must be on the local machine (maybe NFS-mounted), sourcedir can be on a remote machine via the ssh syntax host:dir.
SIZE in name is the disk space taken that this backup, in megabytes, before an optional compression via --cbf, to help you find the good value of N m (e.g: the max of past SIZEs), as this size is hard to find in --snap mode SIZE is rounded to upper bound: 0m means 0 bytes, 2m less than 2m

This a simple script, making backups usables by standard rsync (no need for a dedicated restore script). It has 2 basic modes of operation:

• default: make a perfect copy, of all hard links, devices, sparse files, and just stores in dirs the previous versions of only the changed files. This makes it easy to find the different states a files went through, but make it harder to get a perfect snapshot of what was the full state N days before. E.g, if a directory contains files A and B, and B is modified today, yesterday backup will contain only yesterday version of B, and today will contain current versions of A and B.
• --snap: makes full snapshots of what the source was like at backup times as described in Mike Rubel article It is easier to get to full snapshots of previous states, and should run faster than the default. E.g, if a directory contains files A and B, and B is modified today, both yesterday and today backups will contain A and B, but A will be the same file (hard links to the same inode), and the Bs will be two different fles.

If N has "m" appended (2m, 34m, ..) old versions are removed before backup until we have at least N megabytes free on dest, and the max of space taken by previous backups (+ 10%, see --pbsm). Otherwise, just keep the last N backups.

Options are passed to rsync, but must be a single word parts (e.g: use --rsh=ssh, not -e ssh)

• e.g: rsync-incr -z --bwlimit=12 --rsh=ssh server:/home/me /backups/me

(this will create an perfect backup in /backups/me and a series of previous versions as dirs like /backups/me.past/2004-10-26.04:40:20-234 ...) rsync-incr uses rsync with options: -HSax --delete --force

Special non-rsync options:

• --nohl do not use the -H / --hard-links option (do not preserve hard links), faster if you do not need to preserve hard links.
• --cbf compresses (gzip -r) all backuped files (will not compress files with hard links)
• --snap old backups are full snapshots of previous stats, as in Mike Rubel article but this do not preserve hard links
• --grem global remove: with m appended to N, (i.e., N given as a number of megabytes), will remove oldest backups globally on the filesystem (otherwise space-making on a small backup could be wiped out because of bigger backups).
  You should place a list (one per line) of all the absolute paths of LAST_DATE files on the system in the env variable RSYNCINCR_LASTDATES, for instance by a statement:
  • export RSYNCINCR_LASTDATES=`locate LAST_DATE`

otherwise a global find will be used, which can be very slow. It will only remove backups on same filesystem as destdir, so you can list all LAST_DATE paths on all disks.

• --pbsm=P Previous Backups Space Margin: reserve space before backup for at least the max size of previous backups + P% (P default to 10). If P ends with "m" (like 7m) it is taken as P megabytes to add rather as a percentage.

To restore a backup, use standard rsync (trailing slashes are IMPORTANT):

• rsync -HSax --delete --force backup / original /

Rsync errors are propagated (the script exits with rsync exit status), except for the error #24 which is trapped, as this error can happen on backups of live systems (being modified while backuped)

Installation

Just copy the rsync-incr shell script anywherein your PATH, e.g: /usr/local/bin.

Implementation

A 120 lines of shell script (excluding the embedded doc), based on the fantastic rsync, and Mike Rubel article

License

Pure Open source: GPL

More details

See also what inspired me:

• Rsync
• Mike rubel seminal article
• snapback
• O'Reilly hacks

New versions announcements

New releases and important issues will be annonced on the Rsyncincr blog, so I strongly suggest you monitor it, either

• by RSS to be warned of new releases and major changes (recommended)
• by email, by email gateways to the above RSS, for instance by
  Subscribe via
• by email, for instance by Changedetection.com, on:
  • the RsyncIncrLeftBar left column URL (click here to subscribe) which only changes on new stable releases
  • this current page (click here to subscribe), but this is not recommended as you will be emailed on each small typo correction

Examples of use

These are (modified for privacy) real scripts I use daily:

• Example1 a script run daily on the host backserv to archive incrementally various machine partitions and mailing in case of errors at $email
• Example2 this script is auto-run on start of the backup server: it connects to the main server (named "m"), backups, and halts the backup server It does it in 2 paralled processes impacting disks on different controllers for added speed. Only m root partition is done incrementally

History

• v1.5 2008-10-12, nothing changed, only packaging doc, and web page moved here. No need to upgrade
• v1.4 2007-02-26, bug fix by Jeremy Lingmann: on system with long device names wrapping enabled, rsync-incr was unable to compute free space. We now use df -P to fix this. This is the only change, upgrade recommended.
• v1.3 2006-08-08, bug fix: rsync options with metacharacters were not working (e.g the * in: --exclude='/tmp/*')
• v1.2 2006-06-21, bug fix by Jiri Voves: --pbsm option worked only for sizes given in megabytes (with appended "m")
• v1.1 2005-03-25, bug fix: in some cases some old backups were not deleted. if day of month started with 0.
• v1.0 2005-02-19, first public release
• v0.9 2004-12-15, internal beta test

Comments

---

RSS feed of this page comments