Colas.Nahaboo.net - rsync-incr

Rsync-Incr

2025-02-11T00:00:00+00:00

What is rsync-incr🔗</a></h2>
rsync-incr</strong> is a linux wrapper shell (bash) script around rsync</a> to perform automated, unattended, incremental, disk to disk backups, automatically removing old backups to make room for new ones. It produces standard mirror copies browsable and restorable without specific tools.I have been using it in production daily at work and at home since 2004.</p>

Goals 🔗</a></h3>
I wanted to have a backup system with the following properties:</p>

standard</strong> based on standard tools (rsync), and restorable with only standard tools.</li>
simple</strong> as possible.</li>
automatable</strong> to be run daily (or more) by crontab, managing error conditions reliably so we can mail on errors, and making automatically room for new backups.</li> </ul> User Manual 🔗</a></h2>
`rsync-incr [options] N sourcedir destdir</code></p>`
`rsync-incr will create destdir</code> as a perfect mirror, and save in destdir.past</code> a directory per run with copies of changed files old versions. These N</code> (at most) directories of old versions of changed files are named by their dates in the form destdir.past/YYYY-MM-DD.HHhMN-SIZEm</code></p>`
e.g: for rsync-incr 10 /home /backups/home</code> /backups/home.past/2005-01-24.04h23-122m</code></li> </ul> Date of last backup is in the contents of file destdir.past/LAST_DATE</code>. No need to append a trailing / to source and destination. Dest</code> must be on the local machine (maybe NFS-mounted), sourcedir</code> can be on a remote machine via the ssh syntax host:dir</code>. SIZE</code> in name is the disk space taken that this backup, in megabytes, before an optional compression via --cbf</code>, to help you find the good value of N</code> m (e.g: the max of past SIZEs), as this size is hard to find in --snap</code> mode SIZE is rounded to upper bound: 0m means 0 bytes, 2m less than 2m</p> This a simple script, making backups usables by standard rsync (no need for a dedicated restore script). It has 2 basic modes of operation:</p>
`default</strong>: make a perfect copy, of all hard links, devices, sparse files, and just stores in dirs the previous versions of only the changed files. This makes it easy to find the different states a files went through, but make it harder to get a perfect snapshot of what was the full state N days before. E.g, if a directory contains files A and B, and B is modified today, yesterday backup will contain only yesterday version of B, and today will contain current versions of A and B.</li>`--snap</strong>: makes full snapshots of what the source was like at backup times as described in Mike Rubel's article</a> It is easier to get to full snapshots of previous states, and should run faster than the default. E.g, if a directory contains files A and B, and B is modified today, both yesterday and today backups will contain A and B, but A will be the same file (hard links to the same inode), and the Bs will be two different fles.</li> </ul> If N has "m" appended (2m, 34m, ..) old versions are removed before backup until we have at least N megabytes free on dest, and the max of space taken by previous backups (+ 10%, see --pbsm</code>). Otherwise, just keep the last N backups.</p> Options are passed to rsync, but must be a single word parts (e.g: use --rsh=ssh</code>, not -e ssh</code>)</p> e.g: rsync-incr -z --bwlimit=12 --rsh=ssh server:/home/me /backups/me</code></li> </ul> (this will create an perfect backup in /backups/me</code> and a series of previous versions as dirs like /backups/me.past/2004-10-26.04:40:20-234</code> ...) rsync-incr uses rsync</code> with options: -HSax --delete --force</code></p> Special non-rsync options:</p> --nohl</strong> do not use the -H / --hard-links option (do not preserve hard links), faster if you do not need to preserve hard links.</li> --cbf</strong> compresses (gzip -r) all backuped files (will not compress files with hard links)</li> --snap</strong> old backups are full snapshots of previous stats, as in Mike Rubel's article</a> but this do not preserve hard links</li> --grem</strong> global remove: will remove oldest backups globally on the filesystem (otherwise space-making on a small backup could wipe out all backups in order to desperately make room, not noticing that removing a single old backup of a bigger backup could do the job). You should place a list (one per line) of all the absolute paths of LAST_DATE</code> files on the system in the env variable RSYNCINCR_LASTDATES</code>, for instance by a statement: export RSYNCINCR_LASTDATES`locate =LAST_DATE`</code></li> </ul> </li> </ul> otherwise a global find will be used, which can be very slow</strong>. It will only remove backups on same filesystem as destdir, so you can list all LAST_DATE paths on all disks. For instance if all the backups are organized as /backups/host/partition</code>, the following should be included at the start of backup scripts to decrease startup time: RSYNCINCR_LASTDATES=`ls -1 /backups//.past/LAST_DATE`</code></p> --pbsm=P</strong> Previous Backups Space Margin: reserve space before backup for at least the max size of previous backups + P% (P default to 10). If P ends with "m" (like 7m) it is taken as P megabytes to add rather as a percentage.</li> --inodes=N</strong> ensures we have at least N percent of inodes free, and make room if not. Useful for backuping directories with huge number of small files.</li> --clean</strong> Just make enough room for the backups, but do not actually perform the backups</li> --cross-devs</strong> allows to backup directories spanning multiples volumes, mounted on subdirectories. Otherwise rsync-incr uses the -x option of rsync.</li> </ul> To restore a backup, use standard rsync (trailing slashes are IMPORTANT):</p> rsync -HSax --delete --force backup / original /</code></p> Rsync errors are propagated (the script exits with rsync exit status), except for the error #24 which is trapped, as this error can happen on backups of live systems (being modified while backuped)</p> Download🔗</a></h2> From github: https://github.com/ColasNahaboo/rsync-incr</a></p> Installation🔗</a></h2> Just copy the rsync-incr</code> shell script anywherein your PATH, e.g: /usr/local/bin</code>.</p> Implementation🔗</a></h3> A 120 lines of shell script (excluding the embedded doc)</p> License🔗</a></h3> Pure Open source: GPL</a></p> More details🔗</a></h2> See also what inspired me:</p> Rsync</a></li> Mike rubel seminal article</a></li> snapback</a></li> O'Reilly hacks</a></li> </ul> New versions announcements🔗</a></h3> Since 2025, releases are annonced on the Github site, at: https://github.com/ColasNahaboo/rsync-incr/releases</a></p> This documentation pages are going to migrate to Github too.</p> Examples of use🔗</a></h2> See a detailed example of what rsync-incr does</a></p> Also, here are (modified for privacy) real scripts I use daily:</p> Example1</a> a script run daily on the host backserv to archive incrementally various machine partitions and mailing in case of errors at $email</li> Example2</a> this script is auto-run on start of the backup server: it connects to the main server (named "m"), backups, and halts the backup server It does it in 2 paralled processes impacting disks on different controllers for added speed. Only m root partition is done incrementally</li> To connect to sites that use a non-standard port for ssh, let's say 26</code> the trick is to create a shell script, for instance named ssh-p26</code>, containing the line: exec ssh -p 26 "$@"</code> and call rsync-incr with the option --rsh=ssh-p26</code></li> </ul> Other backup systems🔗</a></h2> Rsync-incr is not the only open source smart and simple backup system. It is I think unique in its automated claculation of free space and smart removal of old backups to make room for new ones. But other ones, can be more relevant to your needs, for instance:</p> rdiff-backup</a> that only stores differences in files. For instance for big log files, rsync-incr will archive a full copy of the log (although it would have transferred on the network only the changed lines), whereas rdiff-backup will only store the actual added lines and is much more space-efficient backup-side. However, you do not get the easy browsing, access and comparison on backups that rsync-incr offers</li> dirvish</a> nice fast backup, but do not automate expiration of backups</li> rsnapshot</a> is more powerful and integrated, in perl having features that rsync-incr leave to a wrapper script</li> </ul> History🔗</a></h2> v1.9</strong> 2017-06-13 faster method to remove directories with huge number of files. new option --inodes=N</li> v1.8</strong> 2011-07-11 fix for working across mounted partitions with new option --cross-devs, rsh support dropped, only use ssh now</li> v1.7</strong> 2009-01-01, bug fix: could fail if was restarted on the same dirs in the same minute 2009-01-05 this documentation updated: the --no</code> option is in fact --clean</code>. Code is inchanged, no need to re-download. new subpage</a> to give a detailed example</li> v1.6</strong> 2008-12-19, works with destdir on NFS now (before was not removing old backups due to free space calculations not working on NFS), new --no</code> option.</li> v1.5</strong> 2008-10-12, nothing changed, only packaging doc, and web page moved here. No need to upgrade</li> v1.4</strong> 2007-02-26, bug fix by Jeremy Lingmann: on system with long device names wrapping enabled, rsync-incr was unable to compute free space. We now use df -P to fix this. This is the only change, upgrade recommended.</li> v1.3</strong> 2006-08-08, bug fix: rsync options with metacharacters were not working (e.g the * in: --exclude='/tmp/*')</li> v1.2</strong> 2006-06-21, bug fix by Jiri Voves: --pbsm option worked only for sizes given in megabytes (with appended "m")</li> v1.1</strong> 2005-03-25, bug fix: in some cases some old backups were not deleted. if day of month started with 0.</li> v1.0</strong> 2005-02-19, first public release</li> v0.9</strong> 2004-12-15, internal beta test</li> </ul> Comments🔗</a></h2>



Detailed example of rsync-incr operation
2009-01-11T00:00:00+00:00
Let's say we have a source directory, dir</code> that we are going to backup
into backup</code> with default options and backup--snap</code> with --snap</code>. We
will have in it the files:</p>
- On Day 1</span></span>
  - B</span></span>
  - C</span></span>
  - D</span></span>
- On Day 2</span></span>
  - A is Added</span></span>
  - B stay the same in Both days</span></span>
  - C is Changed</span></span>
  - D is Deleted and not there anymore</span></span></code></pre>So, on Day 1:🔗</a></h2>
We set up the source</p>
# mkdir dir; echo bbb >dir/B;echo ccc >dir/C;echo ddd >dir/D</span></span>
# ls -l dir</span></span>
total 12</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 B</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 C</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 D</span></span></code></pre>
We backup normally</p>
# rsync-incr 5 dir backups/dir</span></span>
# ls -lR backups</span></span>
backups:</span></span>
total 8</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:40 dir</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:42 dir.past</span></span>
</span>
backups/dir:</span></span>
total 12</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 B</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 C</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 D</span></span>
</span>
backups/dir.past:</span></span>
total 4</span></span>
-rw-rw-r-- 1 colas local 17 Jan 10 23:42 LAST_DATE</span></span></code></pre>
We backup with --snap</p>
# rsync-incr --snap 5 dir backups--snap/dir</span></span>
# ls -lR backups--snap</span></span>
backups--snap:</span></span>
total 8</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:40 dir</span></span>
drwxrwxr-x 3 colas local 4096 Jan 10 23:44 dir.past</span></span>
</span>
backups--snap/dir:</span></span>
total 12</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 B</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 C</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 D</span></span>
</span>
backups--snap/dir.past:</span></span>
total 8</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:44 2009-01-10.23h44-0m</span></span>
-rw-rw-r-- 1 colas local 17 Jan 10 23:44 LAST_DATE</span></span>
</span>
backups--snap/dir.past/2009-01-10.23h44-0m:</span></span>
total 0 </span></span></code></pre>On day 2🔗</a></h2>
We set up the source</p>
# rm dir/D; echo aaa>dir/A; echo CCC>dir/C</span></span>
# ls -l dir/</span></span>
total 12</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:48 A</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 B</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:48 C</span></span></code></pre>
We backup normally</p>
# rsync-incr 5 dir backups/dir</span></span>
# ls -lR backups</span></span>
backups:</span></span>
total 8</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:48 dir</span></span>
drwxrwxr-x 3 colas local 4096 Jan 10 23:50 dir.past</span></span>
 backups/dir:</span></span>
total 12</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:48 A</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 B</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:48 C</span></span>
 backups/dir.past:</span></span>
total 8</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:50 2009-01-10.23h42-1m</span></span>
-rw-rw-r-- 1 colas local 17 Jan 10 23:50 LAST_DATE</span></span>
 backups/dir.past/2009-01-10.23h42-1m:</span></span>
total 8</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 C</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:40 D </span></span></code></pre>
There, you can see that the "past" backup dir contains an image of the
whole of dir the "day" before. But you see that B has a link count of 2
so is shared between the 2 backup dirs, whereas C is not, the 2 C files
have different contents</p>
We backup with --snap</p>
# rsync-incr --snap 5 dir backups/dir</span></span>
# ls -lR backups--snap/:</span></span>
total 8</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:30 dir</span></span>
drwxrwxr-x 4 colas local 4096 Jan 10 23:30 dir.past</span></span>
</span>
backups--snap/dir:</span></span>
total 12</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:30 A</span></span>
-rw-rw-r-- 2 colas local 4 Jan 10 23:27 B</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:30 C</span></span>
</span>
backups--snap/dir.past:</span></span>
total 12</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:30 2009-01-10.23h30-0m</span></span>
drwxrwxr-x 2 colas local 4096 Jan 10 23:27 2009-01-10.23h30-1m</span></span>
-rw-rw-r-- 1 colas local 17 Jan 10 23:30 LAST_DATE</span></span>
</span>
backups--snap/dir.past/2009-01-10.23h30-0m:</span></span>
total 0</span></span>
</span>
backups--snap/dir.past/2009-01-10.23h30-1m:</span></span>
total 12</span></span>
-rw-rw-r-- 2 colas local 4 Jan 10 23:27 B</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:27 C</span></span>
-rw-rw-r-- 1 colas local 4 Jan 10 23:27 D</span></span></code></pre>
There, you can see that the "past" backup dir contains an image of the whole of dir the "day" before. But you see that B has a link count of 2 so is shared between the 2 backup dirs, whereas C is not, the 2 C files have different contents</p>

License 🔗</a></h3>
Pure Open source: GPL</a></p>

New versions announcements 🔗</a></h3>
Since 2025, releases are annonced on the Github site, at: https://github.com/ColasNahaboo/rsync-incr/releases</a></p>
This documentation pages are going to migrate to Github too.</p>

Comments 🔗</a></h2>

Colas.Nahaboo.net - rsync-incr

Rsync-Incr

Detailed example of rsync-incr operation

Colas.Nahaboo.net - rsync-incr

Rsync-Incr

Goals🔗</a></h3> I wanted to have a backup system with the following properties:</p> standard</strong> based on standard tools (rsync), and restorable with only standard tools.</li> simple</strong> as possible.</li>

License🔗</a></h3> Pure Open source: GPL</a></p>

New versions announcements🔗</a></h3> Since 2025, releases are annonced on the Github site, at: https://github.com/ColasNahaboo/rsync-incr/releases</a></p> This documentation pages are going to migrate to Github too.</p>

History🔗</a></h2> v1.9</strong> 2017-06-13 faster method to remove directories with huge number of files. new option --inodes=N</li> v1.8</strong> 2011-07-11 fix for working across mounted partitions with new option --cross-devs, rsh support dropped, only use ssh now</li>

Comments🔗</a></h2>

Detailed example of rsync-incr operation

License 🔗</a></h3>
Pure Open source: GPL</a></p>

New versions announcements 🔗</a></h3>
Since 2025, releases are annonced on the Github site, at: https://github.com/ColasNahaboo/rsync-incr/releases</a></p>
This documentation pages are going to migrate to Github too.</p>

Comments 🔗</a></h2>