Hard Links Not Working in Rsync Backups

Posted on in System Administration

I recently devoted one of my Mac OS X Leopard systems to performing backups on the rest of my OS X and Linux systems. As long time readers of Slaptijack know, I use a script to run rsync with the --link-dest command line option set in order to conserve space with hard links. After getting this new Leopard system all set up with the backup script, list of directories to back up, and the appropriate crontab entry, I thought everything would be fine. I checked a few days later and noticed that I had several directories worth of backups and decided that everything was working.

<!--more-->

After about a week, I started getting email notifications that backups were failing because the target disk was full. Although this 750 GB disk has a few other things on it, there is no way that it should have filled up already. My first suspicion was that this problem was somehow related to how Mac OS X mounts external drives without enabling ownership. I checked the "Get Info" for the mounted volume and didn't see the "Ignore ownership on this volume" checkbox. Apparently, that option disappeared in Leopard.

I verified that hard links weren't working by checking a file in the /etc save directories from one of my Linux servers. This file hasn't changed since sometime in 2006, so there is no reason it would be hard linked. I first used ls -l to look at the file in each of the five existing backup directories:

# ls -l etc*/xpdfrc
-rw-r--r--  1 _unknown  _unknown  3768 Feb  7  2006 etc-1/xpdfrc
-rw-r--r--  1 _unknown  _unknown  3768 Feb  7  2006 etc-2/xpdfrc
-rw-r--r--  1 _unknown  _unknown  3768 Feb  7  2006 etc-3/xpdfrc
-rw-r--r--  1 _unknown  _unknown  3768 Feb  7  2006 etc-4/xpdfrc
-rw-r--r--  1 _unknown  _unknown  3768 Feb  7  2006 etc-5/xpdfrc

(By the way, I changed the directory names in this output to make them a little easier to read on this site. The actual directory structure produced by my script uses date and time in the directory name.)

We can see immediately that I have two problems. The first is that the link count for these files is 1 when it should be 5. That means that each of these files is separate rather than hard linked. Additionally, we see that the owner and group are both unknown. That's what's causing the --link-dest command line option to fail. Since OS X is reporting a different owner and group than the one the server sees, rsync thinks at least one of the file attributes has changed and downloads a whole new copy of the file rather than hard linking to the existing backup file. I can verify this by looking at which inodes the files are assigned via ls -i:

# ls -i etc*/xpdfrc
37073368 etc-1/xpdfrc
37104392 etc-2/xpdfrc
37181270 etc-3/xpdfrc
37519024 etc-4/xpdfrc
38041542 etc-5/xpdfrc

If these files were hard linked, the inode number would be the same in each listing. So, I'm using five times the amount of space necessary to back up this one file. Multiply that across thousands of files and hundreds of GB and I've filled up my 750 GB hard drive pretty darn quickly.

At this point, I was fairly certain this was a mounting problem in OS X, but the checkbox I had used in Tiger to fix this problem didn't exist any more. We can see from the command line that this volume is mounted differently than the primary OS X volume:

/dev/disk0s3 on / (hfs, local, journaled)
devfs on /dev (devfs, local)
fdesc on /dev (fdesc, union)
map -hosts on /net (autofs, automounted)
map auto_home on /home (autofs, automounted)
/dev/disk1s3 on /Volumes/Space (hfs, local, nodev, nosuid, journaled, noowners)

That noowners flag is the culprit. The owners option can be permanently enabled with the vsdbutil command:

# vsdbutil -a /Volumes/Space
# vsdbutil -c /Volumes/Space
Permissions on '/Volumes/Space' are enabled.
# mount
/dev/disk0s3 on / (hfs, local, journaled)
devfs on /dev (devfs, local)
fdesc on /dev (fdesc, union)
map -hosts on /net (autofs, automounted)
map auto_home on /home (autofs, automounted)
/dev/disk1s3 on /Volumes/Space (hfs, local, nodev, nosuid, journaled)

(According to the man page, vsdbutil has been deprecated. I'm not sure what is going to replace it.)

After that, it was just a matter of testing the backups again and making sure they worked as expected. I cleared out the existing backup directories and ran the backups twice to ensure everything was OK:

# ls -l etc*/xpdfrc
-rw-r--r--  2 root  wheel  3768 Feb  7  2006 etc-1/xpdfrc
-rw-r--r--  2 root  wheel  3768 Feb  7  2006 etc-2/xpdfrc
# ls -1i etc*/xpdfrc
39130911 etc-1/xpdfrc
39130911 etc-2/xpdfrc

Everything is looking great! I'm going to let the backups run for a few days, and I'll update this post if anything has changed.

My Bookshelf

Reading Now

Other Stuff