I recently devoted one of my Mac OS X Leopard systems to performing backups on the rest of my OS X and Linux systems. As long time readers of Slaptijack know, I use a script to run rsync with the --link-dest
command line option set in order to conserve space with hard links. After getting this new Leopard system all set up with the backup script, list of directories to back up, and the appropriate crontab
entry, I thought everything would be fine. I checked a few days later and noticed that I had several directories worth of backups and decided that everything was working.
<!--more-->
After about a week, I started getting email notifications that backups were failing because the target disk was full. Although this 750 GB disk has a few other things on it, there is no way that it should have filled up already. My first suspicion was that this problem was somehow related to how Mac OS X mounts external drives without enabling ownership. I checked the "Get Info" for the mounted volume and didn't see the "Ignore ownership on this volume" checkbox. Apparently, that option disappeared in Leopard.
I verified that hard links weren't working by checking a file in the /etc
save directories from one of my Linux servers. This file hasn't changed since sometime in 2006, so there is no reason it would be hard linked. I first used ls -l
to look at the file in each of the five existing backup directories:
# ls -l etc*/xpdfrc
-rw-r--r-- 1 _unknown _unknown 3768 Feb 7 2006 etc-1/xpdfrc
-rw-r--r-- 1 _unknown _unknown 3768 Feb 7 2006 etc-2/xpdfrc
-rw-r--r-- 1 _unknown _unknown 3768 Feb 7 2006 etc-3/xpdfrc
-rw-r--r-- 1 _unknown _unknown 3768 Feb 7 2006 etc-4/xpdfrc
-rw-r--r-- 1 _unknown _unknown 3768 Feb 7 2006 etc-5/xpdfrc
(By the way, I changed the directory names in this output to make them a little easier to read on this site. The actual directory structure produced by my script uses date and time in the directory name.)
We can see immediately that I have two problems. The first is that the link count for these files is 1 when it should be 5. That means that each of these files is separate rather than hard linked. Additionally, we see that the owner and group are both unknown. That's what's causing the --link-dest
command line option to fail. Since OS X is reporting a different owner and group than the one the server sees, rsync
thinks at least one of the file attributes has changed and downloads a whole new copy of the file rather than hard linking to the existing backup file. I can verify this by looking at which inodes the files are assigned via ls -i
:
# ls -i etc*/xpdfrc
37073368 etc-1/xpdfrc
37104392 etc-2/xpdfrc
37181270 etc-3/xpdfrc
37519024 etc-4/xpdfrc
38041542 etc-5/xpdfrc
If these files were hard linked, the inode number would be the same in each listing. So, I'm using five times the amount of space necessary to back up this one file. Multiply that across thousands of files and hundreds of GB and I've filled up my 750 GB hard drive pretty darn quickly.
At this point, I was fairly certain this was a mounting problem in OS X, but the checkbox I had used in Tiger to fix this problem didn't exist any more. We can see from the command line that this volume is mounted differently than the primary OS X volume:
/dev/disk0s3 on / (hfs, local, journaled)
devfs on /dev (devfs, local)
fdesc on /dev (fdesc, union)
map -hosts on /net (autofs, automounted)
map auto_home on /home (autofs, automounted)
/dev/disk1s3 on /Volumes/Space (hfs, local, nodev, nosuid, journaled, noowners)
That noowners
flag is the culprit. The owners
option can be permanently enabled with the vsdbutil
command:
# vsdbutil -a /Volumes/Space
# vsdbutil -c /Volumes/Space
Permissions on '/Volumes/Space' are enabled.
# mount
/dev/disk0s3 on / (hfs, local, journaled)
devfs on /dev (devfs, local)
fdesc on /dev (fdesc, union)
map -hosts on /net (autofs, automounted)
map auto_home on /home (autofs, automounted)
/dev/disk1s3 on /Volumes/Space (hfs, local, nodev, nosuid, journaled)
(According to the man page, vsdbutil
has been deprecated. I'm not sure what is going to replace it.)
After that, it was just a matter of testing the backups again and making sure they worked as expected. I cleared out the existing backup directories and ran the backups twice to ensure everything was OK:
# ls -l etc*/xpdfrc
-rw-r--r-- 2 root wheel 3768 Feb 7 2006 etc-1/xpdfrc
-rw-r--r-- 2 root wheel 3768 Feb 7 2006 etc-2/xpdfrc
# ls -1i etc*/xpdfrc
39130911 etc-1/xpdfrc
39130911 etc-2/xpdfrc
Everything is looking great! I'm going to let the backups run for a few days, and I'll update this post if anything has changed.