Of logs and non-boot volumes

It’s a beautiful Sunday morning, and as you wait for the coffee to start working, you figure it’s probably about time that you started on those benchmark results that have to be done by Monday. You’re sitting there thinking “MAN I wish I’d provisioned a larger boot volume for these Mac Minis, I keep filling it up with logs! Oh wait, I have a smart idea that I acquired from thinking! Maybe I can use the non-boot volume to hold the computer software logs, since it has SO much more space! Ok then, all I need to do is find the buttons for making a symbolic link from /var/log to /Volumes/foo/log, and then… it’ll work! Yeah!”

And lo, it did work, for a time. Imagine my surprise when 2 out of 20 of my mac mini-ons became unusable for my nefarious purposes of generating ical server load because the data volume used by the load simulator was somehow mounted read-only! The two failing minions also happened to be the two that were recently rebooted, after moving their power cables to different power circuits (to prevent blowing breakers, but that’s a whole other story). I didn’t ask for this crazy read-only nonsense!

Volumes get mounted read-only early at boot time, every time you boot, and that is normal. Usually, they are very quickly re-mounted in read-write mode, which is… ya know, more useful and stuff. My best guess at the fail here is that some part of the logging subsystem tried to open /Volumes/foo while it was still read-only, and SUCCEEDED when it probably should have failed. After all, what is the point of a logging system that mounts its log targets read-only, other than to prevent re-mounting of that filesystem read-write!

It appears that this happens. So, don’t do it, unless you wanted both no logging and a read-only data volume.

Recovering from this was not as easy as it should have been. On the first minion, I actually got beyond nuking /Volumes/foo until I remembered the /var/log symlink, so then I removed the symlink, re-created /var/log, and rebooted – this fixed minion number one. The SECOND minion was weirder. I removed the /var/log symlink, made a local /var/log, then rebooted – but /Volumes/foo still came back read-only! What?! Well I’m on a schedule so I just blew that one away too (diskutil eraseVolume), and then it came back ok – *after another reboot*. Funky.