Success ... I think.

Phew. I spent a long time last night copying data off the old chef. One particularly annoying wrinkle was that after the parity rewrite, chef came up with unlean filesystems, and mounted them read-only. Including /. That meant, no logs could be written, and general unhappiness, which prevented remote logins. copying stuff out of chef via scp yielded a question every ~3.5 minutes, that the "host key of new-chef is unknown, is that ok"? In my 1am haze I dutifully typed yes every 3-4 minutes for an hour until the transfer was done.

This morning I thought about this more and figured that the only reason why it's asking repeatedly was that it couldn't write the host key to /root/.ssh/known_hosts and therefore when the ssh re-keys, it had to ask me again. Chef was still up this morning with read-only file systems and I still had to copy off 20GB of data, I didn't want to type yes all the time. I can't fsck the root filesystem while it's mounted read-only, and I was a bit concerned that chef might not come up again, when rebooting, so I mounted memory file systems over /root and /tmp:

mount_mfs swap /root
mount_mfs swap /tmp

The first parameter is just for the mount command to get some disk parameters, usually the special device of the swap partition on a local disk. Using "swap" forces mount_mfs to use a generic default. Useful for systems with no local disk.

I got all data transfered, decommissioned the old chef in the afternoon, and replaced the Athlon 900 motherboard and the two 160GB with a VIA M10000 board, 512MB of memory, and a 300GB disk. The disk is set up as one half of a raid1 mirror, so when I upgrade grumpy to use a 500GB disk as planned, I can use the 300GB in chef. Provided that disk passes a full-blown offline self-test via smartctl and a 8h stress test using multiple bonnie++ instances. The 500GB disk is going to get the same stress test before I put it in grumpy.

One immediately obvious benefit of the new chef is that it feels fast when using it with IMAP. A modern 1GHz CPU easily smokes a 5+ year old 900MHz Athlon. The memory is 4 times faster, too. And the whole system is quieter (mainly because the CPU fan on the Athlon was showing its age).

There's still lots of work left to do. I added features to Chef in small installments over 2+ years. But the basics (smtp, imap, web) are working reasonably well, so for now I'm happy enough with this that I'm going to watch some TV now.