{"id":1212,"date":"2026-07-27T16:52:55","date_gmt":"2026-07-27T14:52:55","guid":{"rendered":"https:\/\/www.distributed-systems.net\/?page_id=1212"},"modified":"2026-07-28T09:03:40","modified_gmt":"2026-07-28T07:03:40","slug":"cloning-a-server","status":"publish","type":"page","link":"https:\/\/www.distributed-systems.net\/index.php\/cloning-a-server\/","title":{"rendered":"Cloning a server"},"content":{"rendered":"\n

Cloning zappa<\/code> onto coltrane<\/code> sounds like it should be the easy part \u2014 copy the files across and you’re done. Copying the files<\/em> is indeed easy. The hard part is making the copy actually boot<\/strong>, on hardware that isn’t zappa<\/code>‘s.<\/p>\n\n\n\n

Why a clone is not a copy<\/h2>\n\n\n\n

zappa<\/code> boots from an NVMe drive; coltrane<\/code> boots from an older SATA disk. A running Linux system carries a boot stack \u2014 the initramfs and the GRUB bootloader \u2014 that is tailored to the machine it was installed on: which disks exist, how they’re numbered, and, crucially, the UUID of the root filesystem baked into the bootloader. rsync happily copies all of that from zappa<\/code> to coltrane<\/code>, at which point coltrane<\/code> is holding zappa<\/code>‘s boot stack on coltrane<\/code>‘s hardware. It won’t come up.<\/p>\n\n\n\n

So every clone is really two steps: replicate the filesystems, then re-fit the boot stack<\/strong> for coltrane<\/code> \u2014 rebuild the initramfs (update-initramfs<\/code>) and reinstall GRUB (grub-install<\/code>, update-grub<\/code>) so that coltrane<\/code> boots itself, not a memory of zappa<\/code>.<\/p>\n\n\n\n

The replication itself is a set of per-filesystem rsync -a --delete --one-file-system<\/code> passes, pushed from<\/em> zappa<\/code> to coltrane<\/code> so that data only ever flows one way, with an exclude list for the things that must never be cloned (virtual-machine images, transient mount points, and so on).<\/p>\n\n\n\n

The UUID that would not die<\/h2>\n\n\n\n

For the better part of a week, coltrane<\/code> kept booting with zappa<\/code>‘s<\/em> root UUID instead of its own. Every configuration file I could grep looked correct \u2014 the live grub.cfg<\/code> pointed at coltrane<\/code>‘s disk, \/proc\/cmdline<\/code> confirmed it \u2014 and yet a fresh clone would quietly boot the wrong disk again. There turned out to be two hiding places, neither of them an ordinary config file.<\/p>\n\n\n\n

First, the UUID was baked into GRUB’s EFI core image<\/strong> \u2014 the compiled grubx64.efi<\/code> binary itself, not any .cfg<\/code> file. A grub-install<\/code> run in the distant past, while the root disk was momentarily misidentified, had embedded zappa<\/code>‘s UUID directly into the bootloader binary. No amount of editing text files touches that.<\/p>\n\n\n\n

Second, and more insidious, there was an orphan bootloader config<\/strong> sitting in a non-standard directory on the EFI system partition (\/boot\/efi\/grub\/grub.cfg<\/code>) \u2014 a full, stale menu that neither update-grub<\/code> nor grub-install<\/code> ever writes to, and that the clone never touches because the EFI partition isn’t part of the rsync. It just sat there, quietly booting zappa<\/code>‘s UUID on every cycle, while every tool I ran \u201cfixed\u201d a different file.<\/p>\n\n\n\n

The cure for the second one is the kind of thing that only looks obvious afterwards: turn that orphan into a stub<\/em> that simply chains to the live, always-current config, so it no longer matters which config GRUB loads \u2014 they all end up at coltrane<\/code>‘s real grub.cfg<\/code>. That’s clone-proof, because a stub carries no UUID of its own.<\/p>\n\n\n\n

The lesson I took from this: with UEFI, the thing that actually boots your machine can live in more places than you’d think, and some of them are binaries, not text. When a boot problem survives every edit you make, you’re editing the wrong file.<\/p>\n\n\n\n

Deleting what the clone should delete<\/h2>\n\n\n\n

A subtler wrinkle: rsync’s --delete<\/code> is deliberately conservative. It only removes files it has actively accounted for during the comparison, and \u2014 importantly \u2014 an --exclude<\/code> rule doesn’t merely hide a file on the sender, it also protects<\/em> the matching file on the receiver from deletion. So when a directory disappears on zappa<\/code> but coltrane<\/code> still holds excluded files inside it, rsync can’t empty the directory and leaves it standing. For a clone that’s meant to be faithful, that’s a slow accumulation of cruft. I handle it deliberately rather than automatically: a small interactive tool discovers the stuck directories from rsync’s own output and, in a --why<\/code> mode, shows me exactly which exclude rule is protecting each one before I remove anything by hand.<\/p>\n\n\n\n

The power failure<\/h2>\n\n\n\n

Then, one day, the power died in the middle of a clone.<\/p>\n\n\n\n

This is the genuinely nasty case, because a power failure isn’t a clean error \u2014 nothing exits with a non-zero code, no trap fires, no flag gets cleared. The process simply vanishes mid-write. coltrane<\/code> was left with zappa<\/code>‘s boot stack half-written and the re-fit step never run: exactly the unbootable state described above, and now with nothing left running to notice.<\/p>\n\n\n\n

Recovering it by hand, remotely, through a network-KVM was an education in itself. GRUB dropped to a bare shell and greeted me with invalid sector size 65535<\/code> \u2014 which, it turned out, wasn’t coltrane<\/code>‘s disk at all, but the remote repair image I’d attached was presenting bogus geometry that GRUB choked on while enumerating devices. Detaching that image made the error disappear. From the GRUB shell I could then point at coltrane<\/code>‘s real disk and boot it by hand:<\/p>\n\n\n\n

set root=(hd0,gpt2)\nlinux \/boot\/vmlinuz-...  root=\/dev\/sda2 ro\ninitrd \/boot\/initrd.img-...\nboot<\/code><\/pre>\n\n\n\n
\u2014 with a detour to shrink GRUB’s video mode (set gfxmode=800x600<\/code>), because the KVM was only showing me a cropped corner of the screen.<\/p>\n\n\n\n
Making interruption survivable<\/h2>\n\n\n\n
Getting coltrane<\/code> back was satisfying; never wanting to do it again by hand was the real outcome. The scripts now assume a clone can be interrupted at any instant, and are built so that the next boot<\/em> heals itself:<\/p>\n\n\n\n