Cloning zappa onto coltrane sounds like it should be the easy part — copy the files across and you’re done. Copying the files is indeed easy. The hard part is making the copy actually boot, on hardware that isn’t zappa‘s.
Why a clone is not a copy
zappa boots from an NVMe drive; coltrane boots from an older SATA disk. A running Linux system carries a boot stack — the initramfs and the GRUB bootloader — that is tailored to the machine it was installed on: which disks exist, how they’re numbered, and, crucially, the UUID of the root filesystem baked into the bootloader. rsync happily copies all of that from zappa to coltrane, at which point coltrane is holding zappa‘s boot stack on coltrane‘s hardware. It won’t come up.
So every clone is really two steps: replicate the filesystems, then re-fit the boot stack for coltrane — rebuild the initramfs (update-initramfs) and reinstall GRUB (grub-install, update-grub) so that coltrane boots itself, not a memory of zappa.
The replication itself is a set of per-filesystem rsync -a --delete --one-file-system passes, pushed from zappa to coltrane so that data only ever flows one way, with an exclude list for the things that must never be cloned (virtual-machine images, transient mount points, and so on).
The UUID that would not die
For the better part of a week, coltrane kept booting with zappa‘s root UUID instead of its own. Every configuration file I could grep looked correct — the live grub.cfg pointed at coltrane‘s disk, /proc/cmdline confirmed it — and yet a fresh clone would quietly boot the wrong disk again. There turned out to be two hiding places, neither of them an ordinary config file.
First, the UUID was baked into GRUB’s EFI core image — the compiled grubx64.efi binary itself, not any .cfg file. A grub-install run in the distant past, while the root disk was momentarily misidentified, had embedded zappa‘s UUID directly into the bootloader binary. No amount of editing text files touches that.
Second, and more insidious, there was an orphan bootloader config sitting in a non-standard directory on the EFI system partition (/boot/efi/grub/grub.cfg) — a full, stale menu that neither update-grub nor grub-install ever writes to, and that the clone never touches because the EFI partition isn’t part of the rsync. It just sat there, quietly booting zappa‘s UUID on every cycle, while every tool I ran “fixed” a different file.
The cure for the second one is the kind of thing that only looks obvious afterwards: turn that orphan into a stub that simply chains to the live, always-current config, so it no longer matters which config GRUB loads — they all end up at coltrane‘s real grub.cfg. That’s clone-proof, because a stub carries no UUID of its own.
The lesson I took from this: with UEFI, the thing that actually boots your machine can live in more places than you’d think, and some of them are binaries, not text. When a boot problem survives every edit you make, you’re editing the wrong file.
Deleting what the clone should delete
A subtler wrinkle: rsync’s --delete is deliberately conservative. It only removes files it has actively accounted for during the comparison, and — importantly — an --exclude rule doesn’t merely hide a file on the sender, it also protects the matching file on the receiver from deletion. So when a directory disappears on zappa but coltrane still holds excluded files inside it, rsync can’t empty the directory and leaves it standing. For a clone that’s meant to be faithful, that’s a slow accumulation of cruft. I handle it deliberately rather than automatically: a small interactive tool discovers the stuck directories from rsync’s own output and, in a --why mode, shows me exactly which exclude rule is protecting each one before I remove anything by hand.
The power failure
Then, one day, the power died in the middle of a clone.
This is the genuinely nasty case, because a power failure isn’t a clean error — nothing exits with a non-zero code, no trap fires, no flag gets cleared. The process simply vanishes mid-write. coltrane was left with zappa‘s boot stack half-written and the re-fit step never run: exactly the unbootable state described above, and now with nothing left running to notice.
Recovering it by hand, remotely, through a network-KVM was an education in itself. GRUB dropped to a bare shell and greeted me with invalid sector size 65535 — which, it turned out, wasn’t coltrane‘s disk at all, but the remote repair image I’d attached was presenting bogus geometry that GRUB choked on while enumerating devices. Detaching that image made the error disappear. From the GRUB shell I could then point at coltrane‘s real disk and boot it by hand:
set root=(hd0,gpt2)
linux /boot/vmlinuz-... root=/dev/sda2 ro
initrd /boot/initrd.img-...
boot— with a detour to shrink GRUB’s video mode (set gfxmode=800x600), because the KVM was only showing me a cropped corner of the screen.
Making interruption survivable
Getting coltrane back was satisfying; never wanting to do it again by hand was the real outcome. The scripts now assume a clone can be interrupted at any instant, and are built so that the next boot heals itself:
- A “clone in progress” marker is dropped on
coltrane‘s disk at the very start of a clone, and removed only after both the boot re-fit and a safety check have passed. A power failure can’t clear it — so if the marker is still there on the next boot,coltraneknows the previous clone was interrupted, shouts about it, and forces a fresh clone and re-fit rather than trusting the half-written system. - A self-heal check: even with no marker, if the live GRUB config doesn’t reference
coltrane‘s own root UUID,coltraneconcludes it has booted on a stale or foreign boot stack and re-fits itself — turning “unbootable until a human with a KVM shows up” into “sorts itself out on the next wake.” - A safety gate:
coltranerefuses to power itself back off unless it has verified that GRUB really does point at its own root. It will never put itself to sleep in a state it can’t wake up from. - Because
coltrane‘s own mail and logs don’t survive a clone-and-power-off, all of this alerts and logs viazappa, whose disk is stable acrosscoltrane‘s cycles.
What this actually is
Stepping back: what I’ve built is a manual-failover cold standby with tiered replication — a daily full clone for the bulk, hourly incrementals, and much tighter syncs for the handful of things I genuinely can’t lose. That’s disaster recovery, not high availability, and the distinction matters. I’m deliberately trading a little downtime and a human in the loop for a system I can fully understand and repair myself. Real HA — clustered databases, synchronously replicated storage, automatic failover — would be far more machinery and many more ways to fail silently, to buy me minutes I don’t need.
The interesting consequence is that, once the mechanics are hardened, the risk moves. It stops being about rsync or GRUB and becomes operational: a failover path that is exercised almost never will quietly rot. coltrane sleeps and drifts, a promote script references a path that has since moved, a certificate expires while the standby was asleep and couldn’t renew. A recovery plan you’ve never actually run end to end isn’t a plan — it’s a hypothesis. So the single most valuable habit here has nothing to do with code: every so often, actually cut over to coltrane, run on it for a while, and fail back — on a calm afternoon of my own choosing, not during a real outage. Everything that teaches me is something I don’t have to learn the hard way.