What I’ve described so far is, functionally, fairly simple. The difficulty is making it all keep working despite failures. A few things to keep in mind:
- External disks fail — if only because you sometimes can’t mount them.
- Mounting a remote server over sshfs can fail too.
- Servers talk to each other over a secured point-to-point link between the two gateways — in my case far less robust than I’d expected.
- Gateways fail. Mine in Eibergen would hang every now and then, and only a restart would clear it.
- Power outages happen. They will.
- Cloning a server is not as simple as you’d first think. Cloning
zappatocoltraneturned out to be a tough one.
The largest part of every gluing script turned out to be the code for handling failures; some scripts are nothing but failure handling. Let me walk through a few examples.
External disks
I use an 8 TB external disk attached to zappa for audio and video. For reasons still entirely unclear to me, it would occasionally become inaccessible. Sometimes a remount fixed it; sometimes I had to remove it and power-cycle it. No fun when you’re not there. The solution was to put it on a remotely controllable power switch and let the server watch its behavior: try a remount automatically, and if that fails, have zappa switch the disk off, power it back up, and wait a few tens of seconds for it to be detected before remounting.
For the disks on varese and horton I take no special measures beyond the occasional manual fix. They’ve been reliable so far, and a simple reboot usually clears any odd behavior. More to the point: with two separate servers making the same incremental backups, there’s no reason to panic when one of them misbehaves.
Failed server mounts
When a server can’t be mounted, it’s almost always the VPN link between the gateways that’s at fault. It’s rare, and when it happens the cause is usually the network. In those cases I simply get a notification mail and the script that needed the mount stops gracefully. Since many of the scripts run periodically — every 5 minutes — I often see the problem cure itself. As I put it in my farewell lecture, computers are remarkably good at the art of stopping — and, it must be said, at producing intermittent errors.
Misbehaving gateways and power outages
The thing to keep in mind is not just that the power can go out, but that it can come back — automatically. When it does, gateways, servers, disks, and switches all need to recover gracefully. So I’ve made sure the gateways come back to life on their own, along with a separate control unit at each site that drives the controllable power switches. That unit also lets me restore states by hand if it ever comes to that.
I’ve made sure that the power switch feeding a gateway always returns to “on” after an outage. Without the gateway up, there’s very little I can do from a distance.
The Eibergen gateway was a recurring troublemaker: only a reboot after cutting its power would fix it. I handed that job to varese. You notice something’s wrong when zappa can no longer be reached — at which point I start getting mail from horton, which monitors connectivity to Eibergen. Note that horton can’t generally send these messages through the mail server running on zappa, so I’m using a separate, external e-mail address provided by the Leiden ISP.
A hanging gateway means the Eibergen servers themselves are still up, so there’s a chance of fixing it from the inside. varese watches whether it can reach a few high-availability sites; if it can’t, it concludes the gateway is misbehaving and works through a series of reboot attempts. (Note that pinging the gateway tells you nothing useful — it can answer pings while forwarding nothing.) Eventually varese gives up, because the fault may simply be that the Eibergen ISP link is down. One key design point: varese must not depend on anything that assumes the gateway is working. The answer is a separate, independent Zigbee link to the gateway’s power switch.
An unreliable point-to-point VPN link
A secured point-to-point link has one awkward property: timing. In my case the two gateways had a rather poor way of keeping the link alive — each would tear it down after a second or so of idleness, then go through the whole song and dance of bringing it back up the moment it was needed again. The result was latency that fluctuated wildly. It took a while to find the root cause, but once I had, the fix was easy: varese now pings the Leiden gateway often enough that the link never goes idle in the first place.