{"id":1208,"date":"2026-07-27T16:50:26","date_gmt":"2026-07-27T14:50:26","guid":{"rendered":"https:\/\/www.distributed-systems.net\/?page_id=1208"},"modified":"2026-07-28T08:58:07","modified_gmt":"2026-07-28T06:58:07","slug":"the-hard-part-robust-solutions","status":"publish","type":"page","link":"https:\/\/www.distributed-systems.net\/index.php\/the-hard-part-robust-solutions\/","title":{"rendered":"The hard part: robust solutions"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">What I&#8217;ve described so far is, functionally, fairly simple. The difficulty is making it all keep working <em>despite<\/em> failures. A few things to keep in mind:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>External disks fail \u2014 if only because you sometimes can&#8217;t mount them.<\/li>\n\n\n\n<li>Mounting a remote server over sshfs can fail too.<\/li>\n\n\n\n<li>Servers talk to each other over a secured point-to-point link between the two gateways \u2014 in my case far less robust than I&#8217;d expected.<\/li>\n\n\n\n<li>Gateways fail. Mine in Eibergen would hang every now and then, and only a restart would clear it.<\/li>\n\n\n\n<li>Power outages happen. They will.<\/li>\n\n\n\n<li>Cloning a server is not as simple as you&#8217;d first think. Cloning <code>zappa<\/code> to <code>coltrane<\/code> turned out to be a tough one.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The largest part of every gluing script turned out to be the code for handling failures; some scripts are <em>nothing but<\/em> failure handling. Let me walk through a few examples.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">External disks<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I use an 8&nbsp;TB external disk attached to <code>zappa<\/code> for audio and video. For reasons still entirely unclear to me, it would occasionally become inaccessible. Sometimes a remount fixed it; sometimes I had to remove it and power-cycle it. No fun when you&#8217;re not there. The solution was to put it on a remotely controllable power switch and let the server watch its behavior: try a remount automatically, and if that fails, have <code>zappa<\/code> switch the disk off, power it back up, and wait a few tens of seconds for it to be detected before remounting.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For the disks on <code>varese<\/code> and <code>horton<\/code> I take no special measures beyond the occasional manual fix. They&#8217;ve been reliable so far, and a simple reboot usually clears any odd behavior. More to the point: with two separate servers making the same incremental backups, there&#8217;s no reason to panic when one of them misbehaves.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Failed server mounts<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">When a server can&#8217;t be mounted, it&#8217;s almost always the VPN link between the gateways that&#8217;s at fault. It&#8217;s rare, and when it happens the cause is usually the network. In those cases I simply get a notification mail and the script that needed the mount stops gracefully. Since many of the scripts run periodically \u2014 every 5 minutes \u2014 I often see the problem cure itself. As I put it in my farewell lecture, computers are remarkably good at the art of stopping \u2014 and, it must be said, at producing intermittent errors.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Misbehaving gateways and power outages<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The thing to keep in mind is not just that the power can go out, but that it can come <em>back<\/em> \u2014 automatically. When it does, gateways, servers, disks, and switches all need to recover gracefully. So I&#8217;ve made sure the gateways come back to life on their own, along with a separate control unit at each site that drives the controllable power switches. That unit also lets me restore states by hand if it ever comes to that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I&#8217;ve made sure that the power switch feeding a gateway always returns to \u201con\u201d after an outage. Without the gateway up, there&#8217;s very little I can do from a distance.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The Eibergen gateway was a recurring troublemaker: only a reboot <em>after<\/em> cutting its power would fix it. I handed that job to <code>varese<\/code>. You notice something&#8217;s wrong when <code>zappa<\/code> can no longer be reached \u2014 at which point I start getting mail from horton, which monitors connectivity to Eibergen. Note that <code>horton<\/code> can&#8217;t generally send these messages through the mail server running on <code>zappa<\/code>, so I&#8217;m using a separate, external e-mail address provided by the Leiden ISP. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A hanging gateway means the Eibergen servers themselves are still up, so there&#8217;s a chance of fixing it from the inside. <code>varese<\/code> watches whether it can reach a few high-availability sites; if it can&#8217;t, it concludes the gateway is misbehaving and works through a series of reboot attempts. (Note that <em>pinging<\/em> the gateway tells you nothing useful \u2014 it can answer pings while forwarding nothing.) Eventually <code>varese<\/code> gives up, because the fault may simply be that the Eibergen ISP link is down. One key design point: <code>varese<\/code> must not depend on anything that assumes the gateway is working. The answer is a separate, independent Zigbee link to the gateway&#8217;s power switch.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">An unreliable point-to-point VPN link<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A secured point-to-point link has one awkward property: timing. In my case the two gateways had a rather poor way of keeping the link alive \u2014 each would tear it down after a second or so of idleness, then go through the whole song and dance of bringing it back up the moment it was needed again. The result was latency that fluctuated wildly. It took a while to find the root cause, but once I had, the fix was easy: <code>varese<\/code> now pings the Leiden gateway often enough that the link never goes idle in the first place.<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"<p>What I&#8217;ve described so far is, functionally, fairly simple. The difficulty is making it all keep working despite failures. A few things to keep in mind: The largest part of <a class=\"more-link\" href=\"https:\/\/www.distributed-systems.net\/index.php\/the-hard-part-robust-solutions\/\">Continue Reading \u2192<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-1208","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages\/1208","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/comments?post=1208"}],"version-history":[{"count":5,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages\/1208\/revisions"}],"predecessor-version":[{"id":1249,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages\/1208\/revisions\/1249"}],"wp:attachment":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/media?parent=1208"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}