The system is spread across two locations: a main site in Eibergen and a secondary site in Leiden. Each site reaches the Internet through a standard, programmable gateway, in turn backed by a 1 Gbps fibre connection \u2014 from two different ISPs.<\/p>\n\n\n\n
The main site consists of two machines:<\/p>\n\n\n\n
zappa<\/code>: the main machine, running the latest LTS release of Ubuntu on an Intel i7-14700K with 64\u00a0GB of memory and a decently large SSD. It hosts several websites (including this one), a Nextcloud instance, a mail server, and the machinery for serving the personalized copies of my books. Important to me: zappa<\/code> also runs an instance of the Lyrion Music Server (LMS), which has to stay in sync with the LMS in Leiden. On top of that, it drives the on\/off switches for various devices \u2014 more on that later.<\/li>\n\n\n\nvarese<\/code>: an always-on Pi\u00a04 running Debian. It makes incremental backups of the important data on zappa<\/code>, and manages the site as a whole: updating DNS records when my ISP hands out a new IP address and rebooting the gateway when that seems called for. For home automation, it also hosts a Docker-based instance of the Homey server.<\/li>\n<\/ul>\n\n\n\nThe secondary site mirrors the main one closely:<\/p>\n\n\n\n
\ncoltrane<\/code>: the warm standby for zappa<\/code>. In most respects, it’s a genuine clone of zappa<\/code>, designed to take over when the main site is unavailable for any length of time. It’s a relatively old machine, with a much slower CPU, less memory, and a slower disk.<\/li>\n\n\n\nhorton<\/code>: another always-on Pi\u00a04 running Debian, very similar to varese<\/code>. It makes the same incremental backups of zappa<\/code>, at the same cadence and handles DNS updates when the Leiden ISP changes my IP address. And, again for home automation, it hosts its own Docker-based Homey instance.<\/li>\n\n\n\nbeefheart<\/code>: a Pi 4 whose main job is to run an instance of LMS and serve music. It talks to zappa<\/code> to keep the two music databases in sync, and can be switched off when it isn’t needed.<\/li>\n<\/ul>\n\n\n\n(By now you may have guessed that I have a soft spot for experimental music \u2014 classical, popular, or jazz.)<\/p>\n\n\n\n
The two gateways maintain a permanent VPN tunnel, so the servers at one site can reach those at the other without opening any external ports on either gateway. The only ports exposed to the outside world are for HTTP(S) and mail. As admin I can only log in to the servers when I’m on a LAN local to one of the two sites; the same goes for managing the gateways themselves. From anywhere else, I first set up a VPN connection to one of the sites through its gateway.<\/p>\n\n\n\n
One note: to properly monitor\u00a0zappa<\/code> and coltrane<\/code>, each is attached to a network-KVM allowing me to use my browser to see what’s going on even during boots<\/em>. It has proven to be a life-saver.<\/p>\n\n\n\nA simple case: keeping music in sync<\/h2>\n\n\n\n
I’m either in Leiden, in Eibergen, or somewhere else \u2014 but never in two places at once. My music library is almost entirely digital, from (legal!) downloads or from ripping my own CDs and LPs, and wherever I am I want all of it available. Every change I make sets a flag recording where<\/em> the change was made. Once a day the two sites rsync automatically (and if need be I trigger the sync by hand).<\/p>\n\n\n\nA change made in Leiden causes beefheart<\/code> to set the flag on zappa<\/code>, having mounted it over sshfs; a change in Eibergen sets a flag on zappa<\/code> directly. When zappa<\/code> sees either flag set, it starts the rsync \u2014 powering up beefheart<\/code> first if necessary. (If both<\/em> flags are set, I’ve done something wrong, and I get a warning to sort it out manually.) If beefheart<\/code> had to be woken, it goes back to sleep once the sync is done; it hangs off a remote-controllable power switch.<\/p>\n\n\n\nA happy side effect: I never have to worry about backing up my music. It simply exists, in full, at two locations.<\/p>\n\n\n\n
Incremental backups<\/h2>\n\n\n\n
I back up the important data on zappa<\/code> to varese<\/code> and horton automatically and incrementally. The schedule is: once an hour for 7 days, then daily for another 7 days, then weekly for the rest of the year. Old backups are pruned automatically once a day. In the same spirit, I back up my current mailbox incrementally every 5 minutes for up to 2 hours \u2014 by which point a regular backup will have caught it anyway.<\/p>\n\n\n\nBeyond this particular scheme, I’ve been backing up my own data automatically for the past twenty years or so, quite independently of whatever my university was doing. It has proved extremely robust, and extremely valuable.<\/p>\n\n\n\n
The trick is making the backing-up itself<\/em> robust, which is not obvious when Pi4s are involved. I’ll come back to that later.<\/p>\n","protected":false},"excerpt":{"rendered":"The system is spread across two locations: a main site in Eibergen and a secondary site in Leiden. Each site reaches the Internet through a standard, programmable gateway, in turn Continue Reading \u2192<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-1204","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages\/1204","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/comments?post=1204"}],"version-history":[{"count":11,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages\/1204\/revisions"}],"predecessor-version":[{"id":1246,"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/pages\/1204\/revisions\/1246"}],"wp:attachment":[{"href":"https:\/\/www.distributed-systems.net\/index.php\/wp-json\/wp\/v2\/media?parent=1204"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}