Category Archives: Technology

Make the Internet Yours Again With an Instant Mesh Network

I’m going to lead with the technical punch line, and then explain it:

Yggdrasil Network is an opportunistic mesh that can be deployed privately or as part of a global-scale network. Each node gets a stable IPv6 address (or even an entire /64) that is derived from its public key and is bound to that node as long as the node wants it (of course, it can generate a new keypair anytime) and is valid wherever the node joins the mesh. All traffic is end-to-end encrypted.

Yggdrasil will automatically discover peers on a LAN via broadcast beacons, and requires zero configuration to peer in such a way. It can also run as an overlay network atop the public Internet. Public peers serve as places to join the global network, and since it’s a mesh, if one device on your LAN joins the global network, the others will automatically have visibility on it also, thanks to the mesh routing.

It neatly solves a lot of problems of portability (my ssh sessions stay live as I move networks, for instance), VPN (incoming ports aren’t required since local nodes can connect to a public peer via an outbound connection), security, and so forth.

Now on to the explanation:

The Tyranny of IP rigidity

Every device on the Internet, at one time, had its own globally-unique IP address. This number was its identifier to the world; with an IP address, you can connect to any machine anywhere. Even now, when you connect to a computer to download a webpage or send a message, under the hood, your computer is talking to the other one by IP address.

Only, now it’s hard to get one. The Internet protocol we all grew up with, version 4 (IPv4), didn’t have enough addresses for the explosive growth we’ve seen. Internet providers and IT departments had to use a trick called NAT (Network Address Translation) to give you a sort of fake IP address, so they could put hundreds or thousands of devices behind a single public one. That, plus the mobility of devices — changing IPs whenever they change locations — has meant that a fundamental rule of the old Internet is now broken:

Every participant is an equal peer. (Well, not any more.)

Nowadays, you can’t you host your own website from your phone. Or share files from your house. (Without, that is, the use of some third-party service that locks you down and acts as an intermediary.)

Back in the 90s, I worked at a university, and I, like every other employee, had a PC on my desk with an unfirewalled public IP. I installed a webserver, and poof – instant website. Nowadays, running a website from home is just about impossible. You may not have a public IP, and if you do, it likely changes from time to time. And even then, your ISP probably blocks you from running servers on it.

In short, you have to buy your way into the resources to participate on the Internet.

I wrote about these problems in more detail in my article Recovering Our Lost Free Will Online.

Enter Yggdrasil

I already gave away the punch line at the top. But what does all that mean?

  • Every device that participates gets an IP address that is fully live on the Yggdrasil network.
  • You can host a website, or a mail server, or whatever you like with your Yggdrasil IP.
  • Encryption and authentication are smaller (though not nonexistent) worries thanks to the built-in end-to-end encryption.
  • You can travel the globe, and your IP will follow you: onto a plane, from continent to continent, wherever. Yggdrasil will find you.
  • I’ve set up /etc/hosts on my laptop to use the Yggdrasil IPs for other machines on my LAN. Now I can just “ssh foo” and it will work — from home, from a coffee shop, from a 4G tether, wherever. Now, other tools like tinc can do this, obviously. And I could stop there; I could have a completely closed, private Yggdrasil network.

    Or, I can join the global Yggdrasil network. Each device, in addition to accepting peers it finds on the LAN, can also be configured to establish outbound peering connections or accept inbound ones over the Internet. Put a public peer or two in your configuration and you’ve joined the global network. Most people will probably want to do that on every device (because why not?), but you could also do that from just one device on your LAN. Again, there’s no need to explicitly build routes via it; your other machines on the LAN will discover the route’s existence and use it.

    This is one of many projects that are working to democratize and decentralize the Internet. So far, it has been quite successful, growing to over 2000 nodes. It is the direct successor to the earlier cjdns/Hyperboria and BATMAN networks, and aims to be a proof of concept and a viable tool for global expansion.

    Finally, think about how much easier development is when you don’t have to necessarily worry about TLS complexity in every single application. When you don’t have to worry about port forwarding and firewall penetration. It’s what the Internet should be.

    Managing an External Display on Linux Shouldn’t Be This Hard

    I first started using Linux and FreeBSD on laptops in the late 1990s. Back then, there were all sorts of hassles and problems, from hangs on suspend to pure failure to boot. I still worry a bit about suspend on unknown hardware, but by and large, the picture of Linux on laptops has dramatically improved over the last years. So much so that now I can complain about what would once have been a minor nit: dealing with external monitors.

    I have a USB-C dock that provides both power and a Thunderbolt display output over the single cable to the laptop. I think I am similar to most people in wanting the following behavior from the laptop:

    • When the lid is closed, suspend if no external monitor is connected. If an external monitor is connected, shut off the built-in display and use the external one exclusively, but do not suspend.
    • Lock the screen automatically after a period of inactivity.
    • While locked, all connected displays should be powered down.
    • When an external display is connected, begin using it automatically.
    • When an external display is disconnected, stop using it. If the lid is closed when the external display is disconnected, go into suspend mode.

    This sounds so simple. But somehow on Linux we’ve split up these things into a dozen tiny bits:

    • In /etc/systemd/logind.conf, there are settings about what to do when the lid is opened or closed.
    • Various desktop environments have overlapping settings covering the same things.
    • Then there are the display managers (gdm3, lightdm, etc) that also get in on the act, and frequently have DIFFERENT settings, set in different places, from the desktop environments. And, what’s more, they tend to be involved with locking these days.
    • Then there are screensavers (gnome-screensaver, xscreensaver, etc.) that also enter the picture, and also have settings in these areas.

    Problems I’ve Seen

    My problems don’t even begin with laptops, but with my desktop, running XFCE with xmonad and lightdm. My desktop is hooked to a display that has multiple inputs. This scenario (reproducible in both buster and bullseye) causes the display to be unusable until a reboot on the desktop:

    1. Be logged in and using the desktop
    2. Without locking the desktop screen, switch the display input to another device
    3. Keep the display input on another device long enough for the desktop screen to auto-lock
    4. At this point, it is impossible to re-awaken the desktop screen.

    I should not here that the problems aren’t limited to Debian, but also extend to Ubuntu and various hardware.

    Lightdm: which greeter?

    At some point while troubleshooting things after upgrading my laptop to bullseye, I noticed that while both were running lightdm, I had different settings and a different appearance between the two. Upon further investigation, I realized that one hat slick-greeter and lightdm-settings installed, while the other had lightdm-gtk-greeter and lightdm-gtk-greeter-settings installed. Very strange.

    XFCE: giving up

    I eventually gave up on making lightdm work. No combination of settings or greeters would make things work reliably when changing screen configurations. I installed xscreensaver. It doesn’t hang, but it does sometimes take a few tries before it figures out what device to display on.

    Worse, since updating from buster to bullseye, XFCE no longer automatically switches audio output when the docking station is plugged in, and there seems to be no easy way to convince Pulseaudio to do this.

    X-Based Gnome and derivatives… sigh.

    I also tried Gnome, Mate, and Cinnamon, and all of them had various inabilities to configure things to act the way I laid out above.

    I’ve long not been a fan of Gnome’s way of hiding things from the user. It now has a Windows-like situation of three distinct settings programs (settings, tweaks, and dconf editor), which overlap in strange ways and interact with systemd in even stranger ways. Gnome 3 make it quite non-intuitive to make app icons from various programs work, and so forth.

    Trying Wayland

    I recently decided to set up an older laptop that I hadn’t used in awhile. After reading up on Wayland, I decided to try Gnome 3 under Wayland. Both the Debian and Arch wikis note that KDE is buggy on Wayland. Gnome is the only desktop environment that supports it then, unless I want to go with Sway. There’s some appeal to Sway to this xmonad user, but I’ve read of incompatibilities of Wayland software when Gnome’s not available, so I opted to try Gnome.

    Well, it’s better. Not perfect, but better. After finding settings buried in a ton of different Settings and Tweaks boxes, I had it mostly working, except gdm3 would never shut off power to the external display. Eventually I found /etc/gdm3/greeter.dconf-defaults, and aadded:

    sleep-inactive-ac-timeout=60
    sleep-inactive-ac-type='blank'
    sleep-inactive-battery-timeout=120
    sleep-inactive-battery-type='suspend'
    

    Of course, these overlap with but are distinct from the same kinds of things in Gnome settings.

    Sway?

    Running without Gnome seems like a challenge; Gnome is switching audio output appropriately, for instance. I am looking at some of the Gnome Shell tiling window manager extensions and hope that some of them may work for me.

    Excellent Experience with Debian Bullseye

    I’ve appreciated the bullseye upgrade, like most Debian upgrades. I’m not quite sure how, since I was already running a backports kernel, but somehow the entire system is snappier. Maybe newer X or something? I’m really pleased with it. Hardware integration is even nicer now, particularly the automatic driverless support for scanners in addition to the existing support for printers.

    All in all, a very nice upgrade, and pretty painless.

    I experienced a few odd situations.

    For one, I had been using Gnome Flashback. Since xmonad-log-applet didn’t compile there (due to bitrot in the log applet, not flashback), and I had been finding Gnome Flashback to be a rather dusty and forgotten corner of Gnome for a long time, I decided to try Mate.

    Mate just seemed utterly unable to handle a situation with a laptop and an external monitor very well. I want to use only the external monitor with the laptop lid is closed, and it just couldn’t remember how to do the right thing – external monitor on, laptop monitor off, laptop not put into suspend. gdm3 also didn’t seem to be able to put the external monitor to sleep, either, causing a few nights of wasted power.

    So off I went to XFCE, which I had been using for years on my workstation anyhow. Lots more settings available in XFCE, plus things Just Worked there. Odd that XFCE, the thin and light DE, is now the one that has the most relevant settings. It seems the Gnome “let’s remove a bunch of features” approach has extended to MATE as well.

    When I switched to XFCE, I also removed gdm3 from my system, leaving lightdm as the only DM on it. That matched what my desktop machine was using, and also what task-xfce-desktop called for. But strangely, the XFCE settings for lightdm were completely different between the laptop and the desktop. It turns out that with lightdm, you can have the lightdm-gtk-greeter and the accompanying lightdm-gtk-greeter-settings, or slick-greeter and the accompanying lightdm-settings. One machine had one greeter and settings, and the other had the other. Why, I don’t know. But lightdm-gtk-greeter-settings had the necessary options for putting monitors to sleep on the login screen, so I went with it.

    This does highlight a bit of a weakness in Debian upgrades. There is SO MUCH choice in Debian, which I highly value. At some point, almost certainly without my conscious choice, one machine got one greeter and another got the other. Despite both having task-xfce-desktop installed, they got different desktop experiences. There isn’t a great way to say “OK, I know I had a bunch of things installed before, but NOW I want the default bullseye experience”.

    But overall, it is an absolutely fantastic distribution. It is great to see this nonprofit community distribution continue to have such quality on such an immense scale. And hard to believe I’ve been a Debian developer for 25 years. That seems almost impossible!

    Distributed, Asynchronous Git Syncing with NNCP

    I have a problem.

    I have a directory that I use with org-mode and org-roam. I want it to be synced across multiple machines. I also want to keep the history with git. And, I want to use end-to-end encryption (no storing a plain git repo on a remote server), have a serverless setup, not require any two machines to be up simultaneously, and be resilient in the face of races and conflicts.

    Whew.

    I’ve tried a number of setups – git-remote-gcrypt on a remote server (fragile), some complicated scripts around a separate repo in syncthing (requires one machine to be “in charge”), etc. They all were subpar.

    Then NNCP introdoced asynchronous multicast and I was intrigued.

    So, I wrote gitsync-nncp, which uses NNCP to distribute git bundles to all the participating machines. The comprehensive documentation for gitsync-nncp goes into a lot more detail about how it works and what problems it solves. It’s working quite well for me!

    A Simple, Delay-Tolerant, Offline-Capable Mesh Network with Syncthing (+ optional NNCP)

    A little while back, I spent a week in a remote area. It had no Internet and no cell phone coverage. Sometimes, I would drive in to town where there was a signal to get messages, upload photos, and so forth. I had to take several devices with me: my phone, my wife’s, maybe a laptop or a tablet too. It seemed there should have been a better way. And there is.

    I’ll use this example to talk about a mesh network, but it could just as well apply to people wanting to communicate on a 12-hour flight that has no in-flight wifi, or spacecraft with an intermittent connection, or a person traveling.

    Syncthing makes a wonderful solution for things like these. Here are some interesting things about Syncthing:

    • You can think of Syncthing as a serverless, peer-to-peer, open source alternative to Dropbox. Machines sync directly with each other without a server, though you can add a server if you want.
    • It can operate completely without Internet access or any central server, though if Internet access is available, it can readily be used.
    • Syncthing devices connected to the same LAN or Wifi will detect each other’s presence and automatically communicate.
    • Syncthing is capable of handling a constantly-changing topology. It can also, for instance, handle two disconnected clusters of nodes with one node that “travels” between them — perhaps just a phone.
    • Syncthing scales from everything from a phone to thousands of nodes.
    • Syncthing normally performs syncs in every direction, but can also do single-direction syncs
    • An individual Syncthing node can register its interest or disinterest in certain files or directories based on filename patterns

    Syncthing works by having you define devices and folders. You can choose which devices to share folders with. A shared folder has an ID that is unique across Sycnthing. You can share a folder from device A to device B, and then device B can share it with device C, even if A and C don’t know about each other or have no way to communicate. More commonly, though, all the devices would know about each other and will opportunistically communicate the best way they can.

    Syncthing uses something akin to a Bittorrent protocol. Say you’re syncing videos from your phone, and they’re going to 3 machines. It doesn’t mean that Syncthing has to send it three times from the phone. Syncthing will send each block, most likely, just once; the other nodes in the swarm will register the block availability from the first other node to get it and will exchange blocks with themselves.

    Syncthing will typically look for devices on the local LAN. Failing that, it will use an introduction server to see if it can reach them directly using P2P. Failing that, perhaps due to restrictive firewalls or NAT, communication can be relayed through volunteer-run Syncthing servers on the Internet. All Syncthing communications are cryptographically encrypted and verified. You can also configure Syncthing arbitrarily; for instance, to run over ssh or Tor tunnels.

    So, let’s look at how Syncthing might help with the example I laid out up front.

    All the devices at the remote location could communicate with each other. The Android app is quite capable of syncing photos and videos using Syncthing, for instance. Then one device could be taken to the Internet location and it would transmit data on behalf of all the others – perhaps back to a computer at your home, or to a server somewhere. Perhaps a script running on the remote server would then move files out of the syncthing synced folder into permanent storage elsewhere, triggering a deletion to be sent to the phone to free up storage. When the phone gets back to the other devices, the deletion can be propagated to them to free up storage there too.

    Or maybe you have a computer out in a shed or somewhere without Internet access that you go to periodically, and need to get files to it. Again, your phone could be a carrier.

    Taking it a step further

    If you envision a file as a packet, you could, conceivably, do something like tunnel TCP/IP over Syncthing, assuming generous-enough timeouts. It can truly handle communication.

    But you don’t need TCP/IP for this. Consider some other things you could do:

    • Drop a script in a special directory that gets picked up by a remote server and run
    • Drop emails in a special directory that get transmitted and then deleted by a remote system when they’re seen
    • Drop files (eg, photos or videos) in a directory that a remote system will copy or move out of there
    • Drop messages (perhaps gpg-encrypted) — which could be text files — for someone to see and process.
    • Drop NNTP bundles for group communication

    You can start to see how there are a lot of possibilities here that extend beyond just file synchronization, though they are built upon a file synchronization tool.

    Enter NNCP

    Let’s look at a tool that’s especially suited for this: NNCP, which I’ve been writing about a lot lately.

    NNCP is designed to handle file exchange and remote execution with remote computers in an asynchronous, store-and-forward manner. NNCP packets are themselves encrypted and authenticated. NNCP traditionally is source-routed (that is, you configure it so that machine A reaches machine D by relaying through B and C), and the packets are onion-routed. NNCP packets can be exchanged by a TCP call, a tar-like stream, copying files to something like a USB stick and physically transporting it to the remote, etc.

    This works really well and I’ve been using it myself. But it gets complicated if the network topology isn’t fixed; it is difficult to reroute packets due to the onion routing, for instance. There are various workarounds that could be used — but why not just use Syncthing as a transport in those cases?

    nncp-xfer is the command that exchanges packets by writing them to, and reading them from, a directory. It is what you’d use to exchange packets on a USB stick. And what you’d use to exchange packets via Syncthing. It writes packets in a RECIPIENT/SENDER/PACKET directory structure, so it is perfectly fine to have multiple systems exchanging packets in a single Syncthing synced folder tree. This structure also allows leaf nodes to only carry the particular packets they’re interested in. The packets are all encrypted, so they can be freely synced wherever.

    Since Syncthing opportunistically syncs a shared folder with any device the folder is shared with, a phone could very easily be the NNCP transport, even if it has no idea what NNCP is. It could carry NNCP packets back and forth between sites, or to the Internet, or whatever.

    NNCP supports file transmission, file request, and remote execution, all subject to controls, of course. It is easy to integrate with Exim or Postfix to use as a mail transport, Git transport, and so forth. I use it for backups. It would be quite easy to have it send those backups (encrypted zfs send) via nncp-xfer to Syncthing instead of the usual method, and then if I’ve shared the Syncthing folder with my phone, all I need to do is bring the phone into Internet range and they get sent. nncp-xfer will normally remove the packets out of the xfer directory as it ingests them, so the space will only be consumed on the phone (and laptop) until we know the packets made it to their destination.

    Pretty slick, eh?

    Roundup of Secure Messengers with Off-The-Grid Capabilities (Distributed/Mesh Messengers)

    Amid all the conversation about Signal, and the debate over decentralization, one thing has often not been raised: all of these things require an Internet connection.

    “Of course,” you might say. “Internet is everywhere these days.” Well, not so much, and it turns out there are some very good reasons that people might want messengers that work offline. Here are some examples:

    • Internet-using messengers leak certain metadata (eg, that a person is using it, or perhaps a sophisticated adversary could use timing analysis to determine that two people are talking using it)
    • Cell signal outages due to natural disaster, large influx of people (protests, unusual sporting events, festivals, etc), or other factors
    • Locations where cell signals are not available (rural areas, camping locations, wilderness areas, etc.)
    • Devices that don’t have cell data capability (many tablets, phones that have had service expire, etc.)

    How do they work?

    These all use some form of local radio signal. Some, such as Briar, may use short-range Bluetooth and Wifi, while others use radios such as LoRa that can reach several miles with low power. I’ve written quite a bit about LoRa before, and its unique low-speed but extreme-distance radio capabilities even on low power.

    One common thread through these is that most of them are Android-only, though many are compatible with F-Droid and privacy-enhanced Android distributions.

    Every item on this list uses full end-to-end encryption (E2EE).

    Let’s dive on in.

    Briar

    Of all the options mentioned here, Briar is the one that bridges the traditional Internet-based approach with alternative options the best. It offers three ways for distributing data:

    • Over the Internet, via Tor onion services
    • Via Bluetooth to nearby devices
    • Via Wifi, to other devices connected to the same access point, even if Internet isn’t wokring on that AP

    As far as I can tell, there is no centralized server in Briar at all. Your “account”, such as it is, lives entirely within your device; if you wipe your device, you will have to make a new account and re-establish contacts. The use of Tor is also neat to see; it ensures that an adversary can’t tell, just from that, that you’re using Briar at all, though of course timing analysis may still be possible (and Bluetooth and Wifi uses may reval some of who is communicating).

    Briar features several types of messages (detailed in the manual), which really are just different spins on communication, which they liken to metaphors people are familiar with:

    • Basic 1-to-1 private messaging
    • “Private groups”, in which one particular person invites people to the chat group, and can dissolve it at any time
    • “Forums”, similar to private groups, but any existing member can invite more people to them, and they continue to exist until the last member leaves (founder isn’t special)
    • “Blogs”, messages that are automatically shared with all your contacts

    By default, Briar raises an audible notification for incoming messages of all types. This is configurable for each type.

    “Blogs” have a way to reblog (even a built-in RSS reader to facilitate that), but framed a different way, they are broadcast messages. They could, for instance, be useful for a “send help” message to everyone (assuming that people haven’t all shut off notifications of blogs due to others using them different ways).

    Briar’s how it works page has an illustration specifically of how blogs are distributed. I’m unclear on some of the details, and to what extent this applies to other kinds of messages, but one thing that you can notice from this is that a person A could write a broadcast message without Internet access, person B could receive it via Bluetooth or whatever, and then when person B gets Internet access again, the post could be distributed more widely. However, it doesn’t appear that Briar is really a full mesh, since only known contacts in the distribution path for the message would repeat it.

    There are some downsides to Briar. One is that, since an account is fully localized to a device, one must have a separate account for each device. That can lead to contacts having to pick a specific device to send a message to. There is an online indicator, which may help, but it’s definitely not the kind of seamless experience you get from Internet-only messengers. Also, it doesn’t support migrating to a new phone, live voice/video calls, or attachments, but attachments are in the works.

    All in all, a solid communicator, and is the only one on this list that works 100% with the hardware everyone already has. While Bluetooth and Wifi have far more limited range than the other entries, there is undeniably convenience in not needing any additional hardware, and it may be particularly helpful when extra bags/pockets aren’t available. Also, Briar is fully Open Source.

    Meshtastic

    Meshtastic is a radio-first LoRa mesh project. What do I mean by radio-first? Well, basically cell phones are how you interact with Meshtastic, but they are optional. The hardware costs about $30 and the batteries last about 8 days. Range between nodes is a few miles in typical conditions (up to 11km / 7mi in ideal conditions), but nodes act as repeaters, so it is quite conceivable to just drop a node “in the middle” if you and contacts will be far apart. The project estimates that around 2000 nodes are in operation, and the network is stronger the more nodes are around.

    The getting started site describes how to build one.

    Most Meshtastic device builds have a screen and some buttons. They can be used independently from the Android app to display received messages, distance and bearing to other devices (assuming both have a GPS enabled), etc. This video is an introduction showing it off, this one goes over the hardware buttons. So even if your phone is dead, you can at least know where your friends are. Incidentally, the phone links up to the radio board using Bluetooth, and can provide a location source if you didn’t include one in your build. There are ideas about solar power for Meshtastic devices, too.

    Meshtastic doesn’t, as far as I know, have an option for routing communication over the Internet, but the devices appear to be very thoughtfully-engineered and easy enough to put together. This one is definitely on my list to try.

    Ripple-based devices

    This is based on the LoRa Mesh Radio Instructables project, and is similar in concept to Meshtastic. It uses similar hardware, a similar app, but also has an option with a QWERTY hardware keyboard available, for those that want completely phone-free operation while still being able to send messages.

    There are a number of related projects posted at Instructables: a GPS tracker, some sensors, etc. These are variations on the same basic concept.

    These use the Ripple firmware, which is not open source, so I haven’t pursued it further.

    GoTenna

    For people that want less of a DIY model, and don’t mind proprietary solutions, there are two I’ll mention. The first is GoTenna Mesh, which is LoRa-based and sells units for $90 each. However, there are significant community concerns about the longevity of the project, as GoTenna has re-focused on government and corporate work. The Android app hasn’t been updated in 6 monnths despite a number of reviews citing issues, and the iOS app is also crusty.

    Beartooth

    Even more expensive at $125 each is the Beartooth. Also a proprietary option, I haven’t looked into it more, but they are specifically targetting backwoods types of markets.

    Do not use: Bridgefy

    Bridgefy was briefly prominent since it was used during the Hong Kong protests. However, numerous vulnerabilities have been demonstrated, and the developers have said they are re-working the app to address them. I wouldn’t recommend it for now.

    Alternatives: GMRS handhelds

    In the USA, GMRS voice handhelds are widely available. Although a license is required, it is simple (no exam) and cheap ($35) and extends to a whole family. GMRS radios also interoperate with FRS radios, which require no license and share some frequencies, but are limited to lower power (though are often sufficient).

    Handheld GMRS radios that use up to 5W of power are readily available. A voice signal is a lot harder to carry for a long distance than a very low-bandwidth digital one, so even with much more power you will probably not get the same kind of range you will with something like Meshtastic, and they don’t come with any kind of security or encryption at all. However, for basic communication, they are often a useful tool.

    Remote Directory Tree Comparison, Optionally Asynchronous and Airgapped

    Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.

    In the previous installment on store-and-forward backups, I mentioned how easy it is to do with ZFS, and some of the tools that can be used to do it without ZFS. A lot of those tools are a bit less robust, so we need some sort of store-and-forward mechanism to verify backups. To be sure, verifying backups is good with ANY scheme, and this could be used with ZFS backups also.

    So let’s say you have a shiny new backup scheme in place, and you’d like to verify that it’s working correctly. To do that, you need to compare the source directory tree on machine A with the backed-up directory tree on machine B.

    Assuming a conventional setup, here are some ways you might consider to do that:

    • Just copy everything from machine A to machine B and compare locally
    • Or copy everything from machine A to a USB drive, plug that into machine B, and compare locally
    • Use rsync in dry-run mode and see if it complains about anything

    The first two options are not particularly practical for large datasets, though I note that the second is compatible with airgapping. Using rsync requires both systems to be online at the same time to perform the comparison.

    What would be really nice here is a tool that would write out lots of information about the files on a system: their names, sizes, last modified dates, maybe even sha256sum and other data. This file would be far smaller than the directory tree itself, would compress nicely, and could be easily shipped to an airgapped system via NNCP, UUCP, a USB drive, or something similar.

    Tool choices

    It turns out there are already quite a few tools in Debian (and other Free operating systems) to do this, and half of them are named mtree (though, of course, not all mtrees are compatible with each other.) We’ll look at some of the options here.

    I’ve made a simple test directory for illustration purposes with these commands:

    mkdir test
    cd test
    echo hi > hi
    ln -s hi there
    ln hi foo
    touch empty
    mkdir emptydir
    mkdir somethingdir
    cd somethingdir
    ln -s ../there
    

    I then also used touch to set all files to a consistent timestamp for illustration purposes.

    Tool option: getfacl (Debian package: acl)

    This comes with the acl package, but can be used with other than ACL purposes. Unfortunately, it doesn’t come with a tool to directly compare its output with a filesystem (setfacl, for instance, can apply the permissions listed but won’t compare.) It ignores symlinks and doesn’t show sizes or dates, so is ineffective for our purposes.

    Example output:

    $ getfacl --numeric -R test
    ...
    # file: test/hi
    # owner: 1000
    # group: 1000
    user::rw-
    group::r--
    other::r--
    ...
    

    Tool option: fmtree, the FreeBSD mtree (Debian package: freebsd-buildutils)

    fmtree can prepare a “specification” based on a directory tree, and compare a directory tree to that specification. The comparison also is aware of files that exist in a directory tree but not in the specification. The specification format is a bit on the odd side, but works well enough with fmtree. Here’s a sample output with defaults:

    $ fmtree -c -p test
    ...
    # .
    /set type=file uid=1000 gid=1000 mode=0644 nlink=1
    .               type=dir mode=0755 nlink=4 time=1610421833.000000000
        empty       size=0 time=1610421833.000000000
        foo         nlink=2 size=3 time=1610421833.000000000
        hi          nlink=2 size=3 time=1610421833.000000000
        there       type=link mode=0777 time=1610421833.000000000 link=hi
    
    ... skipping ...
    
    # ./somethingdir
    /set type=file uid=1000 gid=1000 mode=0777 nlink=1
    somethingdir    type=dir mode=0755 nlink=2 time=1610421833.000000000
        there       type=link time=1610421833.000000000 link=../there
    # ./somethingdir
    ..
    
    ..
    

    You might be wondering here what it does about special characters, and the answer is that it has octal escapes, so it is 8-bit clean.

    To compare, you can save the output of fmtree to a file, then run like this:

    cd test
    fmtree < ../test.fmtree
    

    If there is no output, then the trees are identical. Change something and you get a line of of output explaining each difference. You can also use fmtree -U to change things like modification dates to match the specification.

    fmtree also supports quite a few optional keywords you can add with -K. They include things like file flags, user/group names, various tipes of hashes, and so forth. I'll note that none of the options can let you determine which files are hardlinked together.

    Here's an excerpt with -K sha256digest added:

        empty       size=0 time=1610421833.000000000 \
                    sha256digest=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        foo         nlink=2 size=3 time=1610421833.000000000 \
                    sha256digest=98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
    

    If you include a sha256digest in the spec, then when you verify it with fmtree, the verification will also include the sha256digest. Obviously fmtree -U can't correct a mismatch there, but of course it will detect and report it.

    Tool option: mtree, the NetBSD mtree (Debian package: mtree-netbsd)

    mtree produces (by default) output very similar to fmtree. With minor differences (such as the name of the sha256digest in the output), the discussion above about fmtree also applies to mtree.

    There are some differences, and the most notable is that mtree adds a -C option which reads a spec and converts it to a "format that's easier to parse with various tools." Here's an example:

    $ mtree -c -K sha256digest -p test | mtree -C
    . type=dir uid=1000 gid=1000 mode=0755 nlink=4 time=1610421833.0 flags=none 
    ./empty type=file uid=1000 gid=1000 mode=0644 nlink=1 size=0 time=1610421833.0 flags=none 
    ./foo type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none 
    ./hi type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none 
    ./there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=hi time=1610421833.0 flags=none 
    ./emptydir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none 
    ./somethingdir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none 
    ./somethingdir/there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=../there time=1610421833.0 flags=none 
    

    Most definitely an improvement in both space and convenience, while still retaining the relevant information. Note that if you want the sha256digest in the formatted output, you need to pass the -K to both mtree invocations. I could have done that here, but it is easier to read without it.

    mtree can verify a specification in either format. Given what I'm about to show you about bsdtar, this should illustrate why I bothered to package mtree-netbsd for Debian.

    Unlike fmtree, the mtree -U command will not adjust modification times based on the spec, but it will report on differences.

    Tool option: bsdtar (Debian package: libarchive-tools)

    bsdtar is a fascinating program that can work with many formats other than just tar files. Among the formats it supports is is the NetBSD mtree "pleasant" format (mtree -C compatible).

    bsdtar can also convert between the formats it supports. So, put this together: bsdtar can convert a tar file to an mtree specification without extracting the tar file. bsdtar can also use an mtree specification to override the permissions on files going into tar -c, so it is a way to prepare a tar file with things owned by root without resorting to tools like fakeroot.

    Let's look at how this can work:

    $ cd test
    $ bsdtar --numeric -cf - --format=mtree .
    
    . time=1610472086.318593729 mode=755 gid=1000 uid=1000 type=dir
    ./empty time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=0
    ./foo nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3
    ./hi nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3
    ./ormat\075mtree time=1610472086.318593729 mode=644 gid=1000 uid=1000 type=file size=5632
    ./there time=1610421833.0 mode=777 gid=1000 uid=1000 type=link link=hi
    ./emptydir time=1610421833.0 mode=755 gid=1000 uid=1000 type=dir
    ./somethingdir time=1610421833.0 mode=755 gid=1000 uid=1000 type=dir
    ./somethingdir/there time=1610421833.0 mode=777 gid=1000 uid=1000 type=link link=../there
    

    You can use mtree -U to verify that as before. With the --options mtree: set, you can also add hashes and similar to the bsdtar output. Since bsdtar can use input from tar, pax, cpio, zip, iso9660, 7z, etc., this capability can be used to create verification of the files inside quite a few different formats. You can convert with bsdtar -cf output.mtree --format=mtree @input.tar. There are some foibles with directly using these converted files with mtree -U, but usually minor changes will get it there.

    Side mention: stat(1) (Debian package: coreutils)

    This tool isn't included because it won't operate recursively, but is a tool in the similar toolbox.

    Putting It Together

    I will still be developing a complete non-ZFS backup system for NNCP (or UUCP) in a future post. But in the meantime, here are some ideas you can reflect on:

    • Let's say your backup scheme involves sending a full backup every night. On the source system, you could pipe the generated tar file through something like tee >(bsdtar -cf bcakup.mtree @-) to generate an mtree file in-band while generating the tar file. This mtree file could be shipped over for verification.
    • Perhaps your backup scheme involves sending incremental backup data via rdup or even ZFS, but you would like to periodically verify that everything is good -- that an incremental didn't miss something. Something like mtree -K sha256 -c -x -p / | mtree -C -K sha256 would let you accomplish that.

    I will further develop at least one of these ideas in a future post.

    Bonus: cross-tool comparisons

    In my mtree-netbsd packaging, I added tests like this to compare between tools:

    fmtree -c -K $(MTREE_KEYWORDS) | mtree
    mtree -c -K $(MTREE_KEYWORDS) | sed -e 's/\(md5\|sha1\|sha256\|sha384\|sha512\)=/\1digest=/' -e 's/rmd160=/ripemd160digest=/' | fmtree
    bsdtar -cf - --options 'mtree:uname,gname,md5,sha1,sha256,sha384,sha512,device,flags,gid,link,mode,nlink,size,time,uid,type,uname' --format mtree . | mtree
    

    The Good, Bad, and Scary of the Banning of Donald Trump, and How Decentralization Makes It All Better

    It is undeniable that banning Donald Trump from Facebook, Twitter, and similar sites is a benefit for the moment. It may well save lives, perhaps lots of lives. But it raises quite a few troubling issues.

    First, as EFF points out, these platforms have privileged speakers with power, especially politicians, over regular users. For years now, it has been obvious to everyone that Donald Trump has been violating policies on both platforms, and yet they did little or nothing about it. The result we saw last week was entirely forseeable — and indeed, WAS forseen, including by elements in those companies themselves. (ACLU also raises some good points)

    Contrast that with how others get treated. Facebook, two days after the coup attempt, banned Benjamin Wittes, apparently because he mentioned an Atlantic article opposed to nutcase conspiracy theories. The EFF has also documented many more egregious examples: taking down documentation of war crimes, childbirth images, black activists showing the racist messages they received, women discussing online harassment, etc. The list goes on; YouTube, for instance, has often been promoting far-right violent videos while removing peaceful LGBTQ ones.

    In short, have we simply achieved legal censorship by outsourcing it to dominant corporations?

    It is worth pausing at this point to recognize two important princples:

    First, that we do not see it as right to compel speech.

    Secondly, that there exist communications channels and other services that nobody is calling on to suspend Donald Trump.

    Let’s dive into those a little bit.

    There have been no prominent calls for AT&T, Verizon, Gmail, or whomever provides Trump and his campaign with cell phones or email to suspend their service to him. Moreover, the gas stations that fuel his vehicles and the airports that service his plane continue to provide those services, and nobody has seriously questioned that, either. Even his Apple phone that he uses to post to Twitter remains, as far as I know, fully active.

    Secondly, imagine you were starting up a small web forum focused on raising tomato plants. It is, and should be, well within your rights to keep tomato-haters out, as well as people that have no interest in tomatoes but would rather talk about rutabagas, politics, or Mars. If you are going to host a forum about tomatoes, you have the right to keep it a forum about tomatoes; you cannot be forced to distribute someone else’s speech. Likewise in traditional media, a newspaper cannot be forced to print every letter to the editor in full.

    In law, there is a notion of a common carrier, that provides services to the general public without discrimination. Phone companies and ISPs fall under this.

    Facebook, Twitter, and tomato sites don’t. But consider what happens if Facebook bans you. You might be using Facebook-owned Whatsapp to communicate with family and friends, and suddenly find yourself unable to ask someone to pick you up. Or your treasured family photos might be in Facebook-owned Instagram, lost forever. It’s not just Facebook; similar things happen with Google, locking people out of their phones and laptops, their emails, even their photos.

    Is it right that Facebook and Google aren’t regulated as common carriers? Perhaps, or perhaps we need some line of demarcation between their speech-to-the-public services (Facebook timeline posts, YouTube) and private communication (Whatsapp, Gmail). It’s a thorny issue; should government be regulating speech instead? That’s also fraught. So is corporate control.

    Decentralization Helps Dramatically

    With email, you get to pick your email provider (yes, there are two or three big ones, but still plenty of others). Each email provider will have its own set of things it considers acceptable, and its own set of other servers and accounts it’s willing to exchange mail with. (It is extremely common for mail providers to choose not to accept mail from various other mail servers based on ISP, IP address, reputation, and so forth.)

    What if we could do something like that for Twitter and Facebook?

    Let you join whatever instance you like. Maybe one instance is all about art and they don’t talk about politics. Or another is all about Free Software and they don’t have advertising. And then there are plenty of open instances that accept anything that’s respectful. And, like email, people of one server can interact with those using another just as easily as if they were using the same one.

    Well, this isn’t hypothetical; it already exists in the Fediverse. The most common option is Mastodon, and it so happens that a month ago I wrote about its benefits for other reasons, and included some links on getting started.

    There is no reason that we must all let our online speech be controlled by companies with a profit motive to keep hate speech on their platforms. There is no reason that we must all have a single set of rules, or accept strong corporate or government control, either. The quality of conversation on Mastodon is far higher than either Twitter or Facebook; decentralization works and it’s here today.

    More Topics on Store-And-Forward (Possibly Airgapped) ZFS and Non-ZFS Backups with NNCP

    Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.

    In my previous post, I introduced a way to use ZFS backups over NNCP. In this post, I’ll expand on that and also explore non-ZFS backups.

    Use of nncp-file instead of nncp-exec

    The previous example used nncp-exec (like UUCP’s uux), which lets you pipe stdin in, then queues up a request to run a given command with that input on a remote. I discussed that NNCP doesn’t guarantee order of execution, but that for the ZFS use case, that was fine since zfs receive would just fail (causing NNCP to try again later).

    At present, nncp-exec stores the data piped to it in RAM before generating the outbound packet (the author plans to fix this shortly) [Update: This is now fixed; use -use-tmp with nncp-exec!). That made it unusable for some of my backups, so I set it up another way: with nncp-file, the tool to transfer files to a remote machine. A cron job then picks them up and processes them.

    On the machine being backed up, we have to find a way to encode the dataset to be received. I chose to do that as part of the filename, so the updated simplesnap-queue could look like this:

    #!/bin/bash
    
    set -e
    set -o pipefail
    
    DEST="`echo $1 | sed 's,^tank/simplesnap/,,'`"
    FILE="bakfsfmt2-`date "+%s.%N".$$`_`echo "$DEST" | sed 's,/,@,g'`"
    
    echo "Processing $DEST to $FILE" >&2
    # stdin piped to this
    zstd -8 - \
      | gpg --compress-algo none --cipher-algo AES256 -e -r 012345...  \
      | su nncp -c "/usr/local/nncp/bin/nncp-file -nice B -noprogress - 'backupsvr:$FILE'" >&2
    
    echo "Queued $DEST to $FILE" >&2
    

    I’ve added compression and encryption here as well; more on that below.

    On the backup server, we would define a different incoming directory for each node in nncp.hjson. For instance:

    host1: {
    ...
       incoming: "/var/local/nncp-bakcups-incoming/host1"
    }
    
    host2: {
    ...
       incoming: "/var/local/nncp-backups-incoming/host2"
    }
    

    I’ll present the scanning script in a bit.

    Offsite Backup Rotation

    Most of the time, you don’t want just a single drive to store the backups. You’d like to have a set. At minimum, one wouldn’t be plugged in so lightning wouldn’t ruin all your backups. But maybe you’d store a second drive at some other location you have access to (friend’s house, bank box, etc.)

    There are several ways you could solve this:

    • If the remote machine is at a location with network access and you trust its physical security (remember that although it will store data encrypted at rest and will transport it encrypted, it will — in most cases — handle un-encrypted data during processing), you could of course send NNCP packets to it over the network at the same time you send them to your local backup system.
    • Alternatively, if the remote location doesn’t have network access or you want to keep it airgapped, you could transport the NNCP packets by USB drive to the remote end.
    • Or, if you don’t want to have any kind of processing capability remotely — probably a wise move — you could rotate the hard drives themselves, keeping one plugged in locally and unplugging the other to take it offsite.

    The third option can be helped with NNCP, too. One way is to create separate NNCP installations for each of the drives that you store data on. Then, whenever one is plugged in, the appropriate NNCP config will be loaded and appropriate packets received and processed. The neighbor machine — the spooler — would just store up packets for the offsite drive until it comes back onsite (or, perhaps, your airgapped USB transport would do this). Then when it’s back onsite, all the queued up ZFS sends get replayed and the backups replicated.

    Now, how might you handle this with NNCP?

    The simple way would be to have each system generating backups send them to two destinations. For instance:

    zstd -8 - | gpg --compress-algo none --cipher-algo AES256 -e -r 07D5794CD900FAF1D30B03AC3D13151E5039C9D5 \
      | tee >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk1:$FILE'") \
            >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk2:$FILE'") \
       > /dev/null
    

    You could probably also more safely use pee(1) (from moreutils) to do this.

    This has an unfortunate result of doubling the network traffic from every machine being backed up. So an alternative option would be to queue the packets to the spooling machine, and run a distribution script from it; something like this, in part:

    INCOMINGDIR="/var/local/nncp-bakfs-incoming"
    LOCKFILE="$INCOMINGDIR/.lock"
    printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
    if dotlockfile -r 0 -l -p "${LOCKFILE}"; then
      logit "Lock obtained at ${LOCKFILE} with dotlockfile"
      trap 'ECODE=$?; dotlockfile -u '"${EVAL_SAFE_LOCKFILE}"'; exit $ECODE' EXIT INT TERM
    else
      logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
      exit 0
    fi
    
    
    logit "Scanning queue directory..."
    cd "$INCOMINGDIR"
    for HOST in *; do
       cd "$INCOMINGDIR/$HOST"
       for FILE in bakfsfmt2-*; do
               if [ -f "$FILE" ]; then
                       for BAKFS in backupdisk1 backupdisk2; do
                               runcommand nncp-file -nice B+5 -noprogress "$FILE" "$BAKFS:$HOST/$FILE"
                       done
                       runcommand rm "$FILE"
               else
                       logit "$HOST: Skipping $FILE since it doesn't exist"
               fi
       done
    
    done
    logit "Scan complete."
    

    Security Considerations

    You’ll notice that in my example above, the encryption happens as the root user, but nncp is called under su. This means that even if there is a vulnerability in NNCP, the data would still be protected by GPG. I’ll also note here that many sites run ssh as root unnecessarily; the same principles should apply there. (ssh has had vulnerabilities in the past as well). I could have used gpg’s built-in compression, but zstd is faster and better, so we can get good performance by using fast compression and piping that to an algorithm that can use hardware acceleration for encryption.

    I strongly encourage considering transport, whether ssh or NNCP or UUCP, to be untrusted. Don’t run it as root if you can avoid it. In my example, the nncp user, which all NNCP commands are run as, has no access to the backup data at all. So even if NNCP were compromised, my backup data wouldn’t be. For even more security, I could also sign the backup stream with gpg and validate that on the receiving end.

    I should note, however, that this conversation assumes that a network- or USB-facing ssh or NNCP is more likely to have an exploitable vulnerability than is gpg (which here is just processing a stream). This is probably a safe assumption in general. If you believe gpg is more likely to have an exploitable vulnerability than ssh or NNCP, then obviously you wouldn’t take this particular approach.

    On the zfs side, the use of -F with zfs receive is avoided; this could lead to a compromised backed-up machine generating a malicious rollback on the destination. Backup zpools should be imported with -R or -N to ensure that a malicious mountpoint property couldn’t be used to cause an attack. I choose to use “zfs receive -u -o readonly=on” which is compatible with both unmounted backup datasets and zpools imported with -R (or both). To access the data in a backup dataset, you would normally clone it and access it there.

    The processing script

    So, put this all together and look at an example of a processing script that would run from cron as root and process the incoming ZFS data.

    #!/bin/bash
    set -e
    set -o pipefail
    
    # Log a message
    logit () {
       logger -p info -t "`basename "$0"`[$$]" "$1"
    }
    
    # Log an error message
    logerror () {
       logger -p err -t "`basename "$0"`[$$]" "$1"
    }
    
    # Log stdin with the given code.  Used normally to log stderr.
    logstdin () {
       logger -p info -t "`basename "$0"`[$$/$1]"
    }
    
    # Run command, logging stderr and exit code
    runcommand () {
       logit "Running $*"
       if "$@" 2> >(logstdin "$1") ; then
          logit "$1 exited successfully"
          return 0
       else
           RETVAL="$?"
           logerror "$1 exited with error $RETVAL"
           return "$RETVAL"
       fi
    }
    
    STORE=backups/simplesnap
    INCOMINGDIR=/backups/nncp/incoming
    
    if ! [ -d "$INCOMINGDIR" ]; then
            logerror "$INCOMINGDIR doesn't exist"
            exit 0
    fi
    
    LOCKFILE="/backups/nncp/.nncp-backups-zfs-scan.lock"
    printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
    if dotlockfile -r 0 -l -p "${LOCKFILE}"; then
      logit "Lock obtained at ${LOCKFILE} with dotlockfile"
      trap 'ECODE=$?; dotlockfile -u '"${EVAL_SAFE_LOCKFILE}"'; exit $ECODE' EXIT INT TERM
    else
      logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
      exit 0
    fi
    
    EXITCODE=0
    
    
    cd "$INCOMINGDIR"
    logit "Scanning queue directory..."
    for HOST in *; do
        HOSTPATH="$INCOMINGDIR/$HOST"
        # files like backupsfmt2-134.13134_dest
        for FILE in "$HOSTPATH"/backupsfmt2-[0-9]*_?*; do
            if [ ! -f "$FILE" ]; then
                logit "Skipping non-existent $FILE"
                continue
            fi
    
            # Now, $DEST will be HOST/DEST.  Strip off the @ also.
            DEST="`echo "$FILE" | sed -e 's/^.*backupsfmt2[^_]*_//' -e 's,@,/,g'`"
    
            if [ -z "$DEST" ]; then
                logerror "Malformed dest in $FILE"
                continue
            fi
            HOST2="`echo "$DEST" | sed 's,/.*,,g'`"
            if [ -z "$HOST2" ]; then
                logerror "Malformed DEST $DEST in $FILE"
                continue
            fi
    
            if [ ! "$HOST" = "$HOST2" ]; then
                logerror "$DIR: $HOST doesn't match $HOST2"
                continue
            fi
    
            logit "Processing $FILE to $STORE/$DEST"
                if runcommand gpg -q -d < "$FILE" | runcommand zstdcat | runcommand zfs receive -u -o readonly=on "$STORE/$DEST"; then
                    logit "Successfully processed $FILE to $STORE/$DEST"
                    runcommand rm "$FILE"
            else
                    logerror "FAILED to process $FILE to $STORE/$DEST"
                    EXITCODE=15
            fi
    

    Applying These Ideas to Non-ZFS Backups

    ZFS backups made our job easier in a lot of ways:

    • ZFS can calculate a diff based on an efficiently-stored previous local state (snapshot or bookmark), rather than a comparison to a remote state (rsync)
    • ZFS "incremental" sends, while less efficient than rsync, are reasonably efficient, sending only changed blocks
    • ZFS receive detects and enforces that the incremental source on the local machine must match the incremental source of the original stream, enforcing ordering
    • Datasets using ZFS encryption can be sent in their encrypted state
    • Incrementals can be done without a full scan of the filesystem

    Some of these benefits you just won't get without ZFS (or something similar like btrfs), but let's see how we could apply these ideas to non-ZFS backups. I will explore the implementation of them in a future post.

    When I say "non ZFS", I am being a bit vague as to whether the source, the destination, or both systems are running a non-ZFS filesystem. In general I'll assume that neither are ZFS.

    The first and most obvious answer is to just tar up the whole system and send that every day. This is, of course, only suitable for small datasets on a fast network. These tarballs could be unpacked on the destination and stored more efficiently via any number of methods (hardlink trees, a block-level deduplicator like borg or rdedup, or even just simply compressed tarballs).

    To make the network trip more efficient, something like rdiff or xdelta could be used. A signature file could be stored on the machine being backed up (generated via tee/pee at stream time), and the next run could simply send an rdiff delta over NNCP. This would be quite network-efficient, but still would require reading every byte of every file on every backup, and would also require quite a bit of temporary space on the receiving end (to apply the delta to the previous tarball and generate a new one).

    Alternatively, a program that generates incremental backup files such as rdup could be used. These could be transmitted over NNCP to the backup server, and unpacked there. While perhaps less efficient on the network -- every file with at least one modified byte would be retransmitted in its entirety -- it avoids the need to read every byte of unmodified files or to have enormous temporary space. I should note here that GNU tar claims to have an incremental mode, but it has a potential data loss bug.

    There are also some tools with algorithms that may apply well in this use care: syrep and fssync being the two most prominent examples, though rdedup (mentioned above) and the nascent asuran project may also be combinable with other tools to achieve this effect.

    I should, of course, conclude this section by mentioning btrfs. Every time I've tried it, I've run into serious bugs, and its status page indicates that only some of them have been resolved. I would not consider using it for something as important as backups. However, if you are comfortable with it, it is likely to be able to run in more constrained environments than ZFS and could probably be processed in much the same way as zfs streams.

    Airgapped / Asynchronous Backups with ZFS over NNCP

    In my previous articles in the series on asynchronous communication with the modern NNCP tool, I talked about its use for asynchronous, potentially airgapped, backups. The first article, How & Why To Use Airgapped Backups laid out the foundations for this. Now let’s dig into the details.

    Today’s post will cover ZFS, because it has a lot of features that make it very easy to support in this setup. Non-ZFS backups will be covered later.

    The setup is actually about as simple as it is for SSH, but since people are less familiar with this kind of communication, I’m going to try to go into more detail here.

    Assumptions

    I am assuming a setup where:

    • The machines being backed up run ZFS
    • The disk(s) that hold the backups are also running ZFS
    • zfs send / receive is desired as an efficient way to transport the backups
    • The machine that holds the backups may have no network connection whatsoever
    • Backups will be sent encrypted over some sort of network to a spooling machine, which temporarily holds them until they are transported to the destination backup system and ingested there. This system will be unable to decrypt the data streams it temporarily stores.

    Hardware

    Let’s start with hardware for the machine to hold the backups. I initially considered a Raspberry Pi 4 with 8GB of RAM. That would probably have been a suitable machine, at least for smaller backup sets. However, none of the Raspberry Pi machines support hardware AES encryption acceleration, and my Pi4 benchmarks as about 60MB/s for AES encryption. I want my backups to be encrypted, and decided this would just be too slow for my purposes. Again, if you don’t need encrypted backups or don’t care that much about performance — may people probably fall into this category — you can have a fully-functional Raspberry Pi 4 system for under $100 that would make a fantastic backup server.

    I wound up purchasing a Qotom-Q355G4 micro PC with a Core i5 for about $315. It has USB 3 ports and is designed as a rugged, long-lasting system. I have been using one of their older Celeron-based models as my router/firewall for a number of years now and it’s been quite reliable.

    For backup storage, you can get a USB 3 external drive. My own preference is to get a USB 3 “toaster” (device that lets me plug in SATA drives) so that I have more control over the underlying medium and can save the expense and hassle of a bunch of power supplies. In a future post, I will discuss drive rotation so you always have an offline drive.

    Then, there is the question of transport to the backup machine. A simple solution would be to have a heavily-firewalled backup system that has no incoming ports open but makes occasional outgoing connections to one specific NNCP daemon on the spooling machine. However, for airgapped operation, it would also be very simple to use nncp-xfer to transport the data across on a USB stick or some such. You could set up automounting for a specific USB stick – plug it in, all the spooled data is moved over, then plug it in to the backup system and it’s processed, and any outbound email traffic or whatever is copied to the USB stick at that point too. The NNCP page has some more commentary about this kind of setup.

    Both are fairly easy to set up, and NNCP is designed to be transport-agnostic, so in this article I’m going to focus on how to integrate ZFS with NNCP.

    Operating System

    Of course, it should be no surprise that I set this up on Debian.

    As an added step, I did all the configuration in Ansible stored in a local git repo. This adds a lot of work, but it means that it is trivial to periodically wipe and reinstall if any security issue is suspected. The git repo can be copied off to another system for storage and takes the system from freshly-installed to ready-to-use state.

    Security

    There is, of course, nothing preventing you from running NNCP as root. The zfs commands, obviously, need to be run as root. However, from a privilege separation standpoint, I have chosen to run everything relating to NNCP as a nncp user. NNCP already does encryption, but if you prefer to have zero knowledge of the data even to NNCP, it’s trivial to add gpg to the pipeline as well, and in fact I’ll be demonstrating that in a future post for other reasons.

    Software

    Besides NNCP, there needs to be a system that generates the zfs send streams. For this project, I looked at quite a few. Most were designed to inspect the list of snapshots on a remote end, compare it to a list on the local end, and calculate a difference from there. This, of course, won’t work for this situation.

    I realized my own simplesnap project was very close to being able to do this. It already used an algorithm of using specially-named snapshots on the machine being backed up, so never needed any communication about what snapshots were present where. All it needed was a few more options to permit sending to a stream instead of zfs receive. I made those changes and they are available in simplesnap 2.0.0 or above. That version has also been uploaded to sid, and will work fine as-is on buster as well.

    Preparing NNCP

    I’m going to assume three hosts in this setup:

    • laptop is the machine being backed up. Of course, you may have quite a few of these.
    • spooler holds the backup data until the backup system picks it up
    • backupsvr holds the backups

    The basic NNCP workflow documentation covers the basic steps. You’ll need to run nncp-cfgnew on each machine. This generates a basic configuration, along with public and private keys for that machine. You’ll copy the public key sets to the configurations of the other machines as usual. On the laptop, you’ll add a via line like this:

    backupsvr: {
      id: ....
      exchpub: ...
      signpub: ...
      noisepub: ...
      via: ["spooler"]
    

    This tells NNCP that data destined for backupsvr should always be sent via spooler first.

    You can then arrange for the nncp-daemon to run on the spooler, and nncp-caller or nncp-call on the backupsvr. Or, alternatively, airgapped between the two with nncp-xfer.

    Generating Backup Data

    Now, on the laptop, install simplesnap (2.0.0 or above). Although you won’t be backing up to the local system, simplesnap still maintains a hostlock in ZFS. Prepate a dataset for it:

    zfs create tank/simplesnap
    zfs set org.complete.simplesnap:exclude=on tank/simplesnap
    

    Then, create a script /usr/local/bin/runsimplesnap like this:

    #!/bin/bash
    
    set -e
    
    simplesnap --store tank/simplesnap --setname backups --local --host `hostname` \
       --receivecmd /usr/local/bin/simplesnap-queue \
       --noreap
    
    su nncp -c '/usr/local/nncp/bin/nncp-toss -noprogress -quiet'
    
    if ip addr | grep -q 192.168.65.64; then
      su nncp -c '/usr/local/nncp/bin/nncp-call -noprogress -quiet -onlinedeadline 1 spooler'
    fi
    

    The call to simplesnap sets it up to send the data to simplesnap-queue, which we’ll create in a moment. The –receivmd, plus –noreap, sets it up to run without ZFS on the local system.

    The call to nncp-toss will process any previously-received inbound NNCP packets, if there are any. Then, in this example, we do a very basic check to see if we’re on the LAN (checking 192.168.65.64), and if so, will establish a connection to the spooler to transmit the data. If course, you could also do this over the Internet, with tor, or whatever, but in my case, I don’t want to automatically do this in case I’m tethered to mobile. I figure if I want to send backups in that case, I can fire up nncp-call myself. You can also use nncp-caller to set up automated connections on other schedules; there are a lot of options.

    Now, here’s what /usr/local/bin/simplesnap-queue looks like:

    #!/bin/bash
    
    set -e
    set -o pipefail
    
    DEST="`echo $1 | sed 's,^tank/simplesnap/,,'`"
    
    echo "Processing $DEST" >&2
    # stdin piped to this
    su nncp -c "/usr/local/nncp/bin/nncp-exec -nice B -noprogress backupsvr zfsreceive '$DEST'" >&2
    echo "Queued for $DEST" >&2
    

    This is a pretty simple script. simplesnap will call it with a path based on the –store, with the hostname after; so, for instance, tank/simplesnap/laptop/root or some such. This script strips off the leading tank/simplesnap (which is a local fragment), leaving the host and dataset paths. Then it just pipes it to nncp-exec. -nice B classifies it as low-priority bulk data (so if you have some more important interactive data, it would be sent first), then passes it to whatever the backupsvr defines as zfsreceive.

    Receiving ZFS backups

    In the NNCP configuration on the recipient’s side, in the laptop section, we define what command it’s allowed to run as zfsreceive:

          exec: {
            zfsreceive: ["/usr/bin/sudo", "-H", "/usr/local/bin/nncp-zfs-receive"]
          }
    

    We authorize the nncp user to run this under sudo in /etc/sudoers.d/local–nncp:

    Defaults env_keep += "NNCP_SENDER"
    nncp ALL=(root) NOPASSWD: /usr/local/bin/nncp-zfs-receive
    

    The NNCP_SENDER is the public key ID of the sending node when nncp-toss processes the incoming data. We can use that for sanity checking later.

    Now, here’s a basic nncp-zfs-receive script:

    #!/bin/bash
    set -e
    set -o pipefail
    
    STORE=backups/simplesnap
    DEST="$1"
    
    # now process stdin
    runcommand zfs receive -o readonly=on -x mountpoint "$STORE/$DEST"
    

    And there you have it — all the basics are in place.

    Update 2020-12-30: An earlier version of this article had “zfs receive -F” instead of “zfs receive -o readonly=on -x mountpoint”. These changed arguments are more robust.
    Update 2021-01-04: I am now recommending “zfs receive -u -o readonly=on”; see my successor article for more.

    Enhancements

    You could enhance the nncp-zfs-receive script to improve logging and error handling. For instance:

    #!/bin/bash
    
    set -e
    set -o pipefail
    
    STORE=backups/simplesnap
    # $1 will be the host/dataset
    
    DEST="$1"
    HOST="`echo "$1" | sed 's,/.*,,g'`"
    if [ -z "$HOST" ]; then
       echo "Malformed command line"
       exit 5
    fi
    
    # Log a message
    logit () {
       logger -p info -t "`basename "$0"`[$$]" "$1"
    }
    
    # Log an error message
    logerror () {
       logger -p err -t "`basename "$0"`[$$]" "$1"
    }
    
    # Log stdin with the given code.  Used normally to log stderr.
    logstdin () {
       logger -p info -t "`basename "$0"`[$$/$1]"
    }
    
    # Run command, logging stderr and exit code
    runcommand () {
       logit "Running $*"
       if "$@" 2> >(logstdin "$1") ; then
          logit "$1 exited successfully"
          return 0
       else
           RETVAL="$?"
           logerror "$1 exited with error $RETVAL"
           return "$RETVAL"
       fi
    }
    exiterror () {
       logerror "$1"
       echo "$1" 1>&2
       exit 10
    }
    
    # Sanity check
    
    if [ "$HOST" = "laptop" ]; then
      if [ "$NNCP_SENDER" != "12345678" ]; then
        exiterror "Host $HOST doesn't match sender $NNCP_SENDER"
      fi
    else
      exiterror "Unknown host $HOST"
    fi
    
    runcommand zfs receive -F "$STORE/$DEST"
    

    Now you’ll capture the ZFS receive output in syslog in a friendly way, so you can look back later why things failed if they did.

    Further notes on NNCP

    nncp-toss will examine the exit code from an invocation. If it is nonzero, it will keep the command (and associated stdin) in the queue and retry it on the next invocation. NNCP does not guarantee order of execution, so it is possible in some cases that ZFS streams may be received in the wrong order. That is fine here; zfs receive will exit with an error, and nncp-toss will just run it again after the dependent snapshots have been received. For non-ZFS backups, a simple sequence number can handle this issue.