Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.
In my previous post, I introduced a way to use ZFS backups over NNCP. In this post, I’ll expand on that and also explore non-ZFS backups.
Use of nncp-file instead of nncp-exec
The previous example used nncp-exec (like UUCP’s uux), which lets you pipe stdin in, then queues up a request to run a given command with that input on a remote. I discussed that NNCP doesn’t guarantee order of execution, but that for the ZFS use case, that was fine since zfs receive would just fail (causing NNCP to try again later).
At present, nncp-exec stores the data piped to it in RAM before generating the outbound packet (the author plans to fix this shortly) [Update: This is now fixed; use -use-tmp with nncp-exec!). That made it unusable for some of my backups, so I set it up another way: with nncp-file, the tool to transfer files to a remote machine. A cron job then picks them up and processes them.
On the machine being backed up, we have to find a way to encode the dataset to be received. I chose to do that as part of the filename, so the updated simplesnap-queue could look like this:
#!/bin/bash set -e set -o pipefail DEST="`echo $1 | sed 's,^tank/simplesnap/,,'`" FILE="bakfsfmt2-`date "+%s.%N".$$`_`echo "$DEST" | sed 's,/,@,g'`" echo "Processing $DEST to $FILE" >&2 # stdin piped to this zstd -8 - \ | gpg --compress-algo none --cipher-algo AES256 -e -r 012345... \ | su nncp -c "/usr/local/nncp/bin/nncp-file -nice B -noprogress - 'backupsvr:$FILE'" >&2 echo "Queued $DEST to $FILE" >&2
I’ve added compression and encryption here as well; more on that below.
On the backup server, we would define a different incoming directory for each node in nncp.hjson. For instance:
host1: {
...
incoming: "/var/local/nncp-bakcups-incoming/host1"
}
host2: {
...
incoming: "/var/local/nncp-backups-incoming/host2"
}
I’ll present the scanning script in a bit.
Offsite Backup Rotation
Most of the time, you don’t want just a single drive to store the backups. You’d like to have a set. At minimum, one wouldn’t be plugged in so lightning wouldn’t ruin all your backups. But maybe you’d store a second drive at some other location you have access to (friend’s house, bank box, etc.)
There are several ways you could solve this:
- If the remote machine is at a location with network access and you trust its physical security (remember that although it will store data encrypted at rest and will transport it encrypted, it will — in most cases — handle un-encrypted data during processing), you could of course send NNCP packets to it over the network at the same time you send them to your local backup system.
- Alternatively, if the remote location doesn’t have network access or you want to keep it airgapped, you could transport the NNCP packets by USB drive to the remote end.
- Or, if you don’t want to have any kind of processing capability remotely — probably a wise move — you could rotate the hard drives themselves, keeping one plugged in locally and unplugging the other to take it offsite.
The third option can be helped with NNCP, too. One way is to create separate NNCP installations for each of the drives that you store data on. Then, whenever one is plugged in, the appropriate NNCP config will be loaded and appropriate packets received and processed. The neighbor machine — the spooler — would just store up packets for the offsite drive until it comes back onsite (or, perhaps, your airgapped USB transport would do this). Then when it’s back onsite, all the queued up ZFS sends get replayed and the backups replicated.
Now, how might you handle this with NNCP?
The simple way would be to have each system generating backups send them to two destinations. For instance:
zstd -8 - | gpg --compress-algo none --cipher-algo AES256 -e -r 07D5794CD900FAF1D30B03AC3D13151E5039C9D5 \
| tee >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk1:$FILE'") \
>(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk2:$FILE'") \
> /dev/null
You could probably also more safely use pee(1) (from moreutils) to do this.
This has an unfortunate result of doubling the network traffic from every machine being backed up. So an alternative option would be to queue the packets to the spooling machine, and run a distribution script from it; something like this, in part:
INCOMINGDIR="/var/local/nncp-bakfs-incoming"
LOCKFILE="$INCOMINGDIR/.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "${LOCKFILE}"; then
logit "Lock obtained at ${LOCKFILE} with dotlockfile"
trap 'ECODE=$?; dotlockfile -u '"${EVAL_SAFE_LOCKFILE}"'; exit $ECODE' EXIT INT TERM
else
logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
exit 0
fi
logit "Scanning queue directory..."
cd "$INCOMINGDIR"
for HOST in *; do
cd "$INCOMINGDIR/$HOST"
for FILE in bakfsfmt2-*; do
if [ -f "$FILE" ]; then
for BAKFS in backupdisk1 backupdisk2; do
runcommand nncp-file -nice B+5 -noprogress "$FILE" "$BAKFS:$HOST/$FILE"
done
runcommand rm "$FILE"
else
logit "$HOST: Skipping $FILE since it doesn't exist"
fi
done
done
logit "Scan complete."
Security Considerations
You’ll notice that in my example above, the encryption happens as the root user, but nncp is called under su. This means that even if there is a vulnerability in NNCP, the data would still be protected by GPG. I’ll also note here that many sites run ssh as root unnecessarily; the same principles should apply there. (ssh has had vulnerabilities in the past as well). I could have used gpg’s built-in compression, but zstd is faster and better, so we can get good performance by using fast compression and piping that to an algorithm that can use hardware acceleration for encryption.
I strongly encourage considering transport, whether ssh or NNCP or UUCP, to be untrusted. Don’t run it as root if you can avoid it. In my example, the nncp user, which all NNCP commands are run as, has no access to the backup data at all. So even if NNCP were compromised, my backup data wouldn’t be. For even more security, I could also sign the backup stream with gpg and validate that on the receiving end.
I should note, however, that this conversation assumes that a network- or USB-facing ssh or NNCP is more likely to have an exploitable vulnerability than is gpg (which here is just processing a stream). This is probably a safe assumption in general. If you believe gpg is more likely to have an exploitable vulnerability than ssh or NNCP, then obviously you wouldn’t take this particular approach.
On the zfs side, the use of -F with zfs receive is avoided; this could lead to a compromised backed-up machine generating a malicious rollback on the destination. Backup zpools should be imported with -R or -N to ensure that a malicious mountpoint property couldn’t be used to cause an attack. I choose to use “zfs receive -u -o readonly=on” which is compatible with both unmounted backup datasets and zpools imported with -R (or both). To access the data in a backup dataset, you would normally clone it and access it there.
The processing script
So, put this all together and look at an example of a processing script that would run from cron as root and process the incoming ZFS data.
#!/bin/bash
set -e
set -o pipefail
# Log a message
logit () {
logger -p info -t "`basename "$0"`[$$]" "$1"
}
# Log an error message
logerror () {
logger -p err -t "`basename "$0"`[$$]" "$1"
}
# Log stdin with the given code. Used normally to log stderr.
logstdin () {
logger -p info -t "`basename "$0"`[$$/$1]"
}
# Run command, logging stderr and exit code
runcommand () {
logit "Running $*"
if "$@" 2> >(logstdin "$1") ; then
logit "$1 exited successfully"
return 0
else
RETVAL="$?"
logerror "$1 exited with error $RETVAL"
return "$RETVAL"
fi
}
STORE=backups/simplesnap
INCOMINGDIR=/backups/nncp/incoming
if ! [ -d "$INCOMINGDIR" ]; then
logerror "$INCOMINGDIR doesn't exist"
exit 0
fi
LOCKFILE="/backups/nncp/.nncp-backups-zfs-scan.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "${LOCKFILE}"; then
logit "Lock obtained at ${LOCKFILE} with dotlockfile"
trap 'ECODE=$?; dotlockfile -u '"${EVAL_SAFE_LOCKFILE}"'; exit $ECODE' EXIT INT TERM
else
logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
exit 0
fi
EXITCODE=0
cd "$INCOMINGDIR"
logit "Scanning queue directory..."
for HOST in *; do
HOSTPATH="$INCOMINGDIR/$HOST"
# files like backupsfmt2-134.13134_dest
for FILE in "$HOSTPATH"/backupsfmt2-[0-9]*_?*; do
if [ ! -f "$FILE" ]; then
logit "Skipping non-existent $FILE"
continue
fi
# Now, $DEST will be HOST/DEST. Strip off the @ also.
DEST="`echo "$FILE" | sed -e 's/^.*backupsfmt2[^_]*_//' -e 's,@,/,g'`"
if [ -z "$DEST" ]; then
logerror "Malformed dest in $FILE"
continue
fi
HOST2="`echo "$DEST" | sed 's,/.*,,g'`"
if [ -z "$HOST2" ]; then
logerror "Malformed DEST $DEST in $FILE"
continue
fi
if [ ! "$HOST" = "$HOST2" ]; then
logerror "$DIR: $HOST doesn't match $HOST2"
continue
fi
logit "Processing $FILE to $STORE/$DEST"
if runcommand gpg -q -d < "$FILE" | runcommand zstdcat | runcommand zfs receive -u -o readonly=on "$STORE/$DEST"; then
logit "Successfully processed $FILE to $STORE/$DEST"
runcommand rm "$FILE"
else
logerror "FAILED to process $FILE to $STORE/$DEST"
EXITCODE=15
fi
Applying These Ideas to Non-ZFS Backups
ZFS backups made our job easier in a lot of ways:
- ZFS can calculate a diff based on an efficiently-stored previous local state (snapshot or bookmark), rather than a comparison to a remote state (rsync)
- ZFS "incremental" sends, while less efficient than rsync, are reasonably efficient, sending only changed blocks
- ZFS receive detects and enforces that the incremental source on the local machine must match the incremental source of the original stream, enforcing ordering
- Datasets using ZFS encryption can be sent in their encrypted state
- Incrementals can be done without a full scan of the filesystem
Some of these benefits you just won't get without ZFS (or something similar like btrfs), but let's see how we could apply these ideas to non-ZFS backups. I will explore the implementation of them in a future post.
When I say "non ZFS", I am being a bit vague as to whether the source, the destination, or both systems are running a non-ZFS filesystem. In general I'll assume that neither are ZFS.
The first and most obvious answer is to just tar up the whole system and send that every day. This is, of course, only suitable for small datasets on a fast network. These tarballs could be unpacked on the destination and stored more efficiently via any number of methods (hardlink trees, a block-level deduplicator like borg or rdedup, or even just simply compressed tarballs).
To make the network trip more efficient, something like rdiff or xdelta could be used. A signature file could be stored on the machine being backed up (generated via tee/pee at stream time), and the next run could simply send an rdiff delta over NNCP. This would be quite network-efficient, but still would require reading every byte of every file on every backup, and would also require quite a bit of temporary space on the receiving end (to apply the delta to the previous tarball and generate a new one).
Alternatively, a program that generates incremental backup files such as rdup could be used. These could be transmitted over NNCP to the backup server, and unpacked there. While perhaps less efficient on the network -- every file with at least one modified byte would be retransmitted in its entirety -- it avoids the need to read every byte of unmodified files or to have enormous temporary space. I should note here that GNU tar claims to have an incremental mode, but it has a potential data loss bug.
There are also some tools with algorithms that may apply well in this use care: syrep and fssync being the two most prominent examples, though rdedup (mentioned above) and the nascent asuran project may also be combinable with other tools to achieve this effect.
I should, of course, conclude this section by mentioning btrfs. Every time I've tried it, I've run into serious bugs, and its status page indicates that only some of them have been resolved. I would not consider using it for something as important as backups. However, if you are comfortable with it, it is likely to be able to run in more constrained environments than ZFS and could probably be processed in much the same way as zfs streams.
In my previous articles in the series on asynchronous communication with the modern NNCP tool, I talked about its use for asynchronous, potentially airgapped, backups. The first article, How & Why To Use Airgapped Backups laid out the foundations for this. Now let’s dig into the details.
Today’s post will cover ZFS, because it has a lot of features that make it very easy to support in this setup. Non-ZFS backups will be covered later.
The setup is actually about as simple as it is for SSH, but since people are less familiar with this kind of communication, I’m going to try to go into more detail here.
Assumptions
I am assuming a setup where:
The machines being backed up run ZFS
The disk(s) that hold the backups are also running ZFS
zfs send / receive is desired as an efficient way to transport the backups
The machine that holds the backups may have no network connection whatsoever
Backups will be sent encrypted over some sort of network to a spooling machine, which temporarily holds them until they are transported to the destination backup system and ingested there. This system will be unable to decrypt the data streams it temporarily stores.
Hardware
Let’s start with hardware for the machine to hold the backups. I initially considered a Raspberry Pi 4 with 8GB of RAM. That would probably have been a suitable machine, at least for smaller backup sets. However, none of the Raspberry Pi machines support hardware AES encryption acceleration, and my Pi4 benchmarks as about 60MB/s for AES encryption. I want my backups to be encrypted, and decided this would just be too slow for my purposes. Again, if you don’t need encrypted backups or don’t care that much about performance — may people probably fall into this category — you can have a fully-functional Raspberry Pi 4 system for under $100 that would make a fantastic backup server.
I wound up purchasing a Qotom-Q355G4 micro PC with a Core i5 for about $315. It has USB 3 ports and is designed as a rugged, long-lasting system. I have been using one of their older Celeron-based models as my router/firewall for a number of years now and it’s been quite reliable.
For backup storage, you can get a USB 3 external drive. My own preference is to get a USB 3 “toaster” (device that lets me plug in SATA drives) so that I have more control over the underlying medium and can save the expense and hassle of a bunch of power supplies. In a future post, I will discuss drive rotation so you always have an offline drive.
Then, there is the question of transport to the backup machine. A simple solution would be to have a heavily-firewalled backup system that has no incoming ports open but makes occasional outgoing connections to one specific NNCP daemon on the spooling machine. However, for airgapped operation, it would also be very simple to use nncp-xfer to transport the data across on a USB stick or some such. You could set up automounting for a specific USB stick – plug it in, all the spooled data is moved over, then plug it in to the backup system and it’s processed, and any outbound email traffic or whatever is copied to the USB stick at that point too. The NNCP page has some more commentary about this kind of setup.
Both are fairly easy to set up, and NNCP is designed to be transport-agnostic, so in this article I’m going to focus on how to integrate ZFS with NNCP.
Operating System
Of course, it should be no surprise that I set this up on Debian.
As an added step, I did all the configuration in Ansible stored in a local git repo. This adds a lot of work, but it means that it is trivial to periodically wipe and reinstall if any security issue is suspected. The git repo can be copied off to another system for storage and takes the system from freshly-installed to ready-to-use state.
Security
There is, of course, nothing preventing you from running NNCP as root. The zfs commands, obviously, need to be run as root. However, from a privilege separation standpoint, I have chosen to run everything relating to NNCP as a nncp user. NNCP already does encryption, but if you prefer to have zero knowledge of the data even to NNCP, it’s trivial to add gpg to the pipeline as well, and in fact I’ll be demonstrating that in a future post for other reasons.
Software
Besides NNCP, there needs to be a system that generates the zfs send streams. For this project, I looked at quite a few. Most were designed to inspect the list of snapshots on a remote end, compare it to a list on the local end, and calculate a difference from there. This, of course, won’t work for this situation.
I realized my own simplesnap project was very close to being able to do this. It already used an algorithm of using specially-named snapshots on the machine being backed up, so never needed any communication about what snapshots were present where. All it needed was a few more options to permit sending to a stream instead of zfs receive. I made those changes and they are available in simplesnap 2.0.0 or above. That version has also been uploaded to sid, and will work fine as-is on buster as well.
Preparing NNCP
I’m going to assume three hosts in this setup:
laptop is the machine being backed up. Of course, you may have quite a few of these.
spooler holds the backup data until the backup system picks it up
backupsvr holds the backups
The basic NNCP workflow documentation covers the basic steps. You’ll need to run nncp-cfgnew on each machine. This generates a basic configuration, along with public and private keys for that machine. You’ll copy the public key sets to the configurations of the other machines as usual. On the laptop, you’ll add a via line like this:
backupsvr: {
id: ….
exchpub: …
signpub: …
noisepub: …
via: [“spooler”]
This tells NNCP that data destined for backupsvr should always be sent via spooler first.
You can then arrange for the nncp-daemon to run on the spooler, and nncp-caller or nncp-call on the backupsvr. Or, alternatively, airgapped between the two with nncp-xfer.
Generating Backup Data
Now, on the laptop, install simplesnap (2.0.0 or above). Although you won’t be backing up to the local system, simplesnap still maintains a hostlock in ZFS. Prepate a dataset for it:
zfs create tank/simplesnap
zfs set org.complete.simplesnap:exclude=on tank/simplesnap
Then, create a script /usr/local/bin/runsimplesnap like this:
#!/bin/bash
set -e
simplesnap –store tank/simplesnap –setname backups –local –host `hostname`
–receivecmd /usr/local/bin/simplesnap-queue
–noreap
su nncp -c ‘/usr/local/nncp/bin/nncp-toss -noprogress -quiet’
if ip addr | grep -q 192.168.65.64; then
su nncp -c ‘/usr/local/nncp/bin/nncp-call -noprogress -quiet -onlinedeadline 1 spooler’
fi
The call to simplesnap sets it up to send the data to simplesnap-queue, which we’ll create in a moment. The –receivmd, plus –noreap, sets it up to run without ZFS on the local system.
The call to nncp-toss will process any previously-received inbound NNCP packets, if there are any. Then, in this example, we do a very basic check to see if we’re on the LAN (checking 192.168.65.64), and if so, will establish a connection to the spooler to transmit the data. If course, you could also do this over the Internet, with tor, or whatever, but in my case, I don’t want to automatically do this in case I’m tethered to mobile. I figure if I want to send backups in that case, I can fire up nncp-call myself. You can also use nncp-caller to set up automated connections on other schedules; there are a lot of options.
Now, here’s what /usr/local/bin/simplesnap-queue looks like:
#!/bin/bash
set -e
set -o pipefail
DEST=”`echo $1 | sed ‘s,^tank/simplesnap/,,’`”
echo “Processing $DEST” >&2
# stdin piped to this
su nncp -c “/usr/local/nncp/bin/nncp-exec -nice B -noprogress backupsvr zfsreceive ‘$DEST'” >&2
echo “Queued for $DEST” >&2
This is a pretty simple script. simplesnap will call it with a path based on the –store, with the hostname after; so, for instance, tank/simplesnap/laptop/root or some such. This script strips off the leading tank/simplesnap (which is a local fragment), leaving the host and dataset paths. Then it just pipes it to nncp-exec. -nice B classifies it as low-priority bulk data (so if you have some more important interactive data, it would be sent first), then passes it to whatever the backupsvr defines as zfsreceive.
Receiving ZFS backups
In the NNCP configuration on the recipient’s side, in the laptop section, we define what command it’s allowed to run as zfsreceive:
exec: {
zfsreceive: [“/usr/bin/sudo”, “-H”, “/usr/local/bin/nncp-zfs-receive”]
}
We authorize the nncp user to run this under sudo in /etc/sudoers.d/local–nncp:
Defaults env_keep += “NNCP_SENDER”
nncp ALL=(root) NOPASSWD: /usr/local/bin/nncp-zfs-receive
The NNCP_SENDER is the public key ID of the sending node when nncp-toss processes the incoming data. We can use that for sanity checking later.
Now, here’s a basic nncp-zfs-receive script:
#!/bin/bash
set -e
set -o pipefail
STORE=backups/simplesnap
DEST=”$1″
# now process stdin
runcommand zfs receive -o readonly=on -x mountpoint “$STORE/$DEST”
And there you have it — all the basics are in place.
Update 2020-12-30: An earlier version of this article had “zfs receive -F” instead of “zfs receive -o readonly=on -x mountpoint”. These changed arguments are more robust.Update 2021-01-04: I am now recommending “zfs receive -u -o readonly=on”; see my successor article for more.
Enhancements
You could enhance the nncp-zfs-receive script to improve logging and error handling. For instance:
#!/bin/bash
set -e
set -o pipefail
STORE=backups/simplesnap
# $1 will be the host/dataset
DEST=”$1″
HOST=”`echo “$1″ | sed ‘s,/.*,,g’`”
if [ -z “$HOST” ]; then
echo “Malformed command line”
exit 5
fi
# Log a message
logit () {
logger -p info -t “`basename “$0″`[$$]” “$1”
}
# Log an error message
logerror () {
logger -p err -t “`basename “$0″`[$$]” “$1”
}
# Log stdin with the given code. Used normally to log stderr.
logstdin () {
logger -p info -t “`basename “$0″`[$$/$1]”
}
# Run command, logging stderr and exit code
runcommand () {
logit “Running $*”
if “$@” 2> >(logstdin “$1”) ; then
logit “$1 exited successfully”
return 0
else
RETVAL=”$?”
logerror “$1 exited with error $RETVAL”
return “$RETVAL”
fi
}
exiterror () {
logerror “$1”
echo “$1” 1>&2
exit 10
}
# Sanity check
if [ “$HOST” = “laptop” ]; then
if [ “$NNCP_SENDER” != “12345678” ]; then
exiterror “Host $HOST doesn’t match sender $NNCP_SENDER”
fi
else
exiterror “Unknown host $HOST”
fi
runcommand zfs receive -F “$STORE/$DEST”
Now you’ll capture the ZFS receive output in syslog in a friendly way, so you can look back later why things failed if they did.
Further notes on NNCP
nncp-toss will examine the exit code from an invocation. If it is nonzero, it will keep the command (and associated stdin) in the queue and retry it on the next invocation. NNCP does not guarantee order of execution, so it is possible in some cases that ZFS streams may be received in the wrong order. That is fine here; zfs receive will exit with an error, and nncp-toss will just run it again after the dependent snapshots have been received. For non-ZFS backups, a simple sequence number can handle this issue.
Why I use #ZFS4/ zfs #encryption is more flexible than LUKS. You can apply it to certain parts of the filesystem, use different keys for different parts, etc. Even better, zfs send lets you choose whether to send raw encrypted or decrypted packets. You can make #backups for which no component ever sees the plaintext data. You can also snapshot, clone, back up, etc. without needing decryption keys.
backups
encryption
zfs
Why I use #ZFS 5/ Built-in #compression with fast algorithms (lz4 and now zstd). In many cases the compression is fast enough that the reduced I/O results in a performance gain.
compression
zfs
Why I use #ZFS6/ zvols. You can do all of this with chunks that are presented as block devices, too. Want to back up that Windows virtual machine running under KVM or Virtualbox? Create /dev/zvol/foo with ZFS, attach it to the VM, and you can snapshot and backup and clone just as you would with a filesystem.
zfs
Why I use #ZFS7/ More flexible volume management (in most but not all ways) than #LVM. By integrating the filesystem and the volume manager, resizing a filesystem is just adjusting its filesystem quota. However, it is difficult to remove devices from a ZFS zpool once set up.
lvm
zfs
Why I use #ZFS 8/ More stable than #btrfs. Although btrfs provides many of these benefits – plus the theoretical benefit of more flexibility with rearranging data – every time I have tried btrfs, I have run into serious bugs. Some of them have been corrected, but not all. https://btrfs.wiki.kernel.org/index.php/Status gives the btrfs status. It still has pathologies with many snapshots, I believe still also with databases, though I think hardlinks have been fixed.
btrfs
zfs
Status – btrfs Wiki
And to conclude, 9/ reasons I don’t use #ZFS.Although I use it on every machine I can, it is somewhat RAM hungry, so I don’t run it on my #RaspberryPi machines or other very old hardware. Though I now have an 8GB Pi on hand that I intend to try it with.It always makes me nervous, though. Running without protection against silent data corruption feels… unsafe 🙂
RaspberryPi
zfs
More Topics on Store-And-Forward (Possibly Airgapped) #ZFS and Non-ZFS Backups with #NNCP | The Changelog ⇨ changelog.complete.org/archives/10186…
Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.
In the previous installment on store-and-forward backups, I mentioned how easy it is to do with ZFS, and some of the tools that can be used to do it without ZFS. A lot of those tools are a bit less robust, so we need some sort of store-and-forward mechanism to verify backups. To be sure, verifying backups is good with ANY scheme, and this could be used with ZFS backups also.
So let’s say you have a shiny new backup scheme in place, and you’d like to verify that it’s working correctly. To do that, you need to compare the source directory tree on machine A with the backed-up directory tree on machine B.
Assuming a conventional setup, here are some ways you might consider to do that:
Just copy everything from machine A to machine B and compare locally
Or copy everything from machine A to a USB drive, plug that into machine B, and compare locally
Use rsync in dry-run mode and see if it complains about anything
The first two options are not particularly practical for large datasets, though I note that the second is compatible with airgapping. Using rsync requires both systems to be online at the same time to perform the comparison.
What would be really nice here is a tool that would write out lots of information about the files on a system: their names, sizes, last modified dates, maybe even sha256sum and other data. This file would be far smaller than the directory tree itself, would compress nicely, and could be easily shipped to an airgapped system via NNCP, UUCP, a USB drive, or something similar.
Tool choices
It turns out there are already quite a few tools in Debian (and other Free operating systems) to do this, and half of them are named mtree (though, of course, not all mtrees are compatible with each other.) We’ll look at some of the options here.
I’ve made a simple test directory for illustration purposes with these commands:
mkdir test
cd test
echo hi > hi
ln -s hi there
ln hi foo
touch empty
mkdir emptydir
mkdir somethingdir
cd somethingdir
ln -s ../there
I then also used touch to set all files to a consistent timestamp for illustration purposes.
Tool option: getfacl (Debian package: acl)
This comes with the acl package, but can be used with other than ACL purposes. Unfortunately, it doesn’t come with a tool to directly compare its output with a filesystem (setfacl, for instance, can apply the permissions listed but won’t compare.) It ignores symlinks and doesn’t show sizes or dates, so is ineffective for our purposes.
Example output:
$ getfacl –numeric -R test
…
# file: test/hi
# owner: 1000
# group: 1000
user::rw-
group::r–
other::r–
…
Tool option: fmtree, the FreeBSD mtree (Debian package: freebsd-buildutils)
fmtree can prepare a “specification” based on a directory tree, and compare a directory tree to that specification. The comparison also is aware of files that exist in a directory tree but not in the specification. The specification format is a bit on the odd side, but works well enough with fmtree. Here’s a sample output with defaults:
$ fmtree -c -p test
…
# .
/set type=file uid=1000 gid=1000 mode=0644 nlink=1
. type=dir mode=0755 nlink=4 time=1610421833.000000000
empty size=0 time=1610421833.000000000
foo nlink=2 size=3 time=1610421833.000000000
hi nlink=2 size=3 time=1610421833.000000000
there type=link mode=0777 time=1610421833.000000000 link=hi
… skipping …
# ./somethingdir
/set type=file uid=1000 gid=1000 mode=0777 nlink=1
somethingdir type=dir mode=0755 nlink=2 time=1610421833.000000000
there type=link time=1610421833.000000000 link=../there
# ./somethingdir
..
..
You might be wondering here what it does about special characters, and the answer is that it has octal escapes, so it is 8-bit clean.
To compare, you can save the output of fmtree to a file, then run like this:
cd test
fmtree
If there is no output, then the trees are identical. Change something and you get a line of of output explaining each difference. You can also use fmtree -U to change things like modification dates to match the specification.
fmtree also supports quite a few optional keywords you can add with -K. They include things like file flags, user/group names, various tipes of hashes, and so forth. I’ll note that none of the options can let you determine which files are hardlinked together.
Here’s an excerpt with -K sha256digest added:
empty size=0 time=1610421833.000000000
sha256digest=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
foo nlink=2 size=3 time=1610421833.000000000
sha256digest=98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
If you include a sha256digest in the spec, then when you verify it with fmtree, the verification will also include the sha256digest. Obviously fmtree -U can’t correct a mismatch there, but of course it will detect and report it.
Tool option: mtree, the NetBSD mtree (Debian package: mtree-netbsd)
mtree produces (by default) output very similar to fmtree. With minor differences (such as the name of the sha256digest in the output), the discussion above about fmtree also applies to mtree.
There are some differences, and the most notable is that mtree adds a -C option which reads a spec and converts it to a “format that’s easier to parse with various tools.” Here’s an example:
$ mtree -c -K sha256digest -p test | mtree -C
. type=dir uid=1000 gid=1000 mode=0755 nlink=4 time=1610421833.0 flags=none
./empty type=file uid=1000 gid=1000 mode=0644 nlink=1 size=0 time=1610421833.0 flags=none
./foo type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none
./hi type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none
./there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=hi time=1610421833.0 flags=none
./emptydir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none
./somethingdir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none
./somethingdir/there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=../there time=1610421833.0 flags=none
Most definitely an improvement in both space and convenience, while still retaining the relevant information. Note that if you want the sha256digest in the formatted output, you need to pass the -K to both mtree invocations. I could have done that here, but it is easier to read without it.
mtree can verify a specification in either format. Given what I’m about to show you about bsdtar, this should illustrate why I bothered to package mtree-netbsd for Debian.
Unlike fmtree, the mtree -U command will not adjust modification times based on the spec, but it will report on differences.
Tool option: bsdtar (Debian package: libarchive-tools)
bsdtar is a fascinating program that can work with many formats other than just tar files. Among the formats it supports is is the NetBSD mtree “pleasant” format (mtree -C compatible).
bsdtar can also convert between the formats it supports. So, put this together: bsdtar can convert a tar file to an mtree specification without extracting the tar file. bsdtar can also use an mtree specification to override the permissions on files going into tar -c, so it is a way to prepare a tar file with things owned by root without resorting to tools like fakeroot.
Let’s look at how this can work:
$ cd test
$ bsdtar –numeric -cf – –format=mtree .
#mtree
. time=1610472086.318593729 mode=755 gid=1000 uid=1000 type=dir
./empty time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=0
./foo nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3
./hi nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3
./ormat