Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.
In the previous installment on store-and-forward backups, I mentioned how easy it is to do with ZFS, and some of the tools that can be used to do it without ZFS. A lot of those tools are a bit less robust, so we need some sort of store-and-forward mechanism to verify backups. To be sure, verifying backups is good with ANY scheme, and this could be used with ZFS backups also.
So let’s say you have a shiny new backup scheme in place, and you’d like to verify that it’s working correctly. To do that, you need to compare the source directory tree on machine A with the backed-up directory tree on machine B.
Assuming a conventional setup, here are some ways you might consider to do that:
- Just copy everything from machine A to machine B and compare locally
- Or copy everything from machine A to a USB drive, plug that into machine B, and compare locally
- Use rsync in dry-run mode and see if it complains about anything
The first two options are not particularly practical for large datasets, though I note that the second is compatible with airgapping. Using rsync requires both systems to be online at the same time to perform the comparison.
What would be really nice here is a tool that would write out lots of information about the files on a system: their names, sizes, last modified dates, maybe even sha256sum and other data. This file would be far smaller than the directory tree itself, would compress nicely, and could be easily shipped to an airgapped system via NNCP, UUCP, a USB drive, or something similar.
Tool choices
It turns out there are already quite a few tools in Debian (and other Free operating systems) to do this, and half of them are named mtree (though, of course, not all mtrees are compatible with each other.) We’ll look at some of the options here.
I’ve made a simple test directory for illustration purposes with these commands:
mkdir test cd test echo hi > hi ln -s hi there ln hi foo touch empty mkdir emptydir mkdir somethingdir cd somethingdir ln -s ../there
I then also used touch to set all files to a consistent timestamp for illustration purposes.
Tool option: getfacl (Debian package: acl)
This comes with the acl package, but can be used with other than ACL purposes. Unfortunately, it doesn’t come with a tool to directly compare its output with a filesystem (setfacl, for instance, can apply the permissions listed but won’t compare.) It ignores symlinks and doesn’t show sizes or dates, so is ineffective for our purposes.
Example output:
$ getfacl --numeric -R test ... # file: test/hi # owner: 1000 # group: 1000 user::rw- group::r-- other::r-- ...
Tool option: fmtree, the FreeBSD mtree (Debian package: freebsd-buildutils)
fmtree can prepare a “specification” based on a directory tree, and compare a directory tree to that specification. The comparison also is aware of files that exist in a directory tree but not in the specification. The specification format is a bit on the odd side, but works well enough with fmtree. Here’s a sample output with defaults:
$ fmtree -c -p test
...
# .
/set type=file uid=1000 gid=1000 mode=0644 nlink=1
. type=dir mode=0755 nlink=4 time=1610421833.000000000
empty size=0 time=1610421833.000000000
foo nlink=2 size=3 time=1610421833.000000000
hi nlink=2 size=3 time=1610421833.000000000
there type=link mode=0777 time=1610421833.000000000 link=hi
... skipping ...
# ./somethingdir
/set type=file uid=1000 gid=1000 mode=0777 nlink=1
somethingdir type=dir mode=0755 nlink=2 time=1610421833.000000000
there type=link time=1610421833.000000000 link=../there
# ./somethingdir
..
..
You might be wondering here what it does about special characters, and the answer is that it has octal escapes, so it is 8-bit clean.
To compare, you can save the output of fmtree to a file, then run like this:
cd test fmtree < ../test.fmtree
If there is no output, then the trees are identical. Change something and you get a line of of output explaining each difference. You can also use fmtree -U to change things like modification dates to match the specification.
fmtree also supports quite a few optional keywords you can add with -K. They include things like file flags, user/group names, various tipes of hashes, and so forth. I'll note that none of the options can let you determine which files are hardlinked together.
Here's an excerpt with -K sha256digest added:
empty size=0 time=1610421833.000000000 \
sha256digest=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
foo nlink=2 size=3 time=1610421833.000000000 \
sha256digest=98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
If you include a sha256digest in the spec, then when you verify it with fmtree, the verification will also include the sha256digest. Obviously fmtree -U can't correct a mismatch there, but of course it will detect and report it.
Tool option: mtree, the NetBSD mtree (Debian package: mtree-netbsd)
mtree produces (by default) output very similar to fmtree. With minor differences (such as the name of the sha256digest in the output), the discussion above about fmtree also applies to mtree.
There are some differences, and the most notable is that mtree adds a -C option which reads a spec and converts it to a "format that's easier to parse with various tools." Here's an example:
$ mtree -c -K sha256digest -p test | mtree -C . type=dir uid=1000 gid=1000 mode=0755 nlink=4 time=1610421833.0 flags=none ./empty type=file uid=1000 gid=1000 mode=0644 nlink=1 size=0 time=1610421833.0 flags=none ./foo type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none ./hi type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none ./there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=hi time=1610421833.0 flags=none ./emptydir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none ./somethingdir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none ./somethingdir/there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=../there time=1610421833.0 flags=none
Most definitely an improvement in both space and convenience, while still retaining the relevant information. Note that if you want the sha256digest in the formatted output, you need to pass the -K to both mtree invocations. I could have done that here, but it is easier to read without it.
mtree can verify a specification in either format. Given what I'm about to show you about bsdtar, this should illustrate why I bothered to package mtree-netbsd for Debian.
Unlike fmtree, the mtree -U command will not adjust modification times based on the spec, but it will report on differences.
Tool option: bsdtar (Debian package: libarchive-tools)
bsdtar is a fascinating program that can work with many formats other than just tar files. Among the formats it supports is is the NetBSD mtree "pleasant" format (mtree -C compatible).
bsdtar can also convert between the formats it supports. So, put this together: bsdtar can convert a tar file to an mtree specification without extracting the tar file. bsdtar can also use an mtree specification to override the permissions on files going into tar -c, so it is a way to prepare a tar file with things owned by root without resorting to tools like fakeroot.
Let's look at how this can work:
$ cd test $ bsdtar --numeric -cf - --format=mtree . #mtree . time=1610472086.318593729 mode=755 gid=1000 uid=1000 type=dir ./empty time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=0 ./foo nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3 ./hi nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3 ./ormat\075mtree time=1610472086.318593729 mode=644 gid=1000 uid=1000 type=file size=5632 ./there time=1610421833.0 mode=777 gid=1000 uid=1000 type=link link=hi ./emptydir time=1610421833.0 mode=755 gid=1000 uid=1000 type=dir ./somethingdir time=1610421833.0 mode=755 gid=1000 uid=1000 type=dir ./somethingdir/there time=1610421833.0 mode=777 gid=1000 uid=1000 type=link link=../there
You can use mtree -U to verify that as before. With the --options mtree: set, you can also add hashes and similar to the bsdtar output. Since bsdtar can use input from tar, pax, cpio, zip, iso9660, 7z, etc., this capability can be used to create verification of the files inside quite a few different formats. You can convert with bsdtar -cf output.mtree --format=mtree @input.tar. There are some foibles with directly using these converted files with mtree -U, but usually minor changes will get it there.
Side mention: stat(1) (Debian package: coreutils)
This tool isn't included because it won't operate recursively, but is a tool in the similar toolbox.
Putting It Together
I will still be developing a complete non-ZFS backup system for NNCP (or UUCP) in a future post. But in the meantime, here are some ideas you can reflect on:
- Let's say your backup scheme involves sending a full backup every night. On the source system, you could pipe the generated tar file through something like tee >(bsdtar -cf bcakup.mtree @-) to generate an mtree file in-band while generating the tar file. This mtree file could be shipped over for verification.
- Perhaps your backup scheme involves sending incremental backup data via rdup or even ZFS, but you would like to periodically verify that everything is good -- that an incremental didn't miss something. Something like mtree -K sha256 -c -x -p / | mtree -C -K sha256 would let you accomplish that.
I will further develop at least one of these ideas in a future post.
Bonus: cross-tool comparisons
In my mtree-netbsd packaging, I added tests like this to compare between tools:
fmtree -c -K $(MTREE_KEYWORDS) | mtree mtree -c -K $(MTREE_KEYWORDS) | sed -e 's/\(md5\|sha1\|sha256\|sha384\|sha512\)=/\1digest=/' -e 's/rmd160=/ripemd160digest=/' | fmtree bsdtar -cf - --options 'mtree:uname,gname,md5,sha1,sha256,sha384,sha512,device,flags,gid,link,mode,nlink,size,time,uid,type,uname' --format mtree . | mtree