Monthly Archives: June 2023

Using dar for Data Archiving

June 16, 2023Software2, archiving, backup, backups, darJohn Goerzen

This is the third post in a series about data archiving to removable media (optical discs and hard drives). In the first, I explained the difference between backing up and archiving, established goals for the project, and said I’d evaluate git-annex and dar. The second post evaluated git-annex, and now it’s time to look at dar. The series will conclude with a post comparing git-annex with dar.

What is dar?

I could open with the same thing I did with git-annex, just changing the name of the program: “[dar] is a fantastic and versatile program that does… well, it’s one of those things that can do so much that it’s a bit hard to describe.” It is, fundamentally, an archiver like tar or zip (makes one file representing a bunch of other files), but it goes far beyond that. dar’s homepage lays out a comprehensive list of features, which I will try to summarize here.

Dar itself is both a library (with C++ and Python bindings) for interacting with data, and a CLI tool (dar itself).
Alongside this, there is an ecosystem of tools around dar, including GUIs for multiple platforms, backup scripts, and FUSE implementations.
Dar is like tar in that it can read and write files sequentially if desired. Dar archives can be streamed, just like tar archives. But dar takes it further; if you have dar_slave on the remote end, random access is possible over ssh (dramatically speeding up certain operations).
Dar is like zip in that a dar archive contains a central directory (called a catalog) which permits random access to the contents of an archive. In other words, you don’t have to read an entire archive to extract just one file (assuming the archive is on disk or something that itself permits random access). Also, dar can compress each file individually, rather than the tar approach of compressing the archive as a whole. This increases archive performance (dar knows not to try to compress already-compressed data), boosts restore resilience (corruption of one part of an archive doesn’t invalidate the entire rest of it), and boosts restore performance (permitting random access).
Dar can split an archive into multiple pieces called slices, and it can even split member files among the slices. The catalog contains information allowing you to know which slice(s) a given file is saved in.
The catalog can also be saved off in a file of its own (dar calls this an “isolated catalog”). Isolated catalogs record just metadata about files archived.
dar_manager can assemble a database by reading archives or isolated catalogs, letting you know where files are stored and facilitating restores using the minimal number of discs.
Dar supports differential/incremental backups, which record changes since the last backup. These backups record not just additions, but also deletions. dar can optionally use rsync-style binary deltas to minimize the space needed to record changes. Dar does not suffer from GNU tar’s data loss bug with incrementals.
Dar can “slice and dice” archives like Perl does strings. The usage notes page shows how you can merge archives, create decremental archives (where the full backup always reflects the current state of the system, and incrementals go backwards in time instead of forwards), etc. You can change the compression algorithm on an existing archive, re-slice it, etc.
Dar is extremely careful about preserving all metadata: hard links, sparse files, symlinks, timestamps (including subsecond resolution), EAs, POSIX ACLs, resource forks on Mac, detecting files being modified while being read, etc. It makes a nice way to copy directories, sort of similar to rsync -avxHAXS.

So to tie this together for this project, I will set up a 400MB slice size (to mimic what I did with git-annex), and see how dar saves the data and restores it.

Isolated cataloges aren’t strictly necessary for this, but by using them (and/or dar_manager), we can build up a database of files and locations and thus directly compare dar to git-annex location tracking.

Walkthrough: Creating the first archive

As with the git-annex walkthrough, I’ll set some variables to make it easy to remember:

$SOURCEDIR is the directory being backed up
$DRIVE is the directory for backups to be stored in. Since dar can split by a specified size, I don’t need to make separate filesystems to simulate the separate drive experience as I did with git-annex.
$CATDIR will hold isolated catalogs
$DARDB points to the dar_manager database

OK, we can run the backup immediately. No special setup is needed. dar supports both short-form (single-character) parameters and long-form ones. Since the parameters probably aren’t familiar to everyone, I will use the long-form ones in these examples.

Here’s how we create our initial full backup. I’ll explain the parameters below:

$ dar \ --verbose \ --create $DRIVE/bak1 \ --on-fly-isolate $CATDIR/bak1 \ --slice 400M \ --min-digits 2 \ --pause \ --fs-root $SOURCEDIR

Let’s look at each of these parameters:

–verbose does what you expect
–create selects the operation mode (like tar -c) and gives the archive basename
–on-fly-isolate says to write an isolated catalog as well, right while making the archive. You can always create an isolated catalog later (which is fast, since it only needs to read the last bits of the last slice) but it’s more convenient to do it now, so we do. We give the base name for the isolated catalog also.
–slice 400M says to split the archive, and create slices 400MB each.
–min-digits 2 pertains to naming files. Without it, dar would create files named bak1.dar.1, bak1.dar.2, bak1.dar.10, etc. dar works fine with this, but it can be annoying in ls. This is just convenience for humans.
–pause tells dar to pause after writing each slice. This would let us swap drives, burn discs, etc. I do this for demonstration purposes only; it isn’t strictly necessary in this situation. For a more powerful option, dar also supports –execute, which can run commands after each slice.
–fs-root gives the path to actually back up.

This same command could have been written with short options as:

$ dar -v -c $DRIVE/bak1 -@ $CATDIR/bak1 -s 400M -9 2 -p -R $SOURCEDIR

What does it look like while running? Here’s an excerpt:

... Adding file to archive: /acrypt/no-backup/jgoerzen/testdata/[redacted] Finished writing to file 1, ready to continue ? [return = YES | Esc = NO] ... Writing down archive contents... Closing the escape layer... Writing down the first archive terminator... Writing down archive trailer... Writing down the second archive terminator... Closing archive low layer... Archive is closed.

-------------------------------------------- 581 inode(s) saved including 0 hard link(s) treated 0 inode(s) changed at the moment of the backup and could not be saved properly 0 byte(s) have been wasted in the archive to resave changing files 0 inode(s) with only metadata changed 0 inode(s) not saved (no inode/file change) 0 inode(s) failed to be saved (filesystem error) 0 inode(s) ignored (excluded by filters) 0 inode(s) recorded as deleted from reference backup -------------------------------------------- Total number of inode(s) considered: 581 -------------------------------------------- EA saved for 0 inode(s) FSA saved for 581 inode(s) -------------------------------------------- Making room in memory (releasing memory used by archive of reference)... Now performing on-fly isolation... ...

That was easy! Let’s look at the contents of the backup directory:

$ ls -lh $DRIVE total 3.7G -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.01.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.02.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.03.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.04.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.05.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.06.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.07.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.08.dar -rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:29 bak1.09.dar -rw-r--r-- 1 jgoerzen jgoerzen 156M Jun 16 19:33 bak1.10.dar

And the isolated catalog:

$ ls -lh $CATDIR total 37K -rw-r--r-- 1 jgoerzen jgoerzen 35K Jun 16 19:33 bak1.1.dar

The isolated catalog is stored compressed automatically.

Well this was easy. With one command, we archived the entire data set, split into 400MB chunks, and wrote out the catalog data.

Walkthrough: Inspecting the saved archive

Can dar tell us which slice contains a given file? Sure:

$ dar --list $DRIVE/bak1 --list-format=slicing | less Slice(s)|[Data ][D][ EA ][FSA][Compr][S]|Permission| Filemane --------+--------------------------------+----------+----------------------------- ... 1 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- [redacted] 1-2 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- [redacted] 2 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- [redacted] ...

This illustrates the transition from slice 1 to slice 2. The first file was stored entirely in slice 1; the second stored partially in slice 1 and partially in slice 2, and third solely in slice 2. We can get other kinds of information as well.

$ dar --list $DRIVE/bak1 | less [Data ][D][ EA ][FSA][Compr][S]| Permission | User | Group | Size | Date | filename --------------------------------+------------+-------+-------+---------+-------------------------------+------------ [Saved][ ] [-L-][ 0%][X] -rwxr--r-- jgoerzen jgoerzen 24 Mio Mon Mar 5 07:58:09 2018 [redacted] [Saved][ ] [-L-][ 0%][X] -rwxr--r-- jgoerzen jgoerzen 16 Mio Mon Mar 5 07:58:09 2018 [redacted] [Saved][ ] [-L-][ 0%][X] -rwxr--r-- jgoerzen jgoerzen 22 Mio Mon Mar 5 07:58:09 2018 [redacted]

These are the same files I was looking at before. Here we see they are 24MB, 16MB, and 22MB in size, and some additional metadata. Even more is available in the XML list format.

Walkthrough: updates

As with git-annex, I’ve made some changes in the source directory: moved a file, added another, and deleted one. Let’s create an incremental backup now:

$ dar \ --verbose \ --create $DRIVE/bak2 \ --on-fly-isolate $CATDIR/bak2 \ --ref $CATDIR/bak1 \ --slice 400M \ --min-digits 2 \ --pause \ --fs-root $SOURCEDIR

This command is very similar to the earlier one. Instead of writing an archive and catalog named bak1, we write one named bak2. What’s new here is --ref $CATDIR/bak1. That says, make an incremental based on an archive of reference. All that is needed from that archive of reference is the detached catalog. --ref $DRIVE/bak1 would have worked equally well here.

Here’s what I did to the $SOURCEDIR:

Renamed a file to file01-unchanged
Deleted a file
Copied /bin/cp to a file named cp

Let’s see if dar’s command output matches this:

... Adding file to archive: /acrypt/no-backup/jgoerzen/testdata/file01-unchanged Saving Filesystem Specific Attributes for /acrypt/no-backup/jgoerzen/testdata/file01-unchanged Adding file to archive: /acrypt/no-backup/jgoerzen/testdata/cp Saving Filesystem Specific Attributes for /acrypt/no-backup/jgoerzen/testdata/cp Adding folder to archive: [redacted] Saving Filesystem Specific Attributes for [redacted] Adding reference to files that have been destroyed since reference backup... ... -------------------------------------------- 3 inode(s) saved including 0 hard link(s) treated 0 inode(s) changed at the moment of the backup and could not be saved properly 0 byte(s) have been wasted in the archive to resave changing files 0 inode(s) with only metadata changed 578 inode(s) not saved (no inode/file change) 0 inode(s) failed to be saved (filesystem error) 0 inode(s) ignored (excluded by filters) 2 inode(s) recorded as deleted from reference backup -------------------------------------------- Total number of inode(s) considered: 583 -------------------------------------------- EA saved for 0 inode(s) FSA saved for 3 inode(s) -------------------------------------------- ...

Yes, it does. The rename is recorded as a deletion and an addition, since dar doesn’t directly track renames. So the rename plus the deletion account for the two deletions. The rename plus the addition of cp count as 2 of the 3 inodes saved; the third is the modified directory from which files were deleted and moved out.

Let’s see the files that were created:

$ ls -lh $DRIVE/bak2* -rw-r--r-- 1 jgoerzen jgoerzen 18M Jun 16 19:52 /acrypt/no-backup/jgoerzen/dar-testing/drive/bak2.01.dar $ ls -lh $CATDIR/bak2* -rw-r--r-- 1 jgoerzen jgoerzen 22K Jun 16 19:52 /acrypt/no-backup/jgoerzen/dar-testing/cat/bak2.1.dar

What does –list look like now?

Slice(s)|[Data ][D][ EA ][FSA][Compr][S]|Permission| Filemane --------+--------------------------------+----------+----------------------------- [ ][ ] [---][-----][X] -rwxr--r-- [redacted] 1 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- file01-unchanged ... [--- REMOVED ENTRY ----][redacted] [--- REMOVED ENTRY ----][redacted]

Here I show an example of:

A file that was not changed from the initial backup. Its presence was simply noted, but because we’re doing an incremental, the data wasn’t saved.
A file that is saved in this incremental, on slice 1.
The two deleted files

Walkthrough: dar_manager

As we’ve seen above, the two archives (or their detached catalog) give us a complete picture of what files were present at the time of the creation of each archive, and what files were stored in a given archive. We can certainly continue working in that way. We can also use dar_manager to build a comprehensive database of these archives, to be able to find what media is necessary to restore each given file. Or, with dar_manager’s –when parameter, we can restore files as of a particular date.

Let’s try it out. First, we create our database:

$ dar_manager --create $DARDB $ dar_manager --base $DARDB --add $DRIVE/bak1 Auto detecting min-digits to be 2 $ dar_manager --base $DARDB --add $DRIVE/bak2 Auto detecting min-digits to be 2

Here we created the database, and added our two catalogs to it. (Again, we could have as easily used $CATDIR/bak1; either the archive or its isolated catalog will work here.) It’s important to add the catalogs in order.

Let’s do some quick experimentation with dar_manager:

$ dar_manager -v --base $DARDB --list Decompressing and loading database to memory...


dar path         :

dar options      :

database version : 6

compression used : gzip

compression level: 9
archive #   |    path      |    basename

------------+--------------+---------------

        1       /acrypt/no-backup/jgoerzen/dar-testing/drive    bak1

        2       /acrypt/no-backup/jgoerzen/dar-testing/drive    bak2

$ dar_manager --base $DARDB --stat archive # | most recent/total data | most recent/total EA --------------+-------------------------+----------------------- 1 580/581 0/0 2 3/3 0/0

The –list option shows the correlation between dar_manager archive number (1, 2) with filenames (bak1, bak2). It is coincidence here that 1/bak1 and 2/bak2 correlate; that’s not necessarily the case. Most dar_manager commands operate on archive number, while dar commands operate on archive path/basename.

Now let’s see just what files are saved in archive #2, the incremental:

$ dar_manager --base $DARDB --used 2 [ Saved ][ ] [redacted] [ Saved ][ ] file01-unchanged [ Saved ][ ] cp

Now we can also where a file is stored. Here’s one that was saved in the full backup and unmodified in the incremental:

$ dar_manager --base $DARDB --file [redacted] 1 Fri Jun 16 19:15:12 2023 saved absent 2 Fri Jun 16 19:15:12 2023 present absent

(The absent at the end refers to extended attributes that the file didn’t have)

Similarly, for files that were added or removed, they’ll be listed only at the appropriate place.

Walkthrough: Restoration

I’m not going to repeat the author’s full restoration with dar page, but here are some quick examples.

A simple way of doing everything is using incrementals for the whole series. To do that, you’d have bak1 be full, bak2 based on bak1, bak3 based on bak2, bak4 based on bak3, etc. To restore from such a series, you have two options:

Use dar to simply extract each archive in order. It will handle deletions, renames, etc. along the way.
Use dar_manager with the backup database to do manage the process. It may be somewhat more efficient, as it won’t bother to restore files that will later be modified or deleted.

If you get fancy — for instance, bak2 is based on bak1, bak3 on bak2, bak4 on bak1 — then you would want to use dar_manager to ensure a consistent restore is completed. Either way, the process is nearly identical. Also, I figure, to make things easy, you can save a copy of the entire set of isolated catalogs before you finalize each disc/drive. They’re so small, and this would let someone with just the most recent disc build a dar_manager database without having to go through all the other discs.

Anyhow, let’s do a restore using just dar. I’ll make a $RESTOREDIR and do it that way.

$ dar \ --verbose \ --extract $DRIVE/bak1 \ --fs-root $RESTOREDIR \ --no-warn \ --execute "echo Ready for slice %n. Press Enter; read foo"

This –execute lets us see how dar works; this is an illustration of the power it has (above –pause); it’s a snippet interpreted by /bin/sh with %n being one of the dar placeholders. If memory serves, it’s not strictly necessary, as dar will prompt you for slices it needs if they’re not mounted. Anyhow, you’ll see it first reading the last slice, which contains the catalog, then reading from the beginning.

Here we go:

Auto detecting min-digits to be 2 Opening archive bak1 ... Opening the archive using the multi-slice abstraction layer... Ready for slice 10. Press Enter ... Loading catalogue into memory... Locating archive contents... Reading archive contents... File ownership will not be restored du to the lack of privilege, you can disable this message by asking not to restore file ownership [return = YES | Esc = NO] Continuing... Restoring file's data: [redacted] Restoring file's FSA: [redacted] Ready for slice 1. Press Enter ... Ready for slice 2. Press Enter ... -------------------------------------------- 581 inode(s) restored including 0 hard link(s) 0 inode(s) not restored (not saved in archive) 0 inode(s) not restored (overwriting policy decision) 0 inode(s) ignored (excluded by filters) 0 inode(s) failed to restore (filesystem error) 0 inode(s) deleted -------------------------------------------- Total number of inode(s) considered: 581 -------------------------------------------- EA restored for 0 inode(s) FSA restored for 0 inode(s) --------------------------------------------

The warning is because I’m not doing the extraction as root, which limits dar’s ability to fully restore ownership data.

OK, now the incremental:

$ dar \ --verbose \ --extract $DRIVE/bak2 \ --fs-root $RESTOREDIR \ --no-warn \ --execute "echo Ready for slice %n. Press Enter; read foo" ... Ready for slice 1. Press Enter ... Restoring file's data: /acrypt/no-backup/jgoerzen/dar-testing/restore/file01-unchanged Restoring file's FSA: /acrypt/no-backup/jgoerzen/dar-testing/restore/file01-unchanged Restoring file's data: /acrypt/no-backup/jgoerzen/dar-testing/restore/cp Restoring file's FSA: /acrypt/no-backup/jgoerzen/dar-testing/restore/cp Restoring file's data: /acrypt/no-backup/jgoerzen/dar-testing/restore/[redacted directory] Removing file (reason is file recorded as removed in archive): [redacted file] Removing file (reason is file recorded as removed in archive): [redacted file]

This all looks right! Now how about we compare the restore to the original source directory?

$ diff -durN $SOURCEDIR $RESTOREDIR

No changes – perfect.

We could instead do this restore via a single dar_manager command, though annoyingly, we’d have to pass all top-level files/directories to dar_manager –restore. But still, it’s one command, and basically automates and optimizes the dar restores shown above.

Conclusions

Dar makes it extremely easy to just Do The Right Thing when making archives. One command makes a backup. It saves things in simple files. You can make an isolated catalog if you want, and it too is saved in a simple file. You can query what is in the files and where. You can restore from all or part of the files. You can simply play the backups forward, in order, to achieve a full and consistent restore. Or you can load data about them into dar_manager for an optimized restore.

A bit of scripting will be necessary to make incrementals; finding the most recent backup or catalog. If backup files are named with care — for instance, by date — then this should be a pretty easy task.

I haven’t touched on resiliency yet. dar comes with tools for recovering archives that have had portions corrupted or lost. It can also rebuild the catalog if it is corrupted or lost. It adds “tape marks” (or “escape sequences”) to the archive along with the data stream. So every entry in the catalog is actually stored in the archive twice: once alongside the file data, and once at the end in the collected catalog. This allows dar to scan a corrupted file for the tape marks and reconstruct whatever is still intact, even if the catalog is lost. dar also integrates with tools like sha256sum and par2 to simplify archive integrity testing and restoration.

This balances against the need to use a tool (dar, optionally with a GUI frontend) to restore files. I’ll discuss that more in the next post.

Using git-annex for Data Archiving

June 15, 2023Technology1, 2, archiving, backup, backups, git-annexJohn Goerzen

In my recent post about data archiving to removable media, I laid out the difference between backing up and archiving, and also said I’d evaluate git-annex and dar. This post evaluates git-annex. The next will look at dar, and then I’ll make a comparison post.

What is git-annex?

git-annex is a fantastic and versatile program that does… well, it’s one of those things that can do so much that it’s a bit hard to describe. Its homepage says:

git-annex allows managing large files with git, without storing the file contents in git. It can sync, backup, and archive your data, offline and online. Checksums and encryption keep your data safe and secure. Bring the power and distributed nature of git to bear on your large files with git-annex.

I think the particularly interesting features of git-annex aren’t actually included in that list. Among the features of git-annex that make it shine for this purpose, its location tracking is key. git-annex can know exactly which device has which file at which version at all times. Combined with its preferred content settings, this lets you very easily say things like:

“I want exactly 1 copy of every file to exist within the set #1 of backup drives. Here’s a drive in that set; copy to it whatever needs to be copied to satisfy that requirement.”
“Now I have another set of backup drives. Periodically I will swap sets offsite. Copy whatever is needed to this drive in the second set, making sure that there is 1 copy of every file within this set as well, regardless of what’s in the first set.”
“Here’s a directory I want to use to track the status of everything else. I don’t want any copies at all here.”

git-annex can be set to allow a configurable amount of free space to remain on a device, and it will fill it up with whatever copies are necessary up until it hits that limit. Very convenient!

git-annex will store files in a folder structure that mirrors the origin folder structure, in plain files just as they were. This maximizes the ability for a future person to access the content, since it is all viewable without any special tool at all. Of course, for things like optical media, git-annex will essentially be creating what amounts to incrementals. To obtain a consistent copy of the original tree, you would still need to use git-annex to process (export) the archives.

git-annex challenges

In my prior post, I related some challenges with git-annex. The biggest of them – quite poor performance of the directory special remote when dealing with many files – has been resolved by Joey, git-annex’s author! That dramatically improves the git-annex use scenario here! The fixing commit is in the source tree but not yet in a release.

git-annex no doubt may still have performance challenges with repositories in the 100,000+-range, but in that order of magnitude it now looks usable. I’m not sure about 1,000,000-file repositories (I haven’t tested); there is a page about scalability.

A few other more minor challenges remain:

git-annex doesn’t really preserve POSIX attributes; for instance, permissions, symlink destinations, and timestamps are all not preserved. Of these, timestamps are the most important for my particular use case.
If your data set to archive contains Git repositories itself, these will not be included.

I worked around the timestamp issue by using the mtree-netbsd package in Debian. mtree writes out a summary of files and metadata in a tree, and can restore them. To save:

mtree -c -R nlink,uid,gid,mode -p /PATH/TO/REPO -X <(echo './.git') > /tmp/spec

And, after restoration, the timestamps can be applied with:

mtree -t -U -e < /tmp/spec

Walkthrough: initial setup

To use git-annex in this way, we have to do some setup. My general approach is this:

There is a source of data that lives outside git-annex. I'll call this $SOURCEDIR.
I'm going to name the directories holding my data $REPONAME.
There will be a "coordination" git-annex repo. It will hold metadata only, and no data. This will let us track where things live. I'll call it $METAREPO.
There will be drives. For this example, I'll call their mountpoints $DRIVE01 and $DRIVE02. For easy demonstration purposes, I used a ZFS dataset with a refquota set (to observe the size handling), but I could have as easily used a LVM volume, btrfs dataset, loopback filesystem, or USB drive. For optical discs, this would be a staging area or a UDF filesystem.

Let's get started! I've set all these shell variables appropriately for this example, and REPONAME to "testdata". We'll begin by setting up the metadata-only tracking repo.

$ REPONAME=testdata $ mkdir "$METAREPO" $ cd "$METAREPO" $ git init $ git config annex.thin true

There is a sort of complicated topic of how git-annex stores files in a repo, which varies depending on whether the data for the file is present in a given repo, and whether the file is locked or unlocked. Basically, the options I use here cause git-annex to mostly use hard links instead of symlinks or pointer files, for maximum compatibility with non-POSIX filesystems such as NTFS and UDF, which might be used on these devices. thin is part of that.

Let's continue:

$ git annex init 'local hub' init local hub ok (recording state in git...) $ git annex wanted . "include=* and exclude=$REPONAME/*" wanted . ok (recording state in git...)

In a bit, we are going to import the source data under the directory named $REPONAME (here, testdata). The wanted command says: in this repository (represented by the bare dot), the files we want are matched by the rule that says eveyrthing except what's under $REPONAME. In other words, we don't want to make an unnecessary copy here.

Because I expect to use an mtree file as documented above, and it is not under $REPONAME/, it will be included. Let's just add it and tweak some things.

$ touch mtree $ git annex add mtree add mtree ok (recording state in git...) $ git annex sync git-annex sync will change default behavior to operate on --content in a future version of git-annex. Recommend you explicitly use --no-content (or -g) to prepare for that change. (Or you can configure annex.synccontent) commit [main (root-commit) 6044742] git-annex in local hub 1 file changed, 1 insertion(+) create mode 120000 mtree ok $ ls -l total 9 lrwxrwxrwx 1 jgoerzen jgoerzen 178 Jun 15 22:31 mtree -> .git/annex/objects/pX/ZJ/...

OK! We've added a file, and it got transformed into a symlink. That's the thing I said we were going to avoid, so:

git annex adjust --unlock-present adjust Switched to branch 'adjusted/main(unlockpresent)' ok $ ls -l total 1 -rw-r--r-- 2 jgoerzen jgoerzen 0 Jun 15 22:31 mtree

You'll notice it transformed into a hard link (nlinks=2) file. Great! Now let's import the source data. For that, we'll use the directory special remote.

$ git annex initremote source type=directory directory=$SOURCEDIR importtree=yes \ encryption=none initremote source ok (recording state in git...) $ git annex enableremote source directory=$SOURCEDIR enableremote source ok (recording state in git...) $ git config remote.source.annex-readonly true $ git config annex.securehashesonly true $ git config annex.genmetadata true $ git config annex.diskreserve 100M $ git config remote.source.annex-tracking-branch main:$REPONAME

OK, so here we created a new remote named "source". We enabled it, and set some configuration. Most notably, that last line causes files from "source" to be imported under $REPONAME/ as we wanted earlier. Now we're ready to scan the source.

$ git annex sync

At this point, you'll see git-annex computing a hash for every file in the source directory.

I can verify with du that my metadata-only repo only uses 14MB of disk space, while my source is around 4GB.

Now we can see what git-annex thinks about file locations:

$ git-annex whereis | less whereis mtree (1 copy) 8aed01c5-da30-46c0-8357-1e8a94f67ed6 -- local hub [here] ok whereis testdata/[redacted] (0 copies) The following untrusted locations may also have copies: 9e48387e-b096-400a-8555-a3caf5b70a64 -- [source] failed ... many more lines ...

So remember we said we wanted mtree, but nothing under testdata, under this repo? That's exactly what we got. git-annex knows that the files under testdata can be found under the "source" special remote, but aren't in any git-annex repo -- yet. Now we'll start adding them.

Walkthrough: removable drives

I've set up two 500MB filesystems to represent removable drives. We'll see how git-annex works with them.

$ cd $DRIVE01 $ df -h . Filesystem Size Used Avail Use% Mounted on acrypt/no-backup/annexdrive01 500M 1.0M 499M 1% /acrypt/no-backup/annexdrive01 $ git clone $METAREPO Cloning into 'testdata'... done. $ cd $REPONAME $ git config annex.thin true $ git annex init "test drive #1" $ git annex adjust --hide-missing --unlock adjust Switched to branch 'adjusted/main(hidemissing-unlocked)' ok $ git annex sync

OK, that's the initial setup. Now let's enable the source remote and configure it the same way we did before:

$ git annex enableremote source directory=$SOURCEDIR enableremote source ok (recording state in git...) $ git config remote.source.annex-readonly true $ git config remote.source.annex-tracking-branch main:$REPONAME $ git config annex.securehashesonly true $ git config annex.genmetadata true $ git config annex.diskreserve 100M

Now, we'll add the drive to a group called "driveset01" and configure what we want on it:

$ git annex group . driveset01 $ git annex wanted . '(not copies=driveset01:1)'

What this does is say: first of all, this drive is in a group named driveset01. Then, this drive wants any files for which there isn't already at least one copy in driveset01.

Now let's load up some files!

$ git annex sync --content

As the messages fly by from here, you'll see it mentioning that it got mtree, and then various files from "source" -- until, that is, the filesystem had less than 100MB free, at which point it complained of no space for the rest. Exactly like we wanted!

Now, we need to teach $METAREPO about $DRIVE01.

$ cd $METAREPO $ git remote add drive01 $DRIVE01/$REPONAME $ git annex sync drive01 git-annex sync will change default behavior to operate on --content in a future version of git-annex. Recommend you explicitly use --no-content (or -g) to prepare for that change. (Or you can configure annex.synccontent) commit On branch adjusted/main(unlockpresent) nothing to commit, working tree clean ok merge synced/main (Merging into main...) Updating d1d9e53..817befc Fast-forward (Merging into adjusted branch...) Updating 7ccc20b..861aa60 Fast-forward ok pull drive01 remote: Enumerating objects: 214, done. remote: Counting objects: 100% (214/214), done. remote: Compressing objects: 100% (95/95), done. remote: Total 110 (delta 6), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (110/110), 13.01 KiB | 1.44 MiB/s, done. Resolving deltas: 100% (6/6), completed with 6 local objects. From /acrypt/no-backup/annexdrive01/testdata * [new branch] adjusted/main(hidemissing-unlocked) -> drive01/adjusted/main(hidemissing-unlocked) * [new branch] adjusted/main(unlockpresent) -> drive01/adjusted/main(unlockpresent) * [new branch] git-annex -> drive01/git-annex * [new branch] main -> drive01/main * [new branch] synced/main -> drive01/synced/main ok

OK! This step is important, because drive01 and drive02 (which we'll set up shortly) won't necessarily be able to reach each other directly, due to not being plugged in simultaneously. Our $METAREPO, however, will know all about where every file is, so that the "wanted" settings can be correctly resolved. Let's see what things look like now:

$ git annex whereis | less whereis mtree (2 copies) 8aed01c5-da30-46c0-8357-1e8a94f67ed6 -- local hub [here] b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01] ok whereis testdata/[redacted] (1 copy) b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01]

The following untrusted locations may also have copies: 9e48387e-b096-400a-8555-a3caf5b70a64 -- [source] ok

If I scroll down a bit, I'll see the files past the 400MB mark that didn't make it onto drive01. Let's add another example drive!

Walkthrough: Adding a second drive

The steps for $DRIVE02 are the same as we did before, just with drive02 instead of drive01, so I'll omit listing it all a second time. Now look at this excerpt from whereis:

whereis testdata/[redacted] (1 copy) b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01]


  The following untrusted locations may also have copies:

        9e48387e-b096-400a-8555-a3caf5b70a64 -- [source]

ok

whereis testdata/[redacted] (1 copy)

        c4540343-e3b5-4148-af46-3f612adda506 -- test drive #2 [drive02]

The following untrusted locations may also have copies: 9e48387e-b096-400a-8555-a3caf5b70a64 -- [source] ok

Look at that! Some files on drive01, some on drive02, some neither place. Perfect!

Walkthrough: Updates

So I've made some changes in the source directory: moved a file, added another, and deleted one. All of these were copied to drive01 above. How do we handle this?

First, we update the metadata repo:

$ cd $METAREPO $ git annex sync $ git annex dropunused all

OK, this has scanned $SOURCEDIR and noted changes. Let's see what whereis says:

$ git annex whereis | less ... whereis testdata/cp (0 copies) The following untrusted locations may also have copies: 9e48387e-b096-400a-8555-a3caf5b70a64 -- [source] failed whereis testdata/file01-unchanged (1 copy) b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01]

The following untrusted locations may also have copies: 9e48387e-b096-400a-8555-a3caf5b70a64 -- [source] ok

So this looks right. The file I added was a copy of /bin/cp. I moved another file to one named file01-unchanged. Notice that it realized this was a rename and that the data still exists on drive01.

Well, let's update drive01.

$ cd $DRIVE01/$REPONAME $ git annex sync --content

Looking at the testdata/ directory now, I see that file01-unchanged has been renamed, the deleted file is gone, but cp isn't yet here -- probably due to space issues; as it's new, it's undefined whether it or some other file would fill up free space. Let's work along a few more commands.

$ git annex get --auto $ git annex drop --auto $ git annex dropunused all

And now, let's make sure metarepo is updated with its state.

$ cd $METAREPO $ git annex sync

We could do the same for drive02. This is how we would proceed with every update.

Walkthrough: Restoration

Now, we have bare files at reasonable locations in drive01 and drive02. But, to generate a consistent restore, we need to be able to actually do an export. Otherwise, we may have files with old names, duplicate files, etc. Let's assume that we lost our source and metadata repos and have to restore from scratch. We'll make a new $RESTOREDIR. We'll begin with drive01 since we used it most recently.

$ mv $METAREPO $METAREPO.disabled $ mv $SOURCEDIR $SOURCEDIR.disabled $ git clone $DRIVE01/$REPONAME $RESTOREDIR $ cd $RESTOREDIR $ git config annex.thin true $ git annex init "restore" $ git annex adjust --hide-missing --unlock

Now, we need to connect the drive01 and pull the files from it.

$ git remote add drive01 $DRIVE01/$REPONAME $ git annex sync --content

Now, repeat with drive02:

$ git remote add drive02 $DRIVE02/$REPONAME $ git annex sync --content

Now we've got all our content back! Here's what whereis looks like:

whereis testdata/file01-unchanged (3 copies) 3d663d0f-1a69-4943-8eb1-f4fe22dc4349 -- restore [here] 9e48387e-b096-400a-8555-a3caf5b70a64 -- source b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [origin] ok ...

I was a little surprised that drive01 didn't seem to know what was on drive02. Perhaps that could have been remedied by adding more remotes there? I'm not entirely sure; I'd thought would have been able to do that automatically.

Conclusions

I think I have demonstrated two things:

First, git-annex is indeed an extremely powerful tool. I have only scratched the surface here. The location tracking is a neat feature, and being able to just access the data as plain files if all else fails is nice for future users.

Secondly, it is also a complex tool and difficult to get right for this purpose (I think much easier for some other purposes). For someone that doesn't live and breathe git-annex, it can be hard to get right. In fact, I'm not entirely sure I got it right here. Why didn't drive02 know what files were on drive01 and vice-versa? I don't know, and that reflects some kind of misunderstanding on my part about how metadata is synced; perhaps more care needs to be taken in restore, or done in a different order, than I proposed. I initially tried to do a restore by using git annex export to a directory special remote with exporttree=yes, but I couldn't ever get it to actually do anything, and I don't know why.

These two cut against each other. On the one hand, the raw accessibility of the data to someone with no computer skills is unmatched. On the other hand, I'm not certain I have the skill to always prepare the discs properly, or to do a proper consistent restore.

The Changelog

Comments on family, technology, and society

Monthly Archives: June 2023

Using dar for Data Archiving

Using git-annex for Data Archiving