How git-annex replaces Dropbox + encfs with untrusted providers

June 6, 2016UncategorizedJohn Goerzen

git-annex has been around for a long time, but I just recently stumbled across some of the work Joey has been doing to it. This post isn’t about it’s traditional roots in git or all the features it has for partial copies of large data sets, but rather for its live syncing capabilities like Dropbox. It takes a bit to wrap your head around, because git-annex is just a little different from everything else. It’s sort of like a different-colored smell.

The git-annex wiki has a lot of great information — both low-level reference and a high-level 10-minute screencast showing how easy it is to set up. I found I had to sort of piece together the architecture between those levels, so I’m writing this all down hoping it will benefit others that are curious.

Ir you just want to use it, you don’t need to know all this. But I like to understand how my tools work.

Overview

git-annex lets you set up a live syncing solution that requires no central provider at all, or can be used with a completely untrusted central provider. Depending on your usage pattern, this central provider could require only a few MBs of space even for repositories containing gigabytes or terabytes of data that is kept in sync.

Let’s take a look at the high-level architecture of the tool. Then I’ll illustrate how it works with some scenarios.

Three Layers

Fundamentally, git-annex takes layers that are all combined in Dropbox and separates them out. There is the storage layer, which stores the literal data bytes that you are interested in. git-annex indexes the data in storage by a hash. There is metadata, which is for things like a filename-to-hash mapping and revision history. And then there is an optional layer, which is live signaling used to drive the real-time syncing.

git-annex has several modes of operation, and the one that enables live syncing is called the git-annex assistant. It runs as a daemon, and is available for Linux/POSIX platforms, Windows, Mac, and Android. I’ll be covering it here.

The storage layer

The storage layer simply is blobs of data. These blobs are indexed by a hash, and can be optionally encrypted at rest at remote backends. git-annex has a large number of storage backends; some examples include rsync, a remote machine with git-annex on it that has ssh installed, WebDAV, S3, Amazon Glacier, removable USB drive, etc. There’s a huge list.

One of the git-annex features is that each client knows the state of each storage repository, as well as the capability set of each storage repository. So let’s say you have a workstation at home and a laptop you take with you to work or the coffee shop. You’d like changes on one to be instantly recognized on another. With something like Dropbox or OwnCloud, every file in the set you want synchronized has to reside on a server in the cloud. With git-annex, it can be configured such that the server in the cloud only contains a copy of a file until every client has synced it up, at which point it gets removed. Think about it – that is often what you want anyhow, so why maintain an unnecessary copy after it’s synced everywhere? (This behavior is, of course, configurable.) git-annex can also avoid storing in the cloud entirely if the machines are able to reach each other directly at least some of the time.

The metadata layer

Metadata about your files includes a mapping from the file names to the storage location (based on hashes), change history, and information about the status of each machine that participates in the syncing. On your clients, git-annex stores this using git. This detail is very useful to some, and irrelevant to others.

Some of the git-annex storage backends can support only storage (S3, for instance). Some can support both storage and metadata (rsync, ssh, local drives, etc.) You can even configure a backend to support only metadata (more on why that may be useful in a bit). When you are working with a git-backed repository for git-annex, it can hold data, metadata, or both.

So, to have a working sync system, you must have a way to transport both the data and the metadata. The transport for the metadata is generally rsync or git, but it can also be XMPP in which Git changesets are basically wrapped up in XMPP presence messages. Joey says, however, that there are some known issues with XMPP servers sometimes dropping or reordering some XMPP messages, so he doesn’t encourage that method currently.

The live signaling layer

So once you have your data and metadata, you can already do syncs via git annex sync --contents. But the real killer feature here will be automatic detection of changes, both on the local and the remote. To do that, you need some way of live signaling. git-annex supports two methods.

The first requires ssh access to a remote machine where git-annex is installed. In this mode of operation, when the git-annex assistant fires up, it opens up a persistent ssh connection to the remote and runs the git-annex-shell over there, which notifies it of changes to the git metadata repository. When a change is detected, a sync is initiated. This is considered ideal.

A substitute can be XMPP, and git-annex actually converts git commits into a form that can be sent over XMPP. As I mentioned above, there are some known reliability issues with this and it is not the recommended option.

Encryption

When it comes to encryption, you generally are concerned about all three layers. In an ideal scenario, the encryption and decryption happens entirely on the client side, so no service provider ever has any details about your data.

The live signaling layer is encrypted pretty trivially; the ssh sessions are, of course, encrypted and TLS support in XMPP is pervasive these days. However, this is not end-to-end encryption; those messages are decrypted by the service provider, so a service provider could theoretically spy on metadata, which may include change times and filenames, though not the contents of files themselves.

The data layer also can be encrypted very trivially. In the case of the “dumb” backends like S3, git-annex can use symmetric encryption or a gpg keypair and all that ever shows up on the server are arbitrarily-named buckets.

You can also use a gcrypt-based git repository. This can cover both data and metadata — and, if the target also has git-annex installed, the live signalling layer. Using a gcrypt-based git repository for the metadata and live signalling is the only way to accomplish live syncing with 100% client-side encryption.

All of these methods are implemented in terms of gpg, and can support symmetric of public-key encryption.

It should be noted here that the current release versions of git-annex need a one-character patch in order to fix live syncing with a remote using gcrypt. For those of you running jessie, I recommend the version in jessie-backports, which is presently 5.20151208. For your convenience, I have compiled an amd64 binary that can drop in over /usr/bin/git-annex if you have this version. You can download it and a gpg signature for it. Note that you only need this binary on the clients; the server can use the version from jessie-backports without issue.

Putting the pieces together: some scenarios

Now that I’ve explained the layers, let’s look at how they fit together.

Scenario 1: Central server

In this scenario, you might have a workstation and a laptop that sync up with each other by way of a central server that also has a full copy of the data. This is the scenario that most closely resembles Dropbox, box, or OwnCloud.

Here you would basically follow the steps in the git-assistant screencast: install git-annex on a server somewhere, and point your clients to it. If you want full end-to-end encryption, I would recommend letting git-annex generate a gpg keypair for you, which you would then need to copy to both your laptop and workstation (but not the server).

Every change you make locally will be synced to the server, and then from the server to your other PC. All three systems would be configured in the “client” transfer group.

Scenario 1a: Central server without a full copy of the data

In this scenario, everything is configured the same except the central server is configured with the “transfer” transfer group. This means that the actual data synced to it is deleted after it has been propagated to all clients. Since git-annex can verify which repository has received a copy of which data, it can easily enough delete the actual file content from the central server after it has been copied to all the clients. Many people use something like Dropbox or OwnCloud as a multi-PC syncing solution anyhow, so once the files have been synced everywhere, it makes sense to remove them from the central server.

This is often a good ideal for people. There are some obvious downsides that are sometimes relevant. For instance, to add a third sync client, it must be able to initially copy down from one of the existing clients. Or, if you intend to access the data from a device such as a cell phone where you don’t intend for it to have a copy of all data all the time, you won’t have as convenient way to download your data.

Scenario 1b: Split data/metadata central servers

Imagine that you have a shell or rsync account on some remote system where you can run git-annex, but don’t have much storage space. Maybe you have a cheap VPS or shell account somewhere, but it’s just not big enough to hold your data.

The answer to this would be to use this shell or rsync account for the metadata, but put the data elsewhere. You could, for instance, store the data in Amazon S3 or Amazon Glacier. These backends aren’t capable of storing the git-annex metadata, so all you need is a shell or rsync account somewhere to sync up the metadata. (Or, as below, you might even combine a fully distributed approach with this.) Then you can have your encrypted data pushed up to S3 or some such service, which presumably will grow to whatever size you need.

Scenario 2: Fully distributed

Like git itself, git-annex does not actually need a central server at all. If your different clients can reach each other directly at least some of the time, that is good enough. Of course, a given client will not be able to do fully automatic live sync unless it can reach at least one other client, so changes may not propagate as quickly.

You can simply set this up by making ssh connections available between your clients. git-annex assistant can automatically generate appropriate ~/.ssh/authorized_keys entries for you.

Scenario 2a: Fully distributed with multiple disconnected branches

You can even have a graph of connections available. For instance, you might have a couple machines at home and a couple machines at work with no ability to have a direct connection between them (due to, say, firewalls). The two machines at home could sync with each other in real-time, as could the two machines at work. git-annex also supports things like USB drives as a transport mechanism, so you could throw a USB drive in your pocket each morning, pop it in to one client at work, and poof – both clients are synced up over there. Repeat when you get home in the evening, and you’re synced there. The USB drive’s repository can, of course, be of the “transport” type so data is automatically deleted from it once it’s been synced everywhere.

Scenario 3: Hybrid

git-annex can support LAN sync even if you have a central server. If your laptop, say, travels around but is sometimes on the same LAN as your PC, git-annex can easily sync directly between the two when they are reachable, saving a round-trip to the server. You can assign a cost to each remote, and git-annex will always try to sync first to the lowest-cost path that is available.

Drawbacks of git-annex

There are some scenarios where git-annex with the assistant won’t be as useful as one of the more traditional instant-sync systems.

The first and most obvious one is if you want to access the files without the git-annex client. For instance, many of the other tools let you generate a URL that you can email to people, and then they can download files without any special client software. This is not directly possible with git-annex. You could, of course, make something like a public_html directory be managed with git-annex, but it wouldn’t provide things like obfuscated URLs, password-protected sharing, time-limited sharing, etc. that you get with other systems. While you can share your repositories with others that have git-annex, you can’t share individual subdirectories; for a given repository, it is all or nothing.

The Android client for git-annex is a pretty interesting thing: it is mostly a small POSIX environment, providing a terminal, git, gpg, and the same web interface that you get on a standalone machine. This means that the git-annex Android client is fully functional compared to a desktop one. It also has a quick setup process for syncing off your photos/videos. On the other hand, the integration with the Android ecosystem is poor compared to most other tools.

Other git-annex features

git-annex has a lot to offer besides the git-annex assistant. Besides the things I’ve already mentioned, any given git-annex repository — including your client repository — can have a partial copy of the full content. Say, for instance, that you set up a git-annex repository for your music collection, which is quite large. You want some music on your netbook, but don’t have room for it all. You can tell git-annex to get or drop files from the netbook’s repository without deleting them remotely. git-annex has quite a few ways to automate and configure this, including making sure that at least a certain number of copies of a file exist in your git-annex ecosystem.

Conclusion

I initially started looking at git-annex due to the security issues with encfs, and the difficulty with setting up ecryptfs in this way. (I had been layering encfs atop OwnCloud). git-annex certainly ticks the box for me security-wise, and obviously anything encrypted with encfs wasn’t going to be shared with others anyhow. I’ll be using git-annex more in the future, I’m sure.

Update 2016-06-27: I had some issues with git-annex in this configuration.

That was satisfying

May 30, 2016UncategorizedJohn Goerzen

It’s been awhile due to all sorts of other stuff going on. Nice to see this clogging my inbox again:

It really is satisfying to close bugs!

The Simple Joys of the Plains

May 8, 2016Aviation, FamilyJohn Goerzen

We love to go exploring as a family. Last year, we gave Jacob and Oliver a theme: “find places older than Grandpa.” They got creative really quick, realizing that any state park counts (“dirt is older than grandpa!”) as did pretty much any museum. Probably our hit from last year was the visit to the tunnels under Ellinwood, KS.

Beatrice, NE

This year, our theme is “places we can fly to”. A couple of weeks ago, Laura had a conference in the beautiful small town of Beatrice, NE. So all four of us flew up, and Jacob, Oliver, and I found fun activities while Laura was at her conference.

We walked around Beatrice a bit, and I noticed this rails-to-trails area. Jacob and Oliver were immediately interested (since it was railroad-related). They quickly turned it into a game of kick-the-dandelion, trying to kick dandelions off their stems and see how high in the air they could get them. The answer: pretty high.

Of course, you can’t go wrong with swimming. Here’s Oliver getting ready for some swimming.

Right near Beatrice is the Homestead National Monument. Of course, the bales decorated like a minion got their attention.

Like the other national parks, this one has a junior ranger program. You complete a few things in an activity book and take a pledge to protect the park, and then you get a badge and some stickers. Here’s Oliver proudly taking his pledge, holding the new raccoon he bought in their gift shop.

Canyon, TX

Laura and I have been to Canyon, TX, twice — the first was for our honeymoon. Yes, we did get some strange looks when we told people we were going to Amarillo for our honeymoon. But it was absolutely perfect for us. We both enjoy the simple gifts of nature.

We kept thinking “we’ve got to take the boys here”. So this weekend, we did. We flew a Cessna out there.

Almost every little general aviation airport seems to have a bowl of candy, a plate of cookies, or some such thing for people that are flying through. I often let Jacob and Oliver choose ONE item.

They hit the jackpot when we stopped at West Woodward, Oklahoma for fuel and a break. Two whole fridges stocked with stuff: cans of pop in one, and all sorts of snacks in the other. In typical GA fashion, there was a jar in the fridge asking for $1 if you took something. And it clearly hadn’t been emptied in awhile.

They also had a nice lounge and a patio. Perfect for munching while watching the activities on the ramp.

After landing at the beautiful little Tradewind Airport in Amarillo, we ate dinner at Feldman’s Wrong-Way Diner in Canyon, TX. Oh my, was that ever popular with the boys.

The eagerly looked around to find anything that was “wrong” — a plane hanging upside down from the ceiling, a direction sign saying “Tattoine – 30 parsecs”, movie posters hung upside down, whatever it might be. The fact that model trains were whirring past overhead certainly didn’t hurt either.

They had a giant bin of crayons by the entrance. Jacob and Oliver each grabbed a fistful, and decided it would be fun to do some math problems while we wait. Oliver particularly got into that, and was quite accurate on his large addition problems. Impressive for a first-grader!

Of course, the big highlight of the area is Palo Duro Canyon. Jacob and Oliver were so eager to explore the canyon that they were just about bubbling over with excitement the night before. They decided that we should explore one of the most difficult trails in the canyon – one that would take us from the bottom of the canyon all the way to the top and back, about 2.5 miles each way.

And they LOVED it. We’d stop every few minutes to climb on some rocks, smash up some pieces of sandstone, munch on a snack, or even watch a lizard scurry past.

At the “trading post” in the canyon, both boys explored the gift shop. Jacob happily purchased a Texas magnet and Palo Duro Canyon keychain, which he carried around the rest of the weekend. Oliver loves stuffed animals, and he bought a cuddly little (but long) snake. When we got back to the hotel, he tied a couple of knots in it, and it became “snake airlines”. Here is the snake airline taking off.

He named it “Rattletail the friendly snake”, which I thought was a pretty nifty name.

The hotel’s waffle maker made Texas-shaped waffles, clearly a hit!

Saturday, we explored the absolutely massive Panhandle-Plains Historical Museum. (How does something that huge wind up in Canyon, TX?) Both boys enjoyed spending hours there. Here’s Oliver in Pioneer Town (an indoor recreation of a 1900s town) sending a telegraph message.

Oliver wanted to help with the plane. He helped me tie it down in Amarillo, helped check it over during preflight, basically got involved in every part of it. Jacob studied aviation maps (sectionals) with me, planning our flight, figuring out how fast we’d go. I loaded Avare (an Android app) on an old tablet for him, so he had aviation maps in the cockpit just like me. He would be telling us how fast we were going every so often, pointing out landmarks, etc.

When it was time to head back home, both boys wanted to stay longer — a sure sign of a good trip. They wanted to hike another trail in the canyon, go back to the museum, and “eat at Feldman’s 18 more times.” (We got there twice, which was plenty for Laura and me!)

On our drive home, Oliver said, “Dad-o, will you teach me to be a pilot? You should be my flight instructor. Then I could fly everywhere with you.”

Now that just makes a dad’s day.

Count me as a systemd convert

April 21, 2016DebianJohn Goerzen

Back in 2014, I wrote about some negative first impressions of systemd. I also had a plea to debian-project to end all the flaming, pointing out that “jessie will still boot”, noting that my preference was for sysvinit but things are what they are and it wasn’t that big of a deal.

Although I still have serious misgivings about the systemd upstream’s attitude, I’ve got to say I find the system rather refreshing and useful in practice.

Here’s an example. I was debugging the boot on a server recently. It mounts a bunch of NFS filesystems and runs a third-party daemon that is started from an old-style /etc/init.d script.

We had a situation where the NFS filesystems the daemon required didn’t mount on boot. The daemon then was started, and unfortunately it basically does a mkdir -p on startup. So it started running and processing requests with negative results.

So there were two questions: why did the NFS filesystems fail to start, and how could we make sure the daemon wouldn’t start without them mounted? For the first, journalctl -xb was immensely helpful. It logged the status of each individual mount, and it turned out that it looked like a modprobe or kernel race condition when a bunch of NFS mounts were kicked off in parallel and all tried to load the nfsv4 module at the same time. That was easy enough to work around by adding nfsv4 to /etc/modules. Now for the other question: refusing to start the daemon if the filesystems weren’t there.

With systemd, this was actually trivial. I created /etc/systemd/system/mydaemon.service.requires (I’ll call the service “mydaemon” here), and in it I created a symlink to /lib/systemd/system/remote-fs.target. Then systemctl daemon-reload, and boom, done. systemctl list-dependencies mydaemon will even show the the dependency tree, color-coded status of each item on it, and will actually show every single filesystem that remote-fs requires and the status of it in one command. Super handy.

In a non-systemd environment, I’d probably be modifying the init script and doing a bunch of manual scripting to check the filesystems. Here, one symlink and one command did it, and I get tools to inspect the status of the mydaemon prerequisites for free.

I’ve got to say, as someone that has occasionally had to troubleshoot boot ordering and update-rc.d symlink hell, troubleshooting this stuff in systemd is considerably easier and the toolset is more powerful. Yes, it has its set of poorly-documented complexity, but then so did sysvinit.

I never thought the “world is falling” folks were right, but by now I can be counted among those that feels like systemd has matured to the point where it truly is superior to sysvinit. Yes, in 2014 it had some bugs, but by here in 2016 it looks pretty darn good and I feel like Debian’s decision has been validated through my actual experience with it.

A Year of Flight

April 15, 2016AviationJohn Goerzen

“Dad-o, I’m so glad you’re a pilot!”

My 9-year-old son Jacob has been saying that, always with a big hug and his fond nickname for me (“dad-o”). It has now been a year since the first time I sat in the pilot’s seat of a plane, taking my first step towards exploring the world from the sky. And now, one year after I first sat in the pilot’s seat of an airborne plane, it’s prompted me to think back to my own memories.

Flying over the airport at Moundridge, KS

Memories

Back when I was a child, maybe about the age my children are now, I’d be outside in the evening and see this orange plane flying overhead. Our neighbor Don had a small ultralight plane and a grass landing strip next to his house. I remember longing to be up in the sky with Don, exploring the world from up there. At that age, I didn’t know all the details of why that wouldn’t work — I just knew I wanted to ride in it.

It wasn’t until I was about 11 that I flew for the first time. I still remember that TWA flight with my grandma, taking off early in the morning and flying just a little ways above the puffy clouds lit up all yellow and orange by the sunrise. Even 25 years later, that memory still holds as one of the most beautiful scenes I have ever seen.

Exploring

I have always been an explorer.

When I go past something interesting, I love to go see what it looks like inside. I enjoy driving around Kansas with Laura, finding hidden waterfalls, old county courthouses, ghost towns, beautiful old churches, even small-town restaurants. I explore things around me, too — once taking apart a lawnmower engine as a child, nowadays building HF antennas in my treetops or writing code for Linux. If there is little to learn about something, it becomes less interesting to me.

I see this starting to build in my children, too. Since before they could walk, if we were waiting for something in a large building, we’d “go exploring.”

A patch of rain over Hillsboro, KS

The New World

A pilot once told me, “Nobody can become a pilot without it changing the way they see the world — and then, changing their life.”

I doubted that. But it was true. One of the most poetic sights I know is flying a couple thousand feet above an interstate highway at night, following it to my destination. All those red and white lights, those metal capsules of thousands of lives and thousands of stories, stretching out as far as the eye can see in either direction.

Kansas sunset from the plane

When you’re in a plane, that small town nowhere near a freeway that always seemed so far away suddenly is only a 15-minute flight away, not even enough time to climb up to a high cruise altitude. Two minutes after takeoff, any number of cities that are an hour’s drive away are visible simultaneously, their unique features already recognizable: a grain elevator, oil refinery, college campus, lake, whatever.

And all the houses you fly over — each with people in them. Some pretty similar to you, some apparently not. But pretty soon you realize that we all are humans, and we aren’t all that different. You can’t tell a liberal from a conservative from the sky, nor a person’s race or religion, nor even see the border between states. Towns and cities are often nameless from the sky, unless you’re really low; only your navigation will tell you where you are.

I’ve had the privilege to fly to small out-of-the-way airports, the kind that have a car that pilots can use for free to go into town and get lunch, and leave the key out for them. There I’ve met many friendly people. I’ve also landed my little Cessna at a big commercial airport where I probably used only 1/10th of the runway, on a grass runway that was barely maintained at all. I’ve flown to towns I’d driven to or through many times, discovering the friendly folks at the small airport out of town. I’ve flown to parts of Kansas I’ve never been to before, discovered charming old downtowns and rolling hills, little bursts of rain and beautiful sunsets that seem to turn into a sea.

Parked at the Smith Center, KS airport terminal, about to meet some wonderful people

For a guy that loves exploring the nooks and crannies of the world that everyone else drives by on their way to a major destination, being a pilot has meant many soul-filling moments.

Hard Work

I knew becoming a pilot would be a lot of hard work, and thankfully I remembered stories like that when I finally concluded it would be worth it. I found that I had an aptitude for a lot of things that many find difficult about being a pilot: my experience with amateur radio made me a natural at talking to ATC, my fascination with maps and navigation meant I already knew how to read aviation sectional maps before I even started my training and knew how to process that information in the cockpit, my years as a system administrator and programmer trained me with a careful and methodical decision-making process. And, much to the surprise of my flight instructor, I couldn’t wait to begin the part of training about navigating using VORs (VHF radio beacons). I guess he, like many student pilots, had struggled with that, but I was fascinated by this pre-GPS technology (which I still routinely use in my flight planning, as a backup in case the GPS constellation or a GPS receiver fails). So that left the reflexes of flight, the “art” of it, as the parts I had to work on the hardest.

The exam with the FAA is not like getting your driver’s license. It’s a multi-stage and difficult process. So when the FAA Designated Pilot Examiner said “congratulations, pilot!” and later told my flight instructor that “you did a really good job with this one,” I felt a true sense of accomplishment.

Some of my prep materials

Worth It

Passengers in a small plane can usually hear all the radio conversations going on. My family has heard me talking to air traffic control, to small and big planes. My 6-year-old son Oliver was playing yesterday, and I saw him pick up a plane and say this:

“Two-four-niner-golf requesting to land on runway one-seven…. Two-four-niner-golf back-taxi on one-seven… Two-four-niner-golf ready to takeoff on runway one-seven!”

That was a surprisingly accurate representation of some communication a pilot might have (right down to the made-up tailnumber with the spelling alphabet!)

It just got more involved from there!

Jacob and Oliver love model train shows. I couldn’t take them to one near us, but there was one in Joplin, MO. So the day before Easter, while Laura was working on her Easter sermon, two excited boys and I (frankly also excited) climbed into a plane and flew to Joplin.

We had a great time at the train show, discovered a restaurant specializing in various kinds of hot dogs (of course they both wanted to eat there), played in a park, explored the city, and they enjoyed the free cookies at the general aviation terminal building while I traded tips on fun places to fly with other pilots.

When it comes right down to it, the smiles of the people I fly with are the most beautiful thing in the air.

Jacob after his first father-son flight with me

Free cars, sunsets, and Kansas

March 23, 2016Aviation, Family, TravelJohn Goerzen

“Will you have a car I can borrow?” I asked.

“Sure. No charge. There’s a sign telling you where to find the key.”

It is pretty common for small airports to have a car for a pilot to borrow when flying in. This lets a person go into town for lunch, or visit friends. And it’s usually free, with a can to donate a few bucks or a polite request to fill up the tank when you’re done.

Still, when I had called ahead to ask about flying into the airport in a small town in north-central Kansas, I hadn’t expected to be told to just waltz into the place and take the key. But they had no staff at the airport most of the time. So, to me — another person from a small town — it made perfect sense. Somehow, because of that phone call, this town I had visited once, maybe 25 years ago, seemed instantly familiar.

My mom grew up in a small town near there. She wanted me to see where she grew up, to meet some people that meant a lot to her. As it’s quite a distance from home, I offered to fly her there. So, Laura, mom, and I climbed into a Cessna one morning for the flight northwest. We touched down at the airport, and I pulled the plane up to the little terminal building.

After I took care of parking the plane, I went to find the car. Except the car was missing. Some other pilot had flown in the same day and was using it, according to the logbook on the desk. I called the number on a sign — which rang to the sheriff’s office — and they confirmed it. According to the logbook, this was only the third time that car had been driven since Thanksgiving. Were we stuck at the airport a few miles out of down?

Nope. Mom called the people we were going to meet, a wonderful couple in their upper 80s. They drove out to pick us up. I’m rather glad the car was gone, because I had such a great time visiting with them. Norris told me about the days when the state highways were gravel — how they’d have to re-blade them every few days due to all the traffic. I heard about what happened when the people in that community heard of some folks in Africa in need of car equipment — they modified a tractor to fit in a shipping container and shipped it to Africa, along with a lot of books, blankets, supplies, and anything else needed to fill up a huge shipping container. Sounds like something people around here would do.

We drove around a couple of the small towns. The town my mom grew up in has seen better days. Its schools closed years ago, the old hotel whose owner gave her piano lessons is condemned, and many houses have been lost. But the town lives on. A new community center was built a few years ago. The grain elevator is expanding. Every time a business on Main Street closes, the grocery store expands a little bit: it’s now a grocery store with a little hardware store and a little restaurant mixed in. “The mall”, as the locals jokingly call it. And, of course, two beautiful small churches still meet every Sunday. Here’s the one my mom attended as a child.

We drove past the marker at the geographic center of the contiguous United States. Norris saw some other visitors, rolled down his windows, and treated them — and us — to an unexpected story of the time thousands of people banded together to completely build a house in a single day, just down the road. Smiles all around.

So here I was, nearly 200 miles from home, in an unfamiliar town – but one where I could just feel the goodness. After spending a few hours with these people, I felt like they were old friends.

As I flew us home, I spotted one of my favorite Kansas sights: a beautiful sunset. From the plane, it almost looks like the land at the horizon turns blue like the ocean, and above it the last hint of sun paints the canvas-sky.

In this week of controversy, politics, and reports of violence, it reminds me that we all get the privilege of sharing this beautiful Earth. I didn’t ask anybody on that trip about their politics, religion, or opinions on any of the divisive issues of the day. Whether they agree with me on those things or not is irrelevant. I got to spend a day with good-hearted and delightful people, so I flew back with a smile.

Bach, Dot Matrix Printers, and Dinner

January 27, 2016Children & Computing, Family, HardwareJohn Goerzen

Dinner last night started out all normal. Then Jacob and Oliver started asking me about printers. First they wanted to know how an ink jet printer works. Then they wanted to know how a laser printer works. Then they wanted to know what would happen if you’d put ink in a laser printer or toner in an ink jet. They were fascinated as I described the various kinds of clogging and ruining that would inevitably occur.

Then these words: “What other kinds of printers are there?”

So our dinner conversation started to resolve around printers. I talked about daisy wheel printers, line printers, dot matrix printers. I explained the type chain of line printers, the pins of dot matrix. “More printers!” I had to dig deeper into my memory: wax transfer printers, thermal printers, dye sublimation, always describing a bit about how each one worked — except for dye sublimation, which I couldn’t remember many details about. “More printers!” So we went onwards towards the printing press, offset printing, screen printing, mimeograph, and photocopiers. Although I could give them plenty of details about most of the printers, I also failed under their barrage of questions about offset printing. So I finally capitulated, and said “should I go get my phone and look it up while you finish eating?” “YEAH!”

So I looked up the misty details of dye sublimation and offset printing and described how they worked. That seemed to satisfy them. Then they asked me what my favorite kind of printer was. I said “dot matrix, because it makes the best sound.” That had their attention. They stopped eating to ask the vitally important question: “Dad, what sound does it make?” At this point, I did my best dot matrix impression at the dinner table, to much laughter and delight.

Before long, they wanted to see videos of dot matrix printers. They were fascinated by them. And then I found this gem of a dot matrix printer playing a famous Bach tune, which fascinated me also:

I guess it must have all sunk in, because this morning before school Jacob all of a sudden begged to see the fuser in my laser printer. So we turned it around, opened up the back panel — to his obvious excitement — and then I pointed to the fuser, with its “hot” label. I even heard a breathy “wow” from him.

Hiking a mountain with Ian Murdock

January 3, 2016DebianJohn Goerzen

“Would you like to hike a mountain?” That question caught me by surprise. It was early in 2000, and I had flown to Tucson for a job interview. Ian Murdock was starting a new company, Progeny, and I was being interviewed for their first hire.

“Well,” I thought, “hiking will be fun.” So we rode a bus or something to the top of the mountain and then hiked down. Our hike was full of — well, everything. Ian talked about Tucson and the mountains, about his time as the Debian project leader, about his college days. I asked about the plants and such we were walking past. We talked about the plans for Progeny, my background, how I might fit in. It was part interview, part hike, part two geeks chatting. Ian had no HR telling him “you can’t go hiking down a mountain with a job candidate,” as I’m sure HR would have. And I am glad of it, because even 16 years later, that is still by far the best time I ever had at a job interview, despite the fact that it ruined the only pair of shoes I had brought along — I had foolishly brought dress shoes for a, well, job interview.

I guess it worked, too, because I was hired. Ian wanted to start up the company in Indianapolis, so over the next little while there was the busy work of moving myself and setting up an office. I remember those early days – Ian and I went computer shopping at a local shop more than once to get the first workstations and servers for the company. Somehow he had found a deal on some office space in a high-rent office building. I still remember the puzzlement on the faces of accountants and lawyers dressed up in suits riding in the elevators with us in our shorts and sandals, or tie-die, next to them.

Progeny’s story was to be a complicated one. We set out to rock the world. We didn’t. We didn’t set out to make lasting friendships, but we often did. We set out to accomplish great things, and we did some of that, too.

We experienced a full range of emotions there — elation when we got hardware auto-detection working well or when our downloads looked very popular, despair when our funding didn’t come through as we had hoped, being lost when our strategy had to change multiple times. And, as is the case everywhere, none of us were perfect.

I still remember the excitement after we published our first release on the Internet. Our little server that could got pegged at 100Mb of outbound bandwidth (that was something for a small company in those days.) The moment must have meant something, because I still have the mrtg chart from that day on my computer, 15 years later.

We made a good Linux distribution, an excellent Debian derivative, but commercial success did not flow from it. In the succeeding months, Ian and the company tried hard to find a strategy that would stick and make our big break. But that never happened. We had several rounds of layoffs when hoped-for funding never materialized. Ian eventually lost control of the company, and despite a few years of Itanium contract work after I left, closed for good.

Looking back, Progeny was life — compressed. During the good times, we had joy, sense of accomplishment, a sense of purpose at doing something well that was worth doing. I had what was my dream job back then: working on Debian as I loved to do, making the world a better place through Free Software, and getting paid to do it. And during the bad times, different people at Progeny experienced anger, cynicism, apathy, sorrow for the loss of our friends or plans, or simply a feeling to soldier on. All of the emotions, good or bad, were warranted in their own way.

Bruce Byfield, one of my co-workers at Progeny, recently wrote a wonderful memoriam of Ian. He wrote, “More than anything, he wanted to repeat his accomplishment with Debian, and, naturally he wondered if he could live up to his own expectations of himself. That, I think, was Ian’s personal tragedy — that he had succeeded early in life, and nothing else he did with his life could quite measure up to his expectations and memories.”

Ian was not the only one to have some guilt over Progeny. I, for years, wondered if I should have done more for the company, could have saved things by doing something more, or different. But I always came back to the conclusion I had at the time: that there was nothing I could do — a terribly sad realization.

In the years since, I watched Ubuntu take the mantle of easy-to-install Debian derivative. I saw them reprise some of the ideas we had, and even some of our mistakes. But by that time, Progeny was so thoroughly forgotten that I doubt they even realized they were doing it.

I had long looked at our work at Progeny as a failure. Our main goal was never accomplished, our big product never sold many copies, our company eventually shuttered, our rock-the-world plan crumpled and forgotten. And by those traditional measurements, you could say it was a failure.

But I have come to learn in the years since that success is a lot more that those things. Success is also about finding meaning and purpose through our work. As a programmer, success is nailing that algorithm that lets the application scale 10x more than before, or solving that difficult problem. As a manager, success is helping team members thrive, watching pieces come together on projects that no one person could ever do themselves. And as a person, success comes from learning from our experiences, and especially our mistakes. As J. Michael Straczynski wrote in a Babylon 5 episode, loosely paraphrased: “Maybe this experience will be a good lesson. Too bad it was so painful, but there ain’t no other kind.”

The thing about Progeny is this – Ian built a group of people that wanted to change the world for the better. We gave it our all. And there’s nothing wrong with that.

Progeny did change the world. As us Progeny alumni have scattered around the country, we benefit from the lessons we learned there. And many of us were “different”, sort of out of place before Progeny, and there we found others that loved C compilers, bootloaders, and GPL licenses just as much as we did. We belonged, not just online but in life, and we went on to pull confidence and skill out of our experience at Progeny and use them in all sorts of ways over the years.

And so did Ian. Who could have imagined the founder of Debian and Progeny would one day lead the cause of an old-guard Unix turning Open Source? I run ZFS on my Debian system today, and Ian is partly responsible for that — and his time at Progeny is too.

So I can remember Ian, and Progeny, as a success. And I leave you with a photo of my best memento from the time there: an original unopened boxed copy of Progeny Linux.

Amtrak Airlines

December 22, 2015UncategorizedJohn Goerzen

I came downstairs this morning and found a surprise waiting for me. Chairs from all over had been gathered up and arranged in rows, airline style. Taped to the wall was a “food court” sign. At the front was a picture of an airplane, decked out with the Amtrak logo of all things, and a timetable taped to our dining room table.

Jacob soon got out string to be seatbelts, too. And, using his copy machine, printed out a picture of a wing to tape to the side of the “airplane”.

And here is the “food court” sign Oliver made:

This plane was, according to the boys, scheduled to leave at 9:30. It left a fashionable 2 hours late or so. They told me I would be the pilot, and had me find headphones to be my “headset”. (I didn’t wear my real headset on the grounds that then I wouldn’t be able to hear them.) Jacob decided he would be a flight attendant, his grandma would be the co-pilot, and Oliver would be the food court worker. The food court somehow seemed to travel with the plane.

Oliver made up a menu for the food court. It consisted of, and I quote: “trail mix, banana, trail mix, half banana, trail mix, trail mix, trail mix”. He’s already got the limited selection of airport food down pat, I can see.

Jacob said the flight would be from Chicago to Los Angeles, and so it was. Since it was Amtrak Airlines, we were supposed to pretend to fly over the train tracks the whole way.

If it’s not Christmas yet, we just invent some fun, eh? Pretty clever.

Where does a person have online discussions anymore?

November 30, 2015Online LifeJohn Goerzen

Back in the day, way back in the day perhaps, there were interesting places to hang out online. FidoNet provided some discussion groups — some local, some more national or international. Then there was Usenet, with the same but on a more grand scale.

There were things I liked about both of them.

They fostered long-form, and long-term, discussion. Replies could be thoughtful, and a person could think about it for a day before replying.

Socially, you would actually get to know the people in the communities you participated in. There would be regulars, and on FidoNet at least, you might bump into them in different groups or even in real life. There was a sense of community. Moreover, there was a slight barrier to entry and that was, perhaps, a good thing; there were quite a lot of really interesting people and not so many people that just wanted answers to homework questions.

Technologically, you got to bring your own client. They were also decentralized, without any one single point of failure, and could be downloaded and used offline. You needed very little in terms of Internet connection.

They both had some downsides; Usenet, in particular, often lacked effective moderation. Not everyone wrote thoughtful posts.

Is there anything like it these days? I’ve sometimes heard people suggest Reddit. It shares some of those aspects, and even has some clients capable of offline operation. However, what it doesn’t really have is long-form discussion. I often find that if I am 6 hours late to a thread, nobody will bother to read my reply because it’s off their radar already. This happens so often that I rarely bother to participate anymore; I am not going to sit at reddit hitting refresh all day long.

There are a few web forums, but they suffer from all sorts of myriad problems; no cohesive community, the “hot topic” vanishing issue of Reddit, the single point of failure, etc.

For awhile, Google+ looked like it might head this way. But I don’t think it really has. I still feel as if there is a vacuum out there.

Any thoughts?

The Changelog

Comments on family, technology, and society