All posts by John Goerzen

Baptism

Uncategorized

Today I was baptized and joined the Mennonite Church. I imagine this might surprise some of you for various reasons, so let me provide a bit of background.

I’ve had a lot of people, both in person and in comments responding to my blog post, express surprise with statements I have made. The view of Christianity that many people have is of a group that devalues scientific inquiry and places a lot of emphasis on things like opposition to gay marriage, evolution, and abortion, and enjoys political leaders that say “bring it on.”

While I know some Christians that fall into that mold, there are quite a few that don’t as well. Some churches, such as the one I attend, have a surprisingly diverse set of people and yet still function and get along well.

Today they accepted me with joy. Nobody was concerned that I started my statement with a reference to ancient Greek philosophy, wound up suggesting that the church ought to make sure to make illegal immigrants, gays and lesbians, and prostitutes feel welcome, and embraced both religion and scientific inquiry, feeling them complimentary.

Mennonites practice adult baptism rather than infant baptism. A traditional age for people to be baptized is during high school, though emphasis is placed more on the individual than their age, so it’s not unheard-of for someone to do so a bit later in life as I have.

Part of the baptism involves the candidate sharing their faith story. These are typically intensely personal, widely varied, heartfelt, and honest. Some people’s stories involve struggles with depression, physical disabilities, or the place of religion in their lives, while others reflect little struggle at all. Mine involves letting go of a lot of things, and also seeing some things, such as serious intellectual inquiry or existential questions about God (such as “is there a god?”), as a positive rather than a negative feature of a Christian life. But it also involves a recognition and deep respect for those that don’t approach things in this way.

Though given publicly in a church, baptism testimonies are rarely published or shared more broadly. But I’m going to share mine here. I have edited it only very lightly to remove a few local references that wouldn’t make sense out of the context of this community. This was delivered in front of a rather different audience than is likely to be reading this post, so if you have questions, do feel free to ask in the comments.

Baptism Testimony

John Goerzen

May 22, 2011

Those of you that know me well will probably not be surprised that I will begin my Christian baptism testimony with a story about an ancient Greek philosopher, and also touch on the philosophical nature of truth. These are key parts of my story.

As legend goes, Socrates famously said, “I know that I know nothing” — in other words, he believed that nothing could be known with absolute certainty. The Greek Oracle — thought to be infallibly wise — said that Socrates, the man that thought he knew nothing, was the wisest man in all Athens. An interesting paradox, and one that sheds light on my own religious story. My story involves coming to grips with the understanding that I know very little, that no person can ultimately know much about God, and finding a way to make peace with that situation.

Growing up in this community, I thought I knew some things about faith and Christianity. At a certain age, it all seemed so simple. We took the Bible to be literally true. We marveled in Sunday School at how the apostles could sometimes appear so blind. And, modeled both implicitly and explicitly, was this notion: the stronger our faith, the fewer inner questions or doubt we have about the nature of God, the literal accuracy of the entire Bible, or our relationship to God. Those I perceived held up as examples never seemed to question any of these things, and showed — outwardly, at least — complete certainty about them. Moreover, holding certain intellectual beliefs was key to Christian identity, and even more importantly, to eternal life.

This model has quite obviously worked well for many people for many years. The good that has happened, and continues to happen, from people that have that kind of certainty is manifest all around us. And yet, it didn’t seem to work out for me.

I’m not the kind of person that accepts a lot of things at face value. It is helpful to be able to examine and challenge ideas — and even more helpful to have other people challenge my ideas. From well before I was in high school, I was questioning some things about the Bible, God, and religion in general. My thoughts ranged from the impact of evolution on religion to the apparently vengeful God of the Old Testament to the very existence of God. In the Christian context, I perceived having these questions as a personal failing, something that I ought to repress.

The more I tried to repress them, the more troublesome they became. Why, for instance, should a loving merciful God decide whether to let us into heaven based on whether we hold certain intellectual beliefs?

In high school, I participated in the catechism class here at this church, and was frustrated because it didn’t tackle deeper meanings or the kinds of questions I had. I wasn’t yet able to articulate all my thoughts and questions very well, and I probably had an overactive case of teenage cynicism. As a result, I didn’t get baptized like most others my age did. As I learned more about the early history of Judaism and Christianity, I only found more reasons to question the model of faith I thought I had received — the one in which Biblical literalism and a “divine guarantee” of sorts was key.

More recently, I gradually became aware that the model of Christianity I had in mind was one of many views. Christians, Mennonites, and even this church are incredibly diverse groups, and in retrospect, I am surprised that it took me so long to realize this. Three major steps led me to baptism.

The first step was the realization that, whatever our understanding of the literal accuracy of the Bible, literal truth is often inferior to metaphorical truth. As an example, many of us have read the works of Shakespeare. They are fictional, but the reason they have been revered for so many centuries is that they are true. They teach us things about ourselves and our world in a way that no history book can.

By placing such an emphasis on literal truth in the Bible, I was missing out on the message right there for me. By concerning myself with creationism or evolution, I missed out on thinking about what the story in Genesis meant for the Jews, and what it could mean for me. Evolution ceased to be a threat to religion; it became simply a tool for learning about a different sort of truth than we get from religion.

N. T. Wright mentions an incident that illustrates this point. A person attending a religious conference asked the speaker, a prominent theologian, “Is it true that the serpent in Genesis actually spoke like you and I speak?” The theologian answered, “It doesn’t matter whether the serpent could speak. What matters is what the serpent said.”

The second step I took towards being baptized was realizing where the real core of Christianity lies. It’s not some debate about Genesis, but rather the death to one’s old self, and the rebirth and continual remaking of oneself in the example of Jesus. Given that, a lot of questions seem unimportant or even irrelevant. Good can be expressed in many ways, and if one person achieves a remarkably Christian life via a literal understanding of the Bible, and another via a more metaphorical reading, then it is my place only to affirm both and say that they both got it right. If we say that the task of remaking ourselves is like climbing a mountain, then what matters is not how we are climbing the mountain, but simply that we are climbing it.

I used to equate faith with an intellectual belief. I have come to see that was a narrow view. Faith, to me now, is more about vision: do we see the world like Christ did? Where do our loyalties and our trust lie — in God or in human institutions? What are our goals in life?

We humans have failed to understand God, and probably always will. I too share in that incomplete understanding, but I have come to accept that it is OK. I know enough to know that I want my loyalty to lie in Jesus, to know what kind of vision of the world I want to have, and I have learned to accept that intellectual questions can even be a form of meditation, enlightenment, and prayer.

The third step toward baptism was moving past my own shortcomings. For a long time, I thought I didn’t believe the right things, didn’t believe them strong enough, wasn’t certain enough about God, didn’t pray enough, didn’t read enough, didn’t understand enough, didn’t love enough — and ultimately, that I wasn’t good enough. I request baptism today understanding that, despite the various imperfections I still have, and we all have, everyone is good enough and deserving of love and peace. God’s love is for everyone. No exceptions!

Following Jesus boils down to this: I too must be an agent of love and peace to everyone, without exception. My vision must be centered around the fact that we are to emulate the God that loves the entire world, sinners all, and therefore I should as well. I hope I can show others the kind of love that has been shown me.

I hope, too, to share with this congregation and the global church in the vision of love. I hope that we can continually strive to re-focus on Christ’s vision. As an example, we all know that many different viewpoints about whether homosexuality is sinful exist in Mennonite Church USA. These opinions are deeply held and personal for many, and have been discussed over and over and over. But ultimately, they aren’t terribly relevant to the church’s mission. The example of Jesus unites us all: he embraced everyone. He accepted criminals and prostitutes and showed them love and kindness. Our difficult task, which is also my difficult task, is to show this same love to absolutely everyone, regardless of our varying opinions about them and their conduct. The forgotten and repressed of our day — perhaps criminals, homeless, Muslims, addicts, gays and lesbians, illegal immigrants, ethnic minorities, and still prostitutes — deserve the love of Christ’s church and all its members. My hope is that any one of the earth’s 7 billion people could step through the doors of our church, or any church, and immediately feel Christ’s love, and the unconditional, non-judgmental, welcome and love of Christ’s followers, no matter what.

This prayer from Ephesians 3 sums up my hope and my vision, for myself and everyone:

I pray that out of his glorious riches he may strengthen you with power through his Spirit in your inner being. I pray that Christ may dwell in your hearts through faith. And I pray that you, being rooted and established in love, may have power, together with all the Lord’s people, to grasp how wide and long and high and deep is the love of Christ. And I pray that you will know this love that surpasses knowledge—that you may be filled to the measure of all the fullness of God. (3.16-19)

The death of one

Uncategorized

”I mourn the loss of thousands of precious lives, but I will not rejoice in the death of one, not even an enemy. Returning hate for hate multiplies hate, adding deeper darkness to a night already devoid of stars. Darkness cannot drive out darkness: only light can do that. Hate cannot drive out hate: only love can do that.”

–anonymous (often attributed to Martin Luther King, Jr.)

Seen many times today.

The Mennonite Mecca

Family, General

I’d like to paint you a picture of the sorts of things that have been going on around here the past few months, and with growing fervor the past few days.

I’ll start with church basements. Possibly a little chilly, but with a bunch of women getting together to make some quilts — along with some conversation, maybe some snacks.

Or perhaps about hundreds of kitchens throughout Kansas, warm with baking pies, cookies, breads, and all sorts of items. Then there are a bunch of wood shops, turning up sawdust, building things from toys to furniture. Or even a body shop donating its time and materials to put some finishing touches on a classic car.

I’ve also seen around 300 men getting together to practice for a men’s chorus concert. I know there was large crowd of people gathering to make verenike. People always wind up running 5km on a particular Saturday morning, rain or shine. Or ride 35 miles on a bicycle on a different Saturday. Or even help build a house using as much volunteer labor and donated materials as possible.

And then, of course, comes this week, where Mennonites from all over Kansas start to converge on Hutchinson, KS. I can imagine you might see some odd sights rolling down the road: a vehicle called the “borscht buggy” for preparing large quantities of the delicious soup. Several old tractors being hauled down the road on trailers. Semi loads of food. Vanloads of pie and cookies. Plants, trees, rugs, quilts, even a lawn mower. And then, yesterday and today, excited people ready to buy all of these things. Families with children excited to get a ride on the largest slide they’ve ever seen. And, of course, so many people wanting to eat the famous food that, despite the many parallel serving areas, lines still can extend for blocks.

It’s all because people are hungry.

Not these Kansans with pie-laden kitchens, though — it’s to help those in need.

It’s all part of the annual Kansas MCC Relief Sale. The idea is that people make, bake, build, sing, or give things to the sale. The items are then sold, and the proceeds go to Mennonite Central Committee, one of the world’s most efficient charities. MCC not only helps with directly bringing people out of hunger, but also supports sustainable projects, such as building hand-operated wells to give those in need a safe source of drinking water. Virtually all of the money you spend on those pies winds up helping someone in poverty.

Each year, the relief sale and related events raise around half a million dollars for charity.

It’s not unusual to see a quilt sell for thousands of dollars. Most of the things sell at more normal prices, but last year someone decided to add a loaf of bread to an auction — and it sold for $100.

And so it was this that we took the boys to today. We started with the “Feeding the Multitude” — a wonderful meal with some traditional Kansas Mennonite food. Here’s a photo:

That bread is zwieback, then there is cherry moos, bohne berrogi, verenike, and of course, sausage.

Here are Jacob and Oliver enjoying their food:

IMG_5548.JPG

After that, we went over to the giant slide. I’m not sure how many stories tall it is, but it’s big enough that they have a strip of carpet there at the bottom to slow people down at the end. Here’s a photo of Jacob on my lap after going down the slide. If you zoom in, you can see the giant smile on his face.

IMG_5562.JPG

After that, we hopped on the “mass transit” at the sale: a wagon being pulled by a tractor. We checked out the general auction, with Jacob wanting to be sure to peer under the hood of each car present. Then after a stop to buy some kettle corn for Jacob, we checked out the plants, quilt auction, and then on to buy cheese curds.

After another stop at the slide, it was time to head home.

And then tomorrow is a concert of the Kansas Mennonite Men’s Chorus (motto: We Sing That Others May Live). You get around 300 men on a stage signing together and wow — no recording can do it justice. It’s amazing to hear the power of the choir at the loud parts, but still more amazing to listen to 300 men signing as softly as they possibly can. I’ve been singing with that choir for 3 years, and we’ve been practicing for a few months now.

If you ever hear some Mennonites boasting about something, it’s probably going to be about how much money they’ve raised for charity. And on that note, I’ll nudge those Indiana folks reading this and point out that you have some catching up to do with the Kansas sale…

Country Car Repair & California-Kansas Culture Shock

Uncategorized

I know this is almost a cliche, but here goes. Just to show you that it still exists.

Today my car needed some work. This morning I dropped it off at the mechanic, who lives right off a well-traveled state highway. They don’t have a dropbox for keys because, well, they don’t need one. I left the keys in the car.

They called to say that they’d have to close early this afternoon, but not to worry — the keys were in the car along with the bill, which I could drop by and pay or put in the family’s box at church.

And sure enough, the car sat there the rest of the afternoon, and until about 9:45PM, with the keys in the ignition, bill tucked in under my ham radio speaker, ready to pick up.

And it wasn’t at risk for being stolen. Things just sort of happen that way out here sometimes.

I remember one time we were taking a train out of Newton, KS, which departs at about 3:30AM. The only other train through there departs at 3:00AM. And back then, there was sometimes no station agent on some days. We got there our usual 15 minutes before the train departed, and struck up a conversation with a couple from California. They were almost beside themselves with shock. First off, they had arrived an hour and a half early, being used to doing that for getting on planes and, apparently, trains.

But what really stunned them was the fact that the station was simply left unlocked all night. There weren’t any cops there, and in fact there was nobody there at all for most of the night. And yet it had no signs of graffiti, no vandalism, and no apparent concern from anybody that it was unlocked. I think they also felt unsafe, having no guards or officials present. And they were literally the only people in the station for an hour, I’m sure.

And sure enough, along came the train. We all heard the whistle and had our things ready to go by the time the conductor got off the train, walked into the station, announced it, and led us all out to it. 3 minutes later, we were all on our way to California and the land of, apparently, locked doors.

Back in 2002, when we bought a house in Kansas, the inspector commented that none of the outside locks worked. At closing, the sellers gave us a key, commenting “We never use it and it took forever to find it.” I guess that’s why they didn’t know the locks were broken.

There’s something good about a setting like this. I know that my car is safe at the mechanic with the keys in it. He knows that I’ll pay his bill. We all take pride in our community institutions. And besides, if anybody did try to spray some graffiti in the train station, they’ll be living with people that will remember it for the next 40 years. Accountability is implicit here.

A Proud Dad

Children & Computing, Debian, Family, Linux

I saw this on my computer screen the other day, and I’ve got to say it really warmed my heart. I’ll explain below if it doesn’t provoke that reaction for you.

Evidence a 4-year-old has been using my computer

So here’s why that made me happy. Well for one, it was the first time Jacob had left stuff on my computer that I found later. And of course he left his name there.

But moreover, he’s learning a bit about the Unix shell. sl is a command that displays an animated steam locomotive. I taught him how to use the semicolon to combine commands. So he has realized that he can combine calls to sl with the semicolon to get a series of a LOT of steam trains all at once. And was very excited about this discovery.

Also he likes how error messages start with the word “bash”.

How do you hire programmers and sysadmins? How should employers evaluate you?

Technology

Reading job listings for any sort of IT job is depressing. It’s been quite some time since I’ve had to do that, but how many times have you seen something like this:

  • “5 years of Java experience required.”
  • “3 years of Java experience with modules X, Y, Z required.”
  • “6 years of experience administering Linux machines running RHEL 4 on a Windows 2000 domain with 1500 clients in an educational setting preferred.”

I could go on and on. As a job seeker, that sort of thing is fundamentally devaluing to someone who has strengths in being adaptable and quickly learning new tools, languages, or even entire environments. As an employer, it sends a message that you’re not interested in more than a surface look at someone’s strengths, and probably don’t care to hire the best and the brightest. After all, would you turn away a rockstar programmer simply because he or she had been writing filesystem code in C the last 3 years instead of the latest whizbang Java web widget that will probably be obsolete in a year and unsupported in two? I am quite certain that there are plenty of managers that do. Even if you are a company large enough to have an entire team of people that do nothing but work on that whizbang app, don’t you still want the best you can find, realizing that some of the best people to work on that app may not have even heard of it yet? (And that when the app goes obsolete in 5 years, you’d rather not have to lay off a large team of single-skill people)

Some of you may know that I work in IT at a manufacturing company. We have a small IT team here, about seven people, and are a heavy Debian shop. And we have a vacancy open up in our development/Linux admin group. I’m the manager of that group, which is why I’m thinking about this right now.

We’re too small for single-subject specialists to make sense, yet we’re big enough to appreciate skill, experience, flexibility, and rigor. Consequently, when the occasion arises for me to look for new employees, I don’t prepare a laundry list of things we use in-house and would like experience with.

The list of almost-required things generally begins with “Linux” and ends with “experienced”, and has nothing else in between. In other words, I’d like it if I don’t have to explain to you what a symlink or a hardlink is, but I’d be willing to do so if I think you’d internalize it quickly. On the “experienced” side, it would be nice if you already have a well-developed sense of fear at running rm when you’re root, or have designed a storage infrastructure for a network before, or are paranoid about security. But again, if people can pick up those traits on the job, we are usually still interested. If learning how to package up software for Debian, fix bugs in software you’ve never seen in a language you’ve never heard of, raise good questions about things you may not have lots of experience with, and write documentation for it all on a wiki sounds like fun, then that’s probably the kind of person I want, even if you’ve never used our particular tools before.

If I were to judge based on the stuff I normally see in job postings, I guess you might conclude I’m nuts. I don’t think I am, but then again I’m also the only person I know that formats his own resume in hand-crafted LaTeX. What do you all think?

The next question is: how should one evaluate candidates given this sort of philosophy? I’m not a fan of canned tests, or even “whiteboard tests” that tend to be some sort of canned topic that may test the applicant’s specific knowledge base more than overall skill and flexibility. Similarly, as an applicant in years past, I’ve struggled with how to present the “I’ve never used $LANGUAGE, but I know I could pick it up quickly and do it very well” vibe. To certain people, that might sound like BS. To the more geeky managers, perhaps it sounds like what they want.

We’ve built a fairly diverse team on the back of this approach, and it’s worked out well for us so far. I’m interested to hear your thoughts.

Oh, and if you’d like to work for us, you should probably be sending me an email. No, I’m not going to list the address here on this blog post. If you can’t figure it out, I don’t want to hear from you <grin>

Unix Password and Authority Management

Linux

One of the things that everyone seems to do different is managing passwords. We haven’t looked at that in quite some time, despite growth both of the company and the IT department.

As I look to us moving some things to the cloud, and shifting offsite backups from carrying tapes to a bank to backups via the Internet, I’m aware that the potential for mischief — whether intentional or not — is magnified. With cloud hosting, a person could, with the press of a button, wipe out the equivalent of racks of machines in a few seconds. With disk-based local and Internet-based offsite backups, the potential for malicious behavior may be magnified; someone could pretty quickly wipe out local and remote backups.

Add to that the mysterious fact that many enterprise-targeted services allow only a single username/password for an account, and make no provision for ACLs to delegate permissions to others. Even Rackspace Cloud has this problem, as do their JungleDisk backup product, and many, many other offsite backup products. Amazon AWS seems to be the only real exception to this rule, and their ACL support is more than a little complicated.

So one of the questions we will have to address is the balance of who has these passwords. Too many people and the probability of trouble, intentional or not, rises. Too few and productivity is harmed, and potentially also the ability to restore. (If only one person has the password, and that person is unavailable, company data may be as well.) The company does have some storage locations, including locked vaults and safe deposit boxes, that no IT people have access to. I am thinking that putting a record of passwords in those locations may be a good first step, as putting the passwords in the control of those that can’t use them seems a reasonable step.

But we’ve been thinking of this as it pertains to our local systems as well. We have, for a number of years now, assigned a unique root password to every server. These passwords are then stored in a password-management tool, encrypted with a master password, and stored on a shared filesystem. Everyone in the department therefore can access every password.

Many places where I worked used this scheme, or some variant of it. The general idea was that if root on one machine was compromised and the attacker got root’s password, it would prevent the person from being able to just try that password on the other servers on the network and achieve a greater level of intrusion.

However, the drawback is that we now have more servers than anyone can really remember the passwords for. So many people are just leaving the password tool running. Moreover, while the attack described above is still possible, these days I worry more about automated intrusion attempts that most likely won’t try that attack vector.

A couple of ways we could go may include using a single root password everywhere, or a small set of root passwords. Another option may be to not log in to root accounts at all — possibly even disabling their password — and requiring the use of user accounts plus sudo. This hasn’t been practical to date. We don’t want to make a dependency on LDAP from a bunch of machines just to be able to use root, and we haven’t been using a tool such as puppet or cfengine to manage this stuff. Using such a tool is on our roadmap and could let us manage that approach more easily. But this approach has risks too. One is that if user accounts can get to root on many machines, then we’re not really more secure than a standard root password. Second is that it makes it more difficult to detect and enforce password expiration and systematic password changes.

I’m curious what approaches other people are taking on this.

rdiff-backup, ZFS, and rsync scripts

Debian,

rdiff-backup vs. ZFS

As I’ve been writing about backups, I’ve gone ahead and run some tests with rdiff-backup. I have been using rdiff-backup personally for many years now — probably since 2002, when I packaged it up for Debian. It’s a nice, stable system, but I always like to look at other options for things every so often.

rdiff-backup stores an uncompressed current mirror of the filesystem, similar to rsync. History is achieved by the use of compressed backwards binary deltas generated by rdiff (using the rsync algorithm). So, you can restore the current copy very easily — a simple cp will do if you don’t need to preserve permissions. rdiff-backup restores previous copies by applying all necessary binary deltas to generate the previous version.

Things I like about rdiff-backup:

  1. Bandwidth-efficient
  2. Reasonably space-efficient, especially where history is concerned
  3. Easily scriptable and nice CLI
  4. Unlike tools such as duplicity, there is no need to periodically run full backups — old backups can be deleted without impacting the ability to restore more current backups

Things I don’t like about it:

  1. Speed. It can be really slow. Deleting 3 months’ worth of old history takes hours. It has to unlink vast numbers of files — and that’s pretty much it, but it does it really slowly. Restores, backups, etc. are all slow as well. Even just getting a list of your increment sizes so you’d know how much space would be saved can take a very long time.
  2. The current backup copy is stored without any kind of compression, which is not at all space-efficient
  3. It creates vast numbers of little files that take forever to delete or summarize

So I thought I would examine how efficient ZFS would be. I wrote a script that would replay the rdiff-backup history — first it would rsync the current copy onto the ZFS filesystem and make a ZFS snapshot. Then each previous version was processed by my script (rdiff-backup’s files are sufficiently standard that a shell script can process them), and a ZFS snapshot created after each. This lets me directly compare the space used by rdiff-backup to that used by ZFS using actual history.

I enabled gzip-3 compression and block dedup in ZFS.

My backups were nearly 1TB in size and the amount of space I had available for ZFS was roughly 600GB, so I couldn’t test all of them. As it happened, I tested the ones that were the worst-case scenario for ZFS: my photos, music collection, etc. These files had very little duplication and very little compressibility. Plus a backup of my regular server that was reasonably compressible.

The total size of the data backed up with rdiff-backup was 583 GB. With ZFS, this came to 498GB. My dedup ratio on this was only 1.05 (meaning 5% or 25GB saved). The compression ratio was 1.12 (60GB saved). The combined ratio was 1.17 (85GB saved). Interestingly 498 + 85 = 583.

Remember that the data under test here was mostly a worst-case scenario for ZFS. It would probably have done better had I had the time to throw the rest of my dataset at it (such as the 60GB backup of my iPod, which would have mostly deduplicated with the backup of my music server).

One problem with ZFS is that dedup is very memory-hungry. This is common knowledge and it is advertised that you need to use roughly 2GB of RAM per TB of disk when using dedup. I don’t have quite that much to dedicate to it, so ZFS got VERY slow and thrashed the disk a lot after the ARC grew to about 300MB. I found some tweakables in zfsrc and the zfs command that let me tweak the ARC cache to grow bigger. But the machine in question only has 2GB RAM, and is doing lots of other things as well, so this barely improved anything. Note that this dedup RAM requirement is not out of line with what is expected from these sorts of solutions.

Even if I got absolutely stellar dedup ratio of 2:1, that would get me at most 1TB. The cost of buying a 1TB disk is less than the cost of upgrading my system to 4GB RAM, so dedup isn’t worth it here.

I think the lesson is: think carefully about where dedup makes sense. If you’re storing a bunch of nearly-identical virtual machine images — the sort of canonical use case for this — go for it. A general fileserver — well, maybe you should just add more disk instead of more RAM.

Then that raises the question: if I don’t need dedup from ZFS, do I bother with it at all, or just use ext4 and LVM snapshots? I think ZFS still makes sense, given its built-in support for compression and very fast snapshots — LVM snapshots are known to cause serious degradation to write performance once enabled, which ZFS doesn’t.

So I plan to switch my backups to use ZFS. A few observations on this:

  1. Some testing suggests that the time to delete a few months of old snapshots will be a minute or two with ZFS compared to hours with rdiff-backup.
  2. ZFS has shown itself to be more space-efficient than rdiff-backup, even without dedup enabled.
  3. There are clear performance and convenience wins with ZFS.

    4. Backup Scripts

    So now comes the question of backup scripts. rsync is obviously a pretty nice choice here — and if used with –inplace perhaps even will play friendly with ZFS snapshots even if dedup is off. But let’s say I’m backing up a few machines at home, or perhaps dozens at work. There is a need to automate all of this. Specifically, there’s a need to:

    1. Provide scheduling, making sure that we don’t hammer the server with 30 clients all at once
    2. Provide for “run before” jobs to do things like snapshot databases
    3. Be silent on success and scream loudly via emails to administrators on any kind of error… and keep backing up other systems when there is an error
    4. Create snapshots and provide an automated way to remove old snapshots (or mount them for reading, as ZFS-fuse doesn’t support the .zfs snapshot directory yet)

    To date I haven’t found anything that looks suitable. I found a shell script system called rsbackup that does a large part of this, but something about using a script whose homepage is a forum makes me less than 100% confident.

    On the securing the backups front, rsync comes with a good-looking rrsync script (inexplicably installed under /usr/share/doc/rsync/scripts instead of /usr/bin on Debian) that can help secure the SSH authorization. GNU rush also looks like a useful restricted shell.

Research on deduplicating disk-based and cloud backups

Debian, Linux,

Yesterday, I wrote about backing up to the cloud. I specifically was looking at cloud backup services. I’ve been looking into various options there, but also various options for disk-based backups. I’d like to have both onsite and offsite backups, so both types of backup are needed. Also, it is useful to think about how the two types of backups can be combined with minimal overhead.

For the onsite backups, I’d want to see:

  1. Preservation of ownership, permissions, etc.
  2. Preservation of symlinks and hardlinks
  3. Space-efficient representation of changes — ideally binary deltas or block-level deduplication
  4. Ease of restoring
  5. Support for backing up Linux and Windows machines

Deduplicating Filesystems for Local Storage

Although I initially thought of block-level deduplicating file systems as something to use for offsite backups, they could also make an excellent choice for onsite disk-based backups.

rsync-based dedup backups

One way to use them would be to simply rsync data to them each night. Since copies are essentially free, we could do (or use some optimized version of) cp -r current snapshot/2011-01-20 or some such to save off historic backups. Moreover, we’d get dedup both across and within machines. And, many of these can use filesystem-level compression.

The real upshot of this is that the entire history of the backups can be browsed as a mounted filesystem. It would be fast and easy to find files, especially when users call about that file that they deleted at some point in the past but they don’t remember when, exactly what it was called, or exactly where it was stored. We can do a lot more with find and grep to locate these things than we could do with tools in Bacula (or any other backup program) restore console. Since it is a real mounted filesystem, we could also do fun things like make tarballs of it at will, zip parts up, scp them back to the file server, whatever. We could potentially even give users direct access to their files to restore things they need for themselves.

The downside of this approach is that rsync can’t store all the permissions unless it’s running as root on the system. Wrappers such as rdup around rsync could help with that. Another downside is that there isn’t a central scheduling/statistics service. We wouldn’t want the backup system to be hammered by 20 servers trying to send it data at once. So there’d be an element of rolling our own scripts, though not too bad. I’d have preferred not to authorize a backup server with root-level access to dozens of machines, but may be inescapable in this instance.

Bacula and dedup

The other alternative I thought of system such as Bacula with disk-based “volumes”. A Bacula volume is normally a tape, but Bacula can just write them to disk files. This lets us use the powerful Bacula scheduling engine, logging service, pre-backup and post-backup jobs, etc. Normally this would be an egregious waste of disk space. Bacula, like most tape-heritage programs, will write out an entire new copy of a file if even one byte changes. I had thought that I could let block-level dedupe reduce the storage size of Bacula volumes, but after looking at the Bacula block format spec, this won’t be possible as each block will have timestamps and such in it.

The good things about this setup revolve around using the central Bacula director. We need only install bacula-fd on each server to be backed up, and it has a fairly limited set of things it can do. Bacula already has built-in support for defining simple or complicated retention policies. Its director will email us if there is a problem with anything. And its logs and catalog are already extensive and enable us to easily find out things such as how long backups take, how much space they consume, etc. And it backs up Windows machines intelligently and comprehensively in addition to POSIX ones.

The downsides are, of course, that we don’t have all the features we’d get from having the entire history on the filesystem all at once, and far less efficient use of space. Not only that, but recovering from a disaster would require a more extensive bootstrapping process.

A hybrid option may be possible: automatically unpacking bacula backups after they’ve run onto the local filesystem. Dedupe should ensure this doesn’t take additional space — if the Bacula blocksize aligns with the filesystem blocksize. This is certainly not a given however. It may also make sense to use Bacula for Windows and rsync/rdup for Linux systems.

This seems, however, rather wasteful and useless.

Evaluation of deduplicating filesystems

I set up and tested three deduplicating filesystems available for Linux: S3QL, SDFS, and zfs-fuse. I did not examine lessfs. I ran a similar set of tests for each:

  1. Copy /usr/bin into the fs with tar -cpf - /usr/bin | tar -xvpf - -C /mnt/testfs
  2. Run commands to sync/flush the disk cache. Evaluate time and disk used at this point.
  3. Rerun the tar command, putting the contents into a slightly different path in the test filesystem. This should consume very little additional space since the files will have already been there. This will validate that dedupe works as expected, and provide a hint about its efficiency.
  4. Make a tarball of both directories from the dedup filesystem, writing it to /dev/zero (to test read performance)

I did not attempt to flush read caches during this, but I did flush write caches. The test system has 8GB RAM, 5GB of which was free or in use by a cache. The CPU is a Core2 6420 at 2.13GHz. The filesystems which created files atop an existing filesystem had ext4 mounted noatime beneath them. ZFS was mounted on an LVM LV. I also benchmarked native performance on ext4 as a baseline. The data set consists of 3232 files and 516MB. It contains hardlinks and symlinks.

Here are my results. Please note the comments below as SDFS could not accurately complete the test.

Test ext4 S3QL SDFS zfs-fuse
First copy 1.59s 6m20s 2m2s 0m25s
Sync/Flush 8.0s 1m1s 0s 0s
Second copy+sync N/A 0m48s 1m48s 0m24s
Disk usage after 1st copy 516MB 156MB 791MB 201MB
Disk usage after 2nd copy N/A 157MB 823MB 208MB
Make tarball 0.2s 1m1s 2m22s 0m54s
Max RAM usage N/A 150MB 350MB 153MB
Compression none lzma none gzip-2

It should be mentioned that these tests pretty much ruled out SDFS. SDFS doesn’t appear to support local compression, and it severely bloated the data store, which was much larger than the original data. Moreover, it permitted any user to create and modify files, even if the permissions bits said that the user couldn’t. tar gave many errors unpacking symlinks onto the SDFS filesystem, and du -s on the result threw up errors as well. Besides that, I noted that find found 10 fewer files than in my source data. Between the huge memory consumption, the data integrity concerns, and inefficient disk storage, SDFS is out of the running for this project.

S3QL is optimized for storage to S3, though it can also store its files locally or on an sftp server — a nice touch. I suspect part of its performance problem stems from being designed for network backends, and using slow compression algorithms. S3QL worked fine, however, and produced no problems. Creating a checkpoint using s3qlcp (faster than cp since it doesn’t have to read the data from the store) took 16s.

zfs-fuse appears to be the most-used ZFS implementation on Linux at the moment. I set up a 2GB ZFS pool for this test, and set dedupe=on and compress=gzip-2. When I evaluated compression in the past, I hadn’t looked at lzjb. I found a blog post comparing lzjb to the gzip options supported by zfs and wound up using gzip-2 for this test.

ZFS really shone here. Compared to S3QL, it took 25s instead of over 6 minutes to copy the data over — and took only 28% more space. I suspect that if I selected gzip -9 compression it would have been closer both in time and space to S3QL. But creating a ZFS snapshot was nearly instantaneous. Although ZFS-fuse probably doesn’t have as many users as ZFS on Solaris, still it is available in Debian, and has a good backing behind it. I feel safer using it than I do using S3QL. So I think ZFS wins this comparison.

I spent quite some time testing ZFS snapshots, which are instantaneous. (Incidentally, ZFS-fuse can’t mount them directly as documented, so you create a clone of the snapshot and mount that.) They worked out as well as could be hoped. Due to dedupe, even deleting and recreating the entire content of the original filesystem resulted in less than 1MB additional storage used. I also tested creating multiple filesystems in the zpool, and confirmed that dedupe even works between filesystems.

Incidentally — wow, ZFS has a ton of awesome features. I see why you OpenSolaris people kept looking at us Linux folks with a sneer now. Only our project hasn’t been killed by a new corporate overlord, so guess that maybe didn’t work out so well for you… <grin>.

The Cloud Tie-In

This discussion leaves another discussion: what to do about offsite backups? Assuming for the moment that I want to back them up over the Internet to some sort of cloud storage facility, there are about 3 options:

  1. Get an Amazon EC2 instance with EBS storage and rsync files to it. Perhaps run ZFS on that thing.
  2. Use a filesystem that can efficiently store data in S3 or Cloud Files (S3QL is the only contender here)
  3. Use a third-party backup product (JungleDisk appears to be the leading option)

There is something to be said for using a different tool for offsite backups — if there is some tool-level issue, that could be helpful.

One of the nice things about JungleDisk is that bandwidth is free, and disk is the same $0.15/GB-mo that RackSpace normally charges. JungleDisk also does block-level dedup, and has a central management interface. This all spells “nice” for us.

The only remaining question would be whether to just use JungleDisk to back up the backup server, or to put it on each individual machine as well. If it just backs up the backup server, then administrative burdens are lower; we can back everything there up by default and just not worry about it. On the other hand, if there is a problem with our main backups, we could be really stuck. So I’d say I’m leaning towards ZFS plus some sort of rsync solution and JungleDisk for offsite.

I had two people suggest CrashPlan Pro on my blog. It looks interesting, but is a very closed product which makes me nervous. I like using standard tools and formats — gives me more peace of mind, control, and recovery options. CrashPlan Pro supports multiple destinations and says that they do cloud hosting, but don’t list pricing anywhere. So I’ll probably not mess with it.

I’m still very interested in what comments people may have on all this. Let me know!

Backing Up to the Cloud

Uncategorized

I’m recently taking some big-picture looks at how we do things, and one thing that I think could be useful would be for us to back up a limited set of data to an offsite location states away. Prices are cheap enough for this to make it useful. Services such as Amazon S3 and Rackspace Cloud Files (I’ve heard particularly good things about that one) seem to be perfect for this. I’m not quite finding software that does what I want, though. Here are my general criteria:

  1. Storage fees of $0.15 per gigabyte-month or less
  2. Free or cheap ($0.20 per gigabyte or less) bandwidth fees
  3. rsync-like protocol to avoid having to re-send those 20GB files that have 20MB of changes in their entirety every night
  4. Open Source and cross-platform (Linux, Windows, Mac, Solaris ideally; Linux and Windows at a minimum)
  5. Compression and encryption
  6. Easy way to restore the entire backup set or individual files
  7. Versatile include/exclude rules
  8. Must be runnable from scripts, cron, etc. without a GUI
  9. Nice to have: block or file-level de-duplication
  10. Nice to have: support for accurately backing up POSIX (user, group, permission bits, symlink, hard links, sparse files) and Windows filesystem attributes
  11. Nice to have: a point-and-click interface for the non-Unix folks to use to restore Windows files and routine restore requests

So far, here’s what I’ve found. I should note that not a single one of these solutions appears to handle hard links or sparse files correctly, meaning I can’t rely on them for complete system-level backups. That doesn’t mean they’re useless — I could still use them to back up critical user data — just less useful.

Of the Free Software solutions, Duplicity is a leading contender. It has built-in support for Amazon S3 and Rackspace Cloud Files storage. It uses rdiff, which is a standalone implementation of the rsync binary delta algorithm. So you send up a full backup, then binary deltas from that for incrementals. That makes it bandwidth-efficient for incremental backups, and storage-efficient. However, periodic full backups will have to be run, which will make it less bandwidth-efficient. (Perhaps not incredibly *often*, but they will still be needed.) Duplicity doesn’t offer block-level de-deuplication or a GUI for the point-and-click folks. But it DOES offer the most Unixy approach and feels like a decent match for the task overall.

The other service relying on Free Software is rsync.net, which supports rsync, sftp, scp, etc. protocols directly. That would be great, as it could preserve hard links and be compatible with any number of rsync-based backup systems. The downside is that it’s expensive — really expensive. Their cheapest rate is $0.32 per GB-month and that’s only realized if you store more than 2TB with them. The base rate is $0.80 per GB-month. They promise premium support and such, but I just don’t think I can justify that for what is, essentially, secondary backup.

On the non-Open Source side, there’s JungleDisk, which has a Server Edition that looks like a good fit. The files are stored on either S3 or Rackspace, and it seems to be a very slick and full-featured solution. The client, however, is proprietary though it does seem to offer a non-GUI command-line interface. They claim to offer block-level de-duplication which could be very nice. The other nice thing is that the server management is centralized, which presumably lets you easily automate things like not running more than one backup at a time in order to not monopolize an Internet link. This can, of course, be managed with something like duplicity with appropriate ssh jobs kicked off from appropriate places, but it would be easier if the agent just handled it automatically.

What are people’s thoughts about this sort of thing?