Monthly Archives: October 2013

Why are we still backing up to hardlink farms?

Linux

A person can find all sorts of implementations of backups using hardlink trees to save space for incrementals. Some of them are fairly rudimentary, using rsync --link-dest. Others, like BackupPC, are more sophisticated, doing file-level dedup to a storage pool indexed by a hash.

While these are fairly space-efficient, they are really inefficient in other ways, because they create tons of directory entries. It would not be surprising to find millions of directory entries consumed very quickly. And while any given backup set can be deleted without impact on the others, the act of doing so can be very time-intensive, since often a full directory tree is populated with every day’s backup.

Much better is possible on modern filesystems. ZFS has been around for quite awhile now, and is stable on Solaris, FreeBSD and derivatives, and Linux. btrfs is also being used for real workloads and is considered stable on Linux.

Both have cheap copy-on-write snapshot operations that would work well with a simple rsync --inplace to achieve the same effect ad hardlink farms, but without all the performance penalties. When creating and destroying snapshots is a virtually instantaneous operation, and the snapshots work at a file block level instead of an entire file level, and preserve changing permissions and such as well (which rsync --link-dest can have issues with), why are we not using it more?

BackupPC has a very nice scheduler, a helpful web interface, and a backend that doesn’t have a mode to take advantage of these more modern filesystems. The only tool I see like this is dirvish, which someone made patches for btrfs snapshots three years ago that never, as far as I can tell, got integrated.

A lot of folks are rolling a homegrown solution involving rsync and snapshots. Some are using zfs send / btrfs send, but those mechanisms require the same kind of FS on the machine being backed up as on the destination, and do not permit excluding files from the backup set.

Is this an area that needs work, or am I overlooking something?

Incidentally, hats off to liw’s obnam. It doesn’t exactly do this, but sort of implements its own filesystem with CoW semantics.

Two Kittens

Uncategorized

Almost every time he got off the bus for the past month and a half, Jacob started his afternoon in the same way. Before toys, before his trains and his toy bus, before anything indoors, he went for our cats. Here he is, cradling his favorite, Tigger:

Laura and I both grew up around cats. We had been talking about kittens, and shortly after we got engaged, one of my relatives offered us some free kittens. We went to his place one evening and selected two of them – one calico and one tiger-colored. Since what is now my place will soon be our place, they came to live with me. Our cats were one of the first things we did to prepare for our lives together.

Oliver wanted to name them some rather impractical sentence-long names (“The Cat Who Always Likes To Run”), so Laura and I suggested some names from one of their favorite books: Tigger and Roo. They both liked the names, but Oliver thought they should be called “Tigger the Digger” and “Roo the Runner”. Never mind that they were just 6 weeks old at the time, and not really old enough to either dig or run. Here’s Oliver with Roo, the day after the kittens arrived here.

I have always had outside cats, both because I’m allergic to cats so I need them to be outside, and because they sometimes literally quiver with joy of being outdoors. Tigger and Roo often chased insects, wrestled with each other, ran up (and slowly came back down) trees, and just loved the outside. Sometimes, I have taken my laptop and wireless headset and work from the back porch. The kittens climb up my jeans, inspect the laptop, and once Roo even fell asleep on my lap at one of those times.

Jacob has been particularly attached to Tigger, calling him “my very best friend.” When Jacob picks him up after school, Tigger often purrs while cradled in Jacob’s arms, and Jacob comments that “Tigger loves me. Oh dad, he knows I am his friend!”

The kittens have been growing, and becoming more and more comfortable with their home in the country. Whenever I go outside, it isn’t long before there are two energetic kittens near my feet, running back and forth, sometimes being very difficult to avoid stepping on. I call and I see little heads looking at me, from up in a tree, or peeking out from the grain elevator door, or from under the grill. They stare for just a second, and then start running, sometimes comically crashing into something in their haste.

Yesterday when I went to give them food, I called and no cats came. I was concerned, and walked around the yard, but at some point either they come or they don’t.

Yesterday afternoon, just after the bus dropped off Jacob, I discovered Tigger on the ground, motionless. Once Jacob was in the house, I went to investigate, and found Tigger was dead. As I was moving his body, I saw Roo was dead, too. Both apparently from some sort of sudden physical injury — a bit mysterious, because neither of them were at a place where they had ever gone before. While all this was happening, I had to also think about how I was going to tell the boys about this.

I tried to minimize what he could see, Jacob had caught an unavoidable glimpse of Tigger as we were walking back from the bus, but didn’t know exactly what had happened. He waited in the house, and when I came back, asked me if Tigger was dead. I said he was. Jacob started crying, saying, “Oh Dad, I am so sad”, and reached up for a hug. I picked him up and held him, then sat down on the couch and let him curl up on my lap. I could quite honestly let him know he wasn’t alone, telling him I am sad, too.

Oliver arrived not long after, and he too was sad, though not as much as Jacob. Both boys pretty soon wanted to see them. I decided this was important for them for closure, and to understand, so while they waited in the house, I went back out to arrange the kittens to hide their faces, the part that looks most unnatural after they die. The boys and I walked out to where I put them, then I carried both of them the last few feet. We stood a little ways back — close enough to see who was there, far enough to not get too much detail — and they were both sniffling. I tried to put voice to the occasion, saying, “Goodbye, Tigger and Roo. We love you.” Oliver asked if they could hear us. I said “No, but I told them what I felt like anyway.” Jacob, through tears, said, “Dad, maybe they are in heaven now.”

We went back inside. Jacob said, “Oh dad, I am so sad. This is the saddest day of my life. My heart is breaking.” Hearing a 7-year-old say that isn’t exactly easy for a dad. Pretty soon he was thinking of sort of comfort activities to do, saying “I think I would feel better if we did…” So they decided to watch a favorite TV program. Jacob asked if Laura knew yet, and when I said no, he got his take-charge voice and said, “Dad, you will start the TV show for us. While we are watching, you will send Laura an email to tell her about Tigger and Roo. OK?” What could I say, it wasn’t a bad idea.

Pretty soon both boys were talking and laughing. It was Big Truck Night last night, at a town about half an hour away. It’s an annual event we were already planning to attend, where all sorts of Big Trucks – firetrucks, school bus, combine, bucket truck, cement truck, etc – show up and are open for kids to climb in and explore. It’s always a highlight for them. They played and sang happily as we drove, excitedly opened and closed the big door on the school bus and yelled “All Aboard!” from the top of the combine. We ate dinner, and drove back home. When we got home, Jacob mentioned the cats again, in a sort of matter-of-fact way, and also wanted to make sure he knew Laura had got the message.

A person never wakes up expecting to have to dump a bowl of un-eaten cat food, or to give an impromptu cat funeral for little boys. As it was happening, I wished they hadn’t been around right then. But in retrospect, I am glad they were. They had been part of life for those kittens, and it is only right that they could be included in being part of death. They got visual closure this way, and will never wonder if the cats are coming back someday. They had a chance to say goodbye.

Here is how I remember the kittens.