December 7th, 2013
The recent news that openSUSE considers btrfs safe for users prompted me to consider using it. And indeed I did. I was already familiar with zfs, so considered this a good opportunity to experiment with btrfs.
btrfs makes an intriguing filesystem for all sorts of workloads. The benefits of btrfs and zfs are well-documented elsewhere. There are a number of features btrfs has that zfs lacks. For instance:
- The ability to shrink a device that’s a member of a filesystem/pool
- The ability to remove a device from a filesystem/pool entirely, assuming enough free space exists elsewhere for its data to be moved over.
- Asynchronous deduplication that imposes neither a synchronous performance hit nor a heavy RAM burden
- Copy-on-write copies down to the individual file level with cp --reflink
- Live conversion of data between different profiles (single, dup, RAID0, RAID1, etc)
- Live conversion between on-the-fly compression methods, including none at all
- Numerous SSD optimizations, including alignment and both synchronous and asynchronous TRIM options
- Proper integration with the VM subsystem
- Proper support across the many Linux architectures, including 32-bit ones (zfs is currently only flagged stable on amd64)
- Does not require excessive amounts of RAM
The feature set of ZFS that btrfs lacks is well-documented elsewhere, but there are a few odd btrfs missteps:
- There is no way to see how much space subvolume/filesystem is using without turning on quotas. Even then, it is cumbersome and not reported with df like it should be.
- When a maxmium size for a subvolume is set via a quota, it is not reported via df; applications have no idea when they are about to hit the maximum size of a filesystem.
btrfs would be fine if it worked reliably. I should say at the outset that I have never lost any data due to it, but it has caused enough kernel panics that I’ve lost count. I several times had a file that produced a panic when I tried to delete it, several times when it took more than 12 hours to unmount a btrfs filesystem, behaviors where hardlink-heavy workloads take days longer to complete than on zfs or ext4, and that’s just the ones I wrote about. I tried to use btrfs balance to change the metadata allocation on the filesystem, and never did get it to complete; it seemed to go into an endless I/O pattern after the first 1GB of metadata and never got past that. I didn’t bother trying the live migration of data from one disk to another on this filesystem.
I wanted btrfs to work. I really, really did. But I just can’t see it working. I tried it on my laptop, but had to turn of CoW on my virtual machine’s disk because of the rm bug. I tried it on my backup devices, but it was unusable there due to being so slow. (Also, the hardlink behavior is broken by default and requires btrfstune -r. Yipe.)
At this point, I don’t think it is really all that worth bothering with. I think the SuSE decision is misguided and ill-informed. btrfs will be an awesome filesystem. I am quite sure it will, and will in time probably displace zfs as the most advanced filesystem out there. But that time is not yet here.
In the meantime, I’m going to build a Debian Live Rescue CD with zfsonlinux on it. Because I don’t ever set up a system I can’t repair.
October 12th, 2013
A person can find all sorts of implementations of backups using hardlink trees to save space for incrementals. Some of them are fairly rudimentary, using rsync --link-dest. Others, like BackupPC, are more sophisticated, doing file-level dedup to a storage pool indexed by a hash.
While these are fairly space-efficient, they are really inefficient in other ways, because they create tons of directory entries. It would not be surprising to find millions of directory entries consumed very quickly. And while any given backup set can be deleted without impact on the others, the act of doing so can be very time-intensive, since often a full directory tree is populated with every day’s backup.
Much better is possible on modern filesystems. ZFS has been around for quite awhile now, and is stable on Solaris, FreeBSD and derivatives, and Linux. btrfs is also being used for real workloads and is considered stable on Linux.
Both have cheap copy-on-write snapshot operations that would work well with a simple rsync --inplace to achieve the same effect ad hardlink farms, but without all the performance penalties. When creating and destroying snapshots is a virtually instantaneous operation, and the snapshots work at a file block level instead of an entire file level, and preserve changing permissions and such as well (which rsync --link-dest can have issues with), why are we not using it more?
BackupPC has a very nice scheduler, a helpful web interface, and a backend that doesn’t have a mode to take advantage of these more modern filesystems. The only tool I see like this is dirvish, which someone made patches for btrfs snapshots three years ago that never, as far as I can tell, got integrated.
A lot of folks are rolling a homegrown solution involving rsync and snapshots. Some are using zfs send / btrfs send, but those mechanisms require the same kind of FS on the machine being backed up as on the destination, and do not permit excluding files from the backup set.
Is this an area that needs work, or am I overlooking something?
Incidentally, hats off to liw’s obnam. It doesn’t exactly do this, but sort of implements its own filesystem with CoW semantics.
October 9th, 2013
Almost every time he got off the bus for the past month and a half, Jacob started his afternoon in the same way. Before toys, before his trains and his toy bus, before anything indoors, he went for our cats. Here he is, cradling his favorite, Tigger:
Laura and I both grew up around cats. We had been talking about kittens, and shortly after we got engaged, one of my relatives offered us some free kittens. We went to his place one evening and selected two of them – one calico and one tiger-colored. Since what is now my place will soon be our place, they came to live with me. Our cats were one of the first things we did to prepare for our lives together.
Oliver wanted to name them some rather impractical sentence-long names (“The Cat Who Always Likes To Run”), so Laura and I suggested some names from one of their favorite books: Tigger and Roo. They both liked the names, but Oliver thought they should be called “Tigger the Digger” and “Roo the Runner”. Never mind that they were just 6 weeks old at the time, and not really old enough to either dig or run. Here’s Oliver with Roo, the day after the kittens arrived here.
I have always had outside cats, both because I’m allergic to cats so I need them to be outside, and because they sometimes literally quiver with joy of being outdoors. Tigger and Roo often chased insects, wrestled with each other, ran up (and slowly came back down) trees, and just loved the outside. Sometimes, I have taken my laptop and wireless headset and work from the back porch. The kittens climb up my jeans, inspect the laptop, and once Roo even fell asleep on my lap at one of those times.
Jacob has been particularly attached to Tigger, calling him “my very best friend.” When Jacob picks him up after school, Tigger often purrs while cradled in Jacob’s arms, and Jacob comments that “Tigger loves me. Oh dad, he knows I am his friend!”
The kittens have been growing, and becoming more and more comfortable with their home in the country. Whenever I go outside, it isn’t long before there are two energetic kittens near my feet, running back and forth, sometimes being very difficult to avoid stepping on. I call and I see little heads looking at me, from up in a tree, or peeking out from the grain elevator door, or from under the grill. They stare for just a second, and then start running, sometimes comically crashing into something in their haste.
Yesterday when I went to give them food, I called and no cats came. I was concerned, and walked around the yard, but at some point either they come or they don’t.
Yesterday afternoon, just after the bus dropped off Jacob, I discovered Tigger on the ground, motionless. Once Jacob was in the house, I went to investigate, and found Tigger was dead. As I was moving his body, I saw Roo was dead, too. Both apparently from some sort of sudden physical injury — a bit mysterious, because neither of them were at a place where they had ever gone before. While all this was happening, I had to also think about how I was going to tell the boys about this.
I tried to minimize what he could see, Jacob had caught an unavoidable glimpse of Tigger as we were walking back from the bus, but didn’t know exactly what had happened. He waited in the house, and when I came back, asked me if Tigger was dead. I said he was. Jacob started crying, saying, “Oh Dad, I am so sad”, and reached up for a hug. I picked him up and held him, then sat down on the couch and let him curl up on my lap. I could quite honestly let him know he wasn’t alone, telling him I am sad, too.
Oliver arrived not long after, and he too was sad, though not as much as Jacob. Both boys pretty soon wanted to see them. I decided this was important for them for closure, and to understand, so while they waited in the house, I went back out to arrange the kittens to hide their faces, the part that looks most unnatural after they die. The boys and I walked out to where I put them, then I carried both of them the last few feet. We stood a little ways back — close enough to see who was there, far enough to not get too much detail — and they were both sniffling. I tried to put voice to the occasion, saying, “Goodbye, Tigger and Roo. We love you.” Oliver asked if they could hear us. I said “No, but I told them what I felt like anyway.” Jacob, through tears, said, “Dad, maybe they are in heaven now.”
We went back inside. Jacob said, “Oh dad, I am so sad. This is the saddest day of my life. My heart is breaking.” Hearing a 7-year-old say that isn’t exactly easy for a dad. Pretty soon he was thinking of sort of comfort activities to do, saying “I think I would feel better if we did…” So they decided to watch a favorite TV program. Jacob asked if Laura knew yet, and when I said no, he got his take-charge voice and said, “Dad, you will start the TV show for us. While we are watching, you will send Laura an email to tell her about Tigger and Roo. OK?” What could I say, it wasn’t a bad idea.
Pretty soon both boys were talking and laughing. It was Big Truck Night last night, at a town about half an hour away. It’s an annual event we were already planning to attend, where all sorts of Big Trucks – firetrucks, school bus, combine, bucket truck, cement truck, etc – show up and are open for kids to climb in and explore. It’s always a highlight for them. They played and sang happily as we drove, excitedly opened and closed the big door on the school bus and yelled “All Aboard!” from the top of the combine. We ate dinner, and drove back home. When we got home, Jacob mentioned the cats again, in a sort of matter-of-fact way, and also wanted to make sure he knew Laura had got the message.
A person never wakes up expecting to have to dump a bowl of un-eaten cat food, or to give an impromptu cat funeral for little boys. As it was happening, I wished they hadn’t been around right then. But in retrospect, I am glad they were. They had been part of life for those kittens, and it is only right that they could be included in being part of death. They got visual closure this way, and will never wonder if the cats are coming back someday. They had a chance to say goodbye.
Here is how I remember the kittens.
September 16th, 2013
Today I link you to a video narrated by the legendary Carl Sagan – The Frontier is Everywhere.
We were hunters and foragers. The frontier was everywhere. We were bounded only by the earth, and the ocean, and the sky. The open road still softly calls. Our little terraqueous globe as the madhouse of those hundred thousand millions of worlds. We, who cannot even put our own planetary home in order, riven with rivalries and hatreds; are we to venture out into space?
By the time we are ready to settle even the nearest other planetary systems, we will have changed… For all our failings, despite our limitations and fallibilities, we humans are capable of greatness… How far will our nomadic species have wandered, by the end of the next century, and the next millennium?
Our remote descendants, safely arrayed on many worlds through the solar system, and beyond, will be unified, by their common heritage, by their regard for their home planet, and by the knowledge that, whatever other life may be, the only humans in all the universe, come from Earth.
They will gaze up and strain to find the blue dot in their skies. They will marvel at how vulnerable the repository of all our potential once was, how perilous our infancy, how humble our beginnings, how many rivers we had to cross, before we found our way.
Can you imagine seeing Earthrise?
August 27th, 2013
Today I have the delightful chance to write about a deep and wonderful joy.
Yes, Laura and I are engaged!
There are no words adequate for this kind of occasion, but there is a picture that gets close:
I never imagined a person could find a friend so wonderful, a person that enjoys so much in common, someone that understands me and that I understand so well. And yet, here I am, engaged to that friend. “Amazing” only begins to describe the feeling.
One of the first things Laura and I talked about was a hymn I was tinkering with, typesetting with GNU LilyPond. That hymn ends like this:
“Since Love is Lord of heav’n and earth, how can I keep from singing?”
It is a wonderful thought, and very true. Even literally true; I often find myself singing, humming, or playing the penny whistle during my day.
One of my good friends once told me, “I am completely sure that your happiest days lie ahead of you.” And he was right. I have already experienced them. This is a wonderful time, and every day brings plentiful reasons to be thankful. And as Laura and I prepare for a life together, I know that it is true not only that my friend was right, but that he is right — my happiest days are yet to come, and our happiest days are yet to come, too.
I have been blessed in many ways, and feel like the luckiest man alive.
To be loved, and to love, a person so wonderful is truly a remarkable gift.
August 1st, 2013
Today, Oliver and Jacob heard about monorails. I showed them a wikipedia article about monorails, and it had a picture of a monorail in Germany.
Oliver, age 4: I have been on that monorail!
Me: Sorry, Oliver, but that monorail is in Germany. You have never been to Germany.
Oliver: But I HAVE been to Germany!
Me: No, you have never been to Germany.
Oliver: Dad, I HAVE been to Germany. I love Germany!
Jacob, taking an interest: Dad, can we go to Germany sometime?
Me: Yes, we probably could sometime.
Jacob: Great! Then we could ride a Deutsche Bahn train!
At this point, we had a brief discussion about the fact that we can’t take a train from the United States to Germany, but we can fly there and then take trains.
Oliver: Dad, we should go the the airport and tell them to take us to Germany right now!
July 25th, 2013
I started writing code when I was somewhere around 1st grade, hacking some rudimentary BASIC on a TRS-80. Since then, I’ve used more programming languages than I can count; everything from C to Prolog, Python to Haskell, and probably a dozen others. I enjoy diversity in programming languages and tools.
The problem with all this is that there’s little room for language innovation on those platforms. If predictions about the decline of development on the PC pan out, how will the next advance in programming languages gain traction if the hot spots for development are locked down to three main languages? Will we find ourselves only willing to consider languages that can compile to Java/Dalvik bytecode? And won’t that put limits on the possible space for innovation to occur within?
Yes, I know about things like Mono and the Android Scripting Environment, which offer some potential for hope, but I think it’s still safe to say that these platforms are pretty closed to development in “unofficial” languages. I don’t know of major apps in the Android market written in Python, for instance.
I know of efforts such as Ubuntu’s, but even they simply lock down development to a different toolkit. There is no choice there of GTK vs. Qt, or such things.
Sadly, I don’t think it has to be this way. My phone has way more power than my computer did just a few years ago, and that computer was capable of not just executing, but even compiling, code written in all sorts of different languages.
But I don’t know if there’s much we can do to change things. Even with native code on Android, plenty still has to go through Dalvik (at least that’s my understanding). If you wanted to write a Python app with a full graphical touch interface on Android, I don’t think this would be all that easy of a task.
July 24th, 2013
“There will come a time when you believe everything is finished. That will be the beginning.”
– Louis L’Amour
Last year, I wrote about the difficult times in my life and hope for future, but never really explained why. I have written little since, because there is little I can write without a bit of explanation. It is partly because of the complexity of the task of telling my story without telling too much of others’ stories.
But it is important I tell this story. So often on the Internet, we hear only the brave face, the positive things that happen. This story involves tears and difficulty. And also we often see only anger and bitterness. This story involves joy and celebration.
One afternoon last year, I was working as usual (I work from home) when two sheriff deputies arrived on my doorstep. They gave me paperwork showing that my now ex-wife had filed for divorce, had asked them to serve me the paperwork, and that she had been given temporary possession of the house. I had 2 hours to gather up clothes and a little computer equipment (there was a list) and leave the house.
Thus began the most difficult time in my life. I went from reading a bedtime story and singing a bedtime song to my young boys every night to seeing them only a little, from living in the house my grandparents and dad had lived in to having no particular plan for where I’d sleep that night, from thinking I had a good idea of what the future held to not knowing when, if ever, I’d ever be back home. I worried about how the boys would fare (they have done well so far). And it was incomprehensible; I couldn’t find answers to “why?”.
In the time since, the divorce became final, I did return home, the boys spend more time with me, and a new normal emerged.
At the time, it seemed like a sudden, deep winter blizzard. I couldn’t see very far down the road, spring seemed far off, and I couldn’t see very well either forward or backward.
But I was determined to find positives in the situation. It started almost immediately; I had never been a person to talk about pain, but just a few hours after the divorce was filed, I knew I needed to talk to someone about it, and did. A week later, I shared about it in church. Amazing friends, locally and all over the world, provided support and encouragement. I had less total time with the boys than before the divorce, but more time with just the three of us, and we used it to play together at home, spend days in town, and even take a train trip to Santa Fe, where none of us had ever been before.
I realized how much I could forgive, and that my ex-wife probably did the best she could with the persistent legacy of difficult life events that happened to her long before she met me. I understood this, and was never angry, just sad, for everyone. I have always known nobody is perfect, myself included, but can be hard on myself when I’m less than perfect. I forgave myself, too, realizing that I did my best to help in the most unfamiliar of waters, and although I sometimes didn’t get it right, my conscience is clear because my heart was in the right place and I tried, very hard.
Most incredibly, I became a person with a deep sense of inner peace. I always tried to work hard to set life on a good path; I got good grades in school, am a good employee, and have a strong set of values. But where courts are involved, there’s a strong sense of powerlessness. At times, there was nothing I could do to make life better for my boys or for me. I finally had to let go of taking on responsibility for all that on my own shoulders. I simply knew that things would be OK, and in fact were OK, and that there is nothing in life that really deserves worries. That doesn’t mean worries are never present, but that mostly they are subdued, like a radio quietly playing in another room. When they aren’t, I can sit down at the piano, play my penny whistle, sing, walk to my creek, talk to friends, or any of so many things that let them melt away. I stopped searching for happiness and peace, and let those things find me.
In religious terms, my faith became not just an intellectual one, but also a spiritual one. An atheist friend asked me, “Just what does religion mean to you anyway?” My answer: “The certainty that spring always comes, for everyone that understands this.”
“People speak of misfortunes and sufferings,” remarked Pierre, “but if at this moment I were asked: ‘Would you rather be what you were before you were taken prisoner, or go through all this again?’ then for heaven’s sake let me again have captivity and horseflesh! We imagine that when we are thrown out of our usual ruts all is lost, but it is only then that what is new and good begins. While there is life there is happiness. There is much, much before us.”
– Tolstoy, War and Peace
By Thanksgiving, I had much to be thankful for. Some of it under my nose waiting to be rediscovered after years of distraction, such as the glorious Kansas sunrises. Some things were new, such as roasting a turkey all by myself (or, rather, with Jacob and Oliver) for the first time, and having it come out absolutely perfectly. And some were just the things of everyday life: that I lived in my own house again, that I could walk out to the creek at the edge of my property whenever I wanted, that I could play piano, that the sounds of laughter and little running feet again could often be heard on my wood floors.
That’s not to say everything was easy; the courts sometimes made decisions sometimes I didn’t think were in the boys’ best interests, legal things dragged on and on, but in the end, peace endured. Happiness endured. I found myself thinking at Thanksgiving that it was the best Thanksgiving ever. Not much later, I considered myself happier than I’ve ever been. I was focusing on the daily gifts of life, marveling at the sunrises, looking forward to life’s next adventure, confident that it would be far better than the last.
And then, to my complete surprise, I found myself in a relationship again. On top of all the wonderful things happening in my life, I met Laura. I never could have imagined a friend so wonderful, a relationship so loving and joyful, something to treasure so deeply. I can look back at events and shake my head in amazement and wonder, that I found myself happier than I’d ever been, and then this wonderful relationship on top of that. I have been blessed to have the life I do, and feel almost embarrassingly lucky.
I share this story because friends that had been through divorce years ago shared their stories. They gave me hope. And if I didn’t share this story in this public way, I would be squandering an opportunity to find more positives from what happened. I hope that this helps, somehow, someone that is in pain know that there is beauty in the valley, and spring always comes, every single year.
And I share it because happiness like this can’t be repressed for long. Tolstoy was right. While there is life, there is happiness.
I’ll end with a story from last Thanksgiving. It was 6:10AM that weekend. I was still asleep, and heard this:
Jacob, yelling from his room: “Dad? Dad!”
Me, groggy: “Yes, Jacob?”
Jacob: “Can I go down and look at the Christmas tree?”
Me: “Sure, and you can turn it on too.”
At that point, I could have gone back to sleep. I was really short on sleep that morning, and Jacob would have been fine. But I gave him a couple of minutes, then I went downstairs too. He was curled up on the piano bench, looking at the tree. I quietly turned up the downstairs thermostat, got a chair, put it next to the piano bench, and sat down by the tree too. Jacob crawled over onto my lap and snuggled up for awhile. Neither of us said anything. Then:
“This is the best Thanksgiving ever.” And he gave me a big hug.
And he was right. Yes, he was RIGHT!
“That will be the beginning.” Spring comes!
July 4th, 2013
Jacob, Oliver, and I were driving back home, about an 8-hour drive. At one point I heard Jacob say from the back seat, “Oliver, you can’t have my book. It has a bookmark in it!”
Aside from the apparent anti-sharing properties of bookmarks, I sort of smiled at Jacob inventing a bookmark.
An hour later, I heard Oliver saying, “Jacob, you can’t have MY book. It has a bookmark too!”
And again I thought that sounded rather cute. The thought of traveling boys — and they did travel very well — bookmarking the books I brought for them was a nice one.
And then I realized: I hadn’t brought bookmarks. Or, as far as I could recall, bookmark-making material.
I really wanted to say, “Uhm, Jacob, just what did you make that bookmark from?” But I didn’t. I figured everyone would be happier not having to deal with that question.
June 2nd, 2013
I recently made a routine analysis of my kitchen. (Of course I make a routine analysis of my kitchen; don’t you?) In it, I discovered these items, still usable, but approaching that magic “throw it out” date:
- Baby carrots
- Green beans
So, I thought, what can I make that would use all of these? And I realized I had some shrimp in the freezer, so: a shrimp boil! I tossed it all, plus some various seasonings and a few other veggies, into the Dutch oven, and boiled.
Jacob and Oliver watched the activity with interest. Well, except for the potato-peeling part. For that, they went and played with their toy school buses. But the rest was good. They carefully observed me adding some spices, some vegetables, the shrimp, and watched it all simmer. Then it was time to eat. Excitement!
Of course, it did take a few minutes to boil, so Jacob got down his whiteboard while Oliver looked on.
They enjoyed learning how to peel the shell from the shrimp and devoured their food.
And another night recently, Jacob unexpectedly showed up in the kitchen at 10PM. He said he was thirsty, so I got him some water. He asked, “Dad, did you make ice cream?”
Earlier that day, I had prepared ice cream with Oliver, but it was a kind that had to be cooked (lemon with pureed strawberries and peaches) and it wasn’t cool enough to finish before their bedtime. I did let them help add the ice and salt to the ice cream freezer just before they went to bed.
So I told Jacob that yes, the ice cream was done.
He stood there, tiredly, considering, with this “oh he’ll never say yes to ice cream at 10PM” look on his face.
So I said, “would you like one bite right now?”
The look of delight on his face was amazing; a broad smile, a twinkle in his eyes, and a clap. So I got out the big bowl of ice cream and scooped up one big spoonful. He loved it. Then I said, “should we go look at the stars?”
I carried Jacob outside to the porch. We stood there, looking up. I used to do this with him periodically, but it had been about a year. So he was thrilled. It was a partially overcast night, but there were still some stars visible. He had no idea there were some stars missing. To him, it was amazing and wonderful and infinite. “Oh dad, there are way too many stars to count!”
He stayed there, arms around my neck, for a minute or two, then was ready to go back inside. I set him down, gave him a hug, said “Goodnight, Jacob.” And off he trotted, back upstairs, wearing a contented smile, and he fell asleep almost immediately.
All it takes to delight children is a bit a shrimp or some stars. And those things delight me, too.