A Dry Spring

Spring in the prairie is a bit of an odd thing this year. Here and there, near ditches and creeks, a short, soft blanket of lush green grass covers the ground. A few feet away, patches of green are visible between the brown shoots of last year’s grass. Some trees are already turning green, purple, red, and white, while others stand still and brown, stubbornly insisting that spring is not here yet. To look at the thermometer may not be much guide either; two days after the temperature was nearly 90, we woke to see a dusting of snow on the ground.

It’s been dry, terribly try in Kansas. Grass next to a gravel driveway or road often has a chalkish layer of dust on it, kicked up by passing cars or even a stiff wind. The earth thirsts.

It is somehow fitting to celebrate Easter, that spring holiday, in the midst of the dry ground, to remember that water is not the only thing that can quench thirst.

Easter morning began sleepily, as we got up early to head to a sunrise service. It was in a pasture just outside a small Kansas town, and we gathered there at about 6:15, wearing only light jackets against the breeze. A fire was burning, and there was water on hand to quickly douse any grass that caught first that wasn’t supposed to — and it was occasionally used.

I was doing the prelude for the service, playing on my penny whistle. I enjoyed being able to do that, and was glad that the wind was calm enough that it didn’t interfere too much with the music.

We sang some hymns, listened to some Bible readings, and just stood in silence, listening to the crackle of the fire, some country dogs playing, and watching the sky to the east transform as the sun came up.

Then it was on to church for breakfast, and a break before the Easter service — the pipe organ ringing, piano playing with it, and deep trombone and full sanctuary of people singing our 4-part Easter hymns celebrating the day. Laura had the idea of pinning carnations onto the cross, and we got to watch everyone come up and add theirs.

Jacob and Oliver enjoyed the sunrise service. They decided they would keep a watchful eye on the first and the dogs, they enjoyed muffins at breakfast and playing in the church after that. But if you are 4 or 7, what is Easter without an Easter egg hunt? And they got in several.

Laura and I hid some eggs around the yard. Jacob asked me to use a radio to tell them when the eggs were ready. Here they are, bounding out the door to begin the hunt!

And, of course, if you are 4 or 7 and have a geek for a dad, you will naturally think to bring radios with you to the next hunt. To tell your brother what you’re finding, of course.

It was a good weekend, and in fact, Jacob even volunteered to put up a “wet floor” sign after he spilled some water:

On the last car ride of the day, Jacob decided he would write a story about his Easter. He decided he would publish a big book, and be a famous author and make other children happy. Oliver, of course, decided he needed an Easter story also. We couldn’t very well publish a book in the car, but I did manage to use my phone to capture their stories.

It’s been a long and busy week, but there is much to be joyful about, even when tired.

Springtime in the Mountains

The scene: early one morning as the sun has just started to rise. Jacob and Oliver, ages 7 and 4, are the first people to wake up in the house — their grandparents’ in California, where the four of us are visiting for the first time as a family.

They have a conversation and decide that would be a good to go “find a mystery.”

They decide to take their flashlights — pink and blue, matching each boy’s favorite color — and slowly, but not very quietly, open their bedroom door and creep out.

“Brother, you forgot your flashlight!” says Oliver.

“Oh, thanks brother! I’ll get it!” says Jacob.

Meanwhile, Laura’s mom wakes up, and notices two boys with flashlights creeping through the living room. Pretty soon they reach the kitchen, open the dishwasher, spy a suspicious-looking bowl, and decide that they have found the mystery — a clean bowl!

Or, at least that’s the story that I pieced together based on what a 4-year-old and the grandma he awakened told me.

We were on our first family trip to the Fresno, California area, to visit Laura’s parents — Jacob and Oliver’s new grandparents. They’ve played together before, but as this was our first visit to their place, there was quite the excitement. The boys had flown before, but it was several years ago and neither of them remember it well, so they were excited about that, too.

The night before, Jacob woke up to tell me “Dad, I am too excited to sleep. I think I will go downstairs and watch some TV.” He didn’t get too far with that plan. But he was excited. We went through security at the laid-back Wichita airport (where the TSA agents smile and there are often no security lines at all). We found our gate with enough time to grab lunch, which we did. The boys and I then did what we often do to kill time: explore. We explored the terminal, watching carpet-layers cut out carpet for the jetway, watching the construction of the new Terminal 3 out the window. And, of course, watching airplanes take off and land from the terminal’s large windows.

Finally it was our turn to board, and we all got on the plane: Jacob and Oliver with their backpacks of on-board activites, Laura and me with the rest of our carryon luggage, for the short trip to Denver.

Jacob and Oliver’s noses were pressed against the windows. Or, well, Jacob’s was. Oliver’s window was a little too high for him, but he was thrilled anyhow. They delighted in the airplane snacks, and the fact that they were allowed to drink pop on the plane. We packed books and some new art supplies for them (colored post-its, pages from a train-themed page-a-day calendar, a notebook, and a set of colored pens really seemed to do the trick.)

We had a choice of 35 minutes or 4 hours between flights in Denver, and I had chosen 4 hours, thinking that would be a lot less stressful with boys. And it was. We found a nice corner of the mezzanine to sit for awhile — they did art projects and played a game with Laura. Then I took them exploring Denver. We rode the moving sidewalks up and down the terminal, took a train ride to another terminal and back, ate supper all together, and flew to Fresno.

We had stopped in the Wichita airport to buy them each a souvenir airplane, and these came out often during the rest of the trip.

They enjoyed the mockups of the sequoias in Fresno Yosemite International Airport, enjoyed their beds and their room at the house, and did actually manage to fall asleep eventually.

We had a few days there, where they played in a park, with bubbles on the patio, or croquet in the yard (I even discovered Jacob happily using the cast his broken arm is in as a hammer to pound the hoops into the ground!)

There are a lot of miniatures in the house, and the boys enjoyed exploring the dollhouses — and especially the N-gauge model train. Jacob enjoyed it so much he asked me to record a video of him playing with the trains.

Evenings often brought book-reading, from the many children’s books in the house. At home, Laura and I and both boys often scrunch onto an oversized chair and read a book and sing a song (one I make up on whatever topic they choose). Over there, we often had Laura, Jacob, Oliver, Laura’s mom, and me scrunched up somewhere while the boys heard a story read to them by their grandma. That happened plenty of times other than bedtime, too. (Or Jacob would take his favorite books and read them to himself.)

Laura’s parents organized a reception Sunday for us, for the people from that area that couldn’t make it to our wedding. Jacob and Oliver, predictably, had fun playing and even talked to some of the adults. The adults that didn’t ask Jacob about his cast, anyhow (he dislikes talking about it).

The boys discovered a live mic at the church where the reception was, and do I detect two future pastors in our midst?

We had a great time at Laura’s uncle and aunt’s place. The boys were happy to discover an orange tree in their backyard, a tetherball post not far away, and an uncle ready to give them a demonstration of a “swimming pool vacuum cleaner” or sit at the piano with them. Jacob’s favorite part, though, was when the hamburger buns his great uncle were toasting were left on the grill during the prayer before the meal, got a bit scorched, and the uncle remarked with a chuckle that “I guess the Lord was tired of listening to me drone on!” Jacob loved his meal, and cackled at the thought of a prayer causing buns to get scorched.

But their highlight was the visit to the sequoias at Kings Canyon National Park the next day. The excitement had been building for that day all weekend. On the way out, we stopped at a fruit stand and bought some delicious strawberries — the fresh, juicy, sweet and tasty kind that are red all the way through. We continued up through the foothills, stopping periodically to get out and stretch, look at the sights, take some photos, or borrow grandpa’s binoculars.

I knew we’d be traveling in two cars, so I had the thought to pack some 2-way radios before we left. I gave one to the boys and one to the grandparents. All weekend long, whenever the six of us went somewhere, the boys (and especially Jacob) would give directions to the car that was following. “Turn right! … The light is green! … Catch up, you’re going too slow!” So all the way into the mountains, Jacob would send back instructions on what to do.

We saw Grant Grove, home to the worlds third-largest tree (267ft/81m tall and 3000 years old). It’s quite the impressive tree — the trunk’s diameter near the ground is 29ft or almost 9m. As we walked the trails, their speed kept increasing as they were hunting for the “tree tunnel” I had told them about — a tree that fell centuries ago and had been hollowed out to make a home. That trunk was easily 8ft or more in diameter, and I could stand up completely in places. We found it, to much delight from the boys — “So this is what it’s like to be inside a tree!”

Our trip home brought a delay in Denver and a missed flight, which excited the boys when I told them “now we get to eat supper in the airport!” I wonder how long that tactic will work… But Jacob was also excited because the plane we were put on in the end was bigger than the one we were scheduled on, so that was another piece of excitement.

We got home, and I carried two sleeping boys in from the car, upstairs, tucked them in, pulled off their shoes, and put their favorite stuffed animals in their arms. They were happy to be home, and with memories to treasure for a long time.

Agile Is Dead (Long Live Agility)

In an intriguing post, PragDave laments how empty the word “agile” has become. To paraphrase, I might say he’s put words to a nagging feeling I’ve had: that there are entire books about agile, conferences about agile, hallway conversations I’ve heard about whether somebody is doing this-or-that agile practice correctly.

Which, when it comes down to it, means that they’re not being agile. If process and tools, even if they’re labeled as “agile” processes and tools, are king, then we’ve simply replaced one productivity-impairing dictator with another.

And he makes this bold statement:

Here is how to do something in an agile fashion:

What to do:

  • Find out where you are
  • Take a small step towards your goal
  • Adjust your understanding based on what you learned
  • Repeat

How to do it:

When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier.

Those four lines and one practice encompass everything there is to know about effective software development.

He goes on to dive into that a bit, of course, but I think this man has a rare gift of expressing something complicated so succinctly. I am inclined to believe he is right.

Good all-day light laptop?

I am trying to find a laptop with all-day battery life that’s fairly light. Perhaps I am dreaming too big here, but I thought I’d toss out my hopes and see if there are any recommendations.

I am hoping for:

  • Battery life around 9 hours powered up
  • Fairly light. 2 or 3 pounds would be good.
  • Small. 10″ to 12″ screen is fine.
  • CPU – needs to be something “real”. No ARM or Atom.
  • Storage – size is less important than performance.
  • Screen – resolution has to be better than 768 vertical lines.
  • Ideally, capable of running Debian well.

The point is that I want to be able to do general web browsing with a real browser (Firefox), run Thunderbird for mail, hopefully even be able to run a VM or two under KVM or VirtualBox (with the understanding that this will kill battery life).

I have been using a Thinkpad T420s for a couple of years now. Its battery life wasn’t great even when it was new, and is worse now, of course. An Asus TF700t tablet has great battery life, but the storage system in it is so slow that I rather suspect that the browser cache is hurting, not helping, performance.

I am also rather disappointed with the Android system. I can’t really develop for it with my usual tools. Enough components are closed that, like Windows or MacOS X, I can’t feel like I can truly trust it with sensitive data. Although it has SSH clients available, the SSH server there wants you to pay to use public key auth, and the SSH clients don’t work all that well. Git can work, sort of. Battery life is great, and the keyboard is fine, but even flashing a different OS won’t fix that terrible performance.

I’ve had people recommend certain Asus laptops, a Microsoft Surface Pro, or a Macbook Air. Any thoughts?

Backing up every few minutes with simplesnap

I’ve written a lot lately about ZFS, and one of its very nice features is the ability to make snapshots that are lightweight, space-efficient, and don’t hurt performance (unlike, say, LVM snapshots).

ZFS also has “zfs send” and “zfs receive” commands that can send the content of the snapshot, or a delta between two snapshots, as a data stream – similar in concept to an amped-up tar file. These can be used to, for instance, very efficiently send backups to another machine. Rather than having to stat() every single file on a filesystem as rsync has to, it sends effectively an intelligent binary delta — which is also intelligent about operations such as renames.

Since my last search for backup tools, I’d been using BackupPC for my personal systems. But since I switched them to ZFS on Linux, I’ve been wanting to try something better.

There are a lot of tools out there to take ZFS snapshots and send them to another machine, and I summarized them on my wiki. I found zfSnap to work well for taking and rotating snapshots, but I didn’t find anything that matched my criteria for sending them across the network. It seemed par for the course for these tools to think nothing of opening up full root access to a machine from others, whereas I would much rather lock it down with command= in authorized_keys.

So I wrote my own, called simplesnap. As usual, I wrote extensive documentation for it as well, even though it is very simple to set up and use.

So, with BackupPC, a backup of my workstation took almost 8 hours. (Its “incremental” might take as few as 3 hours) With ZFS snapshots and simplesnap, it takes 25 seconds. 25 seconds!

So right now, instead of backing up once a day, I back up once an hour. There’s no reason I couldn’t back up every 5 minutes, in fact. The data consumes less space, is far faster to manage, and doesn’t require a nightly hours-long cleanup process like BackupPC does — zfs destroy on a snapshot just takes a few seconds.

I use a pair of USB disks for backups, and rotate them to offsite storage periodically. They simply run ZFS atop dm-crypt (for security) and it works quite well even on those slow devices.

Although ZFS doesn’t do file-level dedup like BackupPC does, and the lz4 compression I’ve set ZFS to use is less efficient than the gzip-like compression BackupPC uses, still the backups are more space-efficient. I am not quite sure why, but I suspect it’s because there is a lot less metadata to keep track of, and perhaps also because BackupPC has to store a new copy of a file if even a byte changes, whereas ZFS can store just the changed blocks.

Incidentally, I’ve packaged both zfSnap and simplesnap for Debian and both are waiting in NEW.

Migrated from Hetzner to OVH hosting

Since August 2011, my sites such as complete.org have been running on a Xen-backed virtual private server (VPS) at Hetzner Online, based in Germany. I had what they called their VQ19 package, which included 2GB RAM, 80GB HDD, 100Mb NIC and 4TB transfer.

Unlike many other VPS hosts, I never had performance problems. However, I did sometimes have hardware problems with the host, and it could take hours to resolve. Their tech support only works business hours German time, which was also a problem.

Meanwhile, OVH, a large European hosting company, recently opened a datacenter in Canada. Although they no longer offer their value-line Kimsufi dedicated servers there — starting at $11.50/mo — they do offer their midrange SoYouStart servers there. $50/mo gets a person a 4-core 3.2GHz Xeon server with 32GB RAM, 2x2TB SATA HDD, 200Mbps bandwidth. Not bad at all! The Kimsufi options are still good for lower-end needs as well.

I signed up for one of the SoYouStart servers. I’ve been pleased with my choice to migrate, and at the possibilities that having hardware like that at my disposal open up, but it is not without its downside.

The primary downside is lack of any kind of KVM console. If the server doesn’t boot, I can’t see the Grub error message (or whatever) behind it. They do provide hardware support and automatic technician dispatching when the server isn’t pingable, but… they state they have no KVM access at all. They support many OS flavors, and have a premade image for them, but there is no using a custom ISO to install; if you want ZFS on Linux, for instance, you can’t very easily build it into root.

My server was promised within 72 hours, but delivered much quicker: within about 1. I had two times they said they had to replace a motherboard within the first day; once they did it in 30 minutes, and the other took them 2.5 hours for some reason. They do have phone support, which answers almost immediately, but the people there are not the people actually in the datacenter. It was frustrating with a server down for hours and nobody really commenting on what was going on.

The server performs quite well, and after the initial issues, I’ve been happy.

I was initially planning an all-ZFS installation. SoYouStart does offer a rescue environment, but it doesn’t support ZFS, so I figured I better stick with an ext4 root at least. The default Debian install uses RAID1 on md-raid, with a 20GB root partition and the rest of the 2TB drive in /home, and then a swap partition on each drive (mysteriously NOT in the RAID!) So I broke the mirror on /home and converted those into the two legs of a mirrored vdev for a zpool.

I run all of the real work inside KVM VMs, so that should minimize the number of times I have to do anything to the root filesystem that could cause trouble.

SoYouStart includes 100GB of space on a separate FTP server for backup purposes. I have scripts that upload nightly tarballs of the root filesystem, plus full “zfs send” streams of everything else. Every hour, it uploads an incremental “zfs send” stream as well. This all works quite nicely; even if the machine is a complete loss, I’d never lose more than an hour’s work, and could restore it completely from a rescue environment. Very nice!

I’ll write more in a few days about the ZFS setup I’m using, and some KVM discoveries as well.

VirtFS isn’t quite ready

Despite claims to the contrary [PDF], VirtFS — the 9P-based virtio KVM/QEMU layer designed to pass through a host’s filesystem to the guest — is quite slow. I have yet to get it to perform at even 1/10 the speed of the virtual block device (VBD). That’s unfortunate, because in theory it should be significantly faster. At this rate, I suspect even NFS will be significantly faster.

Beyond that, it seems impossible to use VirtFS as the root filesystem in a VM, at least with Debian; initramfs-tools doesn’t know how to build an initrd in that situation, and the support is just not there.

It would make a great combination with btrfs or zfs, but unfortunately looks to be just not ready yet.

How to fix “fstrim: Operation not supported” under KVM?

Maybe someone out there will have some ideas.

I have a KVM host running wheezy, with wheezy-backports versions of libvirt and qemu. I have defined a guest, properly set discard=unmap in the domain XML file for it, verified that’s being passed to the guest, but TRIM/DISCARD is just not working.

Mounting the ext4 filesystem with discard has no effect, and fstrim / always reports:

fstrim: /: FITRIM ioctl failed: Operation not supported

Every single time.

I’ve tried with the virtio, IDE, and SCSI (both default and virtio-scsi) backend drivers. The guest is also running wheezy (i386 version; the host is amd64) and I’ve tried the latest 3.12 backported kernel for it. No dice.

If I shut down the VM and mount the filesystem on the host, fstrim works fine.

Everything says this should work. But it doesn’t.

Any ideas?

Why and how to run ZFS on Linux

I’m writing a bit about ZFS these days, and I thought I’d write a bit about why I am using it, why it might or might not be interesting for you, and what you might do about it.

ZFS Features and Background

ZFS is not just a filesystem in the traditional sense, though you can use it that way. It is an integrated storage stack, which can completely replace the need for LVM, md-raid, and even hardware RAID controllers. This permits quite a bit of flexibility and optimization not present when building a stack involving those components. For instance, if a drive in a RAID fails, it needs only rebuild the parts that have actual data stored on them.

Let’s look at some of the features of ZFS:

  • Full checksumming of all data and metadata, providing protection against silent data corruption. The only other Linux filesystem to offer this is btrfs.
  • ZFS is a transactional filesystem that ensures consistent data and metadata.
  • ZFS is copy-on-write, with snapshots that are cheap to create and impose virtually undetectable performance hits. Compare to LVM snapshots, which make writes notoriously slow and require an fsck and mount to get to a readable point.
  • ZFS supports easy rollback to previous snapshots.
  • ZFS send/receive can perform incremental backups much faster than rsync, particularly on systems with many unmodified files. Since it works from snapshots, it guarantees a consistent point-in-time image as well.
  • Snapshots can be turned into writeable “clones”, which simply use copy-on-write semantics. It’s like a cp -r that completes almost instantly and takes no space until you change it.
  • The datasets (“filesystems” or “logical volumes” in LVM terms) in a zpool (“volume group”, to use LVM terms) can shrink or grow dynamically. They can have individual maximum and minimum sizes set, but unlike LVM, where if, say, /usr gets bigger than you thought, you have to manually allocate more space to it, ZFS datasets can use any space available in the pool.
  • ZFS is designed to run well in big iron, and scales to massive amounts of storage. It supports SSDs as L2 cache and ZIL (intent log) devices.
  • ZFS has some built-in compression methods that are quite CPU-efficient and can yield not just space but performance benefits in almost all cases involving compressible data.
  • ZFS pools can host zvols, a block device under /dev that stores its data in the zpool. zvols support TRIM/DISCARD, so are ideal for storing VM images, as they can instantly release space released by the guest OS. They can also be snapshotted and backed up like the rest of ZFS.

Although it is often considered a server filesystem, ZFS has been used in plenty of other situations for some time now, with ports to FreeBSD, Linux, and MacOS. I find it particularly useful:

  • To have faith that my photos, backups, and paperwork archives are intact. zpool scrub at any time will read the entire dataset and verify the integrity of every bit.
  • I can create snapshots of my system before running apt-get dist-upgrade, making it easy to track down issues or roll back to a known-good configuration. Ideal for people tracking sid or testing. One can also easily simply boot from a previous snapshot.
  • Many scripts exist that make frequent snapshots, and retain the for a period of time as a way of protecting work in progress against an accidental rm. There is no reason not to snapshot /home every 5 minutes, for instance. It’s almost as good as storing / in git.

The added level of security in having cheap snapshots available is almost worth it by itself.

ZFS drawbacks

Compared to other Linux filesystems, there are a few drawbacks of ZFS:

  • CDDL will prevent it from ever being part of the Linus kernel tree
  • It is more RAM-hungry than most, although with tuning it can even run on the Raspberry Pi.
  • A 64-bit kernel is strongly preferred, even in low-memory situations.
  • Performance on many small files may be less than ext4
  • The ZFS cache does not shrink and expand in response to changing RAM usage conditions on the system as well as the normal Linux cache does.
  • Compared to btrfs, ZFS lacks some features of btrfs, such as being able to shrink an existing pool or easily change storage allocation on the fly. On the other hand, the features in ZFS have never caused me a kernel panic, and half the things I liked about btrfs seem to have.
  • ZFS is already quite stable on Linux. However, the GRUB, init, and initramfs code supporting booting from a ZFS root and /boot is less stable. If you want to go 100% ZFS, be prepared to tweak your system to get it to boot properly. Once done, however, it is quite stable.

Converting to ZFS

I have written up an extensive HOWTO on converting an existing system to use ZFS. It covers workarounds for all the boot-time bugs I have encountered as well as documenting all steps needed to make it happen. It works quite well.

Additional Hints

If setting up zvols to be used by VirtualBox or some such system, you might be interested in managing zvol ownership and permissions with udev.

Debian-Live Rescue image with ZFS On Linux; Ditched btrfs

I’m a geek. I enjoy playing with different filesystems, version control systems, and, well, for that matter, radios.

I have lately started to worry about the risks of silent data corruption, and as such, looked to switch my personal systems to either ZFS or btrfs, both of which offer built-in checksumming of all data and metadata. I initially opted for btrfs, because of its tighter integration into the Linux kernel and ability to shrink an existing btrfs filesystem.

However, as I wrote last month, that experiment was not a success. I had too many serious performance regressions and one too many kernel panics and decided it wasn’t worth it. And that the SuSE people got it wrong, deeply wrong, when they declared btrfs ready for production. I never lost any data, to its credit. But it simply reduces uptime too much.

That left ZFS. Before I build a system, I always want to make sure I can repair it. So I started with the Debian Live rescue image, and added the zfsonlinux.org repository to it, along with some key packages to enable the ZFS kernel modules, GRUB support, and initramfs support. The resulting image is described, and can be downloaded from, my ZFS Rescue Disc wiki page, which also has a link to my source tree on github.

In future blog posts in the series, I will describe the process of converting existing Debian installations to use ZFS, of getting them to boot from ZFS, some bugs I encountered along the way, and some surprising performance regressions in ZFS compared to ext4 and btrfs.