Category Archives: Uncategorized

Migrated from Hetzner to OVH hosting

Since August 2011, my sites such as complete.org have been running on a Xen-backed virtual private server (VPS) at Hetzner Online, based in Germany. I had what they called their VQ19 package, which included 2GB RAM, 80GB HDD, 100Mb NIC and 4TB transfer.

Unlike many other VPS hosts, I never had performance problems. However, I did sometimes have hardware problems with the host, and it could take hours to resolve. Their tech support only works business hours German time, which was also a problem.

Meanwhile, OVH, a large European hosting company, recently opened a datacenter in Canada. Although they no longer offer their value-line Kimsufi dedicated servers there — starting at $11.50/mo — they do offer their midrange SoYouStart servers there. $50/mo gets a person a 4-core 3.2GHz Xeon server with 32GB RAM, 2x2TB SATA HDD, 200Mbps bandwidth. Not bad at all! The Kimsufi options are still good for lower-end needs as well.

I signed up for one of the SoYouStart servers. I’ve been pleased with my choice to migrate, and at the possibilities that having hardware like that at my disposal open up, but it is not without its downside.

The primary downside is lack of any kind of KVM console. If the server doesn’t boot, I can’t see the Grub error message (or whatever) behind it. They do provide hardware support and automatic technician dispatching when the server isn’t pingable, but… they state they have no KVM access at all. They support many OS flavors, and have a premade image for them, but there is no using a custom ISO to install; if you want ZFS on Linux, for instance, you can’t very easily build it into root.

My server was promised within 72 hours, but delivered much quicker: within about 1. I had two times they said they had to replace a motherboard within the first day; once they did it in 30 minutes, and the other took them 2.5 hours for some reason. They do have phone support, which answers almost immediately, but the people there are not the people actually in the datacenter. It was frustrating with a server down for hours and nobody really commenting on what was going on.

The server performs quite well, and after the initial issues, I’ve been happy.

I was initially planning an all-ZFS installation. SoYouStart does offer a rescue environment, but it doesn’t support ZFS, so I figured I better stick with an ext4 root at least. The default Debian install uses RAID1 on md-raid, with a 20GB root partition and the rest of the 2TB drive in /home, and then a swap partition on each drive (mysteriously NOT in the RAID!) So I broke the mirror on /home and converted those into the two legs of a mirrored vdev for a zpool.

I run all of the real work inside KVM VMs, so that should minimize the number of times I have to do anything to the root filesystem that could cause trouble.

SoYouStart includes 100GB of space on a separate FTP server for backup purposes. I have scripts that upload nightly tarballs of the root filesystem, plus full “zfs send” streams of everything else. Every hour, it uploads an incremental “zfs send” stream as well. This all works quite nicely; even if the machine is a complete loss, I’d never lose more than an hour’s work, and could restore it completely from a rescue environment. Very nice!

I’ll write more in a few days about the ZFS setup I’m using, and some KVM discoveries as well.

Delicious Holiday Recipes

I’ve come up with some new favorites this season. The boys and Laura were around for all three, and I am happy to report there were many kitchen smiles over these!

From-Scratch Hot Chocolate

There’s something about hot chocolate made from scratch, with chocolate melted into milk, instead of a powder stirred in. It takes quite a bit more time, and probably has more calories, but it is quite delicious.

The key to a delicious result where milk is concerned is to take things slow and keep stirring. You don’t want the chocolate to scorch at the bottom of the pan. Heating up the milk before the chocolate should help things mix in more easily as well.

  • Begin with 3 cups milk and 1 cup heavy whipping cream. Heat slowly over moderate to low heat, stirring periodically. Once you see bubbles start to form around the edges, it is plenty hot (or even a bit more hot than it needs to be).
  • Add one cup of semisweet chocolate chips, 1 teaspoon sugar, and 1/2 tsp vanilla extract.
  • Stir constantly until all the chocolate is melted and well mixed. There will still be some small bits of chocolate within, but if it is all done slowly like this, the chocolate should be pretty well melted.

The basis for this recipe was here, and it called for 2 cups milk and 2 cups half-and-half. I trust my heavy whipping cream was fine! There are also some other variations on that site.

This nearly made my little cast iron kettle overflow, so next time I made a 3/4 recipe.

Hot Spiced Cider

We put up a Christmas tree yesterday, so I thought hot spiced cider would be perfect for the occasion. I went searching for recipes, and many of them called for cloves (which have to be sifted out later or put in a spice bag). I wasn’t going to have time to delay two boys from setting up a Christmas tree long enough for that, so I found this basic recipe to work well. However, I, as usual, made some modifications ;-)

  • Warm 4 cups apple cider (not juice, as the recipe suggests) in a pot.
  • Add 1/2 tsp cinnamon
  • Add 1/4 tsp nutmeg or allspice (I used allspice because I was mysteriously out of nutmeg, but will probably use nutmeg next time)
  • Add 1 tbsp brown sugar
  • Stir constantly until sufficiently dissolved. Pour immediately before drinking, as the contents will tend to separate.

Mmmmm…. yum….

Turkey or Chicken Noodle Soup

The annual “what to do with all that leftover turkey” quest strikes again. I like chicken noodle soup, so why not a turkey noodle soup done the same way?

Here’s what I used, roughly, in my large 6-quart cast iron cooking pot (aka “Dutch oven”):

  • 9 cups chicken/turkey broth. Your own if you have it, or the canned variety works. Or make your own with boullion if you have it.
  • 2 chopped yellow onions. (I added half a chopped red onion as well because I had it sitting around. Nobody complained, but 2.5 onions was a little much.)
  • 4 tsp fresh basil or 1 tsp dried basil
  • 4 tsp fresh oregano or 1 tsp dried oregano
  • 1/2 tsp pepper
  • 1/2 tsp salt
  • 1 tsp beef bouillon
  • 2 bay leaves
  • 20 oz frozen mixed vegetables (I’d probably add more than that next year; this wasn’t quite enough)
  • Plenty of wide egg noodles. The recipe I used called for 1 cup, which was laughably inadequate. I just dumped until it looked right, and then the package was almost empty so I dumped the rest in too.
  • 4 cups cooked turkey or chicken, cubed (a kitchen scissors makes quick work of that)
  • Two 14.5-oz cans diced tomatoes (do not drain)

Start with the broth, onion, basil, oregano, pepper, and bay leaf. Heat up the mixture and add the vegetables. Bring it to boiling, then add the uncooked noodles. Return to boiling, then reduce heat, cover, and simmer for 8 minutes. Add the turkey or chicken and diced tomatoes, and simmer until hot enough to serve.

The nice thing about soups is that they freeze well and make great winter leftovers. This recipe makes quite a lot of soup; you may wish to halve it.

This recipe was adapted from one in a Better Homes & Gardens cookbook.

Two Kittens

Almost every time he got off the bus for the past month and a half, Jacob started his afternoon in the same way. Before toys, before his trains and his toy bus, before anything indoors, he went for our cats. Here he is, cradling his favorite, Tigger:

Laura and I both grew up around cats. We had been talking about kittens, and shortly after we got engaged, one of my relatives offered us some free kittens. We went to his place one evening and selected two of them – one calico and one tiger-colored. Since what is now my place will soon be our place, they came to live with me. Our cats were one of the first things we did to prepare for our lives together.

Oliver wanted to name them some rather impractical sentence-long names (“The Cat Who Always Likes To Run”), so Laura and I suggested some names from one of their favorite books: Tigger and Roo. They both liked the names, but Oliver thought they should be called “Tigger the Digger” and “Roo the Runner”. Never mind that they were just 6 weeks old at the time, and not really old enough to either dig or run. Here’s Oliver with Roo, the day after the kittens arrived here.

I have always had outside cats, both because I’m allergic to cats so I need them to be outside, and because they sometimes literally quiver with joy of being outdoors. Tigger and Roo often chased insects, wrestled with each other, ran up (and slowly came back down) trees, and just loved the outside. Sometimes, I have taken my laptop and wireless headset and work from the back porch. The kittens climb up my jeans, inspect the laptop, and once Roo even fell asleep on my lap at one of those times.

Jacob has been particularly attached to Tigger, calling him “my very best friend.” When Jacob picks him up after school, Tigger often purrs while cradled in Jacob’s arms, and Jacob comments that “Tigger loves me. Oh dad, he knows I am his friend!”

The kittens have been growing, and becoming more and more comfortable with their home in the country. Whenever I go outside, it isn’t long before there are two energetic kittens near my feet, running back and forth, sometimes being very difficult to avoid stepping on. I call and I see little heads looking at me, from up in a tree, or peeking out from the grain elevator door, or from under the grill. They stare for just a second, and then start running, sometimes comically crashing into something in their haste.

Yesterday when I went to give them food, I called and no cats came. I was concerned, and walked around the yard, but at some point either they come or they don’t.

Yesterday afternoon, just after the bus dropped off Jacob, I discovered Tigger on the ground, motionless. Once Jacob was in the house, I went to investigate, and found Tigger was dead. As I was moving his body, I saw Roo was dead, too. Both apparently from some sort of sudden physical injury — a bit mysterious, because neither of them were at a place where they had ever gone before. While all this was happening, I had to also think about how I was going to tell the boys about this.

I tried to minimize what he could see, Jacob had caught an unavoidable glimpse of Tigger as we were walking back from the bus, but didn’t know exactly what had happened. He waited in the house, and when I came back, asked me if Tigger was dead. I said he was. Jacob started crying, saying, “Oh Dad, I am so sad”, and reached up for a hug. I picked him up and held him, then sat down on the couch and let him curl up on my lap. I could quite honestly let him know he wasn’t alone, telling him I am sad, too.

Oliver arrived not long after, and he too was sad, though not as much as Jacob. Both boys pretty soon wanted to see them. I decided this was important for them for closure, and to understand, so while they waited in the house, I went back out to arrange the kittens to hide their faces, the part that looks most unnatural after they die. The boys and I walked out to where I put them, then I carried both of them the last few feet. We stood a little ways back — close enough to see who was there, far enough to not get too much detail — and they were both sniffling. I tried to put voice to the occasion, saying, “Goodbye, Tigger and Roo. We love you.” Oliver asked if they could hear us. I said “No, but I told them what I felt like anyway.” Jacob, through tears, said, “Dad, maybe they are in heaven now.”

We went back inside. Jacob said, “Oh dad, I am so sad. This is the saddest day of my life. My heart is breaking.” Hearing a 7-year-old say that isn’t exactly easy for a dad. Pretty soon he was thinking of sort of comfort activities to do, saying “I think I would feel better if we did…” So they decided to watch a favorite TV program. Jacob asked if Laura knew yet, and when I said no, he got his take-charge voice and said, “Dad, you will start the TV show for us. While we are watching, you will send Laura an email to tell her about Tigger and Roo. OK?” What could I say, it wasn’t a bad idea.

Pretty soon both boys were talking and laughing. It was Big Truck Night last night, at a town about half an hour away. It’s an annual event we were already planning to attend, where all sorts of Big Trucks – firetrucks, school bus, combine, bucket truck, cement truck, etc – show up and are open for kids to climb in and explore. It’s always a highlight for them. They played and sang happily as we drove, excitedly opened and closed the big door on the school bus and yelled “All Aboard!” from the top of the combine. We ate dinner, and drove back home. When we got home, Jacob mentioned the cats again, in a sort of matter-of-fact way, and also wanted to make sure he knew Laura had got the message.

A person never wakes up expecting to have to dump a bowl of un-eaten cat food, or to give an impromptu cat funeral for little boys. As it was happening, I wished they hadn’t been around right then. But in retrospect, I am glad they were. They had been part of life for those kittens, and it is only right that they could be included in being part of death. They got visual closure this way, and will never wonder if the cats are coming back someday. They had a chance to say goodbye.

Here is how I remember the kittens.

Earthrise

Today I link you to a video narrated by the legendary Carl Sagan – The Frontier is Everywhere.

Partial quotes:

We were hunters and foragers. The frontier was everywhere. We were bounded only by the earth, and the ocean, and the sky. The open road still softly calls. Our little terraqueous globe as the madhouse of those hundred thousand millions of worlds. We, who cannot even put our own planetary home in order, riven with rivalries and hatreds; are we to venture out into space?

By the time we are ready to settle even the nearest other planetary systems, we will have changed… For all our failings, despite our limitations and fallibilities, we humans are capable of greatness… How far will our nomadic species have wandered, by the end of the next century, and the next millennium?

Our remote descendants, safely arrayed on many worlds through the solar system, and beyond, will be unified, by their common heritage, by their regard for their home planet, and by the knowledge that, whatever other life may be, the only humans in all the universe, come from Earth.

They will gaze up and strain to find the blue dot in their skies. They will marvel at how vulnerable the repository of all our potential once was, how perilous our infancy, how humble our beginnings, how many rivers we had to cross, before we found our way.

Can you imagine seeing Earthrise?

Earthrise, 1968 December 24

Incredible.

Development Freedom in New Platforms

I started writing code when I was somewhere around 1st grade, hacking some rudimentary BASIC on a TRS-80. Since then, I’ve used more programming languages than I can count; everything from C to Prolog, Python to Haskell, and probably a dozen others. I enjoy diversity in programming languages and tools.

I’ve had an interest in both web and mobile development for some time, but have done little with either. Reflecting on that lately, I realize that both platforms severely restrict my freedom to use the tools I choose. Even on an old x86 with 640K of RAM, I had choices; BASIC, Pascal, and C would work under DOS. (Plus assembler, but I had yet to learn that at the time.) On Android, my choices are Java or… Java. (OK, or something that compiles to Java, but those selections are still limited.) Plus it seems like I’d have to use Eclipse, which seems to have taken the kitchen sync crown from Emacs long ago, and not in a good way. On iOS, the choices are Objective C. And on the web, it’s JavaScript.

The problem with all this is that there’s little room for language innovation on those platforms. If predictions about the decline of development on the PC pan out, how will the next advance in programming languages gain traction if the hot spots for development are locked down to three main languages? Will we find ourselves only willing to consider languages that can compile to Java/Dalvik bytecode? And won’t that put limits on the possible space for innovation to occur within?

Yes, I know about things like Mono and the Android Scripting Environment, which offer some potential for hope, but I think it’s still safe to say that these platforms are pretty closed to development in “unofficial” languages. I don’t know of major apps in the Android market written in Python, for instance.

I know of efforts such as Ubuntu’s, but even they simply lock down development to a different toolkit. There is no choice there of GTK vs. Qt, or such things.

Sadly, I don’t think it has to be this way. My phone has way more power than my computer did just a few years ago, and that computer was capable of not just executing, but even compiling, code written in all sorts of different languages.

But I don’t know if there’s much we can do to change things. Even with native code on Android, plenty still has to go through Dalvik (at least that’s my understanding). If you wanted to write a Python app with a full graphical touch interface on Android, I don’t think this would be all that easy of a task.

Sometimes, it’s best not to know

Jacob, Oliver, and I were driving back home, about an 8-hour drive. At one point I heard Jacob say from the back seat, “Oliver, you can’t have my book. It has a bookmark in it!”

Aside from the apparent anti-sharing properties of bookmarks, I sort of smiled at Jacob inventing a bookmark.

An hour later, I heard Oliver saying, “Jacob, you can’t have MY book. It has a bookmark too!”

And again I thought that sounded rather cute. The thought of traveling boys — and they did travel very well — bookmarking the books I brought for them was a nice one.

And then I realized: I hadn’t brought bookmarks. Or, as far as I could recall, bookmark-making material.

I really wanted to say, “Uhm, Jacob, just what did you make that bookmark from?” But I didn’t. I figured everyone would be happier not having to deal with that question.

Amazement

One of the benefits of working from home is that I have a great view of Kansas from my desk. While I work, I have seen sunrises, snowfall, birds, rain, ice, and all sorts of wildlife. I heard this verse of Home on the Range the other day, which reminded me of this:

“How often at night when the heavens are bright
With the light from the glittering stars
Have I stood here amazed and asked as I gazed
If their glory exceeds that of ours.”

One time, I asked Jacob if I should wake him up at night to see the stars. He said an excited yes! So when it was dark outside, I woke him up, carried him outside, and held him while he looked up. He said a long, breathy “Wooooow!” Then he went back to bed, curled up with his butterfly, and fell asleep smiling. Every so often, we repeat this little routine.

There are many opportunities in life to just stand somewhere and be amazed. You don’t have to be in Kansas. Children know this. The rest of us just have to notice.

Milk, Cookies, and Delight

Sometimes an attic is all it takes to delight children.

This afternoon, the boys and I made cookies. Jacob has been talking about setting out milk and cookies for Santa Claus for several days, and of course the fact that we had made cookies reminded him of this – as I figured it would. So after the boys got into their pajamas and all ready for bed, we set out milk and cookies for Santa.

The boys have always known that Santa is pretend, but love the stories and traditions anyhow. Never mind that Christmas was 3 days ago, and they’ve already opened their presents. It’s SANTA! It’s magic! It doesn’t matter!

I asked Jacob, “Would you like me to pretend to be Santa tonight?” A big grin, then “Oh yes, dad! Do it!”

So after I read them their bedtime story, sang them a song (Jacob chose a Latin hymn – that’s my boy!), and tucked them into bed, I pretended to be Santa. I went back downstairs. I drank the milk and ate the cookies. Then I went to my small future present stash, selected a few small items, and put them under the tree. I gave it a few minutes.

Then I crept up to the attic. I snuck along the wood floors quietly, until I was above the boys’ room.

Then I jumped. And I scraped a wood chair along the floor. And then I yelled out – “HO HO HO! Merry Christmas!” I had a brief conversation with Rudolph, then made some sliding noises. I was silent for a few seconds, then made some more noise and said, “Wow, Rudolph, Jacob and Oliver left some great milk and cookies! Let’s go deliver the rest of our presents!” And made some vague sleigh taking off from the roof of a house noises.

I crept back down the stairs. I put my ear to the outside of the closed door to the boys’ bedroom. I heard Jacob excitedly jabbering, “He said milk and cookies! He liked them! He really liked them! Ooo butterfly, he was here!” (Butterfly is a stuffed, er, butterfly that he sleeps with.)

I gave it a minute or two, then I went in. “Jacob, did you hear something?”

“Yes!”

“What was it?”

“Well, it was a loud thud! I sat straight up like this. [ he demonstrates ] Then I heard ‘ho ho ho’! And ‘milk and cookies’! And I was excited like this!” [ more demonstrations ]

“What do you think happened?”

“I don’t know! Dad, what did you do?

I told him. It only increased his delight.

“Did it sound like Santa’s sleight landing?”

“(annoyed) No, dad. It sounded like a crash. (brightening) And then Santa coming down the chimney with presents! Oh, it is so exciting!”

(We don’t have a chimney)

It was still magical, even though he knew exactly what happened.

For his part, Oliver slept through it all. He will still discover the empty plate, empty cup, and slightly less empty area underneath the tree. And neither boy knows about the thank you note from Santa yet. I anticipate smiles in the morning!

Proof Humans Are Capable of Working Magic

What an astonishing thing a book is. It’s a flat object made from a tree with flexible parts on which are imprinted lots of funny dark squiggles. But one glance at it and you’re inside the mind of another person, maybe somebody dead for thousands of years. Across the millennia, an author is speaking clearly and silently inside your head, directly to you. Writing is perhaps the greatest of human inventions, binding together people who never knew each other, citizens of distant epochs. Books break the shackles of time.

A book is proof that humans are capable of working magic.

— Carl Sagan, The Persistence of Memory

Unreported Disk Data Corruption – Kernel Bug?

Well this is new, and I’m utterly baffled. Here’s a file that’s not in use by anything.


$ md5sum xppro.vdi
589cbb5501dcddda047344a3550aaa95 xppro.vdi
$ md5sum xppro.vdi
a69806ec60d39e06473edbb0abd71637 xppro.vdi

Every time I run md5sum on it, I get a different answer. Same story with sha256sum. If I grab just the first 100MB, it gives the same answer each time. dmesg doesn’t show any sort of errors whatsoever during the time I’m running the tools. The file is 13GB, and was copied from one laptop to another (the new one being a Thinkpad T420s). The old laptop gives the same answer every time. The new one doesn’t.

I’ve put the file on different ext4 filesystems on the same machine (one using LUKS encryption, the other not, both under LVM) – same result. This will have also guaranteed different placement on the underlying hard disk.

I verified that nothing is modifying the file by using lsof and inotify. The system is a freshly-installed Debian wheezy running kernel 3.2.0-1-amd64. Any ideas how I go about troubleshooting/fixing this? So far I don’t know if it’s hardware or software, though my gut says software; SMART isn’t showing issues here, and the kernel didn’t log hardware issues, either.