• A blog, which I have had for years. I used to post things like short links, daily thoughts, etc – almost every day. It seems that there is some social pressure to not do that on blogs anymore, so I don’t too much. My blog gets mostly edited, more carefully thought-out, longer-form posts now. I’m not entirely happy with that direction though, since it means I don’t post much on the blog because it takes a lot of time to compose things nicely for it.
• A twitter account, which I sometimes use to post links and such. However, I have noticed a significant decline in the number of actual conversations I have on Twitter since Google+ came out, and I wonder how relevant Twitter will remain to people in the future.
• I also have an identi.ca account, though I almost never have any interactions there anymore.
• A Facebook account, which is mostly used to keep in touch with people I know offline in one way or another. Many of them use Facebook exclusively, sometimes even more than email.
• A Google+ account. I post similar content there as I do on twitter, though probably more of it because it doesn’t have a character limit. I really enjoy the community on Google+ – there are few people I’ve met in person in my circles, but many people I know from various online activities. And many just plain brilliant, engaging, or interesting people. As an example: I follow Edd Dumbill, the (former?) chair of OSCon, on Google+. He started talking about his Fitbit getting broken, which led me to ask him some questions about it – which he, and others, answered – and me ordering one myself. I just don’t have that kind of interaction anywhere else.
• A Diaspora account that I created but honestly haven’t had time to use.

So my problems are:

1. Posting things multiple places. I currently can post on identi.ca, which automatically posts to twitter, which automatically posts to Facebook. But then I’d still have to post to Google+, assuming it’s something that I’d like to share with both my Facebook friends and my Google+ circles – it usually is.
2. The situation is even worse for re-tweeting/re-sharing other people’s posts. That is barely possible between platforms and usually involves cutting and pasting. Though this is somewhat more rare.
3. It’s probably possible to make my blog posts automatically generate a tweet, but not to automatically generate a G+ post.

All the hassle of posting things multiple places leads me to just not bother at all some of the time, which is annoying too. There are some tools that would take G+ content and put it on Twitter, but without a character counter on G+, I don’t think this would be useful.

Anyone else having similar issues? How are you coping?

We recently changed the arrangements for the boys. Instead of them each having their own room in which they sleep and sometimes play, we purchased a bunk bed. Oliver graduated from his crib to the lower bunk, and Jacob has the higher bunk. This has, predictably, created a few opportunities for behavior issues. Overall, it’s going well, and they appreciate their new, more open, “play room.”

Both boys sleep with their stuffed animals. Jacob calls his “my friends”. He still likes his butterfly, which he has had since he was an infant. He sometimes talks about how much he loves his friends, and how they like to get hugs, and how they are happy.

Jacob continues to enjoy reading. He has a toy low-res camera and he even recorded a video of himself reading Brown Bear, Brown Bear.

Oliver’s vocabulary is coming alive and is fun to watch. Jacob has taken to trying to teach Oliver how to say things. One day, Jacob saw a number like 451 on the side of a train, producing a conversation like this: “Oliver, can you say four hundred and fifty-one?” “Four dred iffy on!” “You got it!!!” “YAY!”

Today as I was walking past Oliver’s train track on the floor, he grabbed me by the hand, had me sit down, and kept holding on to make sure I’d stay right where he wanted me as he pointed and talked all about his trains. Aww.

One cold and windy Saturday morning, the boys were getting restless. What to do, we thought? I decided to bring out one of the manual typewriters from my collection. Oliver loved watching it do things as he pressed buttons. Jacob enjoyed spelling “mom”. New problem: boys fighting over how long each one’s turn at the typewriter is. This has gone on for a month now.

Ahh, winter.

A few weeks ago, Jacob informed us that he built an antenna out of blocks. He was REALLY proud of it, and even, incredibly, insisted I take his picture with it!

I’m pleased to have a 5-year-old that calls this structure an antenna instead of a skyscraper or tower or some such thing ;-)

We took a train trip to Portland, OR, recently. That’s about 2.5 days on the train each way. It went pretty well — we had quite a bit of excitement though it got a little long for the boys at times. One evening, Jacob excitedly noted that the sky was “almost really dark blue, just like my song!” Wow. That was a song he made up in New York in summer 2010.

Jacob enjoyed collecting leaves as we walked around in Portland. He would then stash his pile of leaves outside the door of whatever building we’d enter, then hope to find them still there when we got back out. It usually worked out OK for him.

The Parenting Dilemma

So, if you are eating dinner, and your 5-year-old falls asleep with his head on the table, and you know he had been excitedly waiting for raspberry sorbet for dessert, and know there will be tears when he later realizes he missed it, what do you do? Let him sleep, or wake him up (with possible tears right then in the dining car?)

As it turned out, I tried to quietly ask him if he wanted dessert. He woke up with a start, banged his head on the table, and then, yes, there were tears. I asked him if he wanted dessert or if he wanted to keep sleeping, and he gave a pathetic, sniffling, “dessert.”

But by the time dessert arrived, he had fallen asleep again. I finally woke him up again, asked if he wanted to eat, and he just ignored me. I asked twice more and then he all of a sudden realized what he was being asked, sat bolt upright, and dug in.

Graffiti

Jacob noticed graffiti in Los Angeles. He called it “silly words” and kept commenting about it as we passed it.

Bears

Oliver always sleeps with a stuffed bear, which is his favorite comfort animal. But Oliver is 2, and when he says “bear”, it sounds more like “beer”. Plus, the boys still seem to be operating on Central Time. So at 4:15AM yesterday, Oliver awakened us saying:

“Beer! Beer! Beer! Where’s beer? Where’s beer? Need beer! Beeeeeer!”

Butterflies

Jacob always sleeps with a stuffed butterfly, which he likes just like Oliver likes his bear. Jacob had this conversation with me this morning:

Jacob: My butterfly knows all the people in the world. And all the animals, too!

Me: Does butterfly even know Nash? (our cat)

Jacob: Yeah, and he knows Sam and Edna too! (the cats across the road)

Me: Wow. I think butterfly knows a lot more people and animals than I do.

Jacob: *brief pause, then* Oh, silly dad! *collapses laughing*

This is the biggest highlight of our trip to Greece for me. I enjoy having the chance to meet people, visit for awhile, and make new friends — and that certainly happened in Greece.

In one of the shops we happened to step into, I noticed a radio behind the counter. That’s not unusual itself; radios often catch my eye these days. But this radio was tuned to 21.070, an amateur radio frequency. So I asked the shop owner if he happened to be a ham. And indeed he was. He was Lakis (SV5KKU), and Terah and I had a great time visiting with him. Terah took some photos of us, and then we made our purchases and headed out.

I had brought my HT (handheld radio, weighs a few ounces and is powered by batteries) with me, and Lakis told me about the repeaters in the area. I had known about some of that since I had emailed Panos of the amatuer radio club in Rhodes before leaving home (I found his name via Google).

A couple of days later, on Tuesday, we found ourselves back in Lindos. It was mid-afternoon, so the shops were quiet. After a late lunch, I thought it would be nice to drop in on Lakis one more time, since we were scheduled to fly back the following morning. I’m not sure how long we stayed — it must have been at least an hour — and enjoyed the fresh orange juice he prepared.

After we got back to our hotel Tuesday, I learned that our flights on Wednesday were canceled due to a nationwide air traffic controller strike in Greece. After 3 hours on the phone with Delta (more on that experience later), we got rescheduled to fly back Friday.

On Wednesday morning, I remembered that the Rhodes amateur radio club meets every Wednesday evening, and now I would be able to go! I knew how to contact Lakis by email and on the radio, and he kindly offered to pick me up and take me there.

So that evening, I got a tour of his impressive mountaintop installation, and then it was on to the club – the Radio Amateur Association of the Dodecanese (SZ5RDS). There I met Panos, whom I had emailed earlier (and I think surprised him a bit). It was a friendly group, and they translated into English for me every so often so I knew what was being discussed.

When I was about to leave, they gave me this:

The translation, partly from my memory and partly with the help of Google Translate, is:

Radio Amateur Association of the Dodecanese, SZ5RDS

Our friend and radio colleague KR0L JOHN GOERZEN, who visited the island, has our recognition as an HONORARY CLUB MEMBER.

PRESIDENT KAVALAKIS PANAGIOTIS SV5AZK (Panos)

SECRETARY PAPADIMITRIOU CHRISTOS SV5DDT

RHODES – Oct. 5, 2011

(I hope that any Greeks reading this will send me corrections.)

I truly appreciated that gesture – and meeting all the people in the club.

On the way back to the hotel, Lakis and I stopped by a restaurant, which I believe had the best souvlaki I’ve ever tasted — thanks! We brought some back for Terah. She had chosen to stay at the hotel that evening and had a small hotel meal earlier, but enjoyed the souvlaki and pita. Terah had explained to the maître d’ that I wasn’t along that evening because I had gone to an amateur radio club meeting. Judging by the surprised reaction, this was probably the first time they had heard that particular comment!

Experiences like this make travel fun and worthwhile. Thank you very much, Lakis and everyone in the club — I hope to have a chance to visit again.

I am a person that enjoys food that’s different from what’s at home, and Rhodes didn’t disappoint. Terah and I used to live close to an excellent Greek restaurant in Indianapolis, so we were already familiar in some way with the food. But there isn’t any Greek restaurant at all in the Wichita area, so we missed it.

Our favorite restaurant on Rhodes was Kalypso of Lindos. Everything there was just excellent, starting with the saganaki, one of my favorite Greek appetizers. I had yogurt with honey there for dessert both times we visited, a surprisingly tasty desert. Like many restaurants in Lindos, Kalypso had the option of eating on the rooftop, or at ground level. We ate on the roof, which had a nice view of the Lindos Acropolis.

Being outdoors, there were sometimes cats around. This kitten enjoyed playing games with my shoestrings for awhile.

Kalypso is at a 17th century captain’s house. Here’s a view of it from the rooftop:

We, of course, had the chance to eat at quite a few different places during our visit, and I’d go on way too long if I mentioned them all. Terah particularly enjoyed the gelaterie.gr ice cream shop in the square in Rhodes. We liked our lunch at Maria’s Taverna in Lindos and enjoyed chatting with the staff there.

I recently talked about shopping in Mexico, and perhaps learned a thing or two from that. I won’t say we never buy them, but in general we don’t buy souvenirs like t-shirts, plastic things made in China, etc. We prefer to buy local. Those items tend to be higher quality, more interesting, and we like to support the local economy. We also don’t have lots of room for things, so we try to choose carefully.

So it was something of a surprise to Terah, and perhaps even to me, when I suggested we go shopping one day. Terah typically enjoys shopping a lot more than I do. Anyhow, off we went to Lindos.

One of the first things that had caught our eye in Lindos was the shop selling glass. But it wasn’t just any glass; it appeared to be made with some sort of layered process, and has a distinctly three-dimensional feel to it. As you move around, it looks like the background shifts. We wound up with this item, which was made in Athens:

By the time we visited Lindos specifically for shopping, we had a good feel for when the busy times of day were, so we could avoid them. It gave us the opportunity to visit with people — and when they weren’t busy, many shopkeepers liked to chat. I enjoy hearing people’s stories and we heard several.

One ceramics shop – the Musa Shop -caught Terah’s eye. They had such incredible and beautiful pieces outside that we just had to go in. We wound up with two pieces from there, both in shades of blue:

Both remind me of the Aegean Sea and the deep blue sky of Rhodes.

And then, as we were walking along, I pointed inside a shop and said to Terah, “Hey, those look different.” We went in, and eventually wound up buying these:

The appearance, and even feel, of them is unlike anything I’d seen. Quite interesting.

And seeing those particular items in the Lakis Place shop led to making some new friends — I’ll write about it in the next post.

That’s a photo of Vlicha Beach, near our hotel. But before I talk about the Greek beaches, I need to explain something about living in Kansas.

Kansas is in the middle of the United States. The nearest ocean is the Gulf of Mexico, which is 700 miles (1100km) away. That’s roughly the same as the distance between Berlin and Minsk, or New York and Chicago. And I believe it’s farther away from a saltwater body than all (or almost all) of Europe. So we’re not just going to the beach every weekend or something.

Vlicha Beach was all those incredible things you ever hear about beaches. The water was so clear that I could easily see my feat while wading in it. That, and the fish swimming around them. It was peaceful, relaxing, and picturesque. Between the deep blue of the sea and the same in the sky, I suppose it is no wonder that blue is often associated with Greece.

We weren’t exactly the only ones there.

Though I think we were the only ones there with Kindles, which seemed to be much more popular in the USA than in Europe. We got several interested people carrying paper books asking what they were.

Towards dusk, the mist would become more pronounced and the mountains off in the distance started to fade:

The evening before we were set to fly home, we spent some time sitting on our balcony watching dusk set in. One last gaze out over the beautiful Aegean, the misty mountains, and the boats in the distance.

I am one to enjoy history. There is something deeply, well, connecting, about standing in an old place. There is a timeless quality to it – a feeling of being connected to so many people of the past, and yet still being connected to change, visible in things such as weathering of stones. To gaze at pottery that’s 300 years old, walk past 700-year-old walls, or pass through what remains of the grand portico of an ancient temple to Athena stirs a feeling I can barely explain, of timelessness.

Although Rhodes doesn’t have the “famous” Greek sites such as the Parthenon or Delphi, I can’t help but wonder why the Rhodes sites aren’t better known. They were incredible and it is hard to condense all that we saw into a short blog post.

I have to start with the medieval Rhodes Old Town. We got off the bus a few blocks from it one bright morning, and our first task was to find a gate across the moat. Oh yes, A GATE ACROSS THE MOAT.

It’s a dry moat, and that bridge off in the distance is the gate we were headed to. Outside of the outer wall is a nice quiet walking area. The moat and walls completely surround Old Town and, for the most part, date back about 500 years. The round stones you see on that picture, we were told, were likely surplus from catapults and other projectile weapons. Cross one line of walls and you come to another, with original canons still present. The Knights Hospitaller of St. John, which held Rhodes for a few centuries until the Ottomans captured it, sure knew how to build to impress.

The gate we happened to use was Amboise, the Grand Master’s Gate. Right there is the stunningly rebuilt landmark Palace of the Grand Master. It is absolutely impossible for any photograph to begin to do this building justice. Between its imported Greek and Roman floors, to the grand nature of everything in it, and the archaeological museum in one corner, it was a fitting start to a visit to Old Town. Here’s one of the main staircases.

Just near the Palace is quiet courtyard with an old door. Pass through that door and suddenly you’re in the midst of the busy Old Town.

And among the landmarks in Old Town, the most prominent is Ippoton, the Avenue of the Knights. Along this avenue are the buildings built by the various nationalities of knights, many of which are historical sites in their own. Taken together, it is quite clear why Rhodes is said to be one of the world’s best-preserved medieval cities.

Down at the other end of Ippoton is the Knights’ Hospital, which is now part of the archaeological museum.

Step off the Avenue a few blocks and you get to some quieter narrow streets – just as old, in many cases.

On Sunday morning, we were able to visit Mount Filerimos. In contrast to the busy Rhodes, Filerimos had an air of quiet and still to it. It was the site of a monastery, two historic churches, and a landmark Italian cross on the mountaintop.

We arrived, and begin our visit with a walk up the quiet stone path.

When we got to the top, we walked past this peaceful church. As we walked past the outside, we heard the beautiful music of chant from indoors. We got to step in and listen to mass for a few minutes.

In typical fashion, directly in front of the church are two much older sites: one, the ruins of a temple to Athena, and the other a 4th-century Christian bapistery.

Rhodes is a popular tourist destination, and of course we saw plenty of popular sites (such as the grandmaster’s palace). Filerimos had a few tourists too, but not as many.

I frequently like to operate on the plan of going wherever all the tourists aren’t. And so, on Filerimos, that meant seeing what was behind the monastery. It started with this peaceful tree-lined path.

And the deserted, but intentionally open, gate led to the remains of a Byzantine fortress, which had been a staging area for both the Knights and the Ottomans before their campaigns to capture Rhodes. It also provided incredible views of the surrounding countryside.

The first historic site we had visited on our trip was the Acropolis of Lindos, parts of which are 2300 years old. Here’s a view of the mountain from the rooftop of the Kalypso, our favorite restaurant in Lindos.

The columns of the temple to Athena Lindia are visible, and of course so are the walls.

The road up to the acropolis is accessible only on foot or by donkey. It is apparently the only road that has ever been used to get to the acropolis. Here is the partially-restored grand portico to the temple.

There’s an old Christian church (4th century, if memory serves) at the Acropolis too.

The Acropolis makes some pretty good use of natural defenses too. Here’s a view from one level of it. There’s a manmade wall up there at the very top. And, of course, the beautiful Aegean always in the background.

There are lots of cats on Rhodes. Here is a kitten napping at the top of the Lindos Acropolis:

Lindos itself is a beautiful town. Here’s one of the quieter streets:

Notice the pebble steps leading into the houses – those intricate pieces of artwork are all over.

This post won’t be complete without the story of our visit to the Acropolis of Rhodes. We walked there from Old Town. At the Acropolis, there are the remains of a temple to Apollo, an ancient theater, and an ancient stadium where qualifying matches for the Olympics were held.

As we got closer to the area, we were repeatedly passed by people dressed in uniforms of various types. And as we got there, we joined a stream of people entering the area. The ancient stadium had apparently thousands of people in it, country names were being read off over the loudspeakers, policemen wielding machine guns were standing by, and we had absolutely no idea what was going on.

At this point, you can appreciate the difference between Terah and me. Terah thought that we have no idea what is happening, she was tired from the walk, and so thought we should just leave. I thought that we have no idea what is happening, which is a great reason to stay. So Terah opted to sit and read a bit under some trees while I explored.

Here’s a view of the stadium as it was emptying out, seen from the theater:

I explored the temple and theater, and eventually we were ready to head back. We knew there was a bus back to the New Market (from where we could get a bus back to our hotel), but didn’t know where the bus stop was. The obvious place to ask were the policemen, which I thought I would do. Terah thought she would just stay sitting under the trees, on the grounds that the policemen nearest us were all carrying machine guns and perhaps wouldn’t like to be disturbed. This led to my cryptic tweet:

“Only ONE of us is the kind of person that goes up to guys with machine guns to ask what’s happening.” – Me to Terah today

They told me that it was the preparations for the opening ceremony for a global shooting contest, and also gave me directions to the bus stop.

In 1965, I was stationed at the Fleet Anti-Submarine Warfare Center in San Diego. I was a Radioman 2nd. Class in the USN, at the time.

One evening, at our radio club station (W6DCM – different license holder, now), I called CQ and got UAØKKC (it’s no longer around), with Ivan at the mic. After a time of the usual signal reports and such, we asked what each did in their lives – I said that I was US Navy radio operator.

He answered that HE ALSO was a Navy radio operator in the Soviet navy. Then we discovered that his station was at the SUBMARINE base at Vladivostok, and I of course was his DIRECT opponent.

Yes… we really DID laugh at that, and I shall never forget what he said (that I heartily agreed with):

“Thomas, isn’t it shame that we’re supposed to hate each other?”

“Yes, Ivan, it is – someday we’ll share a vodka, da?”

“Da”, he replied.

We’re often told we should hate people. Messages I have heard on the media over the last 10 years have said we ought to hate illegal immigrants, CEOs, radical Muslims, the French, Iranians, Mexicans, presidents, UN diplomats, climate scientists, oil company employees, Chinese people, conservatives, liberals, religious people, atheists, and oh yes, still Russians.

But I get to choose who to hate, and in fact, I choose NO HATE. Not only does it keep my stress level way lower, but it also lets me enjoy life more, and makes the world a better place.

We can all talk to people in other countries and with other backgrounds and viewpoints so easily thanks to the Internet. Sadly we rarely have very deep online conversations to the point of getting to know people. For whatever reason, ham radio lends itself to that better.

Even better: visit other places. I wonder how many people that say they hate some group of people have visited them and made an effort to make a connection? It is, after all, really hard to hate someone that is kind to you. Perhaps they’re afraid to let go of their hate.

Think also about this: for whom is it convenient if you hate people? There is usually a reason that hatred is stoked, and it doesn’t usually lead to good things for individual people.

Tom W0EAJ added:

I actually tried to locate him and the station, but both appear to have vanished. Ivan (his name was pronounced Eee-von) could have, it occured to me later, gotten into trouble for saying such things. I think both of us realized AT THE TIME, what an astounding counterpoint each of us was to the other.

Proof that if it were left up to the simple little guys like us, and not to the politicians, we might actually pull off living in peace.

Some highlights include walking up to a 14th-century church and hearing beautiful chant through the windows, crossing a bridge across the moat to a grand gate into Rhodes built by the Knights of St. John, spotting an amateur radio in a shop and making lots of new friends, standing atop the Acropolis of Lindos and feeling the connection to the ancient, and of course swimming in the clear sea – so clear that I could see fish swimming by my feet as I stood in the water.

I’ll write about all of this yet, of course, but for now I want to thank Jon Dowland for leaving a comment on my blog post suggesting Rhodes and even the particular hotel we stayed in. Jon was right, it was great.

And for now, here’s a photo I took from our hotel room balcony, almost immediately after we arrived. A sign of great things to come, perhaps:

“Dad, will you and mom stay up all night decorating the house for me?”

That was Jacob’s question to me at bedtime the evening before his fifth birthday. Jacob had already had his birthday party a few weeks ago. When scheduling means that parties happen that far away from the boys’ real birthdays, they get a smaller celebration with just Terah and me where we open their presents from us. A low-key thing, so we weren’t planning to decorate the house. I said, “Well, I don’t think we’ll be doing all that since you already had your birthday party.” And the look of eager anticipation on his face turned to a very sad and disappointed look, and made me feel really bad. Uhoh.

So while Jacob was sleeping, I mentioned it to Terah. We decided we should improvise something simple, so she found some old streamers and we taped them up, running them through several rooms in the house and across his door. It took a few minutes using supplies we already had, but the joy the next morning was priceless.

“Oh dad, you said you wouldn’t stay up all night, BUT YOU DID! Oh I am SO HAPPY! YAY YAY YAY!” And he ran through the house to discover what else was set up. Then he ran to find Oliver and gave him a tour of everything.

Then we sat down to open his presents. Here he is, holding a present from Terah and me:

That’s a copy of The Lorax. My friend Jonathan had brought a copy along during our road trip in Mexico, and Jacob was really excited about it. And didn’t really want to give it up, because “You can only get The Lorax in Mexico.” He (and Oliver) really enjoyed all his presents — he also got a train book from us (which he said, “Oh, yay, it’s the book they have at preschool!”), and a game and some other presents from the distant relatives.

But the highlight was something of an impulse buy. I was at the RadioShack in Derby — a place I’ve written about before, It is what a RadioShack should (and used to) be. It has a large amateur radio section, sells all sorts of coax by the foot, and provides astonishingly good post-sale service. Well, I was there with a radio question, and Mark (the owner) — who is an excellent salesman in a positive way — pointed me to a display of snap kits. I noticed their Electronics 101 Snap-Kit (a rebranded Snap Circuits Jr. SC-100). A ham in Pennsylvania had suggested them to me once, and as Jacob’s birthday was coming up, I gave it some thought. The kit said ages 8 and up. I asked Mark what he thought about a boy just turning 5. He said, “Well, probably not normally. But knowing you, if you’re there to work on them with him, I think he’d enjoy it. But I wouldn’t have him work on it by himself.” I agreed and we bought it.

I pondered how to explain the concept of this thing to Jacob. Eventually I decided I would call it a “toy-building kit.” He understood that.

Jacob and I spent hours together working with it. He would flip through the book, either picking circuits that looked interesting or telling me what kind of circuit he wanted to build. Then I would tell him what to put where, and he’d snap them together and play with them. He only played with each finished product a few minutes before he was ready to try another. Once I got him very excited with my offer to show him how to hook up two switches in parallel for the fan he built (and later introduced the parallel vs. series concept by hooking them up in series instead.) Here we are working on it together.

Jacob repeatedly called Terah over to look at the things he built. He was very excited that he assembled it himself. Eventually, Oliver (age 2) came over wanting to help. So he sat on my lap, and handed parts to Jacob, then Jacob put them on the grid. Oliver really enjoyed being involved in this way, even though I had to keep him from doing things like ripping the capacitor off its mount.

We tend to be modest in terms of the number of things we give the boys and their cost, reasoning that we, like many, already have too many toys in our house, and that greater cost doesn’t necessarily equate with a better experience for the boys. I particularly look for things with lasting value and unique experiences for them, and I think we succeeded this year.

But I realized quickly that the greatest value of this kit wasn’t electronics. It was having a great way for me to spend a lot of time doing things with the boys which all of us enjoyed. Those hours building things together were as much a present for me as for Jacob, I’m quite sure.

Jacob’s “real” party was a few weeks ago at the Great Plains Transportation Museum in Wichita. They let people rent a historic caboose to use for a birthday party for children. So we did that for Jacob this year.

That was a huge hit for the boys. Jacob got to help his grandpa make some pie (instead of cake) for the party. He enjoyed eating it, of course.

He enjoyed opening his presents high up on the observation chair in the caboose. And the boys got to play on all the other equipment in the museum. Jacob enjoyed playing tour guide for family since most of them hadn’t been there. He also enjoyed watching freight trains pass on the other side of the fence from the museum — preferably while sitting in one of the museum’s engines.

Oliver certainly didn’t get left out. Train-watching is serious business, after all.

Jacob has long talked about going to the “train museum” and the “airplane museum” (Kansas Aviation Museum) on the same day, so one of his birthday surprises was that we went to the airplane museum after his party. His favorite item there is a retired FedEx 727. Here he is walking down the rear of the plane.

And, of course, they played captain and co-pilot in several different planes.

It’s been about two weeks now since we got back home. Every few days, our boys still talk about Mexico. Jacob talks about what he will want to do “when we go back” and how he’d like to see Jonathan over there again. Jacob, Oliver, and I look at photos from our trip a few times a week. This is all a sure sign that our boys loved the trip. And I keep trying to find Mexican food that tastes as good as the real thing (and, so far, failing). Another sign of a good trip!

I think it is possible to have fun, relax, and enjoy new experiences all at the same time. We did that in Mexico. We had a lazy afternoon or two with the boys taking naps or playing with other children at the house we stayed at. And we stayed in a beautiful hotel without air conditioning, explored old downtown areas and ancient ruins, and bought things from crowded markets and people selling things from a table along a road.

To anyone thinking about visiting: Go. Enjoy it, bring back memories, and live a little more serendipitously than usual. That’s what I hope to do when we visit Greece before too long.

Finally, here’s a photo of the painting I bought from a roadside vendor for $17. Make sure to view it full screen. I think it says more about Mexico than 6 blog posts do.

One piece of advice we got in Mexico went like this: the nicer a place looks, the worse the food and prices will be. Roadside taco stands will be great, and nice-looking restaurants not so much. That seemed to be accurate. We only tried one real nice-looking restaurant and it was very good (though pricy), but it may have been sort of an exception.

But perhaps the most interesting bit about eating in Mexico wasn’t the food. It’s the adventure.

We ate one day at Guadalajara’s San Juan de Dios market. In that huge labyrinth somewhere was a set of restaurants. They’d tend to have a small cooking area, usually just open, and a few tables. We chose one.

And at this point, I have to take brief detour and explain something. There are a lot of people in Mexico that do things for tips, and quite often without being asked. Some other examples might be washing a car’s windshield at a stoplight.

So anyhow, we had ordered our food, and before long, a guy wanders down the aisle and plonks down a boombox. And turns it on. And then he pulled out a microphone, which we quite soon realized was connected to the boombox. (I guess making it more of a karaoke box.) Anyhow, he started singing a song — decently — and seemed to be enjoying it. About 45 seconds into it, a competing boombox man plonked down a competing boombox 25 feet away, turned it on, and — yes, you guessed it — pulled out a microphone and started singing a different song. Worse than the first person but louder.

Eventually the boombox people left and our lunchtime conversation could resume. But pretty soon a drum guy showed up. He had a bunch of drums on a strap so he could just walk around and play them. He apparently decided that an excellent place to play them would be directly behind my head. I did not entirely agree with his decision, but hey, it beat the competing karaoke guys.

Eventually the drum guy left, and somehow between the time I looked down to get out money to pay our bill and the time I had it counted out, a clown had shown up and made several balloon animals for our boys. I tipped him, we paid, and then headed on.

You might think from this story that this would be an annoying series of events. And honestly, if it had happened in a big mall in the USA, it would probably have been both annoying and creepy. But really I enjoyed it. The fact that dueling karaoke happened, despite sounding really awful, was pretty funny and really seeing this whole parade of people was interesting too. It made American restaurants seem a little boring. You always know what’s going to happen here (and if something surprising does happen, the place probably gets a bad review on Yelp.) Interesting things sometimes happen at mealtimes in Mexico and I like it that way.

I had a torta ahogada (drowned sandwich) at that restaurant. And at this point, another brief aside.

I’m the kind of person that can go to an average American restaurant, see items on the menu helpfully indicated as spicy, order one, and genuinely wonder if other people would find them spicy, because I either don’t notice spiciness at all, or maybe notice a tiny bit if I concentrate really hard. Others, meanwhile, might take a bite and lunge for the water. Having said that, I know people that lived in Thailand for awhile and I have nowhere near their tolerance for spiciness.

So, having been in Mexico a whole 24 hours or so, I decided not to follow Jonathan’s wise lead in ordering a torta with the spicy sauce on the side. I figured I hadn’t had anything spicy yet, so maybe this was would be nice and mild for me. Via Jonathan’s translation, I ordered it with the spicy sauce. I believe the phrase I heard him use was “con chile“. The waitress looked at me, gave me an amused “the American is ordering it con chile? Hahaha….” sort of smile, and went off.

Pretty soon our food arrived. (The food always seems to arrive pretty soon in Mexico, by the way.) Oliver was having a bit of a culture shock that day, and mostly refusing to eat (once hunger got the best of him later, he really enjoyed Mexican food.) But the rest of us dug in, including me.

I enjoyed my torta. It was spicy, but not too bad. I took some big bites (it was, after all, a thick sandwich) and was really enjoying it. For about a minute. Slowest-acting spiciness I’ve had in awhile.

Then it hit me. Spiciness, and lots of it. I took a big gulp of my horchata (a creamy sweet rice drink that I found at many restaurants). That helped. A little. I really liked the torta and ate it, but it wouldn’t surprise me if the waitress noticed how extremely quickly a drank my horchata…

Another interesting experience was in Guanajuato. It was raining as we walked towards the Guanajuato market. Their market was large and similar in concept to the Guadalajara one, though a lot smaller. The restaurants were all in a row, in a side of the building that was open to the outside. Most were on the ground level but it looked like a restaurant or two were upstairs.

As we approached, all of a sudden people were yelling at us. First it was a guy on the second story, then pretty soon people at the restaurants on the first floor did so as well. They were yelling rapidly in Spanish, waving their menus around in the air. I’m imagining they were naming foods they sold or reasons to eat there, but I don’t know enough Spanish to know. As we walked down the long row of restaurants, the ones we left behind would quiet down in disgust and other hopeful restaurant owners would take up the yelling and waving cause. I imagine if we did some time-lapse videography and walked up and down that row, we could produce an effect not unlike the sound of a dot-matrix printer going back and forth on the page.

Anyhow, we selected one of the quieter restaurants pretty much random. The others then quieted down until another person chanced to walk past — at which point it would get loud again. The lunch there was good but I think I mainly will remember it for the selection process!

On our way into Guanajuato, we stopped at a wonderful roadside taco place. In typical fashion, they had a large vertical pork thing (I don’t know the proper word for it) from which they would carve off meat on the spot anytime someone ordered something with pastor. We found a table. And we ordered a few tacos and such. They were usually a few pesos each (working out to less than a dollar), small round things on a soft tortilla, with meat, cilantro, and onion on top. And typically delicious. They had very little in common with an American “taco”.

We’d often order a few, and if we wanted more, just order more. They were made quickly enough for that. Tacos were very similar from one restaurant to the next. My favorite flavors were pastor (pork), chorizo (sausage), and bistec (beef steak).

A restaurant in Guadalajara — sadly I’ve forgotten its name, since we kept calling it “the potato place” — had what I might call a Mexican version of the loaded baked potato, with a meat, queso (cheese), a delicious sauce with a flavor unlike anything I’d had before, and some garnish. But really my favorite thing from that restaurant was their amazing juices. I am not much of a juice drinker normally, but in Mexico I went for them whenever they were offered. What passes as fruit juice in the USA has about as much resemblance to a real Mexican fruit juice as Taco Bell has to a real Mexican taco stand. (Very little, in case that wasn’t crystal clear.)

That particular restaurant offered three types of juices, which were, if I’m remembering right, aguas, frescas, and jugos. I has a jugo verde (green juice) on the first visit there. It was good, but the one I can still remember was called, I think, the fresa fresca (fresh strawberry juice). And it was incredible. I’m not sure how to describe it, other than real.

One observation before I end. It seemed a common thread at some Mexican roadside taco stands to not have soap in their restrooms. Instead there would be a plastic cup holding — I kid you not — powder-form Tide laundry detergent. It was amusing anyhow. My hands left those places extremely soft and smelling like laundry.

One of the last restaurants we visited on our trip was in Ajijic, near the Chapala lake. It was actually right on the lake and served seafood. This was the only restaurant with prices as high as I’d be used to in the United States. I ordered a stew served in a stone bowl. It came out sizzling, and since the very thick stone bowl retains heat well, it kept sizzling the entire time I was eating. It was excellent as usual.

Coming up in part 6: some thoughts on returning to the United States, our decision to visit, communication, and tips for anyone else considering a first visit to Mexico.

This post is going to be more a photolog than a narrative, and I apologize in advance for it being a bit disjointed.

I’ve already touched on these themes a bit in the other post, but now it’s time to focus on them. Immediately after leaving the airport, it’s quite clear that things are a little different. Trees are square. People ride around in the backs of pickups — sometimes on top of piles of debris. Left turns are made in front of other lanes of traffic going the same way. But those are just the things obvious from the road. It’s a lot of fun to enjoy the differences. First, the ubiquitous square trees.

They look pretty, and are found all over. I also found carefully-manicured trees in cone shapes, more cylindrical shapes, etc. It seems that tree care is taken seriously in Mexico. It was also not uncommon to see the bottom few feet of a tree painted white. A park in Guanajuato had a whole bunch of trees carefully trimmed.

And from up on the mountain, it still looked impressive (the green area behind the dome).

Driving in Mexico was interesting for a lot of reasons. The highways there aren’t quite as limited access as the freeways in the USA. It was quite common to see bicyclists, walkers, a mule, or some cattle ambling along the side of the road. Roadside taco stands don’t require taking an exit. You just pull off the road because it’s right there.

Some sights were a bit surprising. Cattle in a pickup, with rope, for instance.

Or cattle crossing the highway on the overpass.

Street vendors were everywhere. Stop at a red light and someone might spring from the side of the road and suddenly start washing your windshield (expecting a tip); try to sell you flowers, juice, or bug zappers; or even throw business card-sized advertisements for adult websites into any open windows they can find. One night we saw an incredible fire juggler. I would have tipped him well but he was too far away to do so before the light turned green.

Mexico’s history stretches back into prehistoric times, and we saw the Teuchitlan ruins at Guachimontones one day. It was truly a remarkable feeling to be able to walk down the middle of the ancient ball court, or to climb up one pyramid and see the other from it.

It’s not exactly architecture, but Jacob and Oliver sure enjoyed visiting the hot springs at Bosque de la Primavera. Jacob still remembers that “where the steam is, the water is 200 degrees, and we CAN’T TOUCH IT THERE!”

Back in Guadalajara, here’s a photo from the inside of the grand old cathedral.

Compared to the cathedrals we saw in Europe, this was of a similar general size and design, and perhaps only slightly newer. But one big difference: worshipers outnumbered tourists at every Mexican cathedral I saw, whether in the center of Guadalajara or at a rainy intersection in Guanajuato or a plaza in Tlaquepaque. It made them feel more alive, and perhaps more sacred as well.

One surprise was seeing people sitting on the steps of the cathedral in downtown Guadalajara selling trinkets such as beads. I think the only other place I had seen something like that was in New Orleans.

All of Guadalajara’s Centro was beautiful. Much of it survives from colonial days; I think a person could spend days exploring its museums and buildings. Way too many of my 900 photos were taken in Centro to post on the blog, but just for flavor, here’s one of the less than historic scenes.

Yes, that is a bus shaped like a tequila bottle.

Fountains were beautiful and common across Mexico. A few of them were easily reachable by boys, and ours sure loved those.

There was a lot of public art, including this interesting chair/skeleton/I’m not sure what it is:

And, just for good measure while walking around Centro, they tossed in an apparent Redundant Array of Inexpensive Typewriters.

I don’t know what they were doing, but there were about a dozen guys sitting out in the sun typing on their manual typewriters on their identical tables.

And who can leave Guadalajara without seeing one of North America’s most impressive traffic circles. I’ve got to hand it to the Mexicans for making something that is normally really boring into an interesting work of art.

Over in Guanajuato, a lot of driving takes place in the city’s vast underground tunnel system. Here’s a scene emerging from one of them.

Guanajuato was already getting decorated for Mexican independence day festivities (Sept. 15-16) while we were there.

Here’s a typical Guanajuato street scene.

Many of the streets were closed to traffic — and perhaps not wide enough to handle vehicles anyway. Those streets had a wonderful peaceful and slow feel to them.

I feel that I’ve barely done the trip justice with this post. The feelings of walking down a beautiful Guanajuato street, or stepping into a Spanish cathedral, or even seeing a bunch of guys with typewriters, just can’t be replicated. It’s brimming with history and character, and shouldn’t be missed.

Shopping in Mexico was probably the thing I was least prepared for. I probably had the biggest wins of the trip shopping, and also the biggest fails (though they weren’t all that significant).

It seems to me that shopping is all about serendipity. You almost have to be good at impulse buying. I normally try hard to resist impulse purchases, preferring to research and compare carefully before making a decision. That attitude didn’t serve me well in Mexico, and when I was able to overcome it, I got some great purchases.

Around here, some of the best places to shop are the ones that have been in business for decades. Anderson Office Supply in Newton, KS, for instance, has been around for over 100 years, and has — in stock, even — everything from a ribbon for my slightly obscure 1940s typewriter to local history books. Moler’s Camera in Wichita often meets or beats the online stores prices, and has better service.

But in Mexico the best places seemed to be packed into a large crowded shopping area, or a dusty stand along a road, or a guy selling stuff on a plaza somewhere.

Here is Guadalajara’s market (San Juan de Dios):

It’s an incredible open-air market. We spent quite some time there, even ate lunch, and yet I’m sure we saw only a small fraction of what they had. It was a cramped place, with small booths and tiny aisles, but all sorts of interesting things. Although I did appreciate walking quickly past the raw pork and fish corner. Terah bought some genuine extract of vanilla there, at a good price.

Some of the street vendors were selling what I think were butterfly toys — they had some sort of launcher that would launch them in the air, and they’d flutter and float down to the ground. We saw them mainly in Guadalajara Centro on our first day, and I (unwisely, it turns out) thought, “Hmm, a plastic toy — our boys would love it, but I don’t want to buy it on our first day. Besides, I’m sure we’ll see them all over or online.” Wrong on both counts. So there’s one of my fails.

My greatest win came on the road back from Guachimontones (an ancient pyramid site). Every so often along that country road, there would be a vendor with a table selling something or other. I saw some paintings out my window, thought “neat”, but — and here you can see how terrible an impulse buyer I am — didn’t actually put together that we should turn around and look at them until 10 minutes later after a bathroom stop for Jacob. We went back, and I picked out a beautiful painting on canvas of those pyramids at sunset. And the charge: 200 pesos (about $17 USD). Incredible and incredibly cheap, and there is a great place on my wall for it. The vendor was also the artist. I am kicking myself for only buying one. (No photo yet as it was too big to practically transport by plane.)

Another memorable purchase was this one:

This was another roadside find. We were driving through Ajijic, and I noted a rug vendor along the side of the road. I had walked past a rug vendor in the Guadalajara market, so I was keeping my eye out. I made a note to stop there on our way back out of Ajijic. So we did. This is a rug made in the Zapotec tradition, all hand-made, with natural dyes and wool. It was 550 pesos (about $46 USD), which I considered to also be a pretty good deal for what it was. I have no idea how many hours went into creating it, but I’m sure it was many.

My luck in shops wasn’t so good. We visited Tlaquepaque, which had lots of shops selling beautiful things. But the prices there were higher than I’d pay for similar things back home. Tonala’s shops were too inconveniently located to be practical with what were then tired boys, so we didn’t go in there.

I’m not used to shopping without planning, and perhaps am not very good at it. On the other hand, I really enjoyed making those two purchases, and only regret not buying another 200-peso painting! Maybe next time I’m in Mexico, I’ll even buy something on the first day there. Terah will be so proud..

Today I’ll write about the places we stayed. We spent most of the time in a room we rented in a private home in Guadalajara. My friend Jonathan had found it for us, and it was not too far from his home.

The owner was a grandmother, and across the courtyard was more family, including a granddaughter close in age to our boys. They enjoyed playing together.

It was really a perfect arrangement for us. There’s no better way to be a part of local life when traveling than to stay in someone’s house.

Here’s our bedroom:

Notice the glass slats in the window — it’s a nifty, though not airtight, alternative to our regular windows. More on that later. More of the inside:

We had a bit of a language barrier while in Mexico, though never anything significant. My Spanish vocabulary started with almost nothing and I reached maybe a few dozen words by the time we left. Terah knew some Spanish from high school and college, and my friend was fluent. Our hostess also knew a little English. But we all communicated well enough. Terah or Jonathan would help translate when needed.

As I’d seen before, the children would say things to each other, but never seemed to be bothered that their playmates didn’t understand what was being said. They just had a great time anyhow.

On Sunday afternoon, when we came back from our activities, there was a buzz of activity. Children everywhere outside, running and playing. Adults too, chatting. We didn’t know exactly what was happening, but sent Jacob and Oliver out to play anyhow (which they were eager to do). The yard was enclosed by a wall, so children could pretty much run around without lots of supervision.

Eventually they were invited to have some birthday cake (ah ha!) — it was one of the children’s birthday. Jacob and Oliver actually were served the first two pieces of cake (as the “amigos”). Everyone seemed so friendly, warm, and welcoming.

Each morning started with breakfast at the house, followed by a scene like this:

That’s Jacob and Oliver, looking to see if Jonathan had arrived for the day yet.

I often noticed in Mexico that I was unsure if I was inside or outside. Here in Kansas, we can have a string of summer days that each exceed 110F (44C) — or a few weeks in winter that never get above 15F (-9C), even in daytime. And then we have some pleasant days like right now, too — or rain blowing sideways at 60MPH. In general, we spend a lot of effort keeping the outside, well, out.

It is quite clear that this isn’t a problem in the Guadalajara area. Some restaurants could have been described as buildings with large, open windows so you feel a lot of breeze while inside. Or perhaps as a simple shade roof with a few supports on the edges, so you’re never really “inside” at all (sort of like going under a small shade tent outdoors). To the extent that windows could close, many of them couldn’t be made airtight. It was clear that in that area, people spend more energy finding ways to invite the outdoors in rather than to keep it out, thanks to the year-round moderate climate.

Perhaps the best example of this surprised me one evening. We were arriving in Guanajuato, an old silver mining town in the mountains, and were going to spend the night at Hotel Socavón, which had been recommended to us by a local friend of Jonathan’s. From the street, the hotel looked tiny. But walk in, and you get in this old-looking (and feeling) entry tunnel:

That’s the front desk, apparently cut out of it there on the right.

We asked to see some of the rooms before buying — apparently a normal request around there. The innkeeper agreed, and gave us keys and directions to find them on the third floor. It included going up 2 flights of stairs, passing through a courtyard, and going up another flight.

It was after dark, and the hotel was dimly lit — something I was fine with. I thought we were stepping out into a beautiful atrium with some potted plants in the center of the building — something fairly common in some nicer hotels. Until I felt rain on my head. Then I realized that the courtyard, which began two floors up from the street, was open to the sky. Beautiful!

Here was the view from out room door:

And down the “hall”:

After getting home, a Google happened to turn up some reviews of this hotel. I was so annoyed at what some people wrote! One person gave them only 3 stars because they didn’t have air conditioning, had poor water pressure, and “lots of steps”. Someone else complained of the dark entry tunnel — something I couldn’t help but smiling about the moment I entered.

My review, which should hopefully get posted soon, is certainly different. I gave them 5 stars, because if I wanted a Super 8 with generic fluorescent lighting and the same layout as thousands of other hotels, I would have gone to Nebraska instead of Mexico. Most homes and local hotels in the region don’t have air conditioning because they don’t need it, and that’s just how water pressure is in Mexico (due to needing to pump it from municipal supplies to private storage tanks overhead). And who doesn’t appreciate entering a hotel through a brick tunnel? Ah, sigh…

This should give you some idea of the kind of travel we like: part of the point of traveling is enjoying the differences from home, and I think it is a huge mistake to be annoyed at everything that is different. Enjoy the differences!

Finally, here’s a photo of the staircase in the home we stayed in, which I thought was fascinating:

I’ll be writing about all the fun stuff in the posts to come, so you’ll have to bear with me on this one as I describe why we did something that appears to horrify a segment of Americans.

About a year ago, I wrote a review of Travel as a Political Act by Rick Steves. Rick’s point wasn’t actually directly political in the conventional sense, but summed up as:

I’ve taught people how to travel. I focus mostly on the logistics: finding the right hotel, avoiding long lines… But that’s not why we travel. We travel to have enlightening experiences, to meet inspirational people, to be stimulated, to learn, and to grow. Travel has taught me the fun in having my cultural furniture rearranged and my ethnocentric self-assuredness walloped.

He speaks of giving onesself permission to have a conversation with someone that doesn’t speak a language you know, for instance.

I got an email this spring from my longtime friend Jonathan Hall, who had moved to Mexico a couple of years ago. He invited my family and me to go to Mexico, generously offered to host us and show us around, and specifically mentioned my review of Rick Steves as a reason to do so, this was immediately intriguing. Due to his other plans, it was also something of a limited-time offer.

Those of you that don’t live in the USA may not know what sort of stories we get about Mexico over here. Most of them involve either illegal immigration to the USA or the Mexican drug war. Occasionally there is some sort of drug-related violence on the US side of the border as well, which always makes the news. There are a lot of people that have the perception that Mexico is a dangerous place to be. Terah even knows some people that grew up in Mexico and are too afraid to return.

So we did some research, asked some questions, and it became pretty clear that yes, some parts of Mexico are dangerous, but many parts are really quite safe, and Jonathan had invited us to one of those.

The reactions we got from Americans when we told them of our plan ranged from excitement that we would get to visit a beautiful country to concern and worry about our safety. Besides that, I knew almost no Spanish and Terah had a few high school and college classes years ago to go on. And, we knew that Mexico would, in many ways, be more different from the USA than Germany was.

We concluded that this would get us out of our comfort zone in a significant way, have lots of great things to do, be a good experience for the boys, and something that we wouldn’t do without Jonathan. So we bought the tickets for it!

As you might notice, I’m quite glad we did. I’ll follow up with the details in the next stories, but for today I’ll end with the story of getting there. It involves two cute boys, so of course it’ll be interesting. Jacob is almost 5 and Oliver is 2, and neither one of them had flown for over a year. Despite leaving home at 5AM to drive to the airport (about an hour away), they were both awake and alert. Jacob was jabbering away the whole way there. He enjoyed the security process and found it interesting — I had to explain that they were checking to make sure everyone was following the airplane rules.

Then as we walked to our gate, he pressed his face against every window, looking at the parked airplanes in the dark. Oliver would see him doing this and run over to join him. With a bit of help, he noticed some airplanes had “United Express” painted on them, and others had “American” with an “AA” on them. He would talk about “American AA” and United Express airplanes for the rest of the week whenever he heard one in the sky.

I sat between the boys on our flight to Dallas, and Terah was in the row ahead of us. This is how Jacob spent most of the flight.

And Oliver, who had the aisle seat, enjoyed paging through the inflight magazines, safety brochure, wifi instruction card, and airsickness bag.

The real highlight came at the Dallas/Ft. Worth (DFW) airport, where we had a connection. And the reason: we needed to take a train to get to our connecting gate. The DFW Skylink system really works very well — but the boys cared most that it’s a TRAIN. It was difficult to get them both away from it when we got off. They wanted to see it leave, stay and watch the next one come, etc. Jacob was only happy when he realized he could see the Skylink trains running high above the window at our departure gate for Guadalajara.

The 2.5-hour flight to Guadalajara got a little long for them, but they managed OK. We struck up several conversations with friendly people that knew English as we waited in various lines. It seems to be something of a rarity for American families with young children and no Mexican heritage to visit Mexico. People went out of their way to be friendly and welcoming — even the customs officials. It felt like a great start to the visit. One Mexican man who was chatting with me encouraged me to learn some Spanish. I said my wife knows some, and that I had tried to learn some German back when I took foreign language classes. Laughing, he said, “Why would you do that?” Not as an insult; it just genuinely didn’t seem useful to him. I think they were happy and proud that someone wanted to visit Mexico and was excited about it!

More to come.

Anyhow, here’s the sensor suite.

The station is a Davis Vantage Vue, and they have pictures of the included receiving console on their site.

On the upper left is the anemometer (wind speed sensor). Next to that is the rain collector for the rain sensor. There is a solar panel on the front for power, and underneath that is the wind direction sensor. The white thing underneath the back is the temperature and humidity sensor, which is inside a radiation shield. There is also a barometer in there someplace.

The indoor console receives the data via a 900MHz wireless link. The console can also be connected via a serial, USB, or Ethernet link to a PC. All sorts of software can then do all sorts of things with it. The console itself, though, keeps a history, has hi/low reports, graphs, and current condition display that is quite useful already. But if course I’d want it hooked up to a computer.

I figured I couldn’t put the console in the basement next to the server due to wireless signal strength issues, but also didn’t want to require my desktop PC to be up for this to work. So I ordered the serial version, and some cheap DB9 to RJ45 adapter jacks. I have CAT6 strung throughout the house, so I simply converted a surplus Ethernet port to a serial port and it worked beautifully.

There are a ton of weather-related programs out there. My requirements said that I wanted something that runs on Linux. Options there include wview, vanprod, RRD Weather Graphs, Weewx, Meteo, and the commercial non-free Weather Display.

That’s quite a few choices. Of those, wview had the best support for various devices from various manufacturers, which I figure might be important down the road.

It can directly generate detailed weather webpages. It can also submit data to Weather Underground. Weather Underground has a feature called “rapid fire” which lets viewers’ screens update with current observations every few seconds. None of the noncommercial programs supported it, so I added it to wview. With my tree, it’s now sending Rapid Fire updates.

It also can submit data to the Citizen Weather Observer Program, which provides a way to both share it with others and provide it to forecasters at the National Weather Service and university researchers. It gets processed by their ingest system and is quality checked against their computerized QC metrics.

Jacob has enjoyed this — he got into assembling it, and often wants to go see if the anemometer is spinning.

And I was surprised to kind of enjoy working with C again. Perhaps that was because I didn’t have to touch malloc() even once?

We all trooped outside, and for a minute, the boys kind of ran around while I found a saw. We needed to saw off a few branches that were down but still connected. I did one. Jacob wanted to try, but it was a little too hard. But on the second one, which went a little easier, he helped out and with my guidance actually did some sawing. He was very proud of his work.

Then we got into the pickup — always a highlight for the boys, even if we only drove it around on the yard. I brought it up close and started loading brush onto it. Jacob went around finding whatever pieces of wood he could carry, and happily threw them onto the pickup.

Oliver, on the other hand, meticulously picked up smaller pieces of wood and carefully set them on the rear tire — then shoved them off the back. This really annoyed Jacob because Oliver wasn’t doing it right, so Jacob kept trying to find the wood Oliver was dropping. And then, of course, they got to play in the pickup bed.

Next it was off to the brush pile (also on our yard) to unload our 5ft high pile. I got the really heavy stuff mostly off. I told the boys they could throw whatever they wanted off the pickup. Jacob threw things onto the brush pile, and Oliver, who doesn’t really have any aim yet, just threw things and enjoyed trying to figure out where they went. But the big highlight came when the boys got to sit in the back while I pulled the truck a few feet forward and the rest of the brush just fell off the end “all by itself”!

I complimented both of them on their good work – they both really did contribute. Terah commented that when they came in, they both kind of strutted in, looking very tall, proud, and like they did some Good Work with their dad. And later, when Terah said to Oliver, “Dad tells me that you were very helpful, Oliver,” Jacob overheard and came running in to say, “I was very helpful too!”

Jacob: “These pancakes have sprinkles in them! Parties have sprinkles. I think we are having a party! Right, mom?”

Terah: “Sure, this could be a pancake party.”

Jacob: “And all parties also have ICE CREAM! So we should have some ice cream.”

Terah: “Uh, Jacob, I’m not so–”

Jacob: “Oliver, we are having ice cream!”

Oliver: “YAY!”

It’s amazing how fast sprinkles can get out of hand.

Later that day…

I had gone downstairs, pulled out one of the wonderful old hymnbooks I have in my collection but have rarely looked at, and sat down at the piano to play a bit. Oliver came in, sat down next to me, played a few notes, and promptly got down.

Jacob came in, sat down next to me, and took over our digital piano. He set it to provide some automatic accompaniment, selected a salsa beat, then hopped down as I continued to play a 4-part Mennonite hymn in my best imitation salsa style (not so good, in case you’re wondering). Jacob happily announced, “IT’S DANCING TIME!” He then picked up a large blanket and ran around the house, reminding us every minute or so that it’s dancing time at the top of his voice, while Oliver ran around randomly yelling “AAAAAA!”, and I went for maximum dissonance by attempting to play “Praise the Lord, Sing Hallelujah” to a salsa beat. Meanwhile Terah was sitting on the couch looking, well, a bit stunned.

Just another quiet evening at home.

It’s not because of a problem with Debian or with apt-get. It’s because of things upstream authors are doing.

It seems that upstreams, for some reason, like to remove features from new versions of software.

The two recent examples to bite me are the removal of the Gnome Terminal features to play the bell through sound card rather than the console speaker, and the recent upgrade to gtkpod — the best iPod music manager out there — which removed the ability to actually, well, play music. Wha…?

Earlier examples where when Gnome removed the ability to type a filename in the Open dialog box (remember that? thankfully reverted eventually.) KDE 4 was infamous for this too, with maybe thousands of instances (and they broke Amarok so badly that it didn’t actually play music from my devices either.)

I don’t understand this. The Gnome people somehow thought that they knew better than I what options I might want, I guess. But I am totally baffled about gtkpod. All it ever did was call xmms (or something like it such as qmmp) with appropriate arguments to play a file. Simple, configurable, and supremely useful. Probably just a few lines of code. And even that is gone.

There is a post by Ingo Molnar complaining that perhaps Gnome and KDE are trying to mimic the Apple and Google result without internalizing the process. Perhaps. It all seems so baffling to me though.

Updated: rewrote introduction.

I was able to do this without console access to the source machine, though console access wasn’t really required for either. This will work with any basic Linux install on the destination, or it could be prepared directly from a rescue environment.

Before I explain how I did it, I thought it was interesting to think of the different places that the machine that was known as complete.org has lived over the years. Here’s a map (click for detail):

Here’s the general process.

First, well in advance of the move, crank the TTL on the domains way down. This way, old IPs won’t be cached for very long once the system moves.

Next, start rsyncing data from the old machine to the new. Do not yet shut down daemons on the old. Shut down as much as you can on the new. You’ll want to focus on static data, such as /home. /usr is a good candidate as well. /var if you are selective — databases may be a good candidate, or may not. The idea is to seed the destination with data so that when we do the “real” rsync, most of the data will be there; it will have to tidy things up after daemons are stopped, and update some things, but the bulk of the work should be done.

Next, start preparing some exclude lists for the final rsync, which will copy an entire machine to the other. You will likely want to exclude files such as /etc/fstab, /etc/network/interfaces, /etc/resolv.conf, /etc/hosts, /etc/default/grub, /sys/*, /proc/*, /dev/*, and the like. I used rsync -v -P -a -H -A -X -S –numeric-ids –delete-after.

Now, start editing config files for the new IP, but do it on the old server (these will be synced over to the new one). Start with the bind config files, and touch anything else that needs it — maybe Apache configs, whatever.

Next, get ready to do the final sync. In an ideal environment, we’d just shut both machines down to single-user mode, but that’s not going to quite cut it here. Use ps and shut down all daemons except sshd and udev on the destination. Yes, including even syslog. This is to prevent anything actively accessing the disk during the rsync, and also to prevent any issues with clients accessing server daemons that aren’t ready yet.

Now, on the source machine, reload bind. This will start answering DNS queries with the new machine’s IP, and should also propagate the changes to your secondary DNS. Next, kill off all the daemons on the source machine, except for ssh and bind. You might want to set /etc/nologin to prevent regular users from logging in on ssh, if you have them. Now rsync things over to the new machine. Do any final tweaking over there (merging in /etc/hosts maybe, dealing with the udev rules.d persistent net thing, etc.). Reboot the new server and you should be up and running.

But his PC was mute. We opened it up and discovered it didn’t have a console speaker. So we traipsed downstairs, dug out an external speaker, and I figured out how to enable the on-board audio chipset in the BIOS. So now the cw command worked, but also there were a lot of other possibilities. We also brought up a microphone.

While Jacob was busy with other things, I set to work getting things hooked up, volume levels adjusted, and wrote some shell scripts for him. I also printed out this reference sheet for Jacob:

He is good at reading but not so good at spelling. I intentionally didn’t write down what the commands do, hoping that this would provide some avenue for exploration for him. He already is generally familiar with the ones under the quiet category.

I wrote a shell script called “record”. It simply records from the microphone and drops a timestamped WAV file in a holding directory. He can then type “play” to simply play back whatever he recorded most recently. Easy enough.

But what he really wanted was sound for his ASCII steam locomotive. So with the help of a Google search for “steam train mp3″, I wrote a script “ssl” (sound steam locomotive) that starts playing the sound in the background if it isn’t already going, and then runs sl to show the animation. This was a big hit.

I also set it up so he can type “play train” to hear that audio, or “play song” to play our favorite train song (Always a Train in My Dreams by Steve Gillette). Jacob typed that in and sat still for the entire 3 minutes listening to it.

I had to hook up an Ethernet cable to his machine to do all this, and he was very interested that I was hooking his computer up to mine in some way. He thought all the stuff about cables in the walls was quite exciting.

The last thing I did was install flite, a speech synthesis program. I wrote a small shell script called “talk” which reads a line at a time from stdin and invokes flite for each one (to give more instant feedback rather than not starting playback until after having read a large block from stdin). He had some fun hearing it say his name and other favorite words, but predictably the most fun was when he typed gibberish at it, and heard it try to pronounce or spell nonsense words.

In all, he was so excited about this new world of computer sound opened up to him. I’m sure there will be lots of happy experimentation and discovery going on.

And in that $8 package is not a DVD, or a book, or a toy, but rather 12 rolls of adding machine tape (totaling 1/3 of a mile). Oh yes, our four-year-old can hardly contain his excitement over office supplies. Let me back up a bit to explain.

I am always on the lookout for a certain kind of toy. The sort of thing that you don’t find in the toy section of a store. The sort of thing that is interesting and engages the curiosity of our boys, that most people would consider would toss, and that usually costs less than $5. Along those lines, Jacob has an old manual typewriter he can use, he and I built a switchbox full of switches and light bulbs that the boys love to play with, we built him a command-line-only PC out of spare parts, the boys have some radios they can use, and one day I salvaged a decades-old adding machine that was going to be thrown out.

Jacob liked it — he enjoys number and letter games — and had some fun, but didn’t zealously engage with it until just a few days ago.

He sat down at his table and started making “cards”. He will print out a bunch of numbers, making a strip a couple of feet long, and then say, “Dad, I made another card for you!” We will then happily accept the card and he will continue making more. Here he is, holding up one of his cards:

Jacob had a ton of fun making them, and Oliver wanted in on the action too. Oliver was somewhat interested in the adding machine, but not as much as Jacob, and Jacob didn’t want to share. So somehow they decided that Oliver would run and get cards from Jacob and then take them to whatever parent Jacob said. Today Jacob said, “Dad, the Oliver delivery service will bring you a card!”

Saturday morning we had some time to hang around and play, so we played some adding machine tape games. I tucked some long “cards” under my waistband, and the boys happily chased me all over the house trying to get them back. We also decorated:

And here is what the room looked like — after some cleanup:

That orange bucket on the desk is my “card bucket” where I can put all the cards he gives me. He even helpfully empties it out into the recycle bin when there are too many “old cards” in it!

But then tragedy struck — it was out of paper! I ordered a 12-roll pack off Amazon, and boy was it a long couple of days until it arrived. Today before the UPS truck got here, I showed him how we could feed old paper through and print on the back side — an incredible innovation, it seemed!

Terah had a conversation with him earlier that went like this:

Terah: “Jacob, we need to clean up the house.”

Jacob: “Why?”

Terah: “Because some people are coming over.”

Jacob: “Who is… OH YES! The UPS delivery man! Yes, let’s clean!”

Terah: “Ah, well yes he will be here too, but your great uncle and aunt are coming.”

Jacob: “Oh yay! And the UPS man!”

It is hard to describe Jacob’s excitement when the package arrived. I quickly opened it up, and at his great uncle’s prompting, he counted to see if he really did get 12 rolls of paper. “IT IS TWELVE ROLLS! YAY!” And then we quickly installed it in the machine and off he went. At his great uncle’s prompting, he started writing a J on each “card”. This meant “Jacob”, or maybe he’d put two J’s, meaning “To John From Jacob”, or perhaps the two J’s meant “To Terah From Jacob” – he’d always have to clue us in.

Anyhow, for an $8 investment in paper this has brought hours of joy to our boys. Even though Jacob isn’t yet really using it to perform arithmetic, it is setting him up to enjoy that sort of thing — and encouraging his curiosity. Both good things.

Rupert Murdoch’s corporation (owner of FOX News, Wall Street Journal, News of the World, etc) would have been found to have illegally accessed other people’s voicemails.

These included voicemails from a cellphone belonging to a recently-murdered girl, which interfered with the police.

And they had also bribed Scotland Yard officers for information, and actively covered it up.

The story would close one of Britain’s biggest newspapers, and force the resignation of various government and Scotland Yard officials.

That it would lead to the first high-profile investigation under the Foreign Corrupt Practices Act of an American corporation bribing British officials.

That one of the central figures in the scandal would suddenly die at a young age.

That there was a question of how the former editor of Murdoch’s newspaper — whom he steadfastly defended until one day he didn’t — would be taken from jail to testify before Parliament.

That CNN would live stream a hearing of a British Parliamentary subcommittee instead of the news conference given by the President of the United States.

And that during this committee hearing, some guy would attack Rupert Murdoch — who, until then, looked like he had a few drinks too many the night before — with a shaving cream pie.

And that Rupert Murdoch’s newspapers and TV channels would portray News Corp. as a victim of the liberal press in all this, and that they should just be left alone.

So maybe the last one was believable, but my goodness. Just when it couldn’t get any stranger, a SHAVING CREAM PIE?!

Friday morning, we woke the boys at 2:30AM to get to the train station.  The only Amtrak trains through our area are middle of the night departures.  Jacob is normally hard to wake up, but when I tell him that I’m waking him up to go to the train station, he wakes up faster than I’ve ever seen him before.  It takes about 3 seconds for him to process that in his groggy state, and then he sits up straight, throws off the covers, and is instantly ready to go.

Both boys were excited even in the station waiting room.  Oliver has been on an Amtrak train before, but it’s been awhile and he probably doesn’t remember it.    He constantly talked about it, jabbering as much as his vocabulary lets him.

Once we were on the train, Terah and I would have liked to get some more sleep.  The boys, on the other hand, were now wide awake, and didn’t fall back asleep until about 5:30.  That means Terah and I didn’t, either. Here’s what it usually looks like:

When it came time for breakfast, we went to the dining car as usual.  Jacob had already been telling us for days what he would eat for breakfast on the train: “I always have French toast on the train, dad.”  And so he did.  A few minutes later, I heard the waitress telling other people they were out of French toast, so I was glad I didn’t have to disappoint Jacob over that!

But it was Oliver that really came alive on this trip. I don’t think I’ve ever seen him so excited. It was almost constant. He happily talked about anything he could see or touch. But he also listened to other conversations, and would frequently pick out one word from a sentence and try to say it. He does that at home too, but not nearly so often as on the train. Both boys wanted to “go exploring” a lot — my word for taking them for a walk in the train and seeing what they might find. We walked up and down the train several times, Jacob excited over opening the doors between the cars, and Oliver excited just to be there.

Oliver’s excitement kept him from sleeping well. He did eventually get a short nap, but that wasn’t quite enough to avert a couple of tantrums later in the day.

The reason for the trip was the 80th birthday party for Terah’s grandpa. And it so happened that Oliver’s 2nd birthday would be over the same time. So, Saturday morning, we all went for breakfast at Das Dutchman Essenhaus, one of the favorite local restaurants in northern Indiana. After that, it was over to a relative’s place for some birthday festivities. The children got mini cakes to decorate as rail cars. There was frosting and all sorts of toppings. Great fun was had by all, and it was wise that this activity took place in a garage rather than indoors.

Yes, that is a marshmallow stuck to Jacob’s nose.

Then in the evening, it was off to another relative’s place for some more family time. Jacob had a great time all day, and was in high spirits. He asked me to sit by him at dinner, and started one of the longest conversations I’ve had with him in some time. We just talked about the things that happened in the day, but it was nice when I told him, “I like sitting by you, Jacob,” and he said, “Dad, me too!”

The big highlight for the day happened in the evening. We were gathered around a fire with a guitar to do some singing. Jacob was happily perched in his lawn chair, but got very excited when he saw some lightweight airplanes flying overhead. These kept flying at some distance, and he kept pointing them out to us. But that wasn’t even the most exciting part.

That came when the fireflies came out. Jacob ran around, catching them in his hands, and excitedly showing them to whatever person happened to be closest. He was laughing with joy for such a long time. At one point, someone asked him if the bugs were tickling his hands. He said, evidentally just realizing it, “Oh yes, they ARE tickling my hands!” He was one very happy boy.

Sunday I had to leave to get back home, while Terah and the boys will return a couple of days later. Although I do sort of look forward to a train trip that I can relax without having to manage two young boys, I do miss them already and will be happy to have everyone back home in a few days.

(This post written during the trip and posted a week later after arriving home)

Dad often had an old AM radio sitting around and would be listening to a baseball game while working. As it got darker, lights would come on, and the bugs would start flying near them. Sometimes dad would be working just inside the barn, and the bugs would start flying in there, while some light poured out the big front door. There’s something about that scratchy AM signal, the evening slowly getting darker, the slow pace of the baseball game, and just being around dad and a peripheral part of whatever he was doing that stirs a wonderfully fond recollection in me.

I don’t remember the specifics of any one of those times, nor do I really remember how often it happened, but it does stick with me.

We’ve had a routine in our house, starting early enough that neither of our boys know anything different, where right before bed, I read a book and sing a song to each of them individually.

Last November, I was looking for some books to challenge Jacob a little more than what we had been reading. I found The Complete Winnie the Pooh used for $4 on Amazon. This contains the original A. A. Milne stories, not the Disney series. It had a few line drawings, but there were many pages without any. It’s 352 pages and written in a rather dated form of British English. So for all these reasons, I wasn’t sure if Jacob would like it. But it was $4 so I bought it.

And Jacob was hooked. Each evening, we start bedtime with looking at the “map” of the 100-acre forest, just inside the cover. He gets to pick out 4 things for me to describe, and then we turn to our story. We usually read somewhere between 2 and 5 pages at bedtime, depending on how well he got ready without wasting time. And then we sing.

A. A. Milne has his Pooh character make up songs throughout the book. They are printed with words only, no tune, so I make up a tune for them as we go. Jacob has taken to requesting these songs for his bedtime song as well.

Jacob always gets to choose his bedtime story, and sometimes he chooses a different one — but about 75% of the time, it’s been Pooh.

A few weeks ago, he started noticing that we were almost to the end. He got very concerned, asking what we’d do next. I suggested a different book, which he didn’t like. Then I pointed out that we could restart the Pooh stories from the beginning, which was exciting for him.

Last night, we finished the book. The very last story was an interesting one, suggesting Christopher Robin growing up and no longer having imaginary adventures with the animals, but making Pooh promise to always be there for him. I don’t think Jacob caught onto that meaning, though. When we finished it, we had this conversation:

Jacob: “Dad, is that the end?”

Me: “Yes.”

Jacob, getting a big smile: “Yay! So can we start back at the beginning tomorrow?”

Me: “Sure!”

Jacob then gave a clap, shouted “Yay!” again, and was a very happy boy.

Sometimes I wonder what our boys will remember in 25 years of their fun times with me. I don’t know if Jacob will remember all the days reading about the animals in the 100-acre wood when he was 4, or maybe he’ll remember watching train and combine videos, or playing radio hide-and-seek, or maybe something entirely different.

But I have no doubt that I will remember sitting on the couch in his room, holding him on my lap, and reading a 350-page book to a loving 4-year-old. As Pooh aptly put it, “Sometimes, the smallest things take up the most room in your heart.”

• Email: 24 years
• Usenet: 13 years
• Facebook: 2 years
• Google Plus: ?

First, an analogy. Facebook reminds me of the sleazy guy selling stuff from his car down the street. They do things like change privacy defaults when they think it’s good for Facebook, rather than good for you.

Google+ reminds me of an Apple product. It’s beautiful, easy to learn, but locks everything down and is lacking some critical features. So here goes with the details.

The good

I think that this can be best summed up by: it’s not Facebook. Facebook has, perhaps, set an incredibly low bar but still, this holds. Facebook is the only website I can ever remember using that changes things so much, so often, and so completely that I keep having this feeling of not knowing how to do things. Try maintaining a Facebook page or two and you’ll especially feel my pain then. But even the basics: how many times have I accidentally posted a partial comment because I pressed Enter to start a new paragraph (that has only been the button used for that for, hmm, let’s see now, decades), and instead it posted the comment. Principle of least surprise, anyone?

Google+ has a simple and, on the surface anyway, intuitive interface. However, it does get muddy; more on that below.

I’m going to spend a lot more space on “the bad” below, but don’t let that diminish my excitement about having a viable alternative to Facebook. I am keen to get rid of that monster.

The Bad

I’ll start of this section with the fact that Google+ is a tightly-controlled walled garden. There is no way to take a copy of your status updates, comments, etc. and back them up with your own devices. If your Google account goes away, so do all your updates about your kids’ first words. Facebook does have this feature these days. It has broken half the time, but it exists, works, and I use it. Twitter doesn’t have the feature built in, but its API makes it pretty easy; you could easily use my twidge program to do this, for instance.

But whatever you put in G+ stays in G+. There is no autoposting it to twitter or Facebook, no backups, nothing. I’m disappointed in that, given Google’s prior attitude about openness to individual data ownership. (Update 7/12/2011: There is a way to get a backup, which I hadn’t noticed; see comments below.)

Almost as bad, there’s also no way to get data in. So I now have identi.ca, twitter, Facebook, and now also G+ accounts. When I post an item in identi.ca, it autoposts to twitter, and then it autoposts from twitter to Facebook. Handy – one place to notify people of my public activities. My uploads to Flickr and blog posts also auto-post to Facebook, so people I’m friends with there get a picture of what I’m doing outside of Facebook without me manually having to link it in.

Not so with G+. There is no way to even add an RSS feed for my blog to auto-post to G+ as there is with Facebook. If you use G+, the only way to get stuff into it is with a keyboard.

Poor integration with other Google apps is also an issue. G+ is nowhere near Google’s first social app. They also have Blogger, Google Reader, Buzz, Gmail, etc. So here’s the rub. These aren’t integrated well, and when they have tried to integrate them, they’ve done an exceedingly poor job of it.

As an example: Buzz is a twitter-like service for posting updates. Very similar to what G+ does, right? Well, the best you can do is link your Buzz account to a separate tab on your profile. You can’t even have your Buzz updates flow straight into G+, from what I can see. You have a separate list of followers in Buzz from G+. Reader is even worse; it has a sharing feature, but it’s not G+ aware, so it goes to a reader inbox.

But what takes the cake is the integration with Gmail Contacts. Sounds easy? Not entirely. When you add someone to G+, apparently it sometimes links the record with their entry in Contacts. And the help gives you the handy warning that when you remove them from G+, you have the option of removing them from Contacts, which could impact what you see on a mobile device. It is unclear what happens when it doesn’t link the record, or how it decides which record to link (some people have more than one person in a family sharing phone numbers and email addresses, for instance.)

One very nice feature of G+ is you can add email addresses, even if they aren’t G+ members. Then when you post updates to the circle you’ve added them to, G+ offers to send them an email with an update. Very handy.

Except the undocumented part is that if you select “Your circles” (meaning all your circles), rather than individually ticking the box by each circle, they don’t get the email. And they also don’t get it if you set it Public. You have to select their specific circle, which is a significant difference from how people with G+ accounts are treated.

And that’s just an example. There are a ton of things that work one way 95% of the time, but have non-obvious exceptions. Some of these exceptions are documented in the help, and some aren’t.

I’m still confused about the integration with Picasa. G+ help says that you can upload unlimited numbers of photos for free (though they will be downscaled at a certain point). Picasa has definite storage limits. But G+ uploads are showing up in Picasa. Do I really get free storage by uploading to the same place via a different tool? Very odd.

And finally, a gripe about web standards. I’ve been using Firefox/Iceweasel 3.5, which came out in Debian squeeze and is still supported there. It is also pretty darn new by corporate IT standards. And Google Plus refuses to let me log in with it, saying it’s “incompatible.” Perhaps it can’t do some fancy animation, but then again I don’t really care. Seems Google has forgotten the old RFC adage: be liberal in what you accept and conservative in what you generate. If you can detect that my browser won’t animate something right, then you could give me a stripped-down version of the page rather than an error message.

Here are some questions I have been thinking of. If you’d like to help give me anecdotal evidence, please post a comment below this post. Identify the question numbers you are answering. It helps me if you can give specific examples, but if you don’t have the time or memory for that, no problem.

I will post my own answers in a day or two, but the point of this post is listening, not talking, so I’ll not post them immediately.

Hobbies (General – any geeks)

• H1: To what degree do you like your hobbies to be challenging vs. easy? If something isn’t challenging, does that make it a good, bad, or indifferent candidate for a hobby
• H2: To what degree do you like your hobbies to be educational or enlightening?
• H3: How do you pick up new hobbies? Do you go looking for them? Do you stumble upon them? What excites you to commit time and/or money to them at the beginning?
• H4: How does your interest wane? What causes you to lose interest in hobbies?
• H5: For how long do you tend to maintain hobbies? Sub-hobbies?
• H6: Are your hobbies or sub-hobbies cyclical? In other words, do you lose interest in a hobby for a time, then regain interest for a time, then lose it again? What is the length of time of these cycles, if any?
• H7: Do you prefer social hobbies or solitary hobbies? (Note that many hobbies, including programming, video gaming, reading, knitting, etc. could be either social or solitary, depending on the inclination of individuals.)
• H8: Have you ever felt guilt about wanting to stop a hobby or sub-hobby? (For instance, from stopping supporting users of your software project, readers of your e-zine, etc) Did the guilt keep you going? Was that a good thing?

Examples: video games might be a challenging hobby (depending on the person) but in most cases aren’t educational.

A hobby might be “video game playing” or “being a Debian developer.” A sub-hobby might be “playing GTA IV”, “playing RPGs”, or “maintaining mutt”.

Free/Open Source Hobbies

• F1: Considering your answers above, do your FLOSS activities follow the same general pattern as your other hobbies/interests, or are there differences? If there are differences, what are they?
• F2: Has concern for being expected to support software longer than you will have an interest in it ever been a factor in a decision whether to release source code publicly, or how public to make a release?
• F3: Has concern over the long-term interest of a submitter in maintaining their patch/contribution ever caused you to consider rejecting it? (Or caused you to avoid using software over the same concern about its author)
• F4: In general, do you find requirements FLOSS projects place on first-time contributors to be too stringent, not stringent enough, or about right?
• F5: Have you ever continued contributing to a project past the point where your interest would otherwise motivate you to do so? If so, what caused you to do this? Do you believe that cause is a general positive or negative force for members of the FLOSS community?
• F6: Have there ever been factors that caused you to stop contributing to a project even though you still had an active interest in doing so? What were they?
• F7: Have you ever wanted to be able to take a break as a contributor or maintainer of a project, and be able to return to contributing to it later? If so, have you found it easy to do so?
• F8: What is your typical length of engagement with FLOSS projects (such as Debian) and sub-projects (such as maintaining a particular package)?
• F9: Does a change in social group ever encourage or discourage you from changing hobbies or sub-hobbies?
• F10: Have you ever wanted to stop working on a project/sub-project because the problems involved were no longer challenging or educational to you?
• F11: Have you ever wanted to stop working on a project/sub-project because of issues with the people involved?

Examples on F9: If, say, you are a long-time Perl user and have gone to Perl conferences, but now you are interested in Ruby, would your involvement with the Perl community cause you to avoid taking up the Ruby programming hobby? Or would it cause you to cut your ties with Perl less quickly than your changing interest might dictate? (This is a completely arbitrary example and isn’t meant to start a $LANGUAGE thread.)

Changes over time

• C1: Do you believe that your answers to any of the above questions have changed over time? If yes, then:
• C2: What kinds of changes have happened?
• C3: What caused the change?
• C4: Do you believe the changes produced positive results for you? For the community?

The Lives of Others is an incredible film set in the former East Germany (GDR/DDR) mostly in 1984. The authenticity of it is incredible and so is the story. It’s subtitled, but if you’re an American wary of subtitled European films, don’t be wary of this one. It is easy to watch and worth every minute.

The story revolves around the Stasi, the GDR Ministry for State Security (“secret police”). It is an incredible picture of what living in a police state was like, and how many of the informants were victims of the regime too.

My breath caught near the beginning of the film, showing the inside of a Stasi building. A prisoner was being interrogated for helping someone attempt to escape to the west. But the reason my breath caught was this incredible feeling of “I was there”. Last year, Terah and I were in Leipzig and visited the Stasi museum there, Museum in der “Runden Ecke”. I always have an incredible sense of history when being in a preserved place, and this building was literally the Stasi headquarters for Leipzig. Much of it was preserved intact, and seeing it in the film brought home even more vividly the terrible things that happened in that building, and others like it, not so very long ago.

We watched the special features on the Blu-Ray disc, and one of them was an interview with director Florian Henckel von Donnersmarck. He described how he spent a lot of time interviewing both victims of the Stasi, as well as ex-Stasi officers. One of the most disturbing things to me was his almost offhand comment that most of the former Stasi officers still had some “pride” in performing their jobs well. Even now, freed of the state’s ideology, they were proud of the work they did — which could be put most charitably as ruining people’s lives.

What leads a person to view life that way? How can we try to make sure it doesn’t happen again elsewhere?

I am happy to say that most of us have never experienced anything like the Stasi. And yet, small reflections of that mindset can be seen almost everywhere. Societies at wartime or feeling under threat, even Western democracies, can drum up those feelings. In the USA, for instance, the McCarthyism era saw people’s careers ruined for alleged anti-state behavior. Contemporary examples include the indefinite “detention” (I hate that word; shouldn’t we say “imprisonment”?) of terrorism suspects at Guantanamo Bay, and the terrible treatment of Bradley Manning, who revealed some true but embarrassing things about the US military — which really needed to be revealed. Even tobacco farmers and companies are selling a product they know ruins lives, but somehow keep doing it.

And there are still members of the public that try to make life difficult for people that don’t think like they do. From organizing campaigns of telephone harassment of colleges that don’t perform the American national anthem before sporting events, to tossing about the term “un-American” (a loaded McCarthyist one, which many may not even be aware) at an inflated rate, we are not immune from attempts at forcing conformity or silence in others, and blind loyalty to state.

I am never in a particularly celebratory mood on July 4, the biggest day for American boasting, faux patriotism, militarism, and general flag-waving. We do have a lot to be proud of and thankful for, but it seems that we celebrate all the wrong things on July 4, and see it as an occasion to proclaim American exceptionalism rather than as one to see how far we’ve come and bolster hope for how far we can, and should, yet go.

No, I don’t think that the “land of the free” ought to have operated secret prisons in Europe (nor the Europeans to have been complicit in it), or that the American military was “defending our freedom” 100% of the time they were deployed, or that it is right for governments to mandate daily recitation of an untrue document (the pledge of allegiance) in schools.

And yet, I am mindful that I have a lot to be thankful for — stability, lack of much internal violent conflict, etc. And this particular day I am happy that a post like this is not something that gets the attention of some government agency – and mostly that I will have a handful of angry emails to delete.

When Jacob is all alone in his room, sometimes he will sing. Maybe it’s a song he knows, or fragments of a song he knows. Or maybe it’s something he just made up on the fly. It might be about trains, cats, letters, or who knows what. Maybe it will crack him up and he’ll keep singing and then laugh. Or maybe it will stop in 30 seconds and he’ll go on to something else. Brief and out of tune though it may be, it is still one of the most beautiful sounds I know and I love to hear it.

Jacob loves to have other people sing, too. Sometimes I will make up songs while we drive: “We are driving, we are driving, we are driving down the road; we are driving down the old country road… and we’re making some dust, yes a lot of dust…” Or Jacob likes me to sing the songs in the Winnie the Pooh book he has (the original, pre-Disney version, which has the words for the songs but not a tune, so I make up a little bit different one each time). Very occasionally he will sing with me, but if he does, he doesn’t want anybody to comment about it.

I sing with the Kansas Mennonite Men’s Chorus. We’re a group of about 300 singers (all men, though not all Kansan and not all Mennonite) that sing for charity. We’ve raised over $600,000 for charity so far.

I still remember the first time I went to a KMMC practice 3 years ago. It was my first year singing, and the first practice of the season. We started singing the first song, and wow – what a powerful moment. Even at its imperfect first practice state, hearing 300 people sing quietly is a powerful sound – and when they get to the top of their lungs, it’s indescribable. No CD can ever quite do that justice.

Jacob and Oliver come to KMMC concerts sometimes, but this year they took a special interest. Oliver pointed to, hm, perhaps 300 men and said “dad” for each one. (He was quite far away and probably couldn’t pick me out specifically.) Jacob, for the first time, sat quietly engrossed in the concert – until we got to Dry Bones (a song involving surprise and lots of homemade instruments) which made him laugh out loud. Then he would hide under his seat from the applause.

Last week, KMMC went on tour. This was my first choir tour, and Terah came along while the boys spent time with grandparents in Kansas. We sang in Bloomington, IL, and then in Goshen, IN at the fairly new Sauder Concert Hall. That hall is a few years old, and has been reviewed as once of the best concert halls in the world acoustically. It seats 1100 people and was perfect for this. The 85 or so singers from KMMC that went on tour joined the Indiana and Ohio Men’s Choruses to make a combined group of about 200 singers. And what an experience that was. I learned later this was the first event to completely pack Sauder Concert Hall.

We ended the concert singing the “Mennonite Anthem” – a souped-up version of the doxology. (You can see a video of a different group singing it to get an idea of what it was). This is often joked about as being part of Mennonite DNA. You can gather a random group of Mennonites and ask to hear “606″ (its number in, well, an old hymnbook that we don’t use anymore) and you can probably get a fairly well-sounding rendition, from memory, a cappella, complete with 4-part harmony, at the drop of the hat. Mennonite youth have been known to sing it in train stations, airports, soccer stadiums, and with kazoos.

At Goshen, we had 200 men on stage, plus a brass band, plus 1100 people in the audience facing us. They were, of course, invited to sing along. (That’s one of the rules of 606: you can’t exclude anybody – because even if you tried, they’d still sing, because how can you resist?)

So I was up there on stage, singing a song I love, and hearing the 1100 people in the audience sing it to me. I could glance out over the front rows and saw the smiles on so many faces as they sang, and what a moving moment that was. The excitement and thrill of it all hit me so much that I could barely finish the song.

The numbers aren’t all in yet, but I’m guessing that we raised over $15,000 for charity last weekend (the choir members paid their own expenses in advance, so 100% of the money raised goes to international relief efforts.)

A 4-year-old singing alone in his room is really heartwarming. I have to say that 1300 people singing a song that touches each one of them is a pretty close second.

APRS is the Automatic Packet Reporting System. It’s a system for exchanging brief packets of information. It is most frequently used for mapping applications, but it really does a lot more than that. It has its biggest home in the amateur radio world, but isn’t limited to that, either.

The most common way to use APRS is to have some device hooked up to a GPS transmit packets with the GPS information in them. These packets can then be plotted on a map in real-time or with history. That in itself isn’t particularly newsworthy these days.

An interesting thing about APRS is that it’s not just positioning. Let’s say that there was a search-and-rescue operation. A person could draw a rectangle on the map indicating the search area, and within about 3 seconds everyone else’s map also shows that rectangle. People have even been known to play chess by sharing and moving objects on APRS!

The next piece that makes this interesting is that APRS is an ad-hoc mesh network. In its traditional implementation, VHF amateur radio, a radio emits a packet with a geolocation in it (a “beacon”) and any other radio within direct range of that can receive it. Radios can display basic information (such as distance to the other radio, heading, etc.) or hooked up to a laptop or mapping device for a better display. So if everyone is within a few miles, APRS works without any pre-existing infrastructure at all. This makes it wonderful for use in disaster areas, and was put to heavy use in Joplin after the tornado there.

But what about radios that are too far away? Any APRS station could also be a digipeater. When a packet is transmitted, it has a maximum hop count. A digipeater hears the packet, decrements the maximum hop count, and re-transmits it. With this mechanism, packets can travel hundreds of miles. It creates a highly resilient network, one that can route around trouble without even having to have an explicit backup route. I could bring in a digipeater in my car — it can be small enough to hold in my hand — and instantly improve APRS reception in an area.

One interesting aspect is that packets can be digipeated more than the maximum hop count. For instance, if a packet leaves my radio and is picked up by a digipeater to my west and one to my east, it can keep on traveling in both directions. This is part of what leads to resiliency.

APRS also functions over the Internet. There is a large network of interconnected Internet servers that exchange all global APRS traffic amongst themselves. Gateways between the radio (RF) and Internet (APRS-IS) services exist, and are called iGates. They are not generally required, but make useful websites like aprs.fi and email gateways possible. As long as an iGate is within a reasonable number of hops from you, you’re effectively linked. And again, if one iGate drops off, another iGate is probably monitoring your traffic too and you never notice. It’s an ad-hoc mesh network that is actually reliable – how about that?

On the PC side, there are many programs for using APRS. The most common one for Windows is called UI-View, but I don’t use Windows so I can’t comment. On Linux, there are programs (such as aprx) for running your digipeater, but the best-known program is Xastir. Xastir lets you download map files to your local disk, and can interface with the APRS-IS Internet service, radios, weather stations, or simply other arbitrary machines to exchange information. Xastir is a very nice program and is well worth the install, despite its somewhat dated-looking interface.

APRS clients, such as APRSDroid, exist for Android and iOS platforms as well.

So let’s say you’re doing something like helping handle food/water stations for a long bike ride. Even if you don’t have anybody with an amateur radio license, you can use APRS to great effect. At your headquarters, you can run Xastir and turn on its “server mode”. This puts everyone on a map. Then you can have everyone turn on APRS on their phone, and have it report to your custom server instead of APRS-IS. Now you have instant visibility into your entire team’s location and status. If you have transport people driving supplies between locations, that’s especially helpful.

In an amateur radio scenario, you would instead have people with radios at each location, and one laptop hooked up to a radio at HQ. This provides an added bonus of not relying on third-party infrastructure such as cellphone towers.

APRS also has a messaging system, similar in concept to text messaging. It works the same as other things. If I want to send a message to Jane, my radio simply emits a packet that lists the message and Jane as a recipient. It’s digipeated up to its maximum hop count. If Jane is within RF range of one of those digipeaters, she gets the message and her radio ACKs it. Otherwise, it’s delivered into the APRS-IS network — probably several times, which isn’t a problem — and the APRS-IS network delivers it to the iGate closest to her, and from there it gets digipeated the rest of the way to her.

Here’s an example of something created with APRS. While I was on a bus choir tour last weekend, I had a radio with me that was beaconing all the while. Now it was a small handheld radio inside a large metal bus, so it didn’t always have a digipeater in range. But still, you can go see a detailed map with the trail and even see exactly what path each packet took before it hit the Internet.

If you want to try out Xastir, please grab at least version 2.0 – the version in squeeze has some bugs.

The biggest image that remains in my mind is of the first time I saw a person picking through a large pile of rubble. The person was standing on top of what used to be a house. Now it was a pile of wood, glass, carpet, siding, and roofing material. I’m sure there was hope for finding some treasure or other — maybe a photo album or videos of children. In any case, it made me feel so lucky, even unfairly lucky, to have not had to go through that.

This scene was repeated several times, but mostly the houses that devastated appeared abandoned by the time we were there, now two weeks after the event.

But I heard stories, and lots of them. The victims of the storm, who were perhaps trying to rebuild that part of their house that got smashed by a tree or a pickup, or trying to get their intact belongings out before abandoning the house, or whatever, were mostly surprisingly upbeat. They were working out in 95-degree heat, many without electricity, running water, or sewer service. Almost every person I met that suffered a loss from the tornado wanted to tell their story. Many also told of their plans for the future, which were full of hope and even upbeat. These were people doing a hard job in terrible conditions and still showing hope.

Another testament to the disaster was the most unusual set of vehicles you’ve ever seen parked at any hotel you can care to think of, for at least an hour-and-a-half radius in the direction I came from. Besides the usual cars and minivans, there might be FEMA vehicles, electric company trucks, Red Cross vans, construction trucks of every kind, police and law enforcement from all over, etc. There was quite obviously an influx of people helping out in Joplin.

My primary task there was to provide communications support for the effort as an amateur radio operator. Amateurs (or “hams”) are something of a volunteer first responder of sorts during times of crisis; most of us own and are very familiar with operating equipment that can communicate over very long distances without the need for any on-site infrastructure. Amateur radio was the only method of communication for some Joplin hospitals in the immediate aftermath.

The communications emergency is over, but the response isn’t. I was assigned to work with the Salvation Army. They were doing a lot of things in Joplin, and had hundreds of volunteers working with them. I don’t think I even know what many of them were doing – I do know they had set up several warehouses across the city working with donating clothing, food, etc.

The part I was involved with was primarily the canteen operation. The SA sent in food service trucks from several parts of the country. These trucks would roam up and down the streets in the damaged area, trying to get past every single street several times a day. Anybody that we could see would be offered food and water. No strings attached, no questions asked. This included homeowners, electric line workers, construction crews, sanitation workers, and quite a few nonprofit groups that sent well-meaning and useful volunteers into the area but didn’t think to provide them with a large supply of water due to sending them into an area without any. Oops. In any case, with extreme heat and no running water, conditions were dangerous. The canteens also knew of certain at-risk families that were living in homes that were mostly intact in the disaster areas, and made a special point to check in on them. They also generally looked to make sure that people looked like they were healthy. Each canteen also had a counselor on board that would visit with people while we quickly prepared their meal — they all seemed to welcome that.

Amateur Radio’s Role

The operation of this size had quite a logistics challenge. I’d hear of things like an unexpected need of 70 lunches, or a semi showing up with donations before there was space, or an unexpected but very welcome donation of a large quantity of ice cream without a place to store it (so the canteen trucks, which have freezers, needed to pick it up quickly). That’s where us hams came in. Each canteen had an amateur radio operator on board. Each major location also had a ham stationed there, and the head of operations also often had a “shadow” — a ham that would follow him around wherever he went to relay messages back and forth. We also had hams with pickups (with radios in them, of course) that could transport things around the city to places that needed them, hams at headquarters managing all the communication and generally investigating questions that didn’t have immediately obvious answers, etc.

Radios were used instead of cellphones for a few reasons. One big one is that everybody on the operation can hear what everyone else’s needs are, since it’s a group communications situation rather than one-to-one. It’s easy to give a general alert to everyone (“come get your ice cream now please!”) and people that have suggestions can chime in. This came in extremely handy more than once. Also, it frees the people doing other jobs from having to spend time chasing someone’s voicemail, finding phone numbers, etc; that gets delegated to us in some cases. I heard from the head of canteen operations, for whom this was the first disaster he’d worked that had amateur radio support, how wonderful it was to have this going on. I also heard a secondhand report that some police officers that were also amateurs had listened to our operation and reported that “we sound more professional than 911 dispatch and do a better job.”

On Sunday I was assigned to a canteen. This meant I didn’t have a lot of radio traffic to pass, so although I had it in my ear all day, I wasn’t actively talking on the radio very much. So I rode in back, helping hand out water, carry meals to people, and so forth.

On Saturday, I was the shadow for the head of operations. That was a difficult task, because he barely ever moved at a pace slower than a run, sometimes would abruptly zip out somewhere, etc. But it was also enlightening and vital. He was a real “get it done” sort of guy, and was the key to quite a few things. Having someone available to relay questions to and from him was a good thing.

And today I worked as a transport person and at headquarters. Due to not having a pickup there, I didn’t actually get called on much to transport things, but in general between jobs the whole time I’d act as a runner if needed, or simply try to figure out the details of how things were run for next time. I wound up taking “net control” (being the control operator at headquarters, and generally managing communications so that people don’t talk over each other and such) for about an hour. So I got to do a little of just about every amateur radio task.

Thoughts

I am thankful for the opportunity to go, and the good feeling of helping people in need — the first I’ve ever had the chance to do that in a disaster. It’s a good feeling to have a skill that is useful and appreciated. Sometimes it felt like handing out food and water is something pretty small in the scheme of things. But on the other hand, it gives people a chance to have contact with someone that cares, an opportunity to have people that can notice problems drive by a few times a day, and an opportunity to help meet people’s basic needs. And sometimes in a fluid situation, there might be more volunteers than are needed, so I did spend some time sitting waiting for the next task.

But overall, I’m convinced that the work I helped facilitate was a good thing and provided a good and needed service in Joplin. This has been quite the experience and I’m sure it’s changed me too, though I don’t yet know how.

Someone, upon seeing me reading my Kindle, would ask what I am reading.

“War and Peace.”

“Oh. . . uhm, wow.”

“Yeah, it’s great. I’m reading it for fun.”

“Uhm, OK then, see you later. . .”

It seems to be so revered, and also so loathed, that I had to read it. At 1400 pages, it took me 8 months to finish, though I read several other entire books in that time.

I feel rather unequal to the task of expressing how this book impacted me, let alone a review of it. And nonetheless, I also feel I would be remiss if I let it go past saying nothing.

I was struck by so many things as I read War and Peace. Some of them I won’t mention here and hope to turn into their own blog posts.

Of the others, I felt I gained some sense of how the nobility and serfs in Russia (and, to a certain extent, Europe) thought about life, their position, and how things ran. Being a modern Kansan, this thought process was not familiar to me, and though I head read about it in history texts before, felt far more informed having read it in Tolstoy’s novel.

Although much of the novel centers around Napoleon’s invasion of Russia in 1812, the work as a whole spans nearly two decades of time. Tolstoy’s characters aren’t static; they change over time. Some, yes, die; others go through hardships and triumphs that change them to their core. It evoked a feeling of nostalgia in me at times — for the younger, childlike Natasha who was so full of simple delight in life. But then, a thousand pages later, the older Pierre finally was able to find simple delight in life too.

Sometimes I have missed on the simple joy of being, and Jacob or Oliver or Terah remind me of that. Today Oliver and I read a book together, one that we read often, and we discovered an illustration of a tiny worm we had never noticed before. And the worm had a red hat (“hat” is one of Oliver’s favorite words right now.) The happy laughter as he pointed at the tiny hat, saying “hat” over and over, reminds me that sometimes children know how to live better than adults. Jacob later asked me how my day was, and I told him how I read a book on my Kindle, where I sat, and how I even read it lying down on the couch for a bit. At that he too laughed.

As with some other wonderful, engrossing books, I was sad to reach the end of this one. I felt as if I was leaving a conversation early; fictional characters, yes, but their story wasn’t over. And really, that was part of Tolstoy’s point: things don’t happen in isolation, and stories don’t have clearly-defined start and end points.

The novel touched on politics, religion, philosophy, free will, and just about every topic imaginable. It is, really, unfair to call it a just a novel.

Here are some random quotes from the book, which I highlighted:

his heart was now overflowing with love, and by loving people without cause he discovered indubitable causes for loving them.

How often we sin, how much we deceive, and all for what?… All will end in death, all!

A pleasant humming and whistling of bullets were often heard.

Looking into Napoleon’s eyes Prince Andrew thought of the insignificance of greatness, the unimportance of life which no one could understand, and the still greater unimportance of death, the meaning of which no one alive could understand or explain.

“If there is a God and future life, there is truth and good, and man’s highest happiness consists in striving to attain them. We must live, we must love, and we must believe that we live not only today on this scrap of earth, but have lived and shall live forever, there, in the Whole,” said Pierre, and he pointed to the sky. Prince Andrew stood leaning on the railing of the raft listening to Pierre, and he gazed with his eyes fixed on the red reflection of the sun gleaming on the blue waters. There was perfect stillness. Pierre became silent. The raft had long since stopped and only the waves of the current beat softly against it below. Prince Andrew felt as if the sound of the waves kept up a refrain to Pierre’s words, whispering: “It is true, believe it.” He sighed, and glanced with a radiant, childlike, tender look at Pierre’s face, flushed and rapturous, but yet shy before his superior friend.

In Natasha Prince Andrew was conscious of a strange world completely alien to him and brimful of joys unknown to him, a different world, that in the Otradnoe avenue and at the window that moonlight night had already begun to disconcert him. Now this world disconcerted him no longer and was no longer alien to him, but he himself having entered it found in it a new enjoyment.

All the kings, except the Chinese, wear military uniforms, and he who kills most people receives the highest rewards.

“But what is war? What is needed for success in warfare? What are the habits of the military? The aim of war is murder; the methods of war are spying, treachery, and their encouragement, the ruin of a country’s inhabitants, robbing them or stealing to provision the army, and fraud and falsehood termed military craft. The habits of the military class are the absence of freedom, that is, discipline, idleness, ignorance, cruelty, debauchery, and drunkenness. And in spite of all this it is the highest class, respected by everyone.

From the moment Pierre had witnessed those terrible murders committed by men who did not wish to commit them, it was as if the mainspring of his life, on which everything depended and which made everything appear alive, had suddenly been wrenched out and everything had collapsed into a heap of meaningless rubbish. Though he did not acknowledge it to himself, his faith in the right ordering of the universe, in humanity, in his own soul, and in God, had been destroyed. He had experienced this before, but never so strongly as now. When similar doubts had assailed him before, they had been the result of his own wrongdoing, and at the bottom of his heart he had felt that relief from his despair and from those doubts was to be found within himself. But now he felt that the universe had crumbled before his eyes and only meaningless ruins remained, and this not by any fault of his own. He felt that it was not in his power to regain faith in the meaning of life.

When Princess Mary began to cry, he understood that she was crying at the thought that little Nicholas would be left without a father. With a great effort he tried to return to life and to see things from their point of view. “Yes, to them it must seem sad!” he thought. “But how simple it is. “The fowls of the air sow not, neither do they reap, yet your Father feedeth them,” he said to himself and wished to say to Princess Mary; “but no, they will take it their own way, they won’t understand! They can’t understand that all those feelings they prize so—all our feelings, all those ideas that seem so important to us, are unnecessary. We cannot understand one another,” and he remained silent.

“Love hinders death. Love is life. All, everything that I understand, I understand only because I love. Everything is, everything exists, only because I love. Everything is united by it alone. Love is God, and to die means that I, a particle of love, shall return to the general and eternal source.” These thoughts seemed to him comforting.

“People speak of misfortunes and sufferings,” remarked Pierre, “but if at this moment I were asked: ‘Would you rather be what you were before you were taken prisoner, or go through all this again?’ then for heaven’s sake let me again have captivity and horseflesh! We imagine that when we are thrown out of our usual ruts all is lost, but it is only then that what is new and good begins. While there is life there is happiness. There is much, much before us. I say this to you,” he added, turning to Natasha.

During that twenty-year period an immense number of fields were left untilled, houses were burned, trade changed its direction, millions of men migrated, were impoverished, or were enriched, and millions of Christian men professing the law of love of their fellows slew one another. What does all this mean? Why did it happen? What made those people burn houses and slay their fellow men? What were the causes of these events? What force made men act so? These are the instinctive, plain, and most legitimate questions humanity asks itself when it encounters the monuments and tradition of that period.

I’ve had a lot of people, both in person and in comments responding to my blog post, express surprise with statements I have made. The view of Christianity that many people have is of a group that devalues scientific inquiry and places a lot of emphasis on things like opposition to gay marriage, evolution, and abortion, and enjoys political leaders that say “bring it on.”

While I know some Christians that fall into that mold, there are quite a few that don’t as well. Some churches, such as the one we attend, have a surprisingly diverse set of people and yet still function and get along well.

Today they accepted me with joy. Nobody was concerned that I started my statement with a reference to ancient Greek philosophy, wound up suggesting that the church ought to make sure to make illegal immigrants, gays and lesbians, and prostitutes feel welcome, and embraced both religion and scientific inquiry, feeling them complimentary.

Mennonites practice adult baptism rather than infant baptism. A traditional age for people to be baptized is during high school, though emphasis is placed more on the individual than their age, so it’s not unheard-of for someone to do so a bit later in life as I have.

Part of the baptism involves the candidate sharing their faith story. These are typically intensely personal, widely varied, heartfelt, and honest. Some people’s stories involve struggles with depression, physical disabilities, or the place of religion in their lives, while others reflect little struggle at all. Mine involves letting go of a lot of things, and also seeing some things, such as serious intellectual inquiry or existential questions about God (such as “is there a god?”), as a positive rather than a negative feature of a Christian life. But it also involves a recognition and deep respect for those that don’t approach things in this way.

Though given publicly in a church, baptism testimonies are rarely published or shared more broadly. But I’m going to share mine here. I have edited it only very lightly to remove a few local references that wouldn’t make sense out of the context of this community. This was delivered in front of a rather different audience than is likely to be reading this post, so if you have questions, do feel free to ask in the comments.

But before I begin, I just have to pass along a joke a friend told me: “John, you picked quite a day for your baptism — the day after the rapture.”

Baptism Testimony

John Goerzen

May 22, 2011

Those of you that know me well will probably not be surprised that I will begin my Christian baptism testimony with a story about an ancient Greek philosopher, and also touch on the philosophical nature of truth. These are key parts of my story.

As legend goes, Socrates famously said, “I know that I know nothing” — in other words, he believed that nothing could be known with absolute certainty. The Greek Oracle — thought to be infallibly wise — said that Socrates, the man that thought he knew nothing, was the wisest man in all Athens. An interesting paradox, and one that sheds light on my own religious story. My story involves coming to grips with the understanding that I know very little, that no person can ultimately know much about God, and finding a way to make peace with that situation.

Growing up in this community, I thought I knew some things about faith and Christianity. At a certain age, it all seemed so simple. We took the Bible to be literally true. We marveled in Sunday School at how the apostles could sometimes appear so blind. And, modeled both implicitly and explicitly, was this notion: the stronger our faith, the fewer inner questions or doubt we have about the nature of God, the literal accuracy of the entire Bible, or our relationship to God. Those I perceived held up as examples never seemed to question any of these things, and showed — outwardly, at least — complete certainty about them. Moreover, holding certain intellectual beliefs was key to Christian identity, and even more importantly, to eternal life.

This model has quite obviously worked well for many people for many years. The good that has happened, and continues to happen, from people that have that kind of certainty is manifest all around us. And yet, it didn’t seem to work out for me.

I’m not the kind of person that accepts a lot of things at face value. It is helpful to be able to examine and challenge ideas — and even more helpful to have other people challenge my ideas. From well before I was in high school, I was questioning some things about the Bible, God, and religion in general. My thoughts ranged from the impact of evolution on religion to the apparently vengeful God of the Old Testament to the very existence of God. In the Christian context, I perceived having these questions as a personal failing, something that I ought to repress.

The more I tried to repress them, the more troublesome they became. Why, for instance, should a loving merciful God decide whether to let us into heaven based on whether we hold certain intellectual beliefs?

In high school, I participated in the catechism class here at this church, and was frustrated because it didn’t tackle deeper meanings or the kinds of questions I had. I wasn’t yet able to articulate all my thoughts and questions very well, and I probably had an overactive case of teenage cynicism. As a result, I didn’t get baptized like most others my age did. As I learned more about the early history of Judaism and Christianity, I only found more reasons to question the model of faith I thought I had received — the one in which Biblical literalism and a “divine guarantee” of sorts was key.

More recently, I gradually became aware that the model of Christianity I had in mind was one of many views. Christians, Mennonites, and even this church are incredibly diverse groups, and in retrospect, I am surprised that it took me so long to realize this. Three major steps led me to baptism.

The first step was the realization that, whatever our understanding of the literal accuracy of the Bible, literal truth is often inferior to metaphorical truth. As an example, many of us have read the works of Shakespeare. They are fictional, but the reason they have been revered for so many centuries is that they are true. They teach us things about ourselves and our world in a way that no history book can.

By placing such an emphasis on literal truth in the Bible, I was missing out on the message right there for me. By concerning myself with creationism or evolution, I missed out on thinking about what the story in Genesis meant for the Jews, and what it could mean for me. Evolution ceased to be a threat to religion; it became simply a tool for learning about a different sort of truth than we get from religion.

N. T. Wright mentions an incident that illustrates this point. A woman attending a religious conference asked the speaker, a prominent theologian, “Is it true that the serpent in Genesis actually spoke like you and I speak?” The theologian answered, “It doesn’t matter whether the serpent could speak. What matters is what the serpent said.”

The second step I took towards being baptized was realizing where the real core of Christianity lies. It’s not some debate about Genesis, but rather the death to one’s old self, and the rebirth and continual remaking of oneself in the example of Jesus. Given that, a lot of questions seem unimportant or even irrelevant. Good can be expressed in many ways, and if one person achieves a remarkably Christian life via a literal understanding of the Bible, and another via a more metaphorical reading, then it is my place only to affirm both and say that they both got it right. If we say that the task of remaking ourselves is like climbing a mountain, then what matters is not how we are climbing the mountain, but simply that we are climbing it.

I used to equate faith with an intellectual belief. I have come to see that was a narrow view. Faith, to me now, is more about vision: do we see the world like Christ did? Where do our loyalties and our trust lie — in God or in human institutions? What are our goals in life?

We humans have failed to understand God, and probably always will. I too share in that incomplete understanding, but I have come to accept that it is OK. I know enough to know that I want my loyalty to lie in Jesus, to know what kind of vision of the world I want to have, and I have learned to accept that intellectual questions can even be a form of meditation, enlightenment, and prayer.

The third step toward baptism was moving past my own shortcomings. For a long time, I thought I didn’t believe the right things, didn’t believe them strong enough, wasn’t certain enough about God, didn’t pray enough, didn’t read enough, didn’t understand enough, didn’t love enough — and ultimately, that I wasn’t good enough. I request baptism today understanding that, despite the various imperfections I still have, and we all have, everyone is good enough and deserving of love and peace. God’s love is for everyone. No exceptions!

Following Jesus boils down to this: I too must be an agent of love and peace to everyone, without exception. My vision must be centered around the fact that we are to emulate the God that loves the entire world, sinners all, and therefore I should as well. I hope I can show others the kind of love that has been shown me.

I hope, too, to share with this congregation and the global church in the vision of love. I hope that we can continually strive to re-focus on Christ’s vision. As an example, we all know that many different viewpoints about whether homosexuality is sinful exist in Mennonite Church USA. These opinions are deeply held and personal for many, and have been discussed over and over and over. But ultimately, they aren’t terribly relevant to the church’s mission. The example of Jesus unites us all: he embraced everyone. He accepted criminals and prostitutes and showed them love and kindness. Our difficult task, which is also my difficult task, is to show this same love to absolutely everyone, regardless of our varying opinions about them and their conduct. The forgotten and repressed of our day — perhaps criminals, homeless, Muslims, addicts, gays and lesbians, illegal immigrants, ethnic minorities, and still prostitutes — deserve the love of Christ’s church and all its members. My hope is that any one of the earth’s 7 billion people could step through the doors of our church, or any church, and immediately feel Christ’s love, and the unconditional, non-judgmental, welcome and love of Christ’s followers, no matter what.

This prayer from Ephesians 3 sums up my hope and my vision, for myself and everyone:

I pray that out of his glorious riches he may strengthen you with power through his Spirit in your inner being. I pray that Christ may dwell in your hearts through faith. And I pray that you, being rooted and established in love, may have power, together with all the Lord’s people, to grasp how wide and long and high and deep is the love of Christ. And I pray that you will know this love that surpasses knowledge—that you may be filled to the measure of all the fullness of God. (3.16-19)

This week started out with temperatures as high at 99F. Today we didn’t even reach 60, and tonight will be cold almost to freezing. It’s Kansas, so while we remark on the strange weather, we’re not particularly surprised by it.

This morning, Jacob and Oliver went over to play across the street. Jacob then went with Terah to a birthday party for one of his classmates, while Oliver helped plant some trees with his grandparents. I was at home this morning, which gave me a small but welcome break from the past week: we had something going on 6 out of the last 7 evenings, and little time to do anything else.

So I got out the Kindle and read from War and Peace. I started at the kitchen table while eating some granola, and then sat on the couch. The day was cold, windy, and overcast, and I like to use natural light, so there were only a few places to sit with enough light.

I sat there reading the account of Napoleon’s capture of Moscow in 1812, the characters lives uprooted by those events, and stories of violent mobs forming. And, of course, Tolstoy’s essays on the mistake of overstating the importance of famous figures in history were common. I am now about 3/4 through War and Peace, which means I’ve read 1,044 pages and have (only) 348 to go.

As I got up to try to find slippers or a blanket for my cold feet at one point, I remembered a conversation I had while eating lunch this week. I was reading on the Kindle while reading, and a person I knew happened to walk past. She asked what I was reading. I said simply, “War and Peace.” There was this stunned silence, so I added, “for fun. It’s great!” Which is true. She said, “Oooookay, John, you just enjoy that then” and went on her way. This isn’t the first time I’ve had a conversation like that about this book. It seems to be held up as some example of long, dry, literature that people are made to read but don’t want to. I think that only “long” is accurate here.

Anyhow, pretty soon Oliver arrived and, much to my surprise, said “Doh!” (meaning “no”) when I asked him if he was ready for lunch. Instead he stood there looking at me sweetly. Taking a guess, I said, “Oliver, would you like to watch train videos?” “Yeah yeah yeah yeah!” So we did. He particularly enjoyed the steam engines today, and eventually was ready for lunch at 12:30. Jacob and Terah got home just as he was finishing. Oliver got his nap, and I played with Jacob for a bit. Ever since he had heard of the British wedding in preschool, he’s been interested in weddings, so I showed him pictures from Terah and me at our wedding. This was a big hit. Then he went with Terah to decorate a Sunday School room at church, so I read a bit more Tolstoy.

Tolstoy’s characters sometimes are prone to not just changes in mood but changes in outlook and worldview — changing from cold, logical, and uncaring to warm and loving, or from despair at the vanity and shallowness of humanity to elation at the simple honesty of a peasant and renewed trust in the human spirit. At first I marveled at this tendency in certain characters, but on reflection, and in consideration of the timeline of the novel, which spans several years, it didn’t seem quite so strange. Things can happen to change the mood of individuals, or even a country; I can remember the national mood changing after 9/11, and again after Obama’s election, for instance. I also felt a contrast with some of the characters who, due to their decisions or circumstances, lost things that mattered while chasing things that weren’t as important. I felt fortunate to be at this place, with Jacob, Oliver, and Terah, and decided that we all ought to do something fun in the evening, but hadn’t quite figured out what.

And about that, the clouds started to break and Oliver woke up from his nap. I went up to his room, and much to his delight, played peekaboo with him for quite awhile. Eventually he was done with that game, but spotted my phone, so I helped guide his finger to launch Youtube and watch a train video over the wifi. This was certainly a Big Deal to him.

After Terah and Jacob got back, I figured it was time figure out something fun to do. And what could be more fun, I decided on the spur of the moment, than a drive into town to pick up some cans of Silly String for the boys, neither of which had ever used it before.

So we went to a restaurant, then bought the “surprise toy” (I hadn’t told them what it would be beforehand), then went to an ice cream place, and finally back home. Oliver loved it. He needed my help to press the nozzle to make it come out, but still had a great time and shrieked with delight.

Jacob then tried out his can, and stood in one place spraying his bright green strings all over the place. Then he took to running around the yard spraying string randomly in every direction until he finally completely emptied out the can. And after that, he gathered up some of the different colors of string and rolled them up into a ball which he was very proud of.

A nice day.

”I mourn the loss of thousands of precious lives, but I will not rejoice in the death of one, not even an enemy. Returning hate for hate multiplies hate, adding deeper darkness to a night already devoid of stars. Darkness cannot drive out darkness: only light can do that. Hate cannot drive out hate: only love can do that.”

–anonymous (often attributed to Martin Luther King, Jr.)

Seen many times today.

I said that I’d love to go back to Europe — enjoy some more history, have a fun adventure together, see more of the countryside perhaps. This didn’t sound all that relaxing to Terah. So I commented that Europe also has plenty of water, beaches, and resorts and perhaps we could find a place in Europe with history and adventure, plus also a relaxing beach — so we can have some adventure but not too much <grin>

We’ll probably be looking to travel in September or October and would hope for reasonably warm weather — though if we need to go at a different time of year, that option is still open.

I’m at a loss for where to start looking here, so I’m hoping readers of this blog can give a suggestion.

I very much appreciated the great tips last time I asked. Our first trip to Europe last year was great — it was nice to stay with some old friends and make some new ones, to see some amazing cities and buildings, and live for a few days someplace with a culture different from home.

Our boys would not be coming with us on this trip. We did get passports for them recently and are looking at some options for a trip with them, but that wouldn’t be this one.

Today was Terah’s day off. Initially I thought of taking the boys to The Cosmosphere, one of the world’s premier space museums, or to the Kansas Aviation Museum. But then I realized that the Midland Railway, a historic excursion line in Baldwin City, KS, was having their first train trip of the year. And it’s the Easter special: you take the train a few miles out of town, get off, the kids do an Easter egg hunt, then also get to get their picture taken with the Easter bunny. Kid paradise, right? What could be better?

So this is how my alarm came to be set for 5:15 on a Saturday morning. Oliver woke up early, but I had to get Jacob up. He was groggy, trying to cover his face and ears, and annoyed at me for trying to wake him up. Until I said, “Jacob, it’s time to get ready for the car trip to the train ride. Can you go get dressed for the day?” Silence. Then about 5 seconds later, he jumped up, said, “OK dad, I will do it,” and he did.

We drove our 2.5 hours. I had set out a non-dirty breakfast in the car for them the night before, so they munched while we drove. Then had to wait in a huge line for our tickets. An hour later we had them — got the last 3 tickets for the first departure of the day, 10AM — and ran to get on the train before it left. Jacob and Oliver both got very quiet. Sometimes Jacob especially does that when he’s very excited about things. He doesn’t always look it right then, but I knew I’d be hearing about it later. He had his face almost pressed to the window the whole way.

Oh, and yes, he dressed himself and I didn’t check to make sure he had the correct shirt, or that it wasn’t on inside-out, until it was too late. Oops.

We rode in an ex-CP car built in 1936. One of the cars on the train was built in the 1890s — quite some neat equipment.

Anyhow, next came the hunt. There were two large lawns covered in eggs, separated by age group. Jacob and Oliver were in the younger group, but still only got a few eggs (there were lots of other parents, uhm, “helping” their kids by picking up eggs the kids were too slow to get and putting them in baskets…) But they were thrilled with the experience so that’s what counts.

They moved the locomotive to the other end of the train for the trip back using an adjacent track. Lots of the kids watched them couple it back on. Jacob and Oliver were particularly interested — nothing could have distracted them from watching that train — though they didn’t particularly like the noise of the horn.

On the way back, someone in an Easter bunny suit was on the train for pictures. I took the boys there. They usually refuse to sit on laps of Santa and such, so I didn’t figure much success. They actually agreed to do it. And despite their expressions in this photo, Jacob later told me over and over how much fun it was to be with the Easter bunny.

After we got back, we headed for the car. Jacob insisted on carrying his own Easter bag, and predictably wound up carrying it upside down for awhile. So his bag had some grass in it from picking the candy back up. Oliver discovered some rocks by a ditch and had fun throwing them in to make a splash, so we paused for that for awhile. By this point it was about noon.

So at this point, what does a dad who has managed to keep two active boys in line for an hour, help them with an egg hunt and giant rabbit, and keep them from falling into the ditch they’re throwing rocks into, do? Say that it’s time to eat lunch and then head for home so we can be there in time for supper? Or go for MORE ADVENTURE?

I said, “Jacob, how would you like to eat lunch inside a big train station in Kansas City?” Pause for a couple of seconds to think. “YEAH! I have never had lunch in a train station before! Let’s go! Shall we go NOW?”

So I gave them the leftover breakfast to eat during the hour drive on towards the beautiful Kansas City Union Station. It’s one of the few grand American train stations that survives intact, and is just stunning.

Jacob and Oliver both fell asleep on the way there. We parked, walked inside, and Jacob just froze. “Dad… WOW! That ceiling is SO HIGH! I can’t even reach it!” He then raised his hand as high as it would go and said, “See dad, it is so high I can’t touch it!” (Most ceilings are, but this one is indeed much, much taller than most.)

We ate lunch in the second level of a restaurant right in the Grand Hall, which Jacob really enjoyed. Here’s Jacob, annoyed that I pulled out the camera again:

Then we went exploring for an hour. Jacob talked about his “adventure”. We went up the elevators and checked out the balconies. Then we found the “Model Railroad Experience.” And oh my. The boys were visibly the most excited they had been all day. Oliver just froze at the spot he walked through the door, not caring to move and see the rest of the exhibit because he was so fascinated by what he already saw. There was a huge layout in the middle of the room, and smaller ones in adjacent rooms. We finally made our way there, and Oliver just ran round and round one of them, following a train, excitedly yelling “There it is! There it is! There it is!” Here’s a photo from the big layout:

Jacob cried when it was time to go, but I had promised them some dessert before we left, so after a bite of ice cream we were on our way. On the way home, every 15 minutes or so, Jacob would make a remark like this:

“Dad, the train ride was SO fun!”

or

“Dad, I really liked Kansas City Union Station!”

and so forth, naming every highlight of the day for him several times over: the Easter bunny, the egg hunt, train ride, and everything about Union Station.

On the way home, we stopped at our favorite restaurant in Emporia: BobbyD’s Merchant St. BBQ. This is always a hit for everyone in the family. A down-home place that has excellent food, is not far off our path, and never seems to have a wait. Yum.

Oliver has really started to form up words. We can pick out words like lettuce, truck, book, look, and “dad John”. He also has a way of saying “there it is”, most particularly when looking at trains.

Jacob still loves trains. In church, he will take his markers, connect them end to end, and have a marker train that will keep him entertained for quite a long while. He will slide it back and forth on the pew, and eventually it will slide in a “tunnel” (behind my back). This actually started with “hymnbook trains” — get a few books in a row, and it’s a train. Anything can be a train to Jacob. Driving down the road, we might see what he calls tree trains, which are just trees in a row.

Last Sunday, the Kansas Mennonite Men’s Chorus had its first concert of the season. I sing with this group, which has nearly 300 men in it. It’s a lot of fun to be a part of that group, and the music is wonderful. Jacob and Oliver were there on Sunday, and both started to take more of an interest in the music that before. Jacob particularly loved the song Dry Bones, which had a number of odd percussion instruments in it. He was laughing at spots, and wanted to hear me sing it for the next several days. He even wanted Terah to read the program from the concert to him before his nap one day. Oliver enjoyed it too, and kept saying “dad John” (his word for me) whenever there was a soloist. I didn’t sing a solo, but apparently to 1-year-old eyes, 300 guys all dressed alike all look like dad.

Jacob and I recently have been reading Winnie the Pooh stories at his bedtime. Not the Disney books, but the original A. A. Milne stories. We read a few pages every night, and have made it to page 214 by now. Jacob loves it. And he’s becoming increasingly interested in reading things. Sometimes we’ve been unsure if he’s reading something or just had it memorized. Last night, I got out the book, and we were on a page he had never seen or heard before. He pointed to a word, and said, “It says ‘Roo’ there!” And indeed it did. Then he added, “It says ‘said Roo.’”. Yep. And then he read me the whole sentence:

“I love jumping,” said Roo.

I believe that’s the first time I’ve been certain Jacob has read me an entire sentence without having memorized any of it beforehand. And he won’t be old enough to start Kindergarten for another year and a half.

We then read the story, and when I came to the sentence Jacob had read to me, he interrupted, saying, “No, dad, I will read that one!” And he did.

I’ll start with church basements. Possibly a little chilly, but with a bunch of women getting together to make some quilts — along with some conversation, maybe some snacks.

Or perhaps about hundreds of kitchens throughout Kansas, warm with baking pies, cookies, breads, and all sorts of items. Then there are a bunch of wood shops, turning up sawdust, building things from toys to furniture. Or even a body shop donating its time and materials to put some finishing touches on a classic car.

I’ve also seen around 300 men getting together to practice for a men’s chorus concert. I know there was large crowd of people gathering to make verenike. People always wind up running 5km on a particular Saturday morning, rain or shine. Or ride 35 miles on a bicycle on a different Saturday. Or even help build a house using as much volunteer labor and donated materials as possible.

And then, of course, comes this week, where Mennonites from all over Kansas start to converge on Hutchinson, KS. I can imagine you might see some odd sights rolling down the road: a vehicle called the “borscht buggy” for preparing large quantities of the delicious soup. Several old tractors being hauled down the road on trailers. Semi loads of food. Vanloads of pie and cookies. Plants, trees, rugs, quilts, even a lawn mower. And then, yesterday and today, excited people ready to buy all of these things. Families with children excited to get a ride on the largest slide they’ve ever seen. And, of course, so many people wanting to eat the famous food that, despite the many parallel serving areas, lines still can extend for blocks.

It’s all because people are hungry.

Not these Kansans with pie-laden kitchens, though — it’s to help those in need.

It’s all part of the annual Kansas MCC Relief Sale. The idea is that people make, bake, build, sing, or give things to the sale. The items are then sold, and the proceeds go to Mennonite Central Committee, one of the world’s most efficient charities. MCC not only helps with directly bringing people out of hunger, but also supports sustainable projects, such as building hand-operated wells to give those in need a safe source of drinking water. Virtually all of the money you spend on those pies winds up helping someone in poverty.

Each year, the relief sale and related events raise around half a million dollars for charity.

It’s not unusual to see a quilt sell for thousands of dollars. Most of the things sell at more normal prices, but last year someone decided to add a loaf of bread to an auction — and it sold for $100.

And so it was this that we took the boys to today. We started with the “Feeding the Multitude” — a wonderful meal with some traditional Kansas Mennonite food. Here’s a photo:

That bread is zwieback, then there is cherry moos, bohne berrogi, verenike, and of course, sausage.

Here are Jacob and Oliver enjoying their food:

After that, we went over to the giant slide. I’m not sure how many stories tall it is, but it’s big enough that they have a strip of carpet there at the bottom to slow people down at the end. Here’s a photo of Jacob on my lap after going down the slide. If you zoom in, you can see the giant smile on his face.

After that, we hopped on the “mass transit” at the sale: a wagon being pulled by a tractor. We checked out the general auction, with Jacob wanting to be sure to peer under the hood of each car present. Then after a stop to buy some kettle corn for Jacob, we checked out the plants, quilt auction, and then on to buy cheese curds.

After another stop at the slide, it was time to head home.

And then tomorrow is a concert of the Kansas Mennonite Men’s Chorus (motto: We Sing That Others May Live). You get around 300 men on a stage signing together and wow — no recording can do it justice. It’s amazing to hear the power of the choir at the loud parts, but still more amazing to listen to 300 men signing as softly as they possibly can. I’ve been singing with that choir for 3 years, and we’ve been practicing for a few months now.

If you ever hear some Mennonites boasting about something, it’s probably going to be about how much money they’ve raised for charity. And on that note, I’ll nudge those Indiana folks reading this and point out that you have some catching up to do with the Kansas sale…

Today my car needed some work. This morning I dropped it off at the mechanic, who lives right off a well-traveled state highway. They don’t have a dropbox for keys because, well, they don’t need one. I left the keys in the car.

They called to say that they’d have to close early this afternoon, but not to worry — the keys were in the car along with the bill, which I could drop by and pay or put in the family’s box at church.

And sure enough, the car sat there the rest of the afternoon, and until about 9:45PM, with the keys in the ignition, bill tucked in under my ham radio speaker, ready to pick up.

And it wasn’t at risk for being stolen. Things just sort of happen that way out here sometimes.

I remember one time we were taking a train out of Newton, KS, which departs at about 3:30AM. The only other train through there departs at 3:00AM. And back then, there was sometimes no station agent on some days. We got there our usual 15 minutes before the train departed, and struck up a conversation with a couple from California. They were almost beside themselves with shock. First off, they had arrived an hour and a half early, being used to doing that for getting on planes and, apparently, trains.

But what really stunned them was the fact that the station was simply left unlocked all night. There weren’t any cops there, and in fact there was nobody there at all for most of the night. And yet it had no signs of graffiti, no vandalism, and no apparent concern from anybody that it was unlocked. I think they also felt unsafe, having no guards or officials present. And they were literally the only people in the station for an hour, I’m sure.

And sure enough, along came the train. We all heard the whistle and had our things ready to go by the time the conductor got off the train, walked into the station, announced it, and led us all out to it. 3 minutes later, we were all on our way to California and the land of, apparently, locked doors.

Back in 2002, when we bought a house in Kansas, the inspector commented that none of the outside locks worked. At closing, the sellers gave us a key, commenting “We never use it and it took forever to find it.” I guess that’s why they didn’t know the locks were broken.

There’s something good about a setting like this. I know that my car is safe at the mechanic with the keys in it. He knows that I’ll pay his bill. We all take pride in our community institutions. And besides, if anybody did try to spray some graffiti in the train station, they’ll be living with people that will remember it for the next 40 years. Accountability is implicit here.

So here’s why that made me happy. Well for one, it was the first time Jacob had left stuff on my computer that I found later. And of course he left his name there.

But moreover, he’s learning a bit about the Unix shell. sl is a command that displays an animated steam locomotive. I taught him how to use the semicolon to combine commands. So he has realized that he can combine calls to sl with the semicolon to get a series of a LOT of steam trains all at once. And was very excited about this discovery.

Also he likes how error messages start with the word “bash”.

When we planed the trip, we had no idea that the Purdue-Ohio State game was going to be such a big one. We walked over from the Union Club Hotel where we were staying on the Purdue campus. People were streaming towards Mackey Arena from all directions. Once inside, it was already loud and buzzing — people cheering, the band playing. I’ve never experienced anything quite so loud.

The game started out badly for Purdue; they were down a few points up front. The entire game was a close one, and the crowd sometimes got so loud that nothing else could be heard — not even the band or the announcer. When it became clear at the end that Purdue won the game, the people behind me — and apparently 12,000 others — began screaming at the top of their lungs. My hearing did eventually return to normal. So did my throat, which had gotten rather sore from from cheering myself.

This was our first road trip since I got my amateur radio license. I had a lot of fun visiting with people as we drove. I talked to a retired railroad engineer that used to take an amateur radio with him in his locomotive. As he went through a certain town where he had friends he liked to talk with on the radio, he’d get their attention by blowing CQ in Morse code with the train’s whistle. Some people in Kansas City had us laughing as we passed through. In Missouri, I talked with some farmers and a World War II vet. In Champaign, IL, I visited with a retired Unix systems administrator that had spent decades working with Unix operating systems.

Our hotel was connected to the Purdue Memorial Union, a large and historic building. Besides having some ice cream at the Sweet Shop one evening, we also spent a bit of time exploring it. I noticed that the Purdue Amateur Radio Club was having a testing session in there. We walked past once it was underway, and one of the students was not very busy. I introduced myself and asked if we could see their shack. It was neat to see all the equipment — some of it quite old — in the room that they must have been using for decades.

Terah, of course, had ideas for visiting a number of her favorite places while we were there. We visited Arni’s and Bruno’s, both pizza places. Bruno’s happens to be a Swiss pizza place, so much to my surprise, I had Wienerschnitzel there, which was excellent. We ate a (week late) Valentine’s dinner at Bistro 501.

On our way back, we also chatted with various people on the radio, though not quite as much. We got helpful suggestions for which route to take, and stopped at the excellent Bobby D’s Merchant St. BBQ in Emporia, KS for supper.

The boys had also enjoyed their weekend with grandparents, but were glad to see us back. They were particularly interested to see a lot of train videos from Youtube with me the next day.

• “5 years of Java experience required.”
• “3 years of Java experience with modules X, Y, Z required.”
• “6 years of experience administering Linux machines running RHEL 4 on a Windows 2000 domain with 1500 clients in an educational setting preferred.”

I could go on and on. As a job seeker, that sort of thing is fundamentally devaluing to someone who has strengths in being adaptable and quickly learning new tools, languages, or even entire environments. As an employer, it sends a message that you’re not interested in more than a surface look at someone’s strengths, and probably don’t care to hire the best and the brightest. After all, would you turn away a rockstar programmer simply because he or she had been writing filesystem code in C the last 3 years instead of the latest whizbang Java web widget that will probably be obsolete in a year and unsupported in two? I am quite certain that there are plenty of managers that do. Even if you are a company large enough to have an entire team of people that do nothing but work on that whizbang app, don’t you still want the best you can find, realizing that some of the best people to work on that app may not have even heard of it yet? (And that when the app goes obsolete in 5 years, you’d rather not have to lay off a large team of single-skill people)

Some of you may know that I work in IT at a manufacturing company. We have a small IT team here, about seven people, and are a heavy Debian shop. And we have a vacancy open up in our development/Linux admin group. I’m the manager of that group, which is why I’m thinking about this right now.

We’re too small for single-subject specialists to make sense, yet we’re big enough to appreciate skill, experience, flexibility, and rigor. Consequently, when the occasion arises for me to look for new employees, I don’t prepare a laundry list of things we use in-house and would like experience with.

The list of almost-required things generally begins with “Linux” and ends with “experienced”, and has nothing else in between. In other words, I’d like it if I don’t have to explain to you what a symlink or a hardlink is, but I’d be willing to do so if I think you’d internalize it quickly. On the “experienced” side, it would be nice if you already have a well-developed sense of fear at running rm when you’re root, or have designed a storage infrastructure for a network before, or are paranoid about security. But again, if people can pick up those traits on the job, we are usually still interested. If learning how to package up software for Debian, fix bugs in software you’ve never seen in a language you’ve never heard of, raise good questions about things you may not have lots of experience with, and write documentation for it all on a wiki sounds like fun, then that’s probably the kind of person I want, even if you’ve never used our particular tools before.

If I were to judge based on the stuff I normally see in job postings, I guess you might conclude I’m nuts. I don’t think I am, but then again I’m also the only person I know that formats his own resume in hand-crafted LaTeX. What do you all think?

The next question is: how should one evaluate candidates given this sort of philosophy? I’m not a fan of canned tests, or even “whiteboard tests” that tend to be some sort of canned topic that may test the applicant’s specific knowledge base more than overall skill and flexibility. Similarly, as an applicant in years past, I’ve struggled with how to present the “I’ve never used $LANGUAGE, but I know I could pick it up quickly and do it very well” vibe. To certain people, that might sound like BS. To the more geeky managers, perhaps it sounds like what they want.

We’ve built a fairly diverse team on the back of this approach, and it’s worked out well for us so far. I’m interested to hear your thoughts.

Oh, and if you’d like to work for us, you should probably be sending me an email. No, I’m not going to list the address here on this blog post. If you can’t figure it out, I don’t want to hear from you <grin>

That’s my car. You might understand why I went home at 1PM Tuesday and worked from home the rest of the afternoon. The county didn’t get the roads opened until 5:30PM Wednesday, so we were stuck. Terah’s dad kindly opened the driveway for us though. Without that, we wouldn’t have even known where it was.

Jacob and Oliver loved playing in the snow. It was a HUGE hit for them. Both of them hated to go back inside when it was time. Oliver had trouble walking, or even standing up, but that didn’t stop him from begging for more every chance he got.

On Saturday, I took Jacob outside to play. We had in mind building a snow tunnel. We went to one of the big piles of snow left by the tractor after our driveway was cleared, and I dug out a space underneath it that Jacob could crawl through. He got a little scoop (he insisted on calling it “my shovel”) and helped dig too. He had a great time with it. He was very anxious to go inside, and it probably didn’t help his eagerness that the cats could freely walk through it long before he could. They were around and taking a great interest in the proceedings. One of then stretched out on my back while I was stretched out pulling snow out from the pile.

Finally Jacob got to go through. He had a little trouble because it wasn’t tall enough for him to crawl like he usually does (though still plenty big for him to get through with lots of room to spare), but he came out beaming. I then went and got the camera and got some photos of his second trip through.

You’d think that this might be the highlight of a 4-year-old’s snow excursion. And given his excitement, you might be very convinced of that. But you’d be wrong.

What Jacob really wanted was a snow train. So I drew some train track-sized lines in the snow with my shovel. Then I was supposed to draw a train engine. He had been planning this for hours before we even went outside, and I knew very well what was supposed to go on the engine: a snowball for a headlight. We then added passenger cars — and a coal car, because apparently that’s what you do with snow trains. And the result:

When we went inside, Terah asked Jacob about his snow experience. Jacob was very happy, until she asked him if his snow train was a real train or a drawing of a train. Then he became very sad when he said it was a drawing of a train, and decided it was not what he wanted AT ALL.

So today we went out again. This time I built a snow train that stands up, complete with snowballs for wheels. I asked him if it was right, and he kept saying, “No, I want a REAL train.” Finally I asked him what the difference between a real snow train and this one was. Apparently a real train would be on tracks, move around, make noise, and take us places. Fortunately he didn’t dwell too long when I explained I couldn’t build one of those.

And then today’s highlight was writing JACOB in the snow. He had started by drawing larger and larger J letters with his arm, and finally asked me to make one. I made footprints in the shape of a very large J, which he loved. He had me make another one, and then the rest of the letters in his name.

As I look to us moving some things to the cloud, and shifting offsite backups from carrying tapes to a bank to backups via the Internet, I’m aware that the potential for mischief — whether intentional or not — is magnified. With cloud hosting, a person could, with the press of a button, wipe out the equivalent of racks of machines in a few seconds. With disk-based local and Internet-based offsite backups, the potential for malicious behavior may be magnified; someone could pretty quickly wipe out local and remote backups.

Add to that the mysterious fact that many enterprise-targeted services allow only a single username/password for an account, and make no provision for ACLs to delegate permissions to others. Even Rackspace Cloud has this problem, as do their JungleDisk backup product, and many, many other offsite backup products. Amazon AWS seems to be the only real exception to this rule, and their ACL support is more than a little complicated.

So one of the questions we will have to address is the balance of who has these passwords. Too many people and the probability of trouble, intentional or not, rises. Too few and productivity is harmed, and potentially also the ability to restore. (If only one person has the password, and that person is unavailable, company data may be as well.) The company does have some storage locations, including locked vaults and safe deposit boxes, that no IT people have access to. I am thinking that putting a record of passwords in those locations may be a good first step, as putting the passwords in the control of those that can’t use them seems a reasonable step.

But we’ve been thinking of this as it pertains to our local systems as well. We have, for a number of years now, assigned a unique root password to every server. These passwords are then stored in a password-management tool, encrypted with a master password, and stored on a shared filesystem. Everyone in the department therefore can access every password.

Many places where I worked used this scheme, or some variant of it. The general idea was that if root on one machine was compromised and the attacker got root’s password, it would prevent the person from being able to just try that password on the other servers on the network and achieve a greater level of intrusion.

However, the drawback is that we now have more servers than anyone can really remember the passwords for. So many people are just leaving the password tool running. Moreover, while the attack described above is still possible, these days I worry more about automated intrusion attempts that most likely won’t try that attack vector.

A couple of ways we could go may include using a single root password everywhere, or a small set of root passwords. Another option may be to not log in to root accounts at all — possibly even disabling their password — and requiring the use of user accounts plus sudo. This hasn’t been practical to date. We don’t want to make a dependency on LDAP from a bunch of machines just to be able to use root, and we haven’t been using a tool such as puppet or cfengine to manage this stuff. Using such a tool is on our roadmap and could let us manage that approach more easily. But this approach has risks too. One is that if user accounts can get to root on many machines, then we’re not really more secure than a standard root password. Second is that it makes it more difficult to detect and enforce password expiration and systematic password changes.

I’m curious what approaches other people are taking on this.

2010 started with a 3-day ordeal of taking out the trash, nearly snapping the hood off my pickup while closing it, and Greek poetry. I then dived right into reading the Odyssey, and my post suggesting we read the Old Testament of the Bible like we read Greek mythology generated almost 100 comments. I guess I said something controversial…

I also read a number of other books last year.

Our family had quite the year. In March, Terah and I got to visit Europe for the first time. A few months later, we visited New York City for the first time and I attended Debconf10. Jacob predictably loved our train trip to NYC, calling our room in the sleeper “the best place of ever.”

Terah had thyroid surgery, and Jacob survived a tractor accident with amazingly minor injuries.

But as I look back at my posts from last year, I’m struck that most of them were about more everyday, ordinary events of life. Jacob loved the switchbox he and I built, Oliver somtimes acts like Yoda, and they both loved camping. I have made probably way too many posts with photos of them last year, but I’m making no promises to slow down this year!

Jacob and I built a computer together, and he loves bashing random words at the Linux shell.

He also developed an interest in some half-broken FRS radios, and had tons of fun with them. Eventually this kindled in interest in amatuer radio in me.

At the end of 2009, I finally completed my computer science degree and mused about what I might do next. I’m a curious, inquisitive person that enjoys both challenge and knowledge. I thought that perhaps I would dive into reading more, or perhaps take some more college classes. Turns out I instead picked up amateur radio — one of the big unexpected twists of the year. Jacob really did kindle that interest in me, and by July I had passed my technician and general exams and was on the air. I continued to study, and passed my extra class exam in October. Since then, I’ve rigged up my bicycle for amateur radio operation, made some fun contacts, set up outdoor antennas, and got going with digital operation. It has been a lot of fun, and will keep providing opportunities for a challenge for me for a very long time.

Also on the technical side, I made my archive of Gopherspace available for download.

It’s hard to cut off the list — I have dozens of other things I could mention — but I suppose I should end it with the Christmas gathering / piano moving episode.

As I’ve been writing about backups, I’ve gone ahead and run some tests with rdiff-backup. I have been using rdiff-backup personally for many years now — probably since 2002, when I packaged it up for Debian. It’s a nice, stable system, but I always like to look at other options for things every so often.

rdiff-backup stores an uncompressed current mirror of the filesystem, similar to rsync. History is achieved by the use of compressed backwards binary deltas generated by rdiff (using the rsync algorithm). So, you can restore the current copy very easily — a simple cp will do if you don’t need to preserve permissions. rdiff-backup restores previous copies by applying all necessary binary deltas to generate the previous version.

Things I like about rdiff-backup:

1. Bandwidth-efficient
2. Reasonably space-efficient, especially where history is concerned
3. Easily scriptable and nice CLI
4. Unlike tools such as duplicity, there is no need to periodically run full backups — old backups can be deleted without impacting the ability to restore more current backups

Things I don’t like about it:

1. Speed. It can be really slow. Deleting 3 months’ worth of old history takes hours. It has to unlink vast numbers of files — and that’s pretty much it, but it does it really slowly. Restores, backups, etc. are all slow as well. Even just getting a list of your increment sizes so you’d know how much space would be saved can take a very long time.
2. The current backup copy is stored without any kind of compression, which is not at all space-efficient
3. It creates vast numbers of little files that take forever to delete or summarize

So I thought I would examine how efficient ZFS would be. I wrote a script that would replay the rdiff-backup history — first it would rsync the current copy onto the ZFS filesystem and make a ZFS snapshot. Then each previous version was processed by my script (rdiff-backup’s files are sufficiently standard that a shell script can process them), and a ZFS snapshot created after each. This lets me directly compare the space used by rdiff-backup to that used by ZFS using actual history.

I enabled gzip-3 compression and block dedup in ZFS.

My backups were nearly 1TB in size and the amount of space I had available for ZFS was roughly 600GB, so I couldn’t test all of them. As it happened, I tested the ones that were the worst-case scenario for ZFS: my photos, music collection, etc. These files had very little duplication and very little compressibility. Plus a backup of my regular server that was reasonably compressible.

The total size of the data backed up with rdiff-backup was 583 GB. With ZFS, this came to 498GB. My dedup ratio on this was only 1.05 (meaning 5% or 25GB saved). The compression ratio was 1.12 (60GB saved). The combined ratio was 1.17 (85GB saved). Interestingly 498 + 85 = 583.

Remember that the data under test here was mostly a worst-case scenario for ZFS. It would probably have done better had I had the time to throw the rest of my dataset at it (such as the 60GB backup of my iPod, which would have mostly deduplicated with the backup of my music server).

One problem with ZFS is that dedup is very memory-hungry. This is common knowledge and it is advertised that you need to use roughly 2GB of RAM per TB of disk when using dedup. I don’t have quite that much to dedicate to it, so ZFS got VERY slow and thrashed the disk a lot after the ARC grew to about 300MB. I found some tweakables in zfsrc and the zfs command that let me tweak the ARC cache to grow bigger. But the machine in question only has 2GB RAM, and is doing lots of other things as well, so this barely improved anything. Note that this dedup RAM requirement is not out of line with what is expected from these sorts of solutions.

Even if I got absolutely stellar dedup ratio of 2:1, that would get me at most 1TB. The cost of buying a 1TB disk is less than the cost of upgrading my system to 4GB RAM, so dedup isn’t worth it here.

I think the lesson is: think carefully about where dedup makes sense. If you’re storing a bunch of nearly-identical virtual machine images — the sort of canonical use case for this — go for it. A general fileserver — well, maybe you should just add more disk instead of more RAM.

Then that raises the question: if I don’t need dedup from ZFS, do I bother with it at all, or just use ext4 and LVM snapshots? I think ZFS still makes sense, given its built-in support for compression and very fast snapshots — LVM snapshots are known to cause serious degradation to write performance once enabled, which ZFS doesn’t.

So I plan to switch my backups to use ZFS. A few observations on this:

1. Some testing suggests that the time to delete a few months of old snapshots will be a minute or two with ZFS compared to hours with rdiff-backup.
2. ZFS has shown itself to be more space-efficient than rdiff-backup, even without dedup enabled.
3. There are clear performance and convenience wins with ZFS.

Backup Scripts

So now comes the question of backup scripts. rsync is obviously a pretty nice choice here — and if used with –inplace perhaps even will play friendly with ZFS snapshots even if dedup is off. But let’s say I’m backing up a few machines at home, or perhaps dozens at work. There is a need to automate all of this. Specifically, there’s a need to:

1. Provide scheduling, making sure that we don’t hammer the server with 30 clients all at once
2. Provide for “run before” jobs to do things like snapshot databases
3. Be silent on success and scream loudly via emails to administrators on any kind of error… and keep backing up other systems when there is an error
4. Create snapshots and provide an automated way to remove old snapshots (or mount them for reading, as ZFS-fuse doesn’t support the .zfs snapshot directory yet)

To date I haven’t found anything that looks suitable. I found a shell script system called rsbackup that does a large part of this, but something about using a script whose homepage is a forum makes me less than 100% confident.

On the securing the backups front, rsync comes with a good-looking rrsync script (inexplicably installed under /usr/share/doc/rsync/scripts instead of /usr/bin on Debian) that can help secure the SSH authorization. GNU rush also looks like a useful restricted shell.

]]> http://changelog.complete.org/archives/5572-rdiff-backup-zfs-and-rsync-scripts/feed 17 http://changelog.complete.org/archives/5547-research-on-deduplicating-disk-based-and-cloud-backups http://changelog.complete.org/archives/5547-research-on-deduplicating-disk-based-and-cloud-backups#comments Thu, 20 Jan 2011 17:16:25 +0000 John Goerzen http://changelog.complete.org/?p=5547 Yesterday, I wrote about backing up to the cloud. I specifically was looking at cloud backup services. I’ve been looking into various options there, but also various options for disk-based backups. I’d like to have both onsite and offsite backups, so both types of backup are needed. Also, it is useful to think about how the two types of backups can be combined with minimal overhead.

For the onsite backups, I’d want to see:

1. Preservation of ownership, permissions, etc.
2. Preservation of symlinks and hardlinks
3. Space-efficient representation of changes — ideally binary deltas or block-level deduplication
4. Ease of restoring
5. Support for backing up Linux and Windows machines

Deduplicating Filesystems for Local Storage

Although I initially thought of block-level deduplicating file systems as something to use for offsite backups, they could also make an excellent choice for onsite disk-based backups.

rsync-based dedup backups

One way to use them would be to simply rsync data to them each night. Since copies are essentially free, we could do (or use some optimized version of) cp -r current snapshot/2011-01-20 or some such to save off historic backups. Moreover, we’d get dedup both across and within machines. And, many of these can use filesystem-level compression.

The real upshot of this is that the entire history of the backups can be browsed as a mounted filesystem. It would be fast and easy to find files, especially when users call about that file that they deleted at some point in the past but they don’t remember when, exactly what it was called, or exactly where it was stored. We can do a lot more with find and grep to locate these things than we could do with tools in Bacula (or any other backup program) restore console. Since it is a real mounted filesystem, we could also do fun things like make tarballs of it at will, zip parts up, scp them back to the file server, whatever. We could potentially even give users direct access to their files to restore things they need for themselves.

The downside of this approach is that rsync can’t store all the permissions unless it’s running as root on the system. Wrappers such as rdup around rsync could help with that. Another downside is that there isn’t a central scheduling/statistics service. We wouldn’t want the backup system to be hammered by 20 servers trying to send it data at once. So there’d be an element of rolling our own scripts, though not too bad. I’d have preferred not to authorize a backup server with root-level access to dozens of machines, but may be inescapable in this instance.

Bacula and dedup

The other alternative I thought of system such as Bacula with disk-based “volumes”. A Bacula volume is normally a tape, but Bacula can just write them to disk files. This lets us use the powerful Bacula scheduling engine, logging service, pre-backup and post-backup jobs, etc. Normally this would be an egregious waste of disk space. Bacula, like most tape-heritage programs, will write out an entire new copy of a file if even one byte changes. I had thought that I could let block-level dedupe reduce the storage size of Bacula volumes, but after looking at the Bacula block format spec, this won’t be possible as each block will have timestamps and such in it.

The good things about this setup revolve around using the central Bacula director. We need only install bacula-fd on each server to be backed up, and it has a fairly limited set of things it can do. Bacula already has built-in support for defining simple or complicated retention policies. Its director will email us if there is a problem with anything. And its logs and catalog are already extensive and enable us to easily find out things such as how long backups take, how much space they consume, etc. And it backs up Windows machines intelligently and comprehensively in addition to POSIX ones.

The downsides are, of course, that we don’t have all the features we’d get from having the entire history on the filesystem all at once, and far less efficient use of space. Not only that, but recovering from a disaster would require a more extensive bootstrapping process.

A hybrid option may be possible: automatically unpacking bacula backups after they’ve run onto the local filesystem. Dedupe should ensure this doesn’t take additional space — if the Bacula blocksize aligns with the filesystem blocksize. This is certainly not a given however. It may also make sense to use Bacula for Windows and rsync/rdup for Linux systems.

This seems, however, rather wasteful and useless.

Evaluation of deduplicating filesystems

I set up and tested three deduplicating filesystems available for Linux: S3QL, SDFS, and zfs-fuse. I did not examine lessfs. I ran a similar set of tests for each:

1. Copy /usr/bin into the fs with tar -cpf - /usr/bin | tar -xvpf - -C /mnt/testfs
2. Run commands to sync/flush the disk cache. Evaluate time and disk used at this point.
3. Rerun the tar command, putting the contents into a slightly different path in the test filesystem. This should consume very little additional space since the files will have already been there. This will validate that dedupe works as expected, and provide a hint about its efficiency.
4. Make a tarball of both directories from the dedup filesystem, writing it to /dev/zero (to test read performance)

I did not attempt to flush read caches during this, but I did flush write caches. The test system has 8GB RAM, 5GB of which was free or in use by a cache. The CPU is a Core2 6420 at 2.13GHz. The filesystems which created files atop an existing filesystem had ext4 mounted noatime beneath them. ZFS was mounted on an LVM LV. I also benchmarked native performance on ext4 as a baseline. The data set consists of 3232 files and 516MB. It contains hardlinks and symlinks.

Here are my results. Please note the comments below as SDFS could not accurately complete the test.

Test	ext4	S3QL	SDFS	zfs-fuse
First copy	1.59s	6m20s	2m2s	0m25s
Sync/Flush	8.0s	1m1s	0s	0s
Second copy+sync	N/A	0m48s	1m48s	0m24s
Disk usage after 1st copy	516MB	156MB	791MB	201MB
Disk usage after 2nd copy	N/A	157MB	823MB	208MB
Make tarball	0.2s	1m1s	2m22s	0m54s
Max RAM usage	N/A	150MB	350MB	153MB
Compression	none	lzma	none	gzip-2

It should be mentioned that these tests pretty much ruled out SDFS. SDFS doesn’t appear to support local compression, and it severely bloated the data store, which was much larger than the original data. Moreover, it permitted any user to create and modify files, even if the permissions bits said that the user couldn’t. tar gave many errors unpacking symlinks onto the SDFS filesystem, and du -s on the result threw up errors as well. Besides that, I noted that find found 10 fewer files than in my source data. Between the huge memory consumption, the data integrity concerns, and inefficient disk storage, SDFS is out of the running for this project.

S3QL is optimized for storage to S3, though it can also store its files locally or on an sftp server — a nice touch. I suspect part of its performance problem stems from being designed for network backends, and using slow compression algorithms. S3QL worked fine, however, and produced no problems. Creating a checkpoint using s3qlcp (faster than cp since it doesn’t have to read the data from the store) took 16s.

zfs-fuse appears to be the most-used ZFS implementation on Linux at the moment. I set up a 2GB ZFS pool for this test, and set dedupe=on and compress=gzip-2. When I evaluated compression in the past, I hadn’t looked at lzjb. I found a blog post comparing lzjb to the gzip options supported by zfs and wound up using gzip-2 for this test.

ZFS really shone here. Compared to S3QL, it took 25s instead of over 6 minutes to copy the data over — and took only 28% more space. I suspect that if I selected gzip -9 compression it would have been closer both in time and space to S3QL. But creating a ZFS snapshot was nearly instantaneous. Although ZFS-fuse probably doesn’t have as many users as ZFS on Solaris, still it is available in Debian, and has a good backing behind it. I feel safer using it than I do using S3QL. So I think ZFS wins this comparison.

I spent quite some time testing ZFS snapshots, which are instantaneous. (Incidentally, ZFS-fuse can’t mount them directly as documented, so you create a clone of the snapshot and mount that.) They worked out as well as could be hoped. Due to dedupe, even deleting and recreating the entire content of the original filesystem resulted in less than 1MB additional storage used. I also tested creating multiple filesystems in the zpool, and confirmed that dedupe even works between filesystems.

Incidentally — wow, ZFS has a ton of awesome features. I see why you OpenSolaris people kept looking at us Linux folks with a sneer now. Only our project hasn’t been killed by a new corporate overlord, so guess that maybe didn’t work out so well for you… <grin>.

The Cloud Tie-In

This discussion leaves another discussion: what to do about offsite backups? Assuming for the moment that I want to back them up over the Internet to some sort of cloud storage facility, there are about 3 options:

1. Get an Amazon EC2 instance with EBS storage and rsync files to it. Perhaps run ZFS on that thing.
2. Use a filesystem that can efficiently store data in S3 or Cloud Files (S3QL is the only contender here)
3. Use a third-party backup product (JungleDisk appears to be the leading option)

There is something to be said for using a different tool for offsite backups — if there is some tool-level issue, that could be helpful.

One of the nice things about JungleDisk is that bandwidth is free, and disk is the same $0.15/GB-mo that RackSpace normally charges. JungleDisk also does block-level dedup, and has a central management interface. This all spells “nice” for us.

The only remaining question would be whether to just use JungleDisk to back up the backup server, or to put it on each individual machine as well. If it just backs up the backup server, then administrative burdens are lower; we can back everything there up by default and just not worry about it. On the other hand, if there is a problem with our main backups, we could be really stuck. So I’d say I’m leaning towards ZFS plus some sort of rsync solution and JungleDisk for offsite.

I had two people suggest CrashPlan Pro on my blog. It looks interesting, but is a very closed product which makes me nervous. I like using standard tools and formats — gives me more peace of mind, control, and recovery options. CrashPlan Pro supports multiple destinations and says that they do cloud hosting, but don’t list pricing anywhere. So I’ll probably not mess with it.

I’m still very interested in what comments people may have on all this. Let me know!

]]> http://changelog.complete.org/archives/5547-research-on-deduplicating-disk-based-and-cloud-backups/feed 18