It’s time to change locations for my server. I’ve been with CoreNetworks for almost 5 years. They provide a good value, with fully dedicated servers a lot cheaper than most places, and good support to boot — targeting people that can handle root on their own box. I’m switching, though, to Hetzner Online (of Germany), primarily because I am needing more than 1GB RAM, and they can provide a 2GB box for less than I pay now for 1GB. I asked for feedback on Google+, and got positive feedback. Today, I migrated from one machine to another, copying a dozen or two GB of data, and only had about 15 minutes downtime.

I was able to do this without console access to the source machine, though console access wasn’t really required for either. This will work with any basic Linux install on the destination, or it could be prepared directly from a rescue environment.

Before I explain how I did it, I thought it was interesting to think of the different places that the machine that was known as complete.org has lived over the years. Here’s a map (click for detail):

Here’s the general process.

First, well in advance of the move, crank the TTL on the domains way down. This way, old IPs won’t be cached for very long once the system moves.

Next, start rsyncing data from the old machine to the new. Do not yet shut down daemons on the old. Shut down as much as you can on the new. You’ll want to focus on static data, such as /home. /usr is a good candidate as well. /var if you are selective — databases may be a good candidate, or may not. The idea is to seed the destination with data so that when we do the “real” rsync, most of the data will be there; it will have to tidy things up after daemons are stopped, and update some things, but the bulk of the work should be done.

Next, start preparing some exclude lists for the final rsync, which will copy an entire machine to the other. You will likely want to exclude files such as /etc/fstab, /etc/network/interfaces, /etc/resolv.conf, /etc/hosts, /etc/default/grub, /sys/*, /proc/*, /dev/*, and the like. I used rsync -v -P -a -H -A -X -S –numeric-ids –delete-after.

Now, start editing config files for the new IP, but do it on the old server (these will be synced over to the new one). Start with the bind config files, and touch anything else that needs it — maybe Apache configs, whatever.

Next, get ready to do the final sync. In an ideal environment, we’d just shut both machines down to single-user mode, but that’s not going to quite cut it here. Use ps and shut down all daemons except sshd and udev on the destination. Yes, including even syslog. This is to prevent anything actively accessing the disk during the rsync, and also to prevent any issues with clients accessing server daemons that aren’t ready yet.

Now, on the source machine, reload bind. This will start answering DNS queries with the new machine’s IP, and should also propagate the changes to your secondary DNS. Next, kill off all the daemons on the source machine, except for ssh and bind. You might want to set /etc/nologin to prevent regular users from logging in on ssh, if you have them. Now rsync things over to the new machine. Do any final tweaking over there (merging in /etc/hosts maybe, dealing with the udev rules.d persistent net thing, etc.). Reboot the new server and you should be up and running.

I would have probably dismissed as not realistic enough for even The Daily Show this kind of story, had someone suggested it a few months ago:

Rupert Murdoch’s corporation (owner of FOX News, Wall Street Journal, News of the World, etc) would have been found to have illegally accessed other people’s voicemails.

These included voicemails from a cellphone belonging to a recently-murdered girl, which interfered with the police.

And they had also bribed Scotland Yard officers for information, and actively covered it up.

The story would close one of Britain’s biggest newspapers, and force the resignation of various government and Scotland Yard officials.

That it would lead to the first high-profile investigation under the Foreign Corrupt Practices Act of an American corporation bribing British officials.

That one of the central figures in the scandal would suddenly die at a young age.

That there was a question of how the former editor of Murdoch’s newspaper — whom he steadfastly defended until one day he didn’t — would be taken from jail to testify before Parliament.

That CNN would live stream a hearing of a British Parliamentary subcommittee instead of the news conference given by the President of the United States.

And that during this committee hearing, some guy would attack Rupert Murdoch — who, until then, looked like he had a few drinks too many the night before — with a shaving cream pie.

And that Rupert Murdoch’s newspapers and TV channels would portray News Corp. as a victim of the liberal press in all this, and that they should just be left alone.

So maybe the last one was believable, but my goodness. Just when it couldn’t get any stranger, a SHAVING CREAM PIE?!

For those of you that aren’t familiar, please read about the Eternal September, which refers to the lasting damage to a community when vast numbers of newbies suddenly enter and dramatically harm the quality and usefulness if discussions.

• Email: 24 years
• Usenet: 13 years
• Facebook: 2 years
• Google Plus: ?

Seems like everybody’s writing Google+ reviews these days, but most of the ones I’ve found seem to be fluff pieces. I haven’t used it long, but have some initial impressions to share.

First, an analogy. Facebook reminds me of the sleazy guy selling stuff from his car down the street. They do things like change privacy defaults when they think it’s good for Facebook, rather than good for you.

Google+ reminds me of an Apple product. It’s beautiful, easy to learn, but locks everything down and is lacking some critical features. So here goes with the details.

The good

I think that this can be best summed up by: it’s not Facebook. Facebook has, perhaps, set an incredibly low bar but still, this holds. Facebook is the only website I can ever remember using that changes things so much, so often, and so completely that I keep having this feeling of not knowing how to do things. Try maintaining a Facebook page or two and you’ll especially feel my pain then. But even the basics: how many times have I accidentally posted a partial comment because I pressed Enter to start a new paragraph (that has only been the button used for that for, hmm, let’s see now, decades), and instead it posted the comment. Principle of least surprise, anyone?

Google+ has a simple and, on the surface anyway, intuitive interface. However, it does get muddy; more on that below.

I’m going to spend a lot more space on “the bad” below, but don’t let that diminish my excitement about having a viable alternative to Facebook. I am keen to get rid of that monster.

The Bad

I’ll start of this section with the fact that Google+ is a tightly-controlled walled garden. There is no way to take a copy of your status updates, comments, etc. and back them up with your own devices. If your Google account goes away, so do all your updates about your kids’ first words. Facebook does have this feature these days. It has broken half the time, but it exists, works, and I use it. Twitter doesn’t have the feature built in, but its API makes it pretty easy; you could easily use my twidge program to do this, for instance.

But whatever you put in G+ stays in G+. There is no autoposting it to twitter or Facebook, no backups, nothing. I’m disappointed in that, given Google’s prior attitude about openness to individual data ownership. (Update 7/12/2011: There is a way to get a backup, which I hadn’t noticed; see comments below.)

Almost as bad, there’s also no way to get data in. So I now have identi.ca, twitter, Facebook, and now also G+ accounts. When I post an item in identi.ca, it autoposts to twitter, and then it autoposts from twitter to Facebook. Handy – one place to notify people of my public activities. My uploads to Flickr and blog posts also auto-post to Facebook, so people I’m friends with there get a picture of what I’m doing outside of Facebook without me manually having to link it in.

Not so with G+. There is no way to even add an RSS feed for my blog to auto-post to G+ as there is with Facebook. If you use G+, the only way to get stuff into it is with a keyboard.

Poor integration with other Google apps is also an issue. G+ is nowhere near Google’s first social app. They also have Blogger, Google Reader, Buzz, Gmail, etc. So here’s the rub. These aren’t integrated well, and when they have tried to integrate them, they’ve done an exceedingly poor job of it.

As an example: Buzz is a twitter-like service for posting updates. Very similar to what G+ does, right? Well, the best you can do is link your Buzz account to a separate tab on your profile. You can’t even have your Buzz updates flow straight into G+, from what I can see. You have a separate list of followers in Buzz from G+. Reader is even worse; it has a sharing feature, but it’s not G+ aware, so it goes to a reader inbox.

But what takes the cake is the integration with Gmail Contacts. Sounds easy? Not entirely. When you add someone to G+, apparently it sometimes links the record with their entry in Contacts. And the help gives you the handy warning that when you remove them from G+, you have the option of removing them from Contacts, which could impact what you see on a mobile device. It is unclear what happens when it doesn’t link the record, or how it decides which record to link (some people have more than one person in a family sharing phone numbers and email addresses, for instance.)

One very nice feature of G+ is you can add email addresses, even if they aren’t G+ members. Then when you post updates to the circle you’ve added them to, G+ offers to send them an email with an update. Very handy.

Except the undocumented part is that if you select “Your circles” (meaning all your circles), rather than individually ticking the box by each circle, they don’t get the email. And they also don’t get it if you set it Public. You have to select their specific circle, which is a significant difference from how people with G+ accounts are treated.

And that’s just an example. There are a ton of things that work one way 95% of the time, but have non-obvious exceptions. Some of these exceptions are documented in the help, and some aren’t.

I’m still confused about the integration with Picasa. G+ help says that you can upload unlimited numbers of photos for free (though they will be downscaled at a certain point). Picasa has definite storage limits. But G+ uploads are showing up in Picasa. Do I really get free storage by uploading to the same place via a different tool? Very odd.

And finally, a gripe about web standards. I’ve been using Firefox/Iceweasel 3.5, which came out in Debian squeeze and is still supported there. It is also pretty darn new by corporate IT standards. And Google Plus refuses to let me log in with it, saying it’s “incompatible.” Perhaps it can’t do some fancy animation, but then again I don’t really care. Seems Google has forgotten the old RFC adage: be liberal in what you accept and conservative in what you generate. If you can detect that my browser won’t animate something right, then you could give me a stripped-down version of the page rather than an error message.

I’m just back from spending a few days volunteering with the tornado recovery effort in Joplin, MO.

The biggest image that remains in my mind is of the first time I saw a person picking through a large pile of rubble. The person was standing on top of what used to be a house. Now it was a pile of wood, glass, carpet, siding, and roofing material. I’m sure there was hope for finding some treasure or other — maybe a photo album or videos of children. In any case, it made me feel so lucky, even unfairly lucky, to have not had to go through that.

This scene was repeated several times, but mostly the houses that devastated appeared abandoned by the time we were there, now two weeks after the event.

But I heard stories, and lots of them. The victims of the storm, who were perhaps trying to rebuild that part of their house that got smashed by a tree or a pickup, or trying to get their intact belongings out before abandoning the house, or whatever, were mostly surprisingly upbeat. They were working out in 95-degree heat, many without electricity, running water, or sewer service. Almost every person I met that suffered a loss from the tornado wanted to tell their story. Many also told of their plans for the future, which were full of hope and even upbeat. These were people doing a hard job in terrible conditions and still showing hope.

Another testament to the disaster was the most unusual set of vehicles you’ve ever seen parked at any hotel you can care to think of, for at least an hour-and-a-half radius in the direction I came from. Besides the usual cars and minivans, there might be FEMA vehicles, electric company trucks, Red Cross vans, construction trucks of every kind, police and law enforcement from all over, etc. There was quite obviously an influx of people helping out in Joplin.

My primary task there was to provide communications support for the effort as an amateur radio operator. Amateurs (or “hams”) are something of a volunteer first responder of sorts during times of crisis; most of us own and are very familiar with operating equipment that can communicate over very long distances without the need for any on-site infrastructure. Amateur radio was the only method of communication for some Joplin hospitals in the immediate aftermath.

The communications emergency is over, but the response isn’t. I was assigned to work with the Salvation Army. They were doing a lot of things in Joplin, and had hundreds of volunteers working with them. I don’t think I even know what many of them were doing – I do know they had set up several warehouses across the city working with donating clothing, food, etc.

The part I was involved with was primarily the canteen operation. The SA sent in food service trucks from several parts of the country. These trucks would roam up and down the streets in the damaged area, trying to get past every single street several times a day. Anybody that we could see would be offered food and water. No strings attached, no questions asked. This included homeowners, electric line workers, construction crews, sanitation workers, and quite a few nonprofit groups that sent well-meaning and useful volunteers into the area but didn’t think to provide them with a large supply of water due to sending them into an area without any. Oops. In any case, with extreme heat and no running water, conditions were dangerous. The canteens also knew of certain at-risk families that were living in homes that were mostly intact in the disaster areas, and made a special point to check in on them. They also generally looked to make sure that people looked like they were healthy. Each canteen also had a counselor on board that would visit with people while we quickly prepared their meal — they all seemed to welcome that.

Amateur Radio’s Role

The operation of this size had quite a logistics challenge. I’d hear of things like an unexpected need of 70 lunches, or a semi showing up with donations before there was space, or an unexpected but very welcome donation of a large quantity of ice cream without a place to store it (so the canteen trucks, which have freezers, needed to pick it up quickly). That’s where us hams came in. Each canteen had an amateur radio operator on board. Each major location also had a ham stationed there, and the head of operations also often had a “shadow” — a ham that would follow him around wherever he went to relay messages back and forth. We also had hams with pickups (with radios in them, of course) that could transport things around the city to places that needed them, hams at headquarters managing all the communication and generally investigating questions that didn’t have immediately obvious answers, etc.

Radios were used instead of cellphones for a few reasons. One big one is that everybody on the operation can hear what everyone else’s needs are, since it’s a group communications situation rather than one-to-one. It’s easy to give a general alert to everyone (“come get your ice cream now please!”) and people that have suggestions can chime in. This came in extremely handy more than once. Also, it frees the people doing other jobs from having to spend time chasing someone’s voicemail, finding phone numbers, etc; that gets delegated to us in some cases. I heard from the head of canteen operations, for whom this was the first disaster he’d worked that had amateur radio support, how wonderful it was to have this going on. I also heard a secondhand report that some police officers that were also amateurs had listened to our operation and reported that “we sound more professional than 911 dispatch and do a better job.”

On Sunday I was assigned to a canteen. This meant I didn’t have a lot of radio traffic to pass, so although I had it in my ear all day, I wasn’t actively talking on the radio very much. So I rode in back, helping hand out water, carry meals to people, and so forth.

On Saturday, I was the shadow for the head of operations. That was a difficult task, because he barely ever moved at a pace slower than a run, sometimes would abruptly zip out somewhere, etc. But it was also enlightening and vital. He was a real “get it done” sort of guy, and was the key to quite a few things. Having someone available to relay questions to and from him was a good thing.

And today I worked as a transport person and at headquarters. Due to not having a pickup there, I didn’t actually get called on much to transport things, but in general between jobs the whole time I’d act as a runner if needed, or simply try to figure out the details of how things were run for next time. I wound up taking “net control” (being the control operator at headquarters, and generally managing communications so that people don’t talk over each other and such) for about an hour. So I got to do a little of just about every amateur radio task.

Thoughts

I am thankful for the opportunity to go, and the good feeling of helping people in need — the first I’ve ever had the chance to do that in a disaster. It’s a good feeling to have a skill that is useful and appreciated. Sometimes it felt like handing out food and water is something pretty small in the scheme of things. But on the other hand, it gives people a chance to have contact with someone that cares, an opportunity to have people that can notice problems drive by a few times a day, and an opportunity to help meet people’s basic needs. And sometimes in a fluid situation, there might be more volunteers than are needed, so I did spend some time sitting waiting for the next task.

But overall, I’m convinced that the work I helped facilitate was a good thing and provided a good and needed service in Joplin. This has been quite the experience and I’m sure it’s changed me too, though I don’t yet know how.