I just read the absolutely hilarious account of a call to AT&T asking them to install a telegraph line. The even funnier part is that nobody at AT&T seemed to realize that he wasn’t talking about a telephone. Or even knew what AT&T stands for.
While at ZUG, I also read the very funny story about impersonating Michael Jackson by getting a credit card in his name, complete with al sorts of media coverage.
I’ve been managing servers professionally for some years now. Support is one of the most important things when you are managing computers for work. You don’t need support to help you out with a printing problem or an e-mail problem. You need support because every minute the machine fails to power up, your company may lose twice the value of the entire machine. Or even more.
My first day job managing servers involved Dell hardware. What a nightmare. I’ve never had a good experience with Dell support, ever.
First off, Dell support never puts me straight through to an intelligent support rep. I don’t care whether I get to the Indian call center or someone in Texas. The first support person I speak to at Dell has less computer aptitude than my grandmother. One conversation I will always remember went like this:
Me: We have a disk in our array that went bad on our Linux server. The red light on the disk itself is on. Please send us a replacement.
Dell: Have you tried rebooting?
Me: No. This is a production server. The reason the disks are redundant is so I don’t HAVE to. Besides, the light ON THE MACHINE ITSELF is on.
Dell: Ah, OK. Have you run scandisk?
Me: No. This is a Linux server, as I told you, and scandisk wouldn’t see a problem anyway since this disk is in an array and the array is still up.
Dell: OK, great. How about you download the diagnostics .exe from our website…
Me: Can’t. We don’t have Windows on this machine. You did say you support Linux when we bought it.
Dell: Ah. Can you right-click on My Computer…
Me: NO. This runs Linux, and the BAD DISK LIGHT ON THE MACHINE IS ON.
Dell: Ah, OK. I wonder if the problem really is that you have a bad disk.
Me: Could be!
Our first HP server purchase happened to be at a time when HP had undercut Dell by several thousand dollars. I liked the hardware, but it wasn’t anything that much more special than Dell.
But what I really like is the support. I haven’t had to call HP support often, but when I do, I am almost always speaking to a live, experienced person within 5 minutes.
With only one exception, all the HP support people I’ve talked to have been very experienced. They all sound like they’ve been working with HP hardware since the late punched card era. They know what is going on and assume that I do too. The HP people don’t make small talk (it *really* ticks me off when someone obviously in Calcutta or something asks me about the weather in Kansas, because you *know* they are reading it off a screen and don’t care). But that’s fine. I’m not calling them to talk about the weather, I’m calling them because my server is down.
We had a bad disk in an array on a HP server once. That conversation went more like this.
HP: Server support. Serial number please?
Me: [gives it to them]
HP: OK. What’s the problem?
Me: The array dropped a disk. The failed disk light is on and the controller logged a disk error.
HP: OK. That’s a 146GB SCSI, right? 15KRPM or 10K?
Me: 15K.
HP: OK. Is 1PM tomorrow good to send out the replacement?
Me: Fine.
HP: OK, your case number is xxxxx. Can you give us directions to your location?
Me: Sure…
So recently we got in our MSL4048 tape library. A very nice unit. And faster than most *disks*. 48 Ultrium3 tapes — 400GB native each — very nice. And a barcode reader built in.
So anyway, one small problem. When you open up a magazine to put tapes in, you can close the unit back up. It says “scanning”, but it doesn’t notice that we’ve changed tapes until we give it a command on the operator panel (yes, this tape drive has a LCD screen built in). This can be worked around, but is annoying and is just waiting to cause confusion. Plus it’s not how it should work.
So I call HP support yesterday.
Turns out this MSL4048 is a brand-new unit. Had only been on the market a few weeks. Our support rep has never seen one or taken calls about one, and they haven’t even given him all the HP technical docs yet. But no matter, he is willing to try to help us out.
He calls me back twice yesterday with tips and questions after speaking to colleagues. He asks intelligent questions, doesn’t bother with the “are you sure you’re putting the tapes in the right way around” or the “is the power cord securely plugged in” crap, and generally doesn’t waste my time. He called me back about four times more today — they duplicated our setup in their lab, right down to the exact firmware version, but didn’t have the problem. Two of those callbacks were apologizing for taking so long, and explaining that they were learning about this machine as they went along. So a HP rep will be out to our location shortly.
Now THAT’S what I call service. No blaming it on someone else, no trying to make me do stupid troubleshooting things, and returning calls.
My *one* bad experience with HP was one time we put a new internal tape drive in the machine, and it was acting flaky. I got the only not-very-experienced HP rep I ever had spoken to that time, and they tried to blame Debian for what turned out to be a bad SCSI cable. (The symptoms weren’t very similar to what I’d expect for a bad SCSI cable, and the cable had been working fine.) Oh HP, you donate to Debian — why don’t you support your hardware under it?
(In fairness, that is the ONLY time they have flinched when I said I run Debian, though it does make them hesitate sometimes)
Lately we’ve been looking at backup solutions at work.
And I’ve got to say that Bacula is looking downright awesome. It’s GPL’d and it has just about every feature a person could ask for.
I am a complete Bacula newbie. Today, after using Bacula for a total of about 30-60 minutes, I added the first client machine to my Linux test box. The client machine was running the Windows bacula client. It took about 10 minutes to install and configure the client and the server. And both backup and restore worked perfectly the first time. Nice. Setting up a *nix client is even easier.
I’ve been using Amanda for many years at home and at various workplaces. Looks like we’re going to be switching.
We’ve also ordered an HP MSL4048, a 48-tape LTO3 library with barcode support. Each tape has a native storage capacity of 400GB. Should be nice when it arrives. With that library and Bacula, we should be able to back up all our servers using a single backup system. And both our Windows and Unix people can manage the system, including running restores to any machine, from any authorized console machine.
At long last, I’ve finally updated Debian From Scratch (DFS). For those of you not familiar with DFS, it’s a single, full rescue CD capable of working with all major filesystems, LVM, software RAID, and even compiling a new kernel. The DFS ISO images also contain a small Debian mirror subset that lets you use cdebootstrap, along with the other utilities on the CD, to perform a manual, “Gentoo-like” installation. It also serves as an excellent rescue CD, with a full compliment of filesystem tools, backup/restore software, and a development environment complete enough to build your own kernels.
DFS also refers to dfsbuild, the tool that generates DFS images. dfsbuild is available as a Debian package. dfsbuild is designed to make it trivial to build your own custom DFS images. You can have your own set of Debian packages on your images, your own kernels, etc. Unlike many other systems, you can go from the example dfs.cfg to a customized DFS build in just a few minutes, even if you’ve never used dfsbuild before.
Version 0.99.0 is a from-scratch rewrite and port to Haskell. You can read the full list of new features in the announcement, but the biggest is that it now supports standard Debian initramfs kernels in addition to ones that have enough drivers statically linked to be able to read the CD-ROM.
You can also download my DFS images or browse the docs online.
I made a shocking discovery today.
Terah called me in from outside to inform me that the washing machine was just sitting there. No action. Just stalled in the middle of the cycle. I poked around a bit, and sure enough. It’s stuck.
I then turned on the water faucet next to the washing machine. Nothing came out. So naturally I thought: the terrorists have attacked our water supply! Run for the hills! (which in Kansas are a 3-hour drive)
(Actually, what I thought was we want to put this house on the market next week, and the last thing we need now is a lack of running water. But I’m sure if I watched the cable news channels, I’d think of terrorists first.)
Now before I can explain how SUICIDE ANTS were the cause of our water problems, we need to take a small diversion into the fascinating realm of rural plumbing. When you have a well supplying water to your house, there’s a pump connected to a pressure tank, which is then connected to the house’s plumbing. When you turn on a faucet, you drain water out of the tank. When the pressure reaches a certain minimum, a pressure switch turns on the pump. The pump pumps water into the tank until the pressure reaches a maximum, at which point the pressure switch turns the pump back off.
Now, inside the pressure switch are little metal plates. To turn on the pump, the plates snap together, completing the circuit. To turn off the pump, the plates snap apart.
So, back to our story.
I went out to the well house (the small outdoor structure that houses the tank and switch) with my usual well-adjusting tools: pliers, screwdriver, gloves. I open up the well house and cut the power. I take the cover off the pressure switch and stare at it for a minute.
Hmmm I think to myself. The system pressure is 0 PSI, so the pump should have been running… And indeed the pressure switch plates had snapped together….
…or had they?
No! There was — and I am not making this up — a cake of DEAD, PARTIALLY SINGED ANTS between the metal plates. In fact, there were ants crawling around all over the pressure switch. So I used my screwdriver to scrape the plates off as well as I could. I couldn’t get them perfectly clean because of the angle, but I figured it’d be enough to make contact.
I turned the power back on. Little blue sparks appeared at the pressure switch (burning up the last of the SUICIDE ANTS), and then half a second later, the pump sprang into action.
Now this raises some alarming questions:
Why didn’t our government do anything to stop the SUICIDE ANTS from attacking our water supply?
And more importantly, who is behind this sudden sinister interest in pressure switches on the part of our hill-building friends?
I can see the headlines now:
“Ants attack water supply in Kansas… Is New York far behind?”
“Gitmo Intelligence Links Suicide Ants to bin Laden”
“Price of Raid Hits $200 A Barrel Amid Fears of Ant Attacks”
“5 Troops Deployed With Cans of RAID to Terrorist Ant Training Camps in the Pakistani Mountains”
“Thirsty Latchkey Children Forced To Scrape Dangerous Terrorist Ants From High-Voltage Switches Before Using Faucet”
“Is Your Water Safe? Watch the story about the latest threat tonight at 11”
“Small-town Kansas man detained for posting instructions for attacking rural water systems using ants”
I’ve posted some pictures of Grandpa’s yard. Follow the link (and note the second page).
Here’s a sample:
So just when I thought the whole Tuttle story had wound down, more starts appearing.
The Register has a new article: Only “freaks” waste their time with Linux in Oklahoma. In it, they posted a video clip from an Oklahoma TV station that called Tuttle “an international laughing stock.” Quite a funny little piece. The Register also mentioned that the Tuttle city manager left work when he heard a TV crew would be there to interview him. Favorite quote:
Instead of a website, Taylor discovered an Apache server configuration page that mentioned CentOS. He’s an Oklahoma man, and the Apache feather must have proved frightening.
Today’s next little piece of fun is this page at the Tuttle Times. Look on the right, under Links. See the link that reads “Example of CentOS test page”? Notice where it points to. Yes, that’s right, www.tuttlechamber.org. Click on it, and sure enough, up comes the infamous test page. So the Tuttle Chamber of Commerce apparently is getting ready for round two…
And while we’re speaking of the Tuttle Times: they have published a letter to the editor from Johnny Hughes, the CentOS lead developer that was on the other end of those exchanges with Tuttle. I think Johnny wrote an excellent little piece there.
And finally, another funny headline from the Tuttle Times homepage: “Prosecution attorney presents complete case.” I am glad to hear that at long last this terrible scourge of incomplete cases has been brought to an end. But beware, this shocking story is not over yet! The next article is “defense raises compelling questions.” Another first for Tuttle?
Seen this morning on the Wordsmith list by Anu Garg:
In late seventeenth century, William III of UK imposed a window tax, levied on each window in a house.
Three hundred years later, William III of US imposed a Windows tax, levied on each personal computer manufactured, whether it had Windows or not.
Aigars Mahinovs has an interesting post about doing a Lessig-style presentation in which most slides that go with the presentation have just a single word. He describes it as a highlighter for your speech.
The London Telegraph’s article: If everything is killing us, why do we live so long? It’s an interesting look at why some people get so concerned about health, and other’s don’t. And it seems plenty relevant on this side of the pond, too.
So remember Jerry Taylor, the man from Tuttle, OK that threatened to call the FBI on a Linux vendor because an unrelated hosting company had misconfigured Apache?
Well, this story is just getting funnier and funnier.
First off is this story from the Tuttle Times. It basically repeats Taylor’s view that the CentOS people were expected to help him with his problem, and that he was somehow entitled to their help. But there are some funny tidbits in the story:
“Phone calls from across the country started coming in to the newspaper and city offices, and e-mails from Switzerland, Australia, Wales and England were received. Many of the web sites discussing the exchange are in foreign languages.” I hate to break it to you, but Australia, Wales, and England all speak English.
(Ok, so they did switch from talking about email to web, but it still sounds funny.)
“In their search to find out more, web surfers discovered that the Tuttle Times online forums were hacked, and theorized that it was in retribution for the e-mails. Those forums, however, were corrupted several months ago, and the newspaper’s now former web hosts did not repair it after numerous requests. New forums should be available at the Times site in the coming weeks.” I’m so glad to know that your forums were merely corrupted for months and not hacked. Sounds like the IT problems in this town extend well past the city building.
Taylor said: “[CentOS is] a free operating system that this guy gives away, which tells you how much time he’s got on his hands.” Grumble.
Quoting Tuttle Mayor Paxton (trying to say what was more important than this): “We have issues with sewer. People here want better park facilities. They want a library.“ I think this has just validated every stereotype people have about Oklahoma.
Tuttle is more than 7 times larger than my hometown in Kansas, and yet my hometown has had a library for years.
Jerry Taylor also reported having 500 e-mails and numerous phone messages when he arrived to work Monday.
There’s a new blurb on the Wikipedia page about Tuttle about all this. And in their talk page about the now-deleted article on Jerry Taylor, one person wrote: “Mr. Taylor’s actions have coined a new term of art “Tuttled”, in reference to the invocation of criminal consequences by one who is ignorant of the true situation. Since this is now a part of the English vernacular the story behind the term should be explained to give it an historical context. It is no longer about the action of a single person and an attempt to publicly vilify him, it is about a world-wide common experience of dealing with a Kafka-esque minor government official who, through ignorance, creates problems far beyond their normal sphere of influence. The page should be returned to the public.”
The Register has two new stories about it. The first reports that Taylor has been interviewed by all sorts of media and says that he did the right thing. The second, Linux conquered, Tuttle man takes on London is a story about the grandson of the namesake of Tuttle, OK — who happens to be the current US ambassador to the UK. This person is refusing to pay the regular London car fees. The mayor of London said: “It would actually be quite nice if the American ambassador in Britain could pay the charge like everybody else and not skive out of it like some chiseling little crook.”
And finally, there’s some incredibly funny photoshop work on this one over at fark.com (you have to scroll down a ways). Also, this comment:
“A small-town American politician wants a British newspaper to turn off the Internet.
Say that to yourself a few times. Please.”