Now I’m a little annoyed at HP

Hardware, ,

So, a little while ago, I wrote about why I like HP. This week, I’m starting to be annoyed at them.

My employer just bought nearly $100,000 worth of HP hardware. We get a new MSA1500cs Fibre Channel SAN (with redundant controllers, FC switches, disks, etc), a new blade enclosure system, three blades to start with (all of them, at minimum, dual dual-core Opterons with 4GB RAM, and some considerably more), a rack to put all this in, etc.

So we’re starting to set all this stuff up. I’ve got Debian installed on an NFS root for testing the blades and how they interact with the SAN.

The blades have an integrated dual-port QLogic QLA2312 Fibre Channel adapter. The Linux kernel has a built-in driver for this (qla2xxx), which detects it and, so far at least, works fine. We want to run kernel 2.6.17 because it’s the first version where XFS has decent semantics for write ordering to prevent corruption after a power failure. Plus we want at least a 2.6.16.x kernel because we want to run the latest Xen 3.0 on these blades. (Live migration of virtual servers from blade to blade — this will be great.)

But we learn that HP does not support the kernel qla2xxx driver. HP does not say WHY they don’t support it, just that their own driver is the only one that they support.

After plowing through several annoying scripts to get to their driver, I realize why it fails to install: it is OLD. At BEST, 2.6.14 is the most recent kernel it would even compile against (release date: October 2005), and I think the most recent version it supports is more like 2.6.8 (almost TWO YEARS OLD now). They reference a whole bunch of kernel symbols and macros that were removed somewhere between 2.6.8 and 2.6.17.

I sent a ticket to HP support. Their first request was to run their system information gathering tool and send them the results. Fine, that’s reasonable. I did so. Next they say, gee, you’re running Debian, and we don’t support that.

Argh…. If they tried to compile it against 2.6.17.1 on RedHat or SuSE, they’d get the exact same problem. I told them what symbols they were erroneously using, and a simple grep would have showed them that.

Besides, how many customers are going to be pleased with no upgrade path available for 2 years? I wouldn’t want our kernel version to be held hostage to HP’s slow driver development process.

Sigh.

Things are a happenin’

General,

So… Our big move (yes, we’re buying a farm) is moving along!

We’re tentatively set to close on it next Tuesday. The last two months have been busy. A lot of phone calls to arrange permits.

The director of planning and zoning retired recently, and they didn’t have a replacement ready. So all building permits were on hold until a replacement arrives. In the meantime, the county clerk’s office was trying to handle basic inquiries. I called it one day, and didn’t realize until later that I was speaking to the county clerk. I grew up in this county but obviously I’ve lived in larger ones too long if I’m surprised about this. Everyone was very friendly and as helpful as possible with an unfortunate situation.

I got a surprise phone call today. Apparently the telephone company is running new lines in rural areas of the county (upgrading to support DSL) and they wanted to know if that place was going to stay vacant. A “not at all!” later, and it looks like we’ll be able to get DSL by the time we move in. WOOHOO! We were bracing for dialup or wireless…

There are still some bits of paperwork that have to get done before Tuesday yet, so it’s not 100% guaranteed that we’ll close then. But we’re getting excited anyway!

New hosting provider: VPSLink

Reviews, Technology, , ,

Thanks to all of the helpful comments people left in response to my last post about hosting. I got some really helpful hints from them — including the one I eventually followed, to VPSLink.

I wound up going with VPSLink. Their prices are amazing and the performance is good, too.

I have to admit — I shot myself in the foot not once, not twice, but three times. The shorewall config I was using on my old VPS (which was hosted under UML) apparently doesn’t work well under OpenVZ (used at VPSLink). By “doesn’t work well”, I mean “blocks all traffic to or from the host on startup”.

So, I had rsynced over everything from my old host to the new, and rebooted the new. But it didn’t come back up. I was pretty sure this was why. I dropped off a ticket to the VPSLink folks asking them to please rm /etc/init.d/shorewall for me.

60 minutes later, they had done it. (And it looks like a bug in their ticket system prevented it from being flagged as “emergency” — they said they would have done it faster otherwise.)

I then tried to fix shorewall, and it looked like it was working, so I put the init script back and rebooted. Same problem! They fixed it again in about 60 minutes. (The ticket still had normal priority)

Finally, I deleted shorewall entirely, then rsynced my old host to the new one. Things looked good, so I rebooted…. and yes, guess what, that rsync brought back shorewall so it got hosed YET AGAIN. ARGH. This was now well into the overnight hours, but they still helped me out.

All in all, I am so far VERY impressed with VPSLink. I heartily recommend them. I’ll be sure to post updates as time goes by.

Regarding Memset, my current provider: Really great company. I heartily recommend them, too. My needs for RAM just wouldn’t be ecnomical under their current plans, but they are usually competitive. I also have experience with JohnCompanies. I used them way back when they only did FreeBSD VPSs, on through the Linux days, and we currently use their services at work. These are also great folks and I would recommend them to anyone. Like Mako’s suggestion of Rimuhost, both memset and JohnCompanies are “quality first” providers — not necessarily cheapest, but their systems work as advertised and are almost never down, and they support you with experienced Unix admins. BTW, Memset’s current plans use Xen and JohnCompanies uses Virtuozzo.

Disappointed with Dreamhost

Technology, , ,

I’ve been using Memset for many years to host my websites, Darcs repositories, etc. They provide virtual private servers. I’ve been happy with them overall, but as my RAM needs increase, things have been getting slower. As adding RAM to VPS plans is expensive no matter who the provider is, I decided to look for some other options.

I decided to try Dreamhost. They are a web hosting company with a clue. I don’t get root on the box, but they do give me the next best thing. ssh access, a decent suite of preinstalled packages, and clueful admins. I signed up earlier this week and started rsyncing some things over. They have a $10/mo plan which gives 1TB of monthly transfer and 20GB of disk space. Much nicer than the going rate of about $40 for 40GB of monthly transfer and 5GB of disk space for a VPS.

The web side of things worked fairly smoothly, and the system seemed to perform well. Until today, that is. I noticed my site being extremely SLOW this morning. I logged into the system to see if it was some problem with my configuration. It wasn’t, but the system load was — get this — over 100. I watched it for awhile and then fired off a support request. The load had been hovering between 20 and 80. It’s now 9.5 hours later, load is hovering between 11 and 30, things are still somewhat slow, and I’ve not heard back from them.

Then, later today, their server started refusing connections on port 80. Turns out this was a sitewide outage, and seems to have lasted for 30-60 minutes. I signed up to receive an e-mail when it was fixed, but I never got it.

Also, their e-mail hosting has never worked for me. At first, it was bouncing my mail. I sent a support request about that, and the next morning they said they fixed it. Well, it was no longer bouncing mail, but it wasn’t actually *delivering* it either. I told them that this morning, and still haven’t heard back from them.

Now, with my memset account, I can’t remember the last time I’ve had downtime. I’ve had slowness — rivaling what dreamhost is having — but actual downtime is exceedingly rare. And they seem to respond to trouble more quickly.

And the scary thing is that from all I’ve read, Dreamhost is really one of the best web hosting companies out there.

So, what do you all think? Should I continue going the “do it myself” route and host my site with a VPS (either with Memset or with JohnCompanies, another clueful host I’ve dealt with)? Or try to stick it out with Dreamhost? Or are there other clueful web hosting companies out there?

Right now, I’m of the “if you want it done right, do it yourself” mindset. Being able to save money, and also time by trusting someone else to keep the host patched and working only saves me angst if I can actually trust them to do that. Right now, I’m not so sure I can trust Dreamhost — or any other web hosting company.

How to solve “The following packages cannot be authenticated”

Debian, , ,

Users of Debian’s testing or unstable distributions may be noticing messages from apt saying things like:

WARNING: The following packages cannot be authenticated!
  foo bar baz
Install these packages without verification [y/N]?

I noticed today that google doesn’t turn up good hits for the fix. The fix is really simple:

apt-get install debian-archive-keyring
apt-get update

That’s it. You now have secure packages from Debian. Nice, eh?

An iPod under Linux

Hardware, Music, Software, ,

I finally purchased my first iPod: a black 60GB iPod video model. I had been holding off for years. The iPod sounded nifty, but I just didn’t quite go there.

The thing that finally won me over was the camera connector. It lets you plug your iPod directly in to a digital camera. The iPod can download photos from the camera to its internal disk without the need for a PC. Very slick.

So anyway, we got the iPod and the camera adapter at the Apple store in Cambridge — a quick subway ride from Usenix. They were out of stock on the FM tuner, so I ordered that online.

The next step was to get the iPod working with Linux. I currently have it working with both music and video. Here’s how I did it.

Today’s Grammar Rant

General

I have read a lot of statements like this lately:

“CNN are reporting…”

“The Free Software Foundation are encouraging…”

Slashdot is a terrible offender, but I’ve been seeing it elsewhere as well.

So what’s wrong with this? CNN is an organization. Singular. The correct construction is “CNN is reporting…” Your subject is singular, so your verb should be singular as well.

“The employees of CNN are reporting” is also correct.

I are annoyed each time I read something like this. Is you annoyed too?

First Thougts on Xen

Software

At work, we’ve been using vserver for virtualization for some years now. Due to various reasons, we’re looking at Xen.

I’ve been trying to switch my workstation to use Xen. I’ve enountered a few issues so far. Probably these will go away as I learn the system.

Overall, my greatest gripe is the documentation. It is outdated and just plain wrong far too often. For instance, there’s a place where it says to run “make ARCH=xen xconfig”, but the Xen kernel patches don’t (any more, at least) provide a xen arch.

The next gripe is the very weird kernel build system. Xen doesn’t ship a diff against a kernel tree. They instead ship whole files to extract atop a particular kernel version. Annoying and unwieldy. There is a command to generate a diff, but you have to download the full kernel tree first.

A couple of other gripes: There is little documentation on memory management (can Xen adjust the RAM usage of running VMs?), on 64-bit systems (can you run a 32-bit kernel under a 64-bit hypervisor? how about a 64-bit kernel that supports 32-bit userspace?)

I’m also having trouble with my forcedeth card locking up under Xen.

However, I’ve heard of lots of people having good luck with it so I’m going to keep trying.

But one would think that basic docs could be actually worked on a bit more.

Announcing HSH, the Haskell Shell

Programming

Following the “release early, release often” motto, I am happy to announce version 0.1.0 of HSH, the Haskell shell.

You may obtain it with:

darcs get --tag 0.1.0 http://darcs.complete.org/hsh

Things are still very rough in many ways, but this version already lets you:

  • Run commands
  • Pipe things between commands
  • Pipe command input/output into and out of pure Haskell functions
  • Pure Haskell functions are as much a first-class citizen as is grep or cat

Here is an example session: (some lines wrapped for readability)

$ ghci -fglasgow-exts HSH

*HSH> run $ ("ls", ["."])
COPYING    HSH        HSH.hs    TODO    announcements  testsrc
COPYRIGHT  HSH.cabal  Makefile  _darcs  test.hs

*HSH> run $ ("ls", ["-l"]) -|- ("wc", ["-l"])
12

*HSH> :m +Text.Printf
*HSH Text.Printf> let countLines = (zipWith (\i line -> printf "%-5d %s" i line) 
       [(1::Int)..])::([String] -> [String])

*HSH Text.Printf> run $ ("ls", ["-l"]) -|- countLines -|- ("grep", ["hs$"])
6     -rw-r--r-- 1 jgoerzen jgoerzen  1285 Jun  6 09:43 HSH.hs
11    -rw-r--r-- 1 jgoerzen jgoerzen   565 Jun  6 09:43 test.hs

*HSH Text.Printf> :m +Data.List
*HSH Text.Printf Data.List> run $ ("ls", ["-l"]) -|- countLines -|- 
         filter (isSuffixOf "hs")
6     -rw-r--r-- 1 jgoerzen jgoerzen  1285 Jun  6 09:43 HSH.hs
11    -rw-r--r-- 1 jgoerzen jgoerzen   565 Jun  6 09:43 test.hs

*HSH Text.Printf Data.List> run $ ("ls", ["-l"]) -|- countLines -|- filter (isSuffixOf "hs") 
       -|- ("tr", ["a-z", "A-Z"])
6     -RW-R--R-- 1 JGOERZEN JGOERZEN  1285 JUN  6 09:43 HSH.HS
11    -RW-R--R-- 1 JGOERZEN JGOERZEN   565 JUN  6 09:43 TEST.HS

*HSH Text.Printf Data.List> let generator = \(_::String) -> unlines . map show $ [1..20]
*HSH Text.Printf Data.List> generator ""
"1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n"
*HSH Text.Printf Data.List> run $ generator -|- ("grep", ["1"])
1
10
11
12
13
14
15
16
17
18
19

Future versions will likely simplify syntax to make it easier to write scripts and introduce a sh to hsh converter. I also plan to add pure Haskell tools for some common shell-ish things that one could do in Haskell.

Usenix

Technology

Well, we’re back from the Usenix ’06 Annual Technical Conference. Overall, a very nice conference. This is the first Linux/Unix conference I’ve been to since the old LinuxExpo (NOT LinuxWorld) events in the late 90s.

I must say it was quite fun to be around so many intelligent *nix folks. I learned quite a bit, and learned how much I had in common with some (and how little with others). A very nice social at the New England Aquarium Friday night.

Some very interesting speakers, too, with lots of experience to share. I’m glad I was there.

I have only one main complaint about Usenix — they tried to cram too much into too few days. The conference sessions ran till as late at 6:45PM, with BoFs after that. That left almost no time to see anything of Boston and meant that very few people could actually stick around for all the BoFs. Next year, they should use more days and schedule fewer things in the evenings.

The highlight of the trip, though, wasn’t Usenix at all, but instead getting to meet some Boston-area Debian folks Saturday night. Lots of fun, and Mako is as crazy in person as he looks online :-)

Our train trip in each direction was nice, too.

More to come…