This evening, I released HDBC 1.0.0.
HDBC is a database API for Haskell. I wrote it after being unsatisfied with HSQL. HDBC at a certain level feels similar to Perl’s DBI. But it is both simpler and more powerful, IMHO, thanks in large measure to Haskell language features.
The HDBC homepage is here.
So, a little while ago, I wrote about why I like HP. This week, I’m starting to be annoyed at them.
My employer just bought nearly $100,000 worth of HP hardware. We get a new MSA1500cs Fibre Channel SAN (with redundant controllers, FC switches, disks, etc), a new blade enclosure system, three blades to start with (all of them, at minimum, dual dual-core Opterons with 4GB RAM, and some considerably more), a rack to put all this in, etc.
So we’re starting to set all this stuff up. I’ve got Debian installed on an NFS root for testing the blades and how they interact with the SAN.
The blades have an integrated dual-port QLogic QLA2312 Fibre Channel adapter. The Linux kernel has a built-in driver for this (qla2xxx), which detects it and, so far at least, works fine. We want to run kernel 2.6.17 because it’s the first version where XFS has decent semantics for write ordering to prevent corruption after a power failure. Plus we want at least a 2.6.16.x kernel because we want to run the latest Xen 3.0 on these blades. (Live migration of virtual servers from blade to blade — this will be great.)
But we learn that HP does not support the kernel qla2xxx driver. HP does not say WHY they don’t support it, just that their own driver is the only one that they support.
After plowing through several annoying scripts to get to their driver, I realize why it fails to install: it is OLD. At BEST, 2.6.14 is the most recent kernel it would even compile against (release date: October 2005), and I think the most recent version it supports is more like 2.6.8 (almost TWO YEARS OLD now). They reference a whole bunch of kernel symbols and macros that were removed somewhere between 2.6.8 and 2.6.17.
I sent a ticket to HP support. Their first request was to run their system information gathering tool and send them the results. Fine, that’s reasonable. I did so. Next they say, gee, you’re running Debian, and we don’t support that.
Argh…. If they tried to compile it against 2.6.17.1 on RedHat or SuSE, they’d get the exact same problem. I told them what symbols they were erroneously using, and a simple grep would have showed them that.
Besides, how many customers are going to be pleased with no upgrade path available for 2 years? I wouldn’t want our kernel version to be held hostage to HP’s slow driver development process.
Sigh.
Thanks to all of the helpful comments people left in response to my last post about hosting. I got some really helpful hints from them — including the one I eventually followed, to VPSLink.
I wound up going with VPSLink. Their prices are amazing and the performance is good, too.
I have to admit — I shot myself in the foot not once, not twice, but three times. The shorewall config I was using on my old VPS (which was hosted under UML) apparently doesn’t work well under OpenVZ (used at VPSLink). By “doesn’t work well”, I mean “blocks all traffic to or from the host on startup”.
So, I had rsynced over everything from my old host to the new, and rebooted the new. But it didn’t come back up. I was pretty sure this was why. I dropped off a ticket to the VPSLink folks asking them to please rm /etc/init.d/shorewall for me.
60 minutes later, they had done it. (And it looks like a bug in their ticket system prevented it from being flagged as “emergency” — they said they would have done it faster otherwise.)
I then tried to fix shorewall, and it looked like it was working, so I put the init script back and rebooted. Same problem! They fixed it again in about 60 minutes. (The ticket still had normal priority)
Finally, I deleted shorewall entirely, then rsynced my old host to the new one. Things looked good, so I rebooted…. and yes, guess what, that rsync brought back shorewall so it got hosed YET AGAIN. ARGH. This was now well into the overnight hours, but they still helped me out.
All in all, I am so far VERY impressed with VPSLink. I heartily recommend them. I’ll be sure to post updates as time goes by.
Regarding Memset, my current provider: Really great company. I heartily recommend them, too. My needs for RAM just wouldn’t be ecnomical under their current plans, but they are usually competitive. I also have experience with JohnCompanies. I used them way back when they only did FreeBSD VPSs, on through the Linux days, and we currently use their services at work. These are also great folks and I would recommend them to anyone. Like Mako’s suggestion of Rimuhost, both memset and JohnCompanies are “quality first” providers — not necessarily cheapest, but their systems work as advertised and are almost never down, and they support you with experienced Unix admins. BTW, Memset’s current plans use Xen and JohnCompanies uses Virtuozzo.
I’ve been using Memset for many years to host my websites, Darcs repositories, etc. They provide virtual private servers. I’ve been happy with them overall, but as my RAM needs increase, things have been getting slower. As adding RAM to VPS plans is expensive no matter who the provider is, I decided to look for some other options.
I decided to try Dreamhost. They are a web hosting company with a clue. I don’t get root on the box, but they do give me the next best thing. ssh access, a decent suite of preinstalled packages, and clueful admins. I signed up earlier this week and started rsyncing some things over. They have a $10/mo plan which gives 1TB of monthly transfer and 20GB of disk space. Much nicer than the going rate of about $40 for 40GB of monthly transfer and 5GB of disk space for a VPS.
The web side of things worked fairly smoothly, and the system seemed to perform well. Until today, that is. I noticed my site being extremely SLOW this morning. I logged into the system to see if it was some problem with my configuration. It wasn’t, but the system load was — get this — over 100. I watched it for awhile and then fired off a support request. The load had been hovering between 20 and 80. It’s now 9.5 hours later, load is hovering between 11 and 30, things are still somewhat slow, and I’ve not heard back from them.
Then, later today, their server started refusing connections on port 80. Turns out this was a sitewide outage, and seems to have lasted for 30-60 minutes. I signed up to receive an e-mail when it was fixed, but I never got it.
Also, their e-mail hosting has never worked for me. At first, it was bouncing my mail. I sent a support request about that, and the next morning they said they fixed it. Well, it was no longer bouncing mail, but it wasn’t actually *delivering* it either. I told them that this morning, and still haven’t heard back from them.
Now, with my memset account, I can’t remember the last time I’ve had downtime. I’ve had slowness — rivaling what dreamhost is having — but actual downtime is exceedingly rare. And they seem to respond to trouble more quickly.
And the scary thing is that from all I’ve read, Dreamhost is really one of the best web hosting companies out there.
So, what do you all think? Should I continue going the “do it myself” route and host my site with a VPS (either with Memset or with JohnCompanies, another clueful host I’ve dealt with)? Or try to stick it out with Dreamhost? Or are there other clueful web hosting companies out there?
Right now, I’m of the “if you want it done right, do it yourself” mindset. Being able to save money, and also time by trusting someone else to keep the host patched and working only saves me angst if I can actually trust them to do that. Right now, I’m not so sure I can trust Dreamhost — or any other web hosting company.
Users of Debian’s testing or unstable distributions may be noticing messages from apt saying things like:
WARNING: The following packages cannot be authenticated! foo bar baz Install these packages without verification [y/N]?
I noticed today that google doesn’t turn up good hits for the fix. The fix is really simple:
apt-get install debian-archive-keyring apt-get update
That’s it. You now have secure packages from Debian. Nice, eh?
I finally purchased my first iPod: a black 60GB iPod video model. I had been holding off for years. The iPod sounded nifty, but I just didn’t quite go there.
The thing that finally won me over was the camera connector. It lets you plug your iPod directly in to a digital camera. The iPod can download photos from the camera to its internal disk without the need for a PC. Very slick.
So anyway, we got the iPod and the camera adapter at the Apple store in Cambridge — a quick subway ride from Usenix. They were out of stock on the FM tuner, so I ordered that online.
The next step was to get the iPod working with Linux. I currently have it working with both music and video. Here’s how I did it.
At work, we’ve been using vserver for virtualization for some years now. Due to various reasons, we’re looking at Xen.
I’ve been trying to switch my workstation to use Xen. I’ve enountered a few issues so far. Probably these will go away as I learn the system.
Overall, my greatest gripe is the documentation. It is outdated and just plain wrong far too often. For instance, there’s a place where it says to run “make ARCH=xen xconfig”, but the Xen kernel patches don’t (any more, at least) provide a xen arch.
The next gripe is the very weird kernel build system. Xen doesn’t ship a diff against a kernel tree. They instead ship whole files to extract atop a particular kernel version. Annoying and unwieldy. There is a command to generate a diff, but you have to download the full kernel tree first.
A couple of other gripes: There is little documentation on memory management (can Xen adjust the RAM usage of running VMs?), on 64-bit systems (can you run a 32-bit kernel under a 64-bit hypervisor? how about a 64-bit kernel that supports 32-bit userspace?)
I’m also having trouble with my forcedeth card locking up under Xen.
However, I’ve heard of lots of people having good luck with it so I’m going to keep trying.
But one would think that basic docs could be actually worked on a bit more.
Following the “release early, release often” motto, I am happy to announce version 0.1.0 of HSH, the Haskell shell.
You may obtain it with:
darcs get --tag 0.1.0 http://darcs.complete.org/hsh
Things are still very rough in many ways, but this version already lets you:
Here is an example session: (some lines wrapped for readability)
$ ghci -fglasgow-exts HSH
*HSH> run $ ("ls", ["."])
COPYING HSH HSH.hs TODO announcements testsrc
COPYRIGHT HSH.cabal Makefile _darcs test.hs
*HSH> run $ ("ls", ["-l"]) -|- ("wc", ["-l"])
12
*HSH> :m +Text.Printf
*HSH Text.Printf> let countLines = (zipWith (\i line -> printf "%-5d %s" i line)
[(1::Int)..])::([String] -> [String])
*HSH Text.Printf> run $ ("ls", ["-l"]) -|- countLines -|- ("grep", ["hs$"])
6 -rw-r--r-- 1 jgoerzen jgoerzen 1285 Jun 6 09:43 HSH.hs
11 -rw-r--r-- 1 jgoerzen jgoerzen 565 Jun 6 09:43 test.hs
*HSH Text.Printf> :m +Data.List
*HSH Text.Printf Data.List> run $ ("ls", ["-l"]) -|- countLines -|-
filter (isSuffixOf "hs")
6 -rw-r--r-- 1 jgoerzen jgoerzen 1285 Jun 6 09:43 HSH.hs
11 -rw-r--r-- 1 jgoerzen jgoerzen 565 Jun 6 09:43 test.hs
*HSH Text.Printf Data.List> run $ ("ls", ["-l"]) -|- countLines -|- filter (isSuffixOf "hs")
-|- ("tr", ["a-z", "A-Z"])
6 -RW-R--R-- 1 JGOERZEN JGOERZEN 1285 JUN 6 09:43 HSH.HS
11 -RW-R--R-- 1 JGOERZEN JGOERZEN 565 JUN 6 09:43 TEST.HS
*HSH Text.Printf Data.List> let generator = \(_::String) -> unlines . map show $ [1..20]
*HSH Text.Printf Data.List> generator ""
"1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n"
*HSH Text.Printf Data.List> run $ generator -|- ("grep", ["1"])
1
10
11
12
13
14
15
16
17
18
19
Future versions will likely simplify syntax to make it easier to write scripts and introduce a sh to hsh converter. I also plan to add pure Haskell tools for some common shell-ish things that one could do in Haskell.
Well, we’re back from the Usenix ’06 Annual Technical Conference. Overall, a very nice conference. This is the first Linux/Unix conference I’ve been to since the old LinuxExpo (NOT LinuxWorld) events in the late 90s.
I must say it was quite fun to be around so many intelligent *nix folks. I learned quite a bit, and learned how much I had in common with some (and how little with others). A very nice social at the New England Aquarium Friday night.
Some very interesting speakers, too, with lots of experience to share. I’m glad I was there.
I have only one main complaint about Usenix — they tried to cram too much into too few days. The conference sessions ran till as late at 6:45PM, with BoFs after that. That left almost no time to see anything of Boston and meant that very few people could actually stick around for all the BoFs. Next year, they should use more days and schedule fewer things in the evenings.
The highlight of the trip, though, wasn’t Usenix at all, but instead getting to meet some Boston-area Debian folks Saturday night. Lots of fun, and Mako is as crazy in person as he looks online :-)
Our train trip in each direction was nice, too.
More to come…
I’ve been managing servers professionally for some years now. Support is one of the most important things when you are managing computers for work. You don’t need support to help you out with a printing problem or an e-mail problem. You need support because every minute the machine fails to power up, your company may lose twice the value of the entire machine. Or even more.
My first day job managing servers involved Dell hardware. What a nightmare. I’ve never had a good experience with Dell support, ever.
First off, Dell support never puts me straight through to an intelligent support rep. I don’t care whether I get to the Indian call center or someone in Texas. The first support person I speak to at Dell has less computer aptitude than my grandmother. One conversation I will always remember went like this:
Me: We have a disk in our array that went bad on our Linux server. The red light on the disk itself is on. Please send us a replacement.
Dell: Have you tried rebooting?
Me: No. This is a production server. The reason the disks are redundant is so I don’t HAVE to. Besides, the light ON THE MACHINE ITSELF is on.
Dell: Ah, OK. Have you run scandisk?
Me: No. This is a Linux server, as I told you, and scandisk wouldn’t see a problem anyway since this disk is in an array and the array is still up.
Dell: OK, great. How about you download the diagnostics .exe from our website…
Me: Can’t. We don’t have Windows on this machine. You did say you support Linux when we bought it.
Dell: Ah. Can you right-click on My Computer…
Me: NO. This runs Linux, and the BAD DISK LIGHT ON THE MACHINE IS ON.
Dell: Ah, OK. I wonder if the problem really is that you have a bad disk.
Me: Could be!
Our first HP server purchase happened to be at a time when HP had undercut Dell by several thousand dollars. I liked the hardware, but it wasn’t anything that much more special than Dell.
But what I really like is the support. I haven’t had to call HP support often, but when I do, I am almost always speaking to a live, experienced person within 5 minutes.
With only one exception, all the HP support people I’ve talked to have been very experienced. They all sound like they’ve been working with HP hardware since the late punched card era. They know what is going on and assume that I do too. The HP people don’t make small talk (it *really* ticks me off when someone obviously in Calcutta or something asks me about the weather in Kansas, because you *know* they are reading it off a screen and don’t care). But that’s fine. I’m not calling them to talk about the weather, I’m calling them because my server is down.
We had a bad disk in an array on a HP server once. That conversation went more like this.
HP: Server support. Serial number please?
Me: [gives it to them]
HP: OK. What’s the problem?
Me: The array dropped a disk. The failed disk light is on and the controller logged a disk error.
HP: OK. That’s a 146GB SCSI, right? 15KRPM or 10K?
Me: 15K.
HP: OK. Is 1PM tomorrow good to send out the replacement?
Me: Fine.
HP: OK, your case number is xxxxx. Can you give us directions to your location?
Me: Sure…
So recently we got in our MSL4048 tape library. A very nice unit. And faster than most *disks*. 48 Ultrium3 tapes — 400GB native each — very nice. And a barcode reader built in.
So anyway, one small problem. When you open up a magazine to put tapes in, you can close the unit back up. It says “scanning”, but it doesn’t notice that we’ve changed tapes until we give it a command on the operator panel (yes, this tape drive has a LCD screen built in). This can be worked around, but is annoying and is just waiting to cause confusion. Plus it’s not how it should work.
So I call HP support yesterday.
Turns out this MSL4048 is a brand-new unit. Had only been on the market a few weeks. Our support rep has never seen one or taken calls about one, and they haven’t even given him all the HP technical docs yet. But no matter, he is willing to try to help us out.
He calls me back twice yesterday with tips and questions after speaking to colleagues. He asks intelligent questions, doesn’t bother with the “are you sure you’re putting the tapes in the right way around” or the “is the power cord securely plugged in” crap, and generally doesn’t waste my time. He called me back about four times more today — they duplicated our setup in their lab, right down to the exact firmware version, but didn’t have the problem. Two of those callbacks were apologizing for taking so long, and explaining that they were learning about this machine as they went along. So a HP rep will be out to our location shortly.
Now THAT’S what I call service. No blaming it on someone else, no trying to make me do stupid troubleshooting things, and returning calls.
My *one* bad experience with HP was one time we put a new internal tape drive in the machine, and it was acting flaky. I got the only not-very-experienced HP rep I ever had spoken to that time, and they tried to blame Debian for what turned out to be a bad SCSI cable. (The symptoms weren’t very similar to what I’d expect for a bad SCSI cable, and the cable had been working fine.) Oh HP, you donate to Debian — why don’t you support your hardware under it?
(In fairness, that is the ONLY time they have flinched when I said I run Debian, though it does make them hesitate sometimes)