There’s a lot of responsibility on the shoulders of system administrators. Unless you’re one, it’s probably hard to grasp the full weight of it. This week has been full of reminders for me.

This afternoon, shortly after lunch, I got word that people were having trouble with phones. A few minutes of testing showed that calls within our city were working fine, but it was completely impossible to place or receive long-distance calls. A little while later, a local newspaper’s website indicated that numerous cities over a multi-county region, all served by the CenturyLink local telephone company, were out of service in this manner. I figure that represents hundreds of square miles, or a patch of rural Kansas roughly the size of Los Angeles.

Adding to the problem was the fact that emergency 911 services are accessed via these long-distance lines that were down. For roughly three hours, 911 was completely down across this entire area. What’s more, many cellphone towers and Internet access options were also taken down, since they feed from these same lines.

This adds up to a situation that could very easily cost lives due to delayed response of emergency medical, fire, or police services.

In the end, the problem was traced to “a bad controller card on a Titan 5500 owned by AT&T.”

Now, here’s the system administration angle: If you worked for the phone company and had to troubleshoot a problem that you knew had taken down emergency services for thousands of people, what kind of pressure would you be feeling? Would you be able to keep your cool? I’m glad that I don’t have that kind of job.

I also wouldn’t like to be the engineer (or, more likely, accountant) that decided that they didn’t need any more redundancy to provide good service to the area. Especially considering this is the second time in the last year or two this has happened.

But, ironically, yesterday I signed a purchase order for a new Asterisk PBX (corporate phone) server. When selecting a machine for that task, I am always completely conscious of the responsibility on my shoulders: several hundred employees rely on the machine that is ultimately my responsibility to select. Our own access to 911 would be cut if the machine were to go down. I never forget that the correct operation of the systems that our team sets up and deploys could help save someone’s life, and that a malfunction could cost the company dearly in terms of revenue, productivity, image — or worse.

Nearly four years ago, we switched from an analog PBX, with outsourced support, to a digital VOIP system running Asterisk. Note that we use VOIP in-house, but do not use it externally. Anyhow, I can not say that the Asterisk PBX has been 100% perfect; I doubt that this could be honestly said of any PBX of any complexity.

I can say, though, that it saved us well over $100,000 AND has proven far more reliable than the system it replaced. Outages are exceptionally rare and brief now. Plus we have internal expertise to fix it, rather than having to wait for a technician to be dispatched from a city 2 hours away to fix anything. I know I don’t have the resources to build a perfect PBX that will never go down (if such a thing is even possible), but I take my responsibility regarding a reliable PBX extremely seriously.

We used to have a frequent problem: someone would call 911, then hang up. We suspected this was often on accident — maybe people hit 9 for an outside line, then misdialed their number. In any case, 911 dispatch would then call our main office, saying they got a hangup. A person or team would then go across our entire campus making sure nobody was in distress — that nobody had managed to dial, then passed out, for instance.

With Asterisk, I was able to help this situation. Whenever somebody calls 911 now, two emails are generated: the first contains details about the call, such as the extension number that called and the duration of the call. This goes to all people that are likely to receive a callback from 911. It may not always pinpoint the source (as with somebody using a wireless phone), but almost every time will give us a very good idea where the call came from. The second email is a recording of the call, and serves as an additional clue, but goes to fewer people.

I am aware that email isn’t a perfect medium, but: it let us make a dramatic (albeit imperfect) solution to a problem that very few institutions our size are able to address nearly so well.

There’s a lot of weight on our shoulders: keeping the accounting system up, the Internet links up, the web store or the sales phone lines, the shipping systems or the document archives up. These things going down can spell deep trouble in many ways.

And sometimes the systems we maintain might save a life. Such as this morning, when someone was feeling symptoms of a heart attack, used our phones to call a colleague for help, that person called 911, and an ambulance was dispatched. I know the system worked exactly as it should, because I had two familiar emails from Asterisk in my mailbox this morning when I got to work.

I read The Iliad recently; see my review of it here.

This article isn’t about The Iliad, but about Shmoop’s coverage of it.

I have never been a person who puts much stock on “study guides” before, and my own prejudice is that they have probably earned their reputation as useful for little more than a tool to help students pass exams without reading the material. However, I felt rather unequal to reading The Iliad; after all, it has been many years since I had been exposed to Greek mythology, and even then it was very brief. I had remembered from Amazon’s blog that they are offering some of the classics with a Shmoop study guide built in. I figured the $2.39 would be worth it. But I’m not sure it was.

The resulting ebook has two sections: the main text, and the Shmoop overview. The overview contains essentially the information available for free on their site. This information is pretty good. It has some overviews of the entire book, plus summaries of each chapter. They helpfully provide the “backstory” in the chapter summaries, so I know why certain gods are on the side of the Trojans and the others on the side of the Achaeans. It helped me figure out the point to some of the actions described on the text. These are linked to at the beginning of each book (chapter) of the Iliad, which was a bit inconvenient because I prefer to read them after reading the main text. Nevertheless, it wasn’t a big problem.

The summaries were written colloquially, sometimes too much so. I grinned as the “less grabbin’, more stabbin'” summary of the Achaeans being told to salvage armor from the dead later and keep fighting now. I cringed as “no way Jose”, and rolled my eyes as it described how one of the heroes got “owned”. That would have been less clear than the original for many.

I would probably have given the Shmoop edition 4 or 5 stars on Amazon were it not for numerous boneheaded mistakes they made.

The commentary uses one translation (Lattimore), while the included text uses a different one (Butler). That’s bad enough, but it gets worse: Lattimore uses the Greek names for the gods and heroes, while Butler uses the Roman ones. So you can be reading through the text, learning about Jove, Ulysses, Juno, and Mars. But the commentary refers instead to Zeus, Odysseus, Hera, and Ares. Now, some might have memorized the Greek and Roman names for everyone, but then such people probably aren’t wanting a study guide. I finally had to print out a page to help me understand the study guide. That is terrible, and completely inexcusable. They should have used the same translation for the text as the commentary, or at least have provided parenthetical notes throughout the commentary.

Next, they really did a poor job of the main text itself. They have apparently grabbed the Butler ASCII text from Project Gutenberg, and cut and pasted it into some editor. I say this because each line renders as a paragraph; that is, the text doesn’t reflow to fit the screen as every single other book does. Worse, it’s too wide for the Kindle. So you get one line full width, the next only half as wide as the display (some of the first having wrapped), then the next full width again, on down. Butler is a prose, not a poetic, translation, so this is pointless. Moreover, the line endings are exactly where they are at Project Gutenberg.

So, what we have here is that somebody at Shmoop cut and pasted from their own website and Project Gutenberg, and nobody bothered to check if the resulting product was crap. At least with the new Kindle firmware you can read in landscape mode to prevent most of the lines from wrapping. Still, I’m very annoyed at this.

There are other flaws. They highlighted some passages in the text. Clicking on them goes to some commentary about it. OK, fine. But why are some passages highlighted in the text, and some mentioned in the chapter summary? There seemed no rhyme or reason. The highlighted passages commentary was iffy in quality. Sometimes it was useful, and sometimes it asked a question of the sort I might expect in a jr. high English question — “When has pride helped you in your life?” or some such. Worse, not all of the highlights linked to the correct place, and they also suffered from the Greek/Roman name issue.

In all, Shmoop has helped me understand The Iliad and place it in context. But they need to spend a lot more effort on copyediting. Even a couple of hours of someone actually looking at their Kindle product on an actual Kindle would have immediately shown these problems. They could have made a far better product with a few hours’ more effort.

Once a month, I take our recycling to the road. This is something far more annoying than most city-dwellers may suspect.

For one thing, the road is about 1/4 mile away. Over a month, we fill up at least three large plastic trash cans full of recycling. If I walked each of those there to be picked up, then walked them back to the house after, I’d have walked 3 miles.

So, I use our $75 pickup to get the trash to the road. The process is a long one, and it starts two days before the recycling is due to be picked up.

Monday, 5:30PM: Find an outdoor extension cord in the basement. Put on snow-suitable shoes. Go outside, plug the extension cord in at the elevator. Grab the battery charger, hook it up to the pickup. Unfurl and untangle the extension cord, plug it in.

Blog about it if desired.

Tuesday, 5:30PM: Go outside. Disconnect the battery charger, coil up the extension cord. Plug in the air compressor and get it going. Run the hose to the tire that keeps going flat and hope it reaches. Fill up the tire. Return the air compressor hose to the elevator, turn off the air compressor.

Go to pickup and actually get in. Remove the bungee cord from the brake pedal that is holding it up, preventing the brake lights from being on 100% of the time and draining the battery even faster.

Start pumping the accelerator. It’s not a fuel-injected engine, after all. Crank the engine. Stop cranking, pump some more, trying to avoid the hole in the floor under the accelerator. Eventually get it started, hopefully.

Roll down the windows to prevent the exhaust fumes from getting too strong.

Drive the stuff to the road. Unload the three cans, plus all the extra cardboard from Christmas that didn’t fit in them and is now soggy and snow-covered. Drive the pickup back to the yard. Park it. Replace the bungee cord on the brake pedal so it’ll start again tomorrow.

Wednesday, 5:30PM: Dress warmly yet again. Go to the pickup, hoping the tire is still inflated and the battery is still charged. I’m going to be optimistic and assume they are. Hope that the pickup starts after only a few minutes of pumping. Go back to the road, only to find that the Kansas wind has scattered the trash cans and their lid all over the county. With luck, I’ll be able to find them on foot. Sometimes, I have had to get in the pickup and drive down the road to find all my lids. Stash all of this in the pickup, drive it back to the yard, park, replace the bungee cord, and close it up for another month.

I’m promising myself that when I upgrade to a better pickup, it’ll be one worth at least $200.

In Egypt’s sandy silence, all alone,
Stands a gigantic Leg, which far off throws
The only shadow that the Desert knows:
“I am great OZYMANDIAS,” saith the stone,
“The King of Kings; this mighty City shows
“The wonders of my hand.” The City’s gone,
Nought but the Leg remaining to disclose
The site of this forgotten Babylon.

We wonder, and some Hunter may express
Wonder like ours, when thro’ the wilderness
Where London stood, holding the Wolf in chace,
He meets some fragments huge, and stops to guess
What powerful but unrecorded race
Once dwelt in that annihilated place.

—Horace Smith

Many of us that can afford an Internet connection still qualify as exceedingly rich compared to, for example, many poor people in Africa. On a scale of centuries, the ancient centers of civilization and wealth tend to collapse. Even Rome, with over a million inhabitants in its day of ancient prominence, shrank to only 20,000 over the course of history. Some other ancient cities no longer dot our maps.

It helps us, perhaps, to have some perspective. Those magnificent working monuments to wealth in New York, London, or Tokyo probably won’t be there 1000 years from now. What will?

Everyone seems to be familiar with ebooks these days. I own a Kindle 2, and of course we’ve spent weeks hearing how great the Nook will be, then weeks hearing how terrible it turned out to be. But nobody seems to be casting an eye back towards paper, so I thought I’d rectify that here, especially since paper books have some serious stability issues that are often overlooked!

Before I begin, I feel it wise to offer this hint to the reader: this review should not be taken too literally. If you have an uncontrollable urge to heave a volume of the Oxford English Dictionary at me as if I am some European prime minister, please plant your tongue more firmly in your cheek and begin again.

Today I picked up a paper book to read just for fun — The Happiest Days of Our Lives by Wil Wheaton. Long-time (since this spring!) Kindle user that I am, I immediately noticed the dashing use of color on its front cover, but when I opened it, I was disappointed that I couldn’t scale the font size down from the default. It seems that paper books have only one font option — what are all these Kindle forum posters complaining about with its six sizes of a single font?

On the very first page, I encountered a word I wasn’t familiar with (Namaste). I thought I knew what it meant from the context clues, and even had the thought that on the Kindle, I could just highlight it and confirm my guess. But my paper dictionary was in the basement, so I didn’t bother looking it up until I wrote this post. (My hunch was reasonably correct.)

Interface-wise, the paper book is solid, and crashes, lockups, or other malfunctions are rare. I have, however, noted severe stability problems when attempting to read outdoors, especially when it’s windy (which, since I live in Kansas, is pretty much always). Pages start turning themselves, even without me making the “turn page” gesture. Sometimes the book will even lose its memory of my last page read. This is rather annoying, and might even involve a lengthy search for a suitable temporary replacement bookmark. Also, I haven’t tried it, but I suspect that the trick of putting a Kindle in a ziplock bag to read at the beach or in the tub without risk of getting it wet would be impractical with a paper book.

Paper does have its advantages. For one, it’s faster to flip rapidly through pages on paper than on an ebook reader. If you know roughly where in the book something was written, but not the precise wording, searching can be faster on paper. On the other had, if you are looking for a particular word or phrase, the ebook reader may win hands-down, especially if the paper book has no index.

Paper is so stable that some would argue that the extreme impracticality of making good backups isn’t really a problem at all. But on the other hand, paper books degrade slightly each time they are used, and this condition can be aggravated by placement in bags for transport. Eventually, they will wear out. If my Kindle wears out, I can always restore David Copperfield from my backup copy to a new one. If my printed edition (all two volumes) of it wear out, then I have to hope that the used bookstore will still sell me another one for $10. Otherwise I’d have to either drive 45 miles to find one for sale, attempt to deal with the DRM for paper books at a library, or wait a couple of days for Amazon to get it to my door. A fire or flood could be devastating.

Paper books also have some advantages for showing photos; no ebook reader is close to the size and resolution of glossy paper photos books in a reasonable price.

The contrast on most paper books is better than that of my Kindle, but some older ones are actually worse, smell dusty, and suffer from occasional display corruption as bits of them actually break off of the book device.

As to cost, it is a mixed bag. Out of copyright classics are free as ebooks from Project Gutenberg and the like, while still costing money on paper. I have found that the accuracy of some of these paper editions can be rather questionable — people have sometimes manually removed important bits of the story to save on printing costs, rather than let Google Books OCR mangle it for them automatically. On the other hand, used paper editions of more current works can be found for a fraction of the cost of the new ebook edition — though you are often limited in selection of these bargains. But you can usually browse a paper book for a few minutes before buying it, which is rarely available for an ebook.

What’s more, libraries might let you borrow paper books for free. But you often have to expel greenhouse gases to get to them, and then they enforce DRM on you — you only get to read it a certain amount of time before they start adding fees. You could easily wind up paying $2 if you keep it a week longer than you should have. With ebooks, of course, there is no free borrowing (and the Nook feature it too limited to count.) And you of course know that most libraries are run by the government, so they have your address. Trying to circumvent a library’s DRM could wind up involving the police, so you had best comply.

Making copies of a paper book is expensive and requires specialized equipment, even if you just want a copy for backup.

Compatibility problems with paper books are rare, and are usually found among readers with poor eyesight. A few works can be found in “large font editions,” but most can’t, so those readers are left needing expensive specialty magnifiers.

All in all, I prefer reading books on my Kindle, but still read on paper when that’s how I have a book.

Thanks everyone for the helpful comments yesterday on Apache vs. lighttpd. I’ve taken the first few steps towards improving things. I’ve eliminated mod_php5, switched all PHP to FastCGI, and switched from the prefork to the event MPM.

The event MPM docs say that it’s incompatible with mod_ssl, but it worked, and googling for that turned up mailing list posts from 2006 saying it was fixed.

The only real glitch was that egroupware inexplicably depends on libapache2-mod-php5. It works fine without it, but I had to create a fakelibapache2-mod-php5 that provides libapache2-mod-php5 to convince the system not to remove egroupware when I switched to the event MPM.

I went from 28 Apache processes to 4 Apache processes plus an average of around 2 php5-cgi processes — which, despite the name, actually do grok FastCGI. Three of my Apache processes now have an RSS of about 20M, and the other of about 30M. The php5-cgi processes are at around 40M. My old Apache processes ranged from 18M to over 40M. So there is some memory savings here, though not drastic.

My Redmine processes, two of them, each use over 50M. Ruby on Rails still is leaving me with a bad taste. At least it’s not a Java app; it seems a lot of those allocate 2GB before they ever start accepting connections.

I’ll see how this goes for awhile, but for now I doubt that moving to lighttpd (or similar) will yield enough benefit to make it worth the effort. But there may be some benefit in inserting a caching proxy in front of Apache, so I may yet do that.