Daily Archives: August 7, 2007

Disasters happen

Software

It’s been one of those days.

It started with me forgetting my laptop bag at home. Then I had to work with an attorney on some things. He’s a nice guy, and does a good job, but you know, it’s not what I’d really like to spend my day doing.

Then in the middle of the afternoon, I hung up the phone and checked my email. About 50 messages from Samba telling me that processes with various PIDs had crashed unexpectedly. Uh-oh. I think we still have some 80ish people using that.

About a minute later, a coworker says, “John, I’ve been bad.” “Is that why I just got 50 emails from Samba?” “No. Well, yes. Well, maybe. I don’t know.”

It turned out he was working on a restore from tape that, out of necessity, grabbed more data than he needed. He meant to type rm -r ./var but typed rm -r /var instead. Oops. He hit Ctrl-C halfway through, so /var was still there enough to send email but not enough for Samba (or, apparently, NFS) to work.

As he dashed off to pull yesterday’s tapes from offsite storage, I prepared the restore and made a plan. We hadn’t installed any software since yesterday, so I restored var to a temporary location, took the server down into single-user mode, overwrote the /var that still existed, and rebooted the Xen instance in question. Everything back to normal. Except, that is, for the potentially dozens of users that will require assistance running SCANPST.EXE because their Outlook PST, being the fragile heap of garbage that it is, will have somehow been corrupted by this little incident.

So, what did we learn from this?

  • Deleting /var was probably the least annoying outage I’ve had to deal with yet. Certainly less nerve-wracking than the time I was working on a live, powered-up server and my wedding ring shorted out something on a circuit board. I didn’t know if that thing would just reboot or if we’d be down for hours waiting for parts…
  • It was really nice knowing what was going on, rather than trying to find that bit out
  • One coworker commented, “if he had to delete part of an operating system, at least it wasn’t Windows. We wouldn’t have recovered in 15 minutes if it was.” True.
  • Bacula is great.
  • Backups are great, even if you don’t use Bacula to make them.
  • I dislike programs that take server load from 0.3 to 9.5 just telling you that there’s something wrong with the server.

I suppose this means I’m a geek

Uncategorized

I work in an open-plan office. Normally I like to listen to some of my iPod’s music, or NPR or something, at some point during the day. It helps me tune out distractions when I’m coding or concentrating on something. My iPod, and my nice Etymotic headphones, get transported to and from work each day in my laptop bag. Today I forgot the laptop bag at home.

What to do? I could just work without headphones. I’d be fine, but you know, I’ve got standards here. My job involves working with computers, so I ought to be able to come up with a workaround, right?

So lesse… what do I have? One binaural (mono sound, but speakers for each hear) telephone headset. One Polycom SIP phone, connected to our corporate Asterisk system. One workstation with sound capabilities. One installation of Asterisk on this workstation for testing purposes. And, a pre-existing path from the corporate system to the workstation system for testing Asterisk. (Very handy that, and used a lot when we were doing active Asterisk work.)

So in less then five minutes I had music going via my telephone headset. Lo-fi, and not noise-dampening like the Etymotics, but I enjoyed it for the simple fact that it was being played *over the phone* at no cost to anyone. My desk phone supports multiple “lines”, so I still could place and receive calls just fine.

Should anyone care to look, they’d find a 5-hour call from me to myself deep in the Asterisk logs. My own workstation logs will show that I put myself on hold for 5 hours (since I used Asterik’s music-on-hold feature to play my own selections).

IP telephony is fun. So is Asterisk.