You know, I really hate it when e-mail inquiries to customer support departments are second-class citizens. The same company that will have me on hold for less than 30 minutes, then resolve my question in 10, won’t even attempt to look at my e-mail inquiries to the same department for 1-2 business days.

Ugh.

What’s the deal here? Why should e-mailers be treated poorly?

Today’s company to shame: Shutterfly, which otherwise has been very good. I’m e-mailing them because my UPS next day air package was delayed by UPS. UPS said they’d issue a refund, but I have to ask for it through Shutterfly.

The fire didn’t do any significant damage to the building, but it did cut off our power supply because it burned it all up. We have the phone system and a skeleton network back by mid-day Tuesday and a full network back by Wednesday morning. Thursday afternoon, temporary utility power was available which provided enough for the office to return to normal.

Our factory was scheduled to shut down for the holidays on Monday. With work, it looks like everything will be back to normal on January 2. So we really will lose only 4 days of production.

The company’s business interruption insurance will be paying the wages for workers that couldn’t work those four days due to fire as well.

So, busy week, but it could have been much worse.

It’s been an interesting 24 hours.

Monday
8:00 PM
I get home after working a regular day and having supper with a friend.

8:15 PM
One of our friend from church — who happens to live a few blocks from my workplace — calls. “Did you know your workplace is on fire?” “Uhh, no….” “Yeah, big fire, flames, dozens of emergency vehicles…” “Hmm…”

So then I call my manager… “Did you know the company is on fire?” “Uhh… no….” “Yeah, sounds bad, I’m going to head in and see what’s going on.”

8:30 PM
I arrive at work. Cops have blocked all the entrances. I identify myself as an I.S. manager and am directed to the “manager area”. (Others are kept across the street.)

An eerie orange glow — and lots of smoke — is visible above the rooftop. This looks bad.

We learn that something in the compressor room caught fire or exploded. Fire spread upward quickly.

Most importantly, everyone got out without any injuries.

8:35 PM
Our T1 provider calls my cellphone. These are really, really good folks. I answer, as I’m standing next to a noisy fire truck.

ISP: Hi. Just calling to let you know that your T1 is down and we’re looking into it.

Me: Hmm. That might be because it’s on fire.

ISP: Uhm, did you just say it’s on fire?

Me: Could be. There’s a fire in the building and we’re not allowed in just yet. Plus the electric company is trying to cut the power.

ISP: OK then, I think we’re going to assume the problem is at your end for now.

This sounds bad. I know the I.S. dept. still has UPS power, since the phone system attendant is answering. But if the T1 is down, it seems that it must have burned up (its run goes very close to the affected area), and taken some phone lines with it.

9 PM
It’s below freezing already, and I didn’t bundle up very well. Several waiting — but unused — ambulances are nearby and the paramedics invite those of us waiting inside the heated vehicles.

9:30 PM
The CEO and CFO are permitted to tour the building with a firefighter. Word is that sprinklers have come on and there’s water in the I.S. dept. This is potentially really bad news. The I.S. dept is in the basement and water has been a problem before, so we’re prepared, but water and computers are never a good combination…

After a little while, we are allowed inside as well. There is 1 to 2 inches of water on the floor in the basement, but fortunately nothing has reached the raised floor yet. The fire department supplies an AC generator to power the sump pump, plus 6 squeeges. We supply 6 people. We keep on the squeege duty until about midnight.

Meanwhile, it turns out that a water pipe broke out in the shop near the fire. Water gets shut off. But the floor up there appears to unfortunately slope towards the staircase that leads to the basement. Water is coming down fast. We find what we can to block the path, and that seemed to just make the collection deeper.

I periodically try to find someone that can bring a pump and a generator — or something — to help this get cleared up.

Midnight
I head home. The water situation is improving, but not resolved yet. Others remain.

Once home, I post a notice on our public website saying that our power, Internet, phone, and FAX lines are down. You just might have a problem reaching us.

I also e-mail some consultants, saying that the reason the VPN doesn’t work is that we had a fire. I also e-mail the new I.S. person who has an hour commute and advise him of the situation.

Our fire was top story on one of the local newscasts. We watch the video of it, sigh at the annoying “on the scene” reporter (motto: “things were interesting an hour ago”), and turn in.

1AM
Time to sleep.

Tuesday
6AM
Get up, in to work by 6:45.

Everything is dead. No power, no water, no phones, no heat, no gas. Also, fortunately, no more water coming down.

I call the other I.S. developer and tell him not to bother coming in right now, but maybe we’ll call later.

8:30 AM
An impromptu meeting happens. The CEO sets communications as the #1 priority: phone and basic Internet access. Plants 2 and 4 have working power, so if we can bring up some servers and network gear on a generator, computers over there will work.

No water in plant 1, where my office is.

So nearest restrooms are a 5-10 minute walk through dark and damp areas. Bottled water, coffee, and donuts appear at the halfway mark to the restrooms ;-)

9:00 AM
Generators start appearing near the I.S. dept. After getting them powered up, proper extension cords found, etc., we start plugging in the phone system.

Power strip 1 works.

I flick the switch on power strip 2, and it all goes dead. This generator’s circuit breaker keeps doing that every time we try to turn on the phone system.

So we try generator #2. Phone system finally powers up. But all incoming lines are ringing at phones in sales. Volunteers go up to answer those phones.

Meanwhile, phone tech shows up, and orders his own, larger generator.

I call our person that’s an hour away and tell him to go ahead and head in.

10:00 AM
Now we turn to the computers. We’re going to have to be careful what we bring up so we don’t overload the generator (and thus have an unclean shutdown on anything that’s already up.)

I recable power to some switches, our T1 router (ISP calls: the T1 just came up! yay!), and our firewall.

Flick the switch on firewall.

Generator stays up. Firewall emits a high-pitched whine. Eeep. Flick the switch again!

After checking things out, we adjust the voltage a bit and determine that things are generally OK. I run the firewall on one of its two power supplies. At least if it burns one out, we can still run it off the other later. Turn it back on — whines again, but it goes away after a few minutes. Phew.

Next step: our main Linux file server. Flick the switch, and there’s a satisfying “whoosh” from the fans. Excellent.

Our developer shows up and observes later: “John, this was odd. When I got here, there were 7 guys standing around watching you boot up machines.”

11:00 AM
Decision time. Our main ERP server is an AIX box. 8U. About a dozen disks. Certain to be a large draw. Can the generator handle it? Yes, it should.

Should we power it up, given that it takes 45 minutes to come up and 20 to go down? We opt to try.

I cable it up and press the power button.

Nothing happens.

But that is normal for AIX. After a few minutes, we start seeing hex codes on the LCD. A few minutes later, disks start coming up. All looks good…

11:15 AM
Orange alert lamp comes on for the AIX box. Uhoh.

Turns out one of the drives in the RAID is bad. Time to call IBM, but at least it’s mirrored to the box is up.

IBM service calls are always “fun” for this. I spent a total of about an hour on the phone with them, 90% of which was spent with them analyzing error logs trying to find out what exact part number to give is, and 10% spent trying to look up our account information.

Plus, the phone guy is now switching the phone system to his genset. He tries to hook it up through a home-type UPS, which is giving him fits. I get disconnected from IBM at least twice.

Noon
My supervisor goes to Subway and brings in lunch for all four of us in I.S. since we don’t really feel like we can leave.

Word arrives that the generator that’s powering the servers is leaking oil. Eeep. But it should be good until evening. On the plus side, the generator will automatically shut itself off when it runs out of oil. Won’t damage itself.

1PM
IBM calls, trying again to get information about what exact part they need to bring out. Only 29 minutes on the phone this time.

Word arrives that a 60kW generator has arrived that they will attempt to hook into the mains supply for the building. The main power distribution center has been burnt. Plus we aren’t allowed to touch large parts of it since fire inspectors and insurance adjustors aren’t done yet.

But no ETA on the big generator.

We make a preliminary decision to cut power to computers & the network at 5PM, and maintain the single phone system generator overnight.

2PM
Local IBM rep calls. He’ll be out between 3:30 and 4. I say that’ll be fine, but warn him that if he’s any later, the power may be down.

3:45PM
IBM rep shows up, replaces the dead disk. It works. Whee.

4:15PM
I start powering down the AIX box.

We then learn that ETA on the big generator is 5PM. So we start shutting down other servers as planned.

5PM
Big generator comes up. It works. We slowly bring up our systems. Everything is good.

BUT — we are basically maxing out the capacity on the generator. People will not be able to just come in and work like usual tomorrow.

We’ll be meeting in the morning to figure it all out.

6PM
We post signs at entrances warning people to not turn *anything* on.

Finally head home at 6:30.

While I’m on the subject, Wichita State University is apparently giving personal information about me to military recruiters. I’ve received numerous calls, from at least two different branches of the military. Some of them know nothing but my name and home phone number. Others appear to know my age, and still others how close I am to graduation and that I’m taking evening classes.

WSU never informed me that they would be sharing this information with the military. I am quite troubled that they do so without my informed consent, and I will be letting them know this.

Wow — this really ought to put Marquette University to shame. They have suspended a student for making comments on his blog, which their own ethicist thought were harmless.

Scary.

Yesterday, I told you about Aunt Viola, choir director at our church for many years, about 50 years ago. Today, I’d like to tell you what she did in church last week.

In our community, the song Nun Ist Sie Erschienen is a traditional Christmas hymn. Even though our church services have been in English since the 60s, and the song isn’t in any current hymnal, it’s still sung around Christmas. (There is a photocopy of this song with each hymnal.) Some of us wish it were sung a little more often.

Our church, like many Mennonite churches, has a sharing time during the service. People share what’s going on in their lives — joys or concerns, or whatever it may be.

Last Sunday, the first day of advent, the hymn wasn’t in the bulletin. The sermon that day was on surprises (very fitting). At the end of sharing time, one of the older people in our church got the mic and said something like “I feel like singing Nun Ist Sie Erschienen, so we’re going to right now!”

Aunt Viola had been in on this — she had checked with the organist in advance. So it was all set. Aunt Viola sprung up and somehow got to the front of the sanctuary very quickly (she is in her 80s), ready to direct the congregation! And it was great. I like the song too. It was one of my grandfather’s favorites (he played it wonderfully on harmonica) and we sang it (in English) at our wedding.

After church, I asked Aunt Viola how long it had been since she had directed for a song. She didn’t know — “a very long time”. But she said that if I ever wanted to request a song during sharing time, she’d be happy to direct again! I think I might have to one of these days.

I’m in the adult choir at church. The current choir director’s aunt, Viola, is also in the choir. So she is Aunt Viola to quite a few people.

Aunt Viola was the director of the choir herself for many years, back in the “old” church. That church was torn down to make room for a new building in 1965.

Not long ago, I heard from someone that she was looking for a picture of the old choir. She thought it was in the church archives, but nobody could find it. But I happened to have a copy. Another person that was in the choir had given it to me to scan for a book project a few weeks before.

So I uploaded it and ordered an 8×10 print for Aunt Viola. I sat behind Viola and her husband the next Sunday. As people were visiting before church started, I handed over the envelope and told her there was a little surprise in it. She opened the envelope, and knew right away what it was. And was she ever excited to have it! It made us very happy to have made her day. She was talking about it for the rest of the day — actually, for a few weeks.

So here it is, the choir of Tabor Mennonite Church in 1946. Aunt Viola is on the far right in the front row.

Thanks to suggestions from readers of this blog, I have ordered a Nikon Coolscan V. I’m really looking forward to it. Once I considered the volume of photos I want to scan, it really makes sense to buy such a thing myself.

First off, Dan Zukowski has some very informative articles about Amtrak. It’s particularly funny listening to Mineta say that the Bush administration wants to support Amtrak. After they proposed $0 for it in their most recent budget proposal, saying that bankrupting Amtrak would be a good thing. Hmm. Doublespeek seems to be in vogue these days.

I’ve watched about half of this week’s 4-hour hearing about Amtrak governance. One very interesting thing about it is that the Amtrak board, which fired David Gunn, may not have had a quorom for months, or even years, including the meeting where they fired Gunn.

Secretary of Transportation Mineta, who holds a seat on the Amtrak Board, appears to have never actually attended an Amtrak Board meeting.

I wonder what will come out next about all this…

Why is it that the media always uses “detained” instead of “arrested”, “jailed”, “tortured”, etc? Jordan isn’t delaying terrorists from reaching their destination… they’re putting them in jail. Let’s call it what it is.