Wednesday, March 19, 2008

You Know the Drill

I had been at work this morning just long enough to eat my cinnamon bun and get halfway through my first cup of coffee, when the phone rang. It was our network architect with a very important message.

"This is a drill. We're having a major weather emergency in the Data Center."

I knew this was coming. It had been three months or so since our last drill, and as a valued member of the Desktop/Network Support Team, I was sure to be on the call tree. (This had been an oversight during our last pseudo-emergency. More on this in a moment.)

Now obviously the best way to conduct these drills is to pretend that it's real. So I tried to get into the spirit of things. I popped up so I could see over my cubicle wall at the weather. It was cloudy, and the forecast was for scattered showers throughout the day. I could work with this.

"Oh my God! What's the situation?"

There was a pause on the other end, then: "Remember, this is just a drill. We don't know if it's a tornado or a microburst, but a tree just crashed through the windows in the Data Center. The power's out so we can't get in to assess the situation, but we know that SAP's down, and it looks like some of the servers are offline."

"What about phones? Do we have phone service?"

Again that annoying pause at the other end of the phone line.

"I think the phones are OK. Ummm. Why don't you come to the command center?"

I refilled my cup and made my way to the conference room. Somebody had brought donuts, so I snagged one on the way in. A man's gotta keep his energy up somehow in the face of disaster.

Unfortunately, there's very little for a Desktop/Network Support Team member to do in a meeting like this. All the damage was in the Data Center. In the case of an actual emergency, I'd be involved in cleanup, but that's for later. I was assigned to the phones. I was responsible for contacting the other techs and getting the word out to the userbase.

I hate being phone boy. My mood was ruined.

The Disaster Manager confirmed the tree through the window, and added that another set of windows had blown out. "The floor is flooded already, and more rain and debris are lashing through the openings. All the servers on the south side of the Data Center are down and probably heavily damaged. The rest of the servers are still online, but that could change any second. Any suggestions?"

The Network Admin was tasked to bring down the remaining servers as gracefully as possible. Manager, Architect, and a small team would inspect the Data Center for safety and to make a preliminary assessment. I pulled my cell and called the lead tech at our other campus as the team filed out to cross the hall to the center.

Once they were gone somebody raised his own concern: "Reckon we should have had Facilities cut the power before they entered a flooded out room with live high-voltage outlets running across the floor?"

All in all, this drill went a lot smoother for me than the last one.

As I might have mentioned, I wasn't notified of the previous drill. So I didn't have a clue that when I barrelled into the Data Center, to my little work area in the corner, that I was entering an inferno.

The security guard stuck his head in the other door. He looked surprised to see me. "You're dead."

"I am? Why?"

Turns out the huge power supplies against the wall had burst into flame, and the entire roomful of very expensive computer equipment was burning with an intensity that was melting the aluminum trackways conveying cables across the room.

I had to hang out in the hallway for a while. I called my wife and informed her that I was dead. She wasn't nearly as upset as I thought she'd be. I sure didn't expect laughter.

At the drill post-mortem (which I thought was a poor choice of words, given my sorry demise) I was mildly ridiculed for leaving a black mark on the drill results.

"Now wait," I said. "Are we assuming that all the alarm systems worked during the emergency? Then you're also assuming that I ignored an evacuation order, didn't notice the flashing alarm lights, didn't hear the sirens, and forged ahead through intense heat and impenetrable smoke, just so I could get to the workbench long enough to finish setting up Tony's laptop.

"I think anybody that dedicated deserves a raise."

1 comment:

Amy Mullis said...

Sit. Stay. Tag. Check out the rules at Mind over Mullis. Good Boy.