Americas

  • United States

Asia

Memory-Lane Monday: An ounce of prevention is worth a pounding headache

opinion
Sep 21, 20202 mins
Backup and RecoveryData Center

Not the greatest decision ever made.

Computerworld  |  Shark Tank
Credit: Computerworld / IDG

A very large thunderstorm is headed for this company’s data center, and there’s reason to worry, says a pilot fish working there.

The company is the product of several mergers back in the late ’90s, when it was decided that the newest data center would be the new company’s production data center. That particular data center was, of course, built for a much smaller company than the merged entity, so it had to be immediately expanded.

Laments fish: “In the intervening years, the quantity of servers and network and disk hardware in the data center increased rapidly, but the UPS and emergency generator didn’t keep up with the growth.”

Nonetheless, with that big storm on the way, a high-level manager overrides the advice of the technical support team and decides to switch the data center over to emergency power, rather than risk the possibility that the city power could be disrupted.

The emergency generator is started up and the switchover is made on a Friday evening without a hitch.

But the generator had never been tested since it was installed, and the diesel fuel it runs on had been sitting in the tank for a long time, so sludge has accumulated in the bottom of the tank.

When the sludge reaches the fuel line, the engine sputters to a stop. Shortly after that, the underpowered UPS runs out of juice.

And the entire data center shuts down catastrophically.

“The important servers and network were back up and running within 24 hours,” reports fish. “But some of the large databases took days to recover due to data corruption.

“And by the way, the city power never had an interruption.”

sharky

Questions that Sharky gets a lot

Q: What's a pilot fish?

A: There are two answers to that question. One is the Mother Nature version: Pilot fish are small fish that swim just ahead of sharks. When the shark changes direction, so do the pilot fish. When you watch underwater video of it, it looks like the idea to change direction occurred simultaneously to shark and pilot fish.

Thing is, sharks go pretty much anywhere they want, eating pretty much whatever they want. They lunge and tear and snatch, but in so doing, leave plenty of smorgasbord for the nimble pilot fish.

The IT version: A pilot fish is someone who swims with the sharks of enterprise IT -- and lives to tell the tale. Just like in nature, a moment's inattention could end the pilot fish's career. That's life at the reef.

Q: Are all the Sharky stories true?

A: Yes, as best we can determine.

Q: Where do the Sharky tales come from?

A: From readers. Sharky just reads and rewrites and basks in the reflected glory of you, our readers. It is as that famous fish-friendly philosopher Spinoza said, "He that can carp in the most eloquent or acute manner at the weakness of the human mind is held by his fellows as almost divine."

Q: Do I have to write my story in Sharky-ese?

A: No. Not at all. Just be sure to give us details. What happened, to whom, what he said, what she said, how it all worked out. If Sharky likes your tale of perfidy, heroism or just plain weirdness at your IT shop, he will supply his particular brand of Shark snark.

Q: I've got a really funny story, but I could get fired if my old trout of a boss found out I told you. How confidential is what I send to Sharky?

A: We don't publish names: yours, your boss's, your trout's, your company's. We try to file off the serial numbers, though there's no absolute guarantee that someone who lived through the incident won't recognize himself. Our aim is to share the outrageous, knee-slapping, milk-squirting-out-your-nose funny tales that abound in the IT world, not to get you fired. That would not be funny.

Q: How do I get each new Shark Tank tale emailed to me?

Easy. Subscribe to the newsletter.

Q: Where are the Sharkives?

Tales of old can be found in Sharky's archive.