Thursday 19 July 2007

Business Continuity, always expect the unexpected

When discussions turn to business continuity talk generally turns to server failures, power outages, fires and the like. However there is always the unexpected.

The centre of Nottingham, where our office is located, was without mains water today Taps Run Dry In City. Seems a leak had developed but Severn Trent Water couldn't find it.

This seemed quite funny at first. We have water bubblers for drinking water anyway and people were happy to hold-on, imagining it to be a temporary situation. By the time lunchtime arrived and still no sign of a fix we had to give people the option of going home.

Fortunately, most people stayed on so the show went on and business carried on as usual. We are lucky in that most people, certainly in sales, operations and support, can do they jobs from home in a crisis situation, but it's not ideal. It's another scenario to be considered in our contigency planning.

Another example was given to me by a friend who is responsible for his company's IT infrastructure, including their datacentre. In the recent floods, their datacentre flooded. The cable void under the floor was filling with water at an alarming rate.

Suddenly he was faced with questions like "Where do I hire a pump?", "How big a pump do I need?", "Where do I pump the water too?", "How quickly can IT engineers pail water?". Questions you don't want to be asking for the first time on a Saturday afternoon when your servers are on the verge of bath.

Preparation, redundancy and contigency planning are key to surviving most incidents. Unfortunately the unexpected laughs in the face of planning so having people with the ability to think effectively on their feet is probably as important.

No comments: