Dave Shore Consulting
Disaster recovery: Hard lessons from September 11
(First published in Tech Republic - May 2002)
The events of 9/11 at the World Trade Centre have had a tremendous technology impact for the financial services firms in that location, as well as for those companies' global locations.
Exploring the technology and business ramifications following the tragic events of September 11. Were the plans adequate to cope with the initial disaster as well as the effectiveness of longer-term business continuity plans.
Approximately 8,000 Intel-based servers and approximately 5,000 UNIX servers were lost at an approximate replacement cost of $370 million. It has also been estimated that 30,000 securities positions (defined as trading, sales, research, and operations desks) were lost in the seven WTC buildings and another 15,000 to 20,000 positions in the adjacent buildings.
The second tier, the business continuity plan, comes into play during a longer-term, major incident, and would provide guidance on how to get the remaining 80 percent of the workforce back to work in the shortest possible time. This latter plan is now where much focus lies following the tragic events in New York City.
Business continuity plans should include the acquisition of alternate premises (possibly a move 11 September created unique challenges for disaster recovery All of the organisations affected by the 9/11 disaster have, to varying degrees, invoked disaster recovery plans. Many of the plans feature agreements with disaster recovery service providers that are contracted to provide emergency desk space.
Yet despite some prearrangements with providers, the events of 11 September brought about some unique difficulties for all parties. Although a company may hold an individual contract with a service provider, the emergency space promised in the agreement is usually shared with other companies.
Disaster recovery service providers base their plans on the probability that multiple clients will not invoke their agreements at the same time for the same space -- an unforeseen issue in the case of the New York City events. While the client organisation is busy restoring business as usual, the disaster recovery service provider is left with as many as six other clients who share that same disaster recovery space and cannot invoke that space if necessary. This leaves the disaster recovery service suppliers looking for alternate space as well, possibly competing with their clients to find suitable housing.
Further, service providers predict an expected duration of occupancy following an invocation. The typical expectation is that recovery from the incident will occur within days, or at worst, within two to three weeks -- clearly not enough time for the organisations affected by the events of 9/11. Given the scale of the destruction in New York, organisations are left with the prospect of having to occupy a disaster recovery site for the full term of their contract, usually a maximum of six months. During this time, business continuity plans must come into effect and should include the acquisition of alternate premises (possibly a move to another location within the same organisation) to allow the full complement of staff to resume normal work.
TowerGroup, a research and advisory firm, estimates that it will cost $3.2 billion to replace technology at the affected securities firms. Of this estimate, $1.7 billion will be spent on hardware -- from trading stations, sales stations, workstations, PCs, servers, printers, minicomputers, storage devices, cabling, and communications hubs to routers and switches. The remaining $1.5 billion will cover services and software to install and connect the necessary networks, operating systems, and applications infrastructures.
TowerGroup's estimates are based on replacement costs of $52,000 each for the 16,000 trading desks (including turrets and multiple workstations outfitted with multiple flat-screen displays) and $5,000 each for the 34,000 general PC workstations (including monitors, extra memory, software, and networking equipment) lost as a result of the terrorist attacks.
A frequently issued statistic regarding disaster recovery is that one in five organisations that suffer a major incident never recover and end up going out of business. Last September's events clearly illustrate why enterprises must plan for the unexpected and put processes in place to ensure that business continues in any possible scenario.
Plans must take all scenarios into account The extreme scenarios that have occurred as a result of the terrorist attacks in New York City provide a good lesson in how sometimes, even the best-laid plans can fail. Organisations must now learn from these types of events and adapt current disaster recovery and business continuity plans to reflect these new issues and needs.
A two-tiered approach to disaster planning Before September 11, anyone who had ever tried to persuade an organisation to part with money to fund disaster recovery and/or business continuity requirements was quickly met with resistance.
The difficulties in acquiring the necessary funding has typically led to a two-tier approach when coping with an immediate incident. The first part of the program, the disaster recovery plan, is put in motion when coping with an immediate (and often short-term) incident and traditionally provides for the most critical elements of the business -- for example, typically no more than 20 percent of an organisation's workforce will have a desk at which to work.
Dave Shore Consulting