Dave Shore Consulting
Lessons Emerging from Disaster
(First published in Waters Magazine and on Waters Online June 2002 edition)
Web access to market data and other tools could trim costs of maintaining redundant office space. Many organisations that were affected on Sept. 11 found themselves in the midst of their worst-case scenario with respect to disaster recovery and business continuity planning.
Business continuity plans should include the acquisition of alternate premises (possibly a move to another location within the same organisation) to allow the full staff to resume normal work. Setting up a new installation from scratch will take time, possibly even longer than the six-month interval specified in the disaster recovery service contract. While the client organisation is busying itself restoring business as usual, the disaster recovery service provider is left with as many as six other clients who share that same disaster recovery space and cannot invoke that space if necessary. This leaves the disaster recovery service suppliers looking for alternate space as well, possibly competing with their clients for suitable housing.
The TowerGroup has estimated that it would cost approximately $3.2 billion to replace the technology destroyed for securities firms impacted in the World Trade Centre disaster. Of this estimate, TowerGroup further anticipated that $1.7 billion would be spent on hardware, consisting of trading stations, sales stations, workstations, PCs, servers, printers, minicomputers, storage devices, cabling, communications hubs, routers, switches and the like. The remaining $1.5 billion would cover the services and software needed to install and connect the network, operating system and applications infrastructures of these securities firms.
A typical disaster recovery plan only provides desk space for the most critical elements of a business. Thus, no more than 20 percent of an organisation’s workforce will usually have a desk to go to. The business continuity plan has to cover how the other 80 percent of the workforce can get back to work in the shortest possible time, thus providing business continuity.
The investment banking sector has a huge investment in disaster recovery and business continuity planning for obvious reasons. Having dealers and traders out of commission for even a few minutes can cost millions of dollars. Despite the size of the potential losses though, even here the disaster recovery plans usually provide sufficient facilities for only 20 percent of the head count to allow dealers/traders to close positions as quickly as possible. The planning assumption is that normal service to the primary office environment is quickly restored. A major factor in the decision to only provide limited recovery facilities is the cost of having “redundant” office space ready to use in case of an incident. This is especially true in the investment banking environment. Here, dealers and traders are always utilising the latest technology and it is very complex to set up. Providing desk space with a workstation at a disaster recovery site for a dealer/trader position is in the region of three times more expensive than that of an ordinary office desk utilising general office systems.
Before Sept. 11, anyone who had ever tried to persuade an organisation to part with money to fund disaster recovery and/or business continuity requirements learned of the difficulties involved. Frequently asked questions such as “what do I get for my money” have to be fended off and justifications put forward. The difficulties in acquiring the necessary funding has led to a two-tier approach with separate plans to cope with an immediate (and often short-term) incident--the disaster recovery plan; and the second-tier of recovery during a longer-term major incident--the business continuity plan.
If dealers and traders could access the exact same information through the Internet as they can at their work location, then conceivably they can work from anywhere, even from home. New products that exploit the Internet offer the promise of helping reduce both the complexity of a dealer/trader desktop and the cost of provision. VistaSource, for example, a Westboro, Massachusetts-based corporation that builds software tools for retrieving and analysing real-time market data, offers a Web-enabled version of its software called the Anyware RealTime Websheet. For a services approach, Rimes Technology Corporation offers Web-based services for traders through RIMES Online. Either way, the effect is to create a mobile or virtual trading desk.
All of the organisations affected by the New York disaster to varying degrees invoked their disaster recovery plans. This involved agreements with disaster recovery service providers contracted to provide emergency desk space previously put into a state of semi-readiness. Although under contract, the space is usually a shared space with multiple companies contracting for the same space.
Herein lies the next layer of problems, not only for the organisations invoking their plans, but also for the service providers. Disaster recovery service providers work on the basis of the probability that multiple clients will not invoke their plans at the same time for the same space. The problem born out of the situation last fall in New York is that clients that invoked their disaster recovery plans occupied the space that might have be required by another organisation should an incident should strike them. The expectation is that recovery from the incident will occur within days, or at worst two to three weeks.
The scale of the massive destruction that took place in New York, left organisations with the prospect of having to occupy their disaster recovery sites for the full term of their contract, usually a maximum of six months. During this time, business continuity plans have to come in to effect.
Solutions such as these would better enable those firms whose dealers and traders were affected by the events of Sept. 11 to simply send them home to work. The disaster recovery site would need only to implement a new server and install the firm’s analytics and models created with the software tools. Dealers and traders are free to work from anywhere that provides suitable Web access. Even better, they will see the same familiar screen image on their PC no matter where they are. For the IT departments, this means that they only have to maintain the server systems and do not have to connect all the data feeds to each dealer/trader desktop as they do currently.
The implications of this technology are that dealers and traders would have been able to go to an office out of town, or even go home, and continue dealing and trading as if nothing had happened. Business resumption would be reduced to the time required for the dealer/trader to connect to the Internet, and the time required for the IT department to have the backup server up and running.
Dave Shore Consulting