Survive the Storm with Disaster Recovery
The term “disaster recovery” evokes images of devastation and destruction, but even mundane system glitches can cripple businesses. Porte Brown, an accounting firm based in Elk Grove Village, Ill., had a glimpse of disaster when its Microsoft Exchange server crashed at the height of the 2010 tax season.
Fortunately, the firm’s IT team was able to remedy the problem before suffering any long-term damage, but it wasn’t pretty, recalls Technology Support Specialist Jon Heller.
“You don’t realize how much you use an Exchange server until it goes down,” Heller says. “I have a lot less hair now than I did before this, and it’s a lot grayer.”
Despite the oft-repeated refrain about the importance of business continuity planning, many companies live with incomplete, out-of-date or inadequate plans until they face potentially devastating scenarios themselves.
The lucky ones, like Porte Brown, escape virtually unscathed, with only memories of stressful close calls. But many aren’t so lucky. Nearly three-fourths of small and medium-size businesses report having disaster recovery plans, according to Rachel Dines, an analyst at Forrester Research. She adds, “I think the question is, are those plans robust enough?”
Of the 74 percent with plans, 17 percent admit that they don’t test them, Dines says, citing the Forrester/Disaster Recovery Journal November 2010 Global Disaster Recovery Preparedness Online Survey.
Many plans are unexpectedly put to the test. At Porte Brown, the IT team spent a week trying to figure out what was wrong with the Exchange server before finally transferring all the data to a new server. All the while, it was fielding complaints from harried accountants working around the clock to finalize returns for their clients. “It failed; it happens,” Heller says of the server. “But it failed at the worst time in the world for our firm.”
Porte Brown, which also has an IT consulting arm, had been thinking about virtualizing its servers for some time, but it was always on the future-projects list. “When [the server crashed], it pushed us to do it,” says Heller.
Last October, the firm went live with eight virtualized servers sitting on two physical boxes. If one physical server goes down, Heller can now easily move everything to the other. He can also control the servers remotely if he can’t get into the office. “I can manage it from my iPhone if I need to,” he says.
There are costs to create a virtualized environment. But it can make disaster recovery a lot less expensive down the road, Dines points out. Having some sort of virtualized recovery system in place probably offers SMBs the most bang for the buck, she says.
Get Some Sleep
Dan Elder knows the value of peace of mind. He has all-too-clear memories of an old job that kept him up at night. “What if that server fails?” he’d wonder at some ungodly hour. “We don’t have a backup.” As with many small companies, business continuity planning was lip service, Elder says.
Since he started as vice president of IT and infrastructure at Sprott in Toronto in 2007, Elder still wonders, “What if …?” But he doesn’t stop there. The asset management company, which has 150 employees in Canada and the United States, has a detailed business continuity plan that clearly identifies which systems are most critical to operations (e-mail, file servers, BlackBerry Enterprise Server and the domain controller).
The plan then spells out potential disaster scenarios and the steps necessary to keep the business running (see “What’s the Worst that Could Happen?”).
“My infomercial line is, ‘It lets me sleep at night,’” says Elder. “In our business, X amount of minutes can equal a lot of dollars, up or down.” Walking through such scenarios can go a long way toward warding off real-life disasters. But as any IT pro knows, plans and real life don’t always mesh. Testing plans is just as important as developing them.
“It might sound a little cliché, but practice makes perfect,” says Jeff Rooney, senior systems engineer at Network Merchants Inc., an electronic-payment facilitator in Schaumburg, Ill. When disaster strikes, many learn the hard way that their business continuity plans are missing pieces. “The devil’s in the details,” Rooney adds.
NMI leases space in a Chicago-area Tier 1 data center, and in 2009, it deployed a mirrored data center in a CDW facility just outside of Madison, Wis. — far enough away that it’s on a separate power grid, but close enough to drive to during an emergency. For testing, NMI pushes all of its real-time traffic to the backup site. “You can test individual pieces,” Rooney says, “but if they don’t communicate with each other, that’s definitely a piece that can bite you.”
Bite Back
Testing can uncover issues such as a lack of capacity, Rooney adds. Any time NMI scales its systems, it makes sure the backup site can handle the increased capacity. Otherwise, there might not be enough power to run them all.
A similar mistake that many businesses make is overlooking application interdependencies, says Forrester’s Dines. If an Exchange server is mission-critical, then all the systems it relies on, such as Active Directory, are critical, too.
But a strong plan goes far beyond the technology, says Dines. It starts with the business requirements. She advises companies to determine what their critical business processes are and how long they can afford to be down, then build their solutions based on the answers. For some, disk-based backup is fine. Others may just have one critical application that would be best to outsource or run in the cloud.
Head in the Clouds
When Elder first joined Sprott, the business was in the midst of a growth spurt. The firm had been in a shared network environment and was ready to shift to its own network. Elder researched collocation spots and data centers, generators, power and servers. He purchased software from Computer Associates that would let him replicate and fail over virtual servers.
Around that time, he got a call from a cloud provider and realized that the vendor already had the data center, the virtual servers and the exact software he was about to deploy. “It was a turnkey solution where they had done exactly what I had planned to do,” he says. “It was a no-brainer.”
When you consider management, hardware licensing and maintenance — not to mention the additional person Elder would have had to hire to help him — the cloud solution was much cheaper than building a network in house, he says.
As a test, Elder unplugged the file server from the network, and it failed over to the cloud within 35 seconds. He walked back to his desk and answered a call about the downed server, but before the caller could complete his sentence, he cut himself off. “Never mind,” the user said to Elder. “I’m back up.” He had no idea he was accessing a server 50 miles away. “Nothing was lost,” Elder says.
Of course, cloud computing isn’t for everyone. Businesses in regulated industries are particularly wary of the cloud because of the potential privacy risks. But along with virtualization, it is one of the growing trends in business continuity planning.
To stay on top of new developments in this area, keep an eye on what large businesses are doing, Rooney recommends. When working on NMI’s disaster recovery plan, he learned a lot about best practices from his CDW representative, who works with large clients in tightly regulated industries.
Then, when you find out what others are doing, challenge yourself to improve upon that, he adds. “From our point of view, if you’re not continually working on the process, continually improving, you’re already behind.”
