Businesses Share Lessons Learned from Backup Storage Challenges
Three years ago, Worthington Federal Bank purchased a new storage system based on a dedicated backup server and LTO tape library. However, the bank’s virtualized infrastructure quickly demanded a more comprehensive, scalable backup solution.
“It’s grown a lot in the last three years,” says Matt Elmore, network administrator at the Huntsville, Ala., bank. “It’s gone from the Stone Age to the modern age of storage very quickly. And our needs continue to grow.”
Worthington’s backup system reached its limit within a year and a half. “We were constantly battling it,” recalls Elmore. “Backups would fail for one reason or another. What it boiled down to was that the solution’s approach to managing the backups was not very scalable.”
The bank’s IT team spent a lot of time and money trying to implement a VMware agent offered by its storage maker, but they couldn’t get it to work, even with the vendor’s assistance. “It was to the point where we just didn’t feel good, from a recovery standpoint, about where we might end up with this product if we continued along,” says Scott Jenkins, senior vice president and chief information and technology officer at Worthington.
It’s a common problem among small and medium-sized businesses struggling to keep pace with the growing storage demands of everything from virtualization and data retention to regulatory requirements. Many learn the hard way that not all storage solutions work for all companies.
In Search of Better Backup
When Jenkins signed a contract for a new backup system, he thought the IT team’s problems were over, but it turned out to be a step backward. “It quickly became apparent that they had oversold the solution,” he says of the vendor.
Jenkins’ team made it clear that the bank backed up to tape, but when staff began testing the new system, they discovered it only worked with disk-to-disk or disk-to-online backups. “It would have to have been a completely manual process — very cumbersome and time consuming — and it left such room for error that it was just not workable,” Jenkins says.
The vendor said it might be able to provide library functionality some time next year, but Jenkins notes, “I can’t operate my IT environment on promises. And I’m not going to spend $20,000-plus on a solution that has any ‘ifs’ tied to it.” It was frustrating, he adds. “I don’t know how many hours we put into this before we came to this showstopper.”
The projected amount of data stored worldwide in 2012, up from 1ZB in 2010
SOURCE: “IDC Predictions 2012: Competing for 2020” (IDC)
Jenkins and Elmore got on the phone with Worthington’s CDW representative, Kim Fiala, to explain the situation. “We said, ‘We need something, and we need it right away,’” recalls Jenkins. He had recently purchased a new tape library from CDW, but he worked directly with the manufacturer on the storage solution. “I wish we had talked to CDW sooner,” he says. “Maybe we would have found this out from them instead of experiencing it.”
After discussing Worthington’s needs with the CDW sales team, they decided to give CommVault Simpana backup software a try. The system was more expensive, says Jenkins, but even from the demo, they could tell it was everything they were looking for and then some. “Although the cost of entry may seem high, you have to consider the value of your data to your company,” he says. “Are you willing to take a chance with it?”
The CommVault product worked with VMware out of the box, and it had all the features the bank needed built in, explains Elmore. The licensing was straightforward, and CommVault’s long history with tape backups left the IT team confident they’d be able to use the software with the new tape library.
Worthington hired CommVault Professional Services to install the system, and everything went smoothly. “Within a day, we had all the major parts — VMware, SQL and all our files — working. I was pretty impressed,” says Elmore.
Their one regret is that they fought for their old system as long as they did. “When you’re doing the piecemeal stuff, you have fewer guarantees than if you’re using an appliance that’s designed for storage,” Elmore points out.
Storage Made Simple
When Nick Saber found the missing piece to his storage woes, he almost dismissed it as too good to be true.
His company, The Pulse Network, produces about 20 to 30 hours of video content per week. Employees were storing content on their local computers, then backing it up to external hard drives at the end of each workday. There was a shelf full of hard drives, and the backup process was unwieldy. “We were spending hours a day making sure our projects were backed up,” Saber says.
The Canton, Mass., digital media company had hoped to expand the enterprise-class storage solution it uses for its VMware infrastructure so that it would have enough space for its video editing and content creation as well. “But we got shell-shocked with quotes in the neighborhood of $50,000 to $100,000,” recalls Saber. “And that would just get us over the hump. Then we’d have to add to that.”
What is your company’s primary backup strategy?
40% SAN or NAS appliances
32% Tape storage
16% iSCSI or SATA drives
8% Cloud storage
4% Solid-state drives
SOURCE: CDW poll of 351 BizTech readers
Last January, at the Consumer Electronics Show, Saber passed the Intel booth, where representatives were demonstrating an Iomega network-attached storage (NAS) device. “I thought, if this really works as advertised, it’s unbelievable,” Saber says. “Where can you get 18 terabytes of storage with RAID 6, all the security measures built into it, and the ability to back up to the cloud — all for around $3,500? It’s crazy.”
Saber ordered the Iomega StorCenter px6-300d, a six-drive, high-performance desktop NAS device that delivers up to 18TB of network storage, figuring that if it didn’t work he’d only be out $3,500. A week later, he bought another. A month later, he purchased a third.
Pulse’s employees noticed a difference right away. The ability to put 18TB of data in one place instead of having to move it from one drive to another — and rifle through drives to find it — was invaluable, Saber says. Plus, they could easily copy data to another backup device.
Another advantage, he adds, is the product’s scalability. Pulse’s content grows so fast that Saber can’t say what the company’s storage needs will be in six months. But the Iomega product can scale from a home network to a major corporation. “It’s the same interface, the same application,” he says.
Saber expected it would take three or four days to implement the device, but it was up and running in just a few hours. “It shocked us, to be quite honest with you,” he says. “You essentially turn it on. It almost needs no configuration.” It’s also easy for IT to manage and for users to access and navigate.
Pulse did have a drive fail, but Iomega sent out a replacement, which Saber’s team was able to pop right into the device. Because it’s RAID 6, he explains, “there’s no loss of data, and it’s an easy fix.”
Saber has yet to find any drawbacks. “I’d call this a must-have device for any organization,” he says. “I don’t care if you’re a one-man firm, a lawyer or accountant — or an enterprise company.”
How long could your business survive without access to its data? A month? A week? A day?
Charlie Mortel has had several discussions with his team about that question. “It’s hard to get a gauge on how fast you need to get systems back up in disaster scenario,” says Mortel, networking administrative director at Chicago-based William Harris Investors. “The ballpark range we’ve talked about is a week. After a week of having our systems down, it may hinder performance.”
The IT team has been working to ensure that all the company’s mission-critical data and systems are stored securely enough that they could be retrieved within that timeframe. But it’s been a long road.
Three years ago, William Harris Investors installed a 3 terabyte Xiotech storage area network. “After a year and a half, we were quickly running out of storage,” says Mortel. The company added a capacity disk shelf with 8TB, but performance wasn’t good enough to support the business’s SharePoint and customer relationship management deployments.
After shopping around for solutions, Mortel and his team settled on the NetApp FAS2240 device because of its scalability. As the company’s needs grow, it can add high-density drive shelves rather than undergo a forklift replacement of the system.