Running out of storage

Last week Jim Stallings, IBM's general manager of enterprise systems, told an audience about a conversation he had with a CEO of a major financial institution (he didn't name names).

The president, according to Stallings, knew how many servers his company had (about 50,000), but he had no idea how much data those machines could store. That, Stallings says, highlights the current crisis that many companies are facing. Thanks to the growing need to keep information for long periods of time, companies are finding it harder and harder to manage their storage infrastructure.

The need for data storage has risen tremendously over the past few years, fuelled by everything from the demands of compliance regulations to a surge in file sharing via Web 2.0. The problem doesn't end with companies just finding a way to store their data either. That extra data is driving up energy costs, can be difficult to access, and requires finding qualified people who can manage these systems.

Tommy Hilfiger, the company founded by the American designer, has seen its storage demands drastically increase over the last few years, thanks to the company's decision to comply with the Sarbanes-Oxley law in the US. The company is private, but wants to keep open the option of issuing an IPO, according to Arthur Vermeer, a technical services director at the company. Three years ago, the company was able to keep all of the relevant financial data on seven terabytes of storage. Today it needs about 50TB of data, or the equivalent of 11,000 DVDs.

Storage issues are not just for the IT managers either, Stallings says. They've now become a Boards of Directors-level problem.

"Board members want technology that will cut costs at data centres. They want to see contributions - and I don't just mean money."

According to IBM, the IT industry will spend over $50 billion on running and cooling its machines, which is small compared to the $150 billion the industry will have to spend maintaining them. Since 1996, that represents an 800 per cent increase in management costs, and a 600 per cent increase in energy costs. Those numbers are going to continue to grow, too. IBM believes that 86 per cent of US companies will expand their data centres in the next year, due to compliance regulations spelled out in the Sarbanes-Oxley law that requires extensive filing of financial records.

Issues

Financial institutions aren't the only ones being hit by compliance regulations.

"Privacy issues for health care providers are onerous," says Krishna S. Nathan, IBM's vice-president of storage systems development. While data usage itself has skyrocketed - by about 55 per cent annually - Nathan estimates that in the medical field, storage needs have grown by as much as 300 per cent as medical centres in the US move to digital record-keeping, which includes everything from patient charts to X-rays.

The need for more storage is also being seen here in the Middle East, according to Omar Shihab, an analyst for IT-research firm IDC.

"The main focus in this region is from governments and the financial service sectors, but to a large degree, the area is just going through the first found of automation," he says, adding that growth here would continue to rise. "Financial lending is one space that is seeing a lot of governance pushed down on it."

However, Shihab said the Middle East is still lagging behind in implementing current storage technology, although he did identify a number of leaders in the area, including the UAE, Qatar and Jordan.

Technology steps in

To cut down on the amount of storage required, many companies are turning to virtualisation, a technology designed to make storage devices more efficient in how they store data. Many companies discovered the way they were storing data - using multiple hard disk drives - resulted in a lot of unused space.

According to Dieter Meünk, who handles storage and business development for IBM, companies without virtualisation are only using 40 to 50 per cent of their storage capacity on average, and to add insult to injury, much of the information being stored is either irrelevant or a duplicate of something else. He estimates that only 10 to 15 per cent of un-virtualised storage systems are being utilised for relevant information.

With virtualisation, those drives are combined into a just a few (the number can actually be determined by the company) virtual drives that cuts down on the amount of unused space. The more towards storage virtualisation - as opposed to desktop and server virtualisation - is being spearheaded by companies like EMC, Hitachi, HP and IBM.

IBM is also focussing on a relatively new technology: de-duplication. Any one looking for a video on YouTube has probably noticed that the same video can be available under several different names. Each copy of that file eats up more disk space.

That doesn't mean de-duplication software takes suicidal measure of deleting potentially valuable corporate info. Instead IBM's software identifies files that are identical with the exception of header information, such as the title. It then keeps a single copy of the identical information while retaining all the relevant differentiating data, which will remain linked to the main file.

To further cut down on energy consumption, companies are also turning to an old technology that many people in IT thought was obsolete: tape. Stallings estimates that 75 per cent of corporate data sits on tape today.

The advantage of tape is, unlike a hard disk drive which can continue to spin even when not in use, tape doesn't move unless specific information is required, resulting in less power consumption. Tape drives are generally slower at retrieving data and the storage capacity of tape is often higher that disk-drives, making the medium a more desirable solution for long-term storage.

"It's still very cost-efficient," said Dieter Meünk. "Overall, tape is a very good step towards long term or ultra long-term storage"

Another alternative to standard hard drives is the same technology behind the ubiquitous USB drives, flash memory. Unlike hard disks and tape drives, flash memory has no moving parts and has a lower latency, the term used to describe the speed at which data can be retrieved.

However, Meünk said it could be three to four years before flash memory is ready for enterprise storage. The cost of flash far exceeds that of drives or tape and the volatility of the memory is also far higher, meaning that every time a computer writes to flash memory, it shortens the memory's lifetime. IBM estimates flash drives working under the heavy work loads associated with enterprise software may need to be replaced in as little as six months. In comparison, corporate hard drives can last for years and tape can last for decades, although most are replaced after a few years.

"I would not invest in another hard disk plant, but I wouldn't shut one down either if I had one," he said, adding that flash memory will be 20 to 30 times more expensive for the foreseeable future.

But while software and hardware solutions can help, companies are also being pushed into being proactive in identifying data they don't need to keep.

Many large corporations are attempting to keep all their data as long as possible, according to Andy Monshaw, general manager of IBM's system storage division.

"That's the state of the nation right now. Companies are saying 'I have to save it because of compliance issues'. That's the wrong approach," Monshaw said, who said IBM is offering to help companies retain the data required by regulations, a challenge that often requires IBM to work with a company to determine exactly just what sometimes nebulous regulation actually require.

But at the same time, companies are trying to avoid some of the problems of the past, he said. Changes in technology have resulted in valuable information being lost of the years, due to changes in hardware, such as the migration from floppy disks to hard drives.

"Companies don't want to have the same problems as previous generations," he said.

To further cut down on energy consumption, companies are also turning to an old technology that many people in IT thought was obsolete: tape.Stallings estimates that 75 per cent of corporate data sits on tape today