Dealing with Big Data: Change Your Thinking, Not Just Your Technology

Finding business insights in a flood of "big data" requires organizations to change their processes, not just their technology.
Organizations have access to more data, from more sources, in more formats, than ever before. This "big data" may contain insights that could save money, uncover customer needs or predict market changes. Until now, the sheer complexity of how to store and index large data stores, as well as the information models required to access them, have made it difficult for organizations to convert this data into insight. Uncovering those insights requires prioritizing and organizing this "big data" based on its ability to deliver business value.

Big Data Challenges

Big data refers to data sets so large they become awkward to capture, store, search, analyze and visualize using conventional tools. Much of this data is in the unstructured form of documents, videos, or text that is difficult to fit into traditional databases. It also contains "multiple versions of the truth" in the form of data organized for different purposes at different times, or similar data obtained from different sources.

Before analysis users must validate data created at different times for different purposes by different sources to determine which are most accurate. Delays may also result from recovering data from "unofficial" locations such as user's desktops. Finally, the increase in the volume of data objects has made access schemes overly complex, so that finding the data that matters is akin to finding a needle in a haystack.

As a result, existing storage management technologies and processes cannot make all this information available in a neatly organized warehouse accessible when the business needs it.

Big Data: Competitive Necessity

Only eight years ago, a 300 to 400 terabyte data warehouse was considered an outlier. Today, multi-petabyte warehouses are common. Failure to derive the most insight at the least cost from the information pouring into the enterprise will become an ever-larger competitive disadvantage.

Quick links