I was on a call the other day when a friend commented,
“What is this Big Data thing? It feels so new and overwhelming. How can I get my arms around it?”
I guess a lot of people feel this way these days. I sure did for a long time, but here are some things I learned about Big Data; and more importantly how I learned it. Big Data isn’t just lots of data in big data warehouses. Its about distributing lots of data warehouses across lots of database servers who each have a dedicated purpose to analyze that data in parallel processing and provide results faster than any one server could in the same amount of time. That extra power gives organizations the ability to analyze massive amounts of information that heretofore just wasn’t possible. Of course there are lots of IBMers who can explain Big Data better than I can, but what I’m interested in is not so much the technological landscape of Big Data but how organizations are doing it – the use cases.
Lately, in the Information Governance Community, we’ve been exploring Big Data Governance Use Cases via teleconferences with members. The use case examples from companies like IDC and Sabre are helping us to understand implementation patterns, opportunities and challenges, as well as anxieties and cultural opposition. We’re taking what we learn and encoding it into a new Big Data Category in the Information Governance Maturity Model. Its an opportunity for many thousands of Community members to contribute small amounts of observed behavior and see those contributions woven into a far larger fabric of Community Insight.
But we shouldn’t just learn from each other. We should also take in what others are doing outside our Community. Because there is a lot of good work being done with Big Data all over the world. Here are a few examples I’ve come across:
Price Discovery outside of Markets. In a blog post two years ago, I remarked how 60% of all equity trades today are done in what are called Dark Pools. Dark Pools are the result of bi-lateral contract agreements between buyers and sellers who consummate transactions outside the boundaries of market exchanges. They use markets like NYSE for price discovery, but execute their own transactions. This is the result of the growing sophistication of technological trading platforms that allow buyers and sellers to find each other without needing to be members of a market. That’s an important development but markets are still essential to discover the going price of things folks want to buy.
But Big Data can change that too. In this example, some folks used Big Data to study “Rice” and “Price” in Twitter streams from Indonesia. They found that doing this long enough allowed them to discover the market price of rice without using a commodity market. These people are using publically available data to do price discovery in real time that is far cheaper and faster than market based discovery.
That’s a revolution.
When businesses, governments, and private citizens can use Big Data technology to analyze massive streams of publicly available information to divine market prices, understand trends, and digest relationships, many existing economic models of value creation will fall – market research, publishing, proprietary trading, consulting. Any field dependent on specialized knowledge can be endangered by super fast data analysis on massive streams from disparate sources.
On April 27, the Information Governance Community will continue the discussion of Big Data Use Cases. I’ll update my blog every week with new ones I find. Bring yours as well. We need all the use cases we can find to inform our discussion and refine our understanding. And join the Community. Its the largest, most vibrant, and active of its kind in the data management world. And its free and welcoming.