The fact that “big data” is the next big thing is hardly new, but there are obstacles to its adoption. Manipulating data to get it to reveal patterns and useable information is hard and requires skills that many do not possess. In companies, it tends to be the preserve of IT departments.
Now, a company that grew out of research in Stanford, Calif., hopes to go some way towoard putting sophisticated and powerful data visualization tools into the hands of anyone comfortable using a spreadsheet.
We caught up with Jock Mackinlay, the director of visual analysis at Tableau software, during his fleeting visit to London. Tableau is a range of software tools that allow pretty much anyone who can navigate Excel to combine large datasets and produce rich interactive data presentations. The people it targets, he calls “data enthusiasts.” (Full disclosure: Some reporters at The Wall Street Journal have been using Tableau to create interactive graphics online.)
“They are a level below data scientists,” he said. “What they do is to use the data to tell a story. The ability to tell stories, he says, is a key part of making big data useful. “The moment you find out something interesting, the very first thing you have to do is go and tell someone about it.”
Like many in the field, Mr. Mackinlay is worried that the lack of skills is hampering the uptake of big data.
“To make it sing you have to be a programmer,” he said. “Right away you have instantly cut out most of the world. Requiring people to be able to program or write advanced database queries is a no go.”
So to, he says, is requiring your data researchers to have specialist presentation skills. “Not only should you not require people to be programmers, they should not be required to be skilled in perceptional psychology as well.”
Software solutions like Tableau do not obviate the need for data scientists, he said, but they do put rich interactive tools into the hands of ordinary people.
One of the promises of big data is that it will help to solve the Rumsfeldian “unknown unknown” problem. It was the then-U.S. defense secretary who grappled with the epistemological problem; there are things that are known, things that are known unknowns, and then a whole set of things that we don’t know we don’t know. One of big data’s promises is that it will allow executives to find out things they didn’t know they didn’t know.
According to Mr. Mackinlay, data visualization can help. “Say you plot the data graphically and you see something you don’t expect to see. You might have some errant point off to the side. If you have this interactive data experience then you can drill down into that data point. Was it just dirty data, or is it actually something you should be looking at?”
One of the other things that data visualization can do is address problems of cognitive bias, the situation where people think they know things that perhaps they don’t, or that they quickly draw conclusions from data that may not be right. “Visual analysis is often used to check if what you think you know is actually correct,” Mr. Mackinlay said.
“Cognitive bias is a problem for smart people. You start with this presumption that the world looks like, The smarter you are the faster you do that because that is what being smart is, forming conclusions from data,” he said. “But the problem is that sometimes you can be really wrong. If you can use data, and particularly if you can visualize that data, then you can test your theories.”
For examples of some interactive graphics see Tableau’s interactive gallery.