Asking the right questions
During Thursday’s Forrester-sponsored Twitter conference (“Tweet Jam”) on “What Business Intelligence Is and Is Not” (search Twitter for #dmjam), I remarked that BI in general faces the problem that “people ask the wrong questions,” and that this will get worse with the impact of social media, applications, platforms, and patterns. I was challenged to provide some examples, but:
What I mean by “the wrong questions” is a problem that afflicts BI already, a peculiarly active form of the more general problem known as “confirmation bias,” that becomes more acute in the presence of “social media” dynamics.
You always come to BI with some sense of understanding how things work, some guesses at what you’re going to find. Often, your BI serves primarily to confirm your suspicions—and that’s a good thing, because it empowers you to move more forthrightly in the right direction. A problem arises, however, when your guesses and suspicions are actually wrong: because you frame the questions and interpret the answers, there’s still a strong tendency for the BI to confirm. Confirmation of your wrong guesses and misplaced suspicions is still going to empower you to move forthrightly, but in these cases, you’ll be moving in the wrong direction! Analysts generally understand this; decision-makers sometimes don’t. A critical part of the business analyst’s job is examining additional analyses that have the potential to disprove current guesses and suspicions. If they hold up, if you fail to disprove, great: you’re on the right track; you might not even show those results to the decision-maker. If you find holes, though, then even better: you’re saved from walking into a pit—or you will be, as soon as you show and explain them.
All that applies already, in “conventional BI.” But “social media” introduce a new problem: cooperative structures (like the OAuth system for delegated authentication) mean that some of the information you’d like is primarily in someone else’s hands. It may be that the protocol and partnerships allow you access to this extra information, but can you afford the performance and programming costs to collect it? If, as an analyst, you’ve routinely been looking at more data than you show the decision-makers (and you really should!), then you have a hard job ahead explaining why you suddenly need to spend so much more to collect information they’ve never seen! There’s a risk that you’ll only have access to the core information, the information that can only confirm guesses and suspicions that were already held at the time the system and partnerships were designed: you might lose richness of the data necessary for those counter-guess tests.
Some examples, since that was the challenge:
Imagine a company that offers a free download of a limited version of their product. For a variety of reasons, you institute a registration system for the downloads. It’s a common enough arrangement: it increases potential customer exposure to the product, it identifies better-qualified leads, and it provides great, instant feedback on the effectiveness of new releases and ad campaigns. You’re going to mine those registrations and downloads for all the BI you can squeeze out. Lots of obvious, guess-confirming questions, there. But here’s a guess-denying question that should be asked (hoping, of course, that the answer is “no”): do people have some other, non-registered way to download the same files? Because if they do, then a lot of them are going to find it. You need the info on registered downloads, but you also need the comparative info on downloads that bypass the registration. Someone who thinks they know the system is likely to assume—without thought—that there aren’t any unregistered downloads; someone else needs to wonder and check.
Filed under: Business Intelligence | 3 Comments