My recent post on broadening the usefulness of statistics presupposed two things about the statistical sophistication of business intelligence tool users:
- It varies a lot.
- In many cases, it isn’t be very high.
Let me now say a little more on the subject. My basic message is — people’s facility with statistics is extremely difficult to predict.
If you DO have to make a point estimate, however, you could do worse than just putting quotation marks around the last four words of that sentence …
Suppose we measure people’s statistical understanding on a 5-point scale:
- People who haven’t clue what a p-value is.
- People who think a p-value of .05 signifies a 95% chance of truth.
- People who know better than that, but who still think that “statistically significant” is pretty close to the same as “true”.
- People who know better yet, but aren’t fluent in using statistical techniques correctly.
- People who are fluent in statistics.
Just knowing somebody’s job description, can you confidently predict their ranking to within, say, +/- 1 point? I suggest you can’t. People differ wildly in general numeracy and in specific statistical knowledge.
Even our guesses about average knowledge may be off, not least because education is changing things. I got to graduate school without even knowing what a conditional probability was;* now a whole generation of kids is growing up with option of taking AP Statistics. On the flip side, a long list of recent studies suggests that research scientists, physicians, et al. are less clueful about statistics than we might have thought. A quick googling on statistical errors by scientists turned up:
- Several stories about a paper uncovering a particular, frequent error in published neuroscience papers.
- A list of 20 common errors in biomedical papers.
- Another paper about common errors in medical research.
- An Elsevier guide to common errors reviewers might find in submitted papers.
No wonder that a large fraction of medical research can’t be reproduced.**
*3 years later I had taught a low-end college course in statistics and written a PhD thesis on game theory …but let’s not over-generalize from that part of the story. Anyhow, these days my ranking would be somewhere in the 4 range.
**Another reason might be HeLa cell contamination, but I digress.