In the context of reporting the latest on hurricanes/himmicanes, Freese comes up with a new one for the lexicon. Considering the latest manipulations performed by the hurricanes/himmicanes people, the sociologist writes:
Like statistical chemotherapy, even though it slightly poisons their key result, it still leaves it alive just below the conventional statistical cutoff (p = .035). But the diseased result in the wrong direction is now above the .05 threshold . . .
Bingo! An excellent contribution from the author of the quip, “more vampirical than empirical—unable to be killed by mere evidence” [we quoted here, for example] which applies to so much published research, on topics ranging from hurricanes to, ummmm, I dunno, ovulation and voting?
I find it distressing that people have so much difficulty admitting they could’ve made a mistake. I do attribute some of my own willingness to admit error to my training as a statistician. Maybe another thing that helped me along was the bad experience I had with my colleagues when I worked at the University of California, which gave me some appreciation for pluralism in a very general sense.
The ethics question
Another question is: at what point does stubborn denial and refusal to accept criticism (which is how I read the behavior of the hurricanes/himmicanes researchers) edge into flat-out unethical scientific behavior? I certainly don’t think it’s unethical for researchers to make mistakes or to publish mistaken work. But at some point, if they keep fighting, it makes me wonder whether at some level they realize what’s going on.
My take on it (without knowing these people personally) is that they started the exchange with Freese and others under the reasonable assumption that they’d done things correctly (after all, their paper got through peer review at the prestigious
tabloid Proceedings of the National Academy of Sciences), but at some point when the criticisms started coming in faster and harsher, they switched to a war footing. They felt they were being attacked. And once you’re in a war (and you’re convinced that “the other side” will use any statistical tactic to get you), you feel that any statistical tactic is allowed in response. Nowhere do they give evidence that they ever stepped back to think: Hey, maybe we’re wrong!
That progression is all understandable to me. Still, as scientists we have some responsibility to the public. And to defend and defend like they’re doing, this seems unethical to me. At some point, as a researcher, you have to shift from ignorance as a defense to a recognition of your limitations. If you don’t do that, you are in some sense violating a public trust.
P.S. This discussion from Bob O’Hara shows a revealing quote from the hurricane/himmicane authors:
The ladder test uses Stata’s sktest to determine normality transforms, based on the work of D’Agostino, Belanger, and D’Agostino Jr (1990), with an adjustment made by Patrick Royston, (1991), one of the leading statisticians worldwide in smoothing and transforms. He is the original author of fractional polynomials, which is much better than GAM at smoothing on complex situations. The results are the same with the Shapiro-Wilk and Shapiro-Francia tests for normality.
This paragraph indicates an unfortunately common attitude, that statistics is a collection of tests which can be applied without serious consideration of the application in question. At some level, of course, statistics is the science of defaults, and I wouldn’t write books full of statistical methods if I didn’t think they could be applied in some generality—but the quote just above give a sense of how this attitude can go wrong.