Proofs based on probabilities are usually meaningless
I want to believe as much as the next person that particle physicists have discovered a Higgs boson, the so-called “God particle,” one with a mass of 125 gigaelectronic volts (GeV). But so far I do not buy the statistical claims being made about the discovery. Since the claims about the evidence are based on “statistical significance” – that is, on the number of standard deviations by which the observed signal departs from a null hypothesis of “no difference” – the physicists’ claims are not believable. Statistical significance is junk science, and its big piles of nonsense are spoiling the research of more than particle physicists.
I’m an economist. So don’t trust me with newfangled junk bonds or the fate of the world financial system. But here is something you can believe, and will want to: Statistical significance stinks. In statistical sciences from economics to medicine, including some parts of physics and chemistry, the ubiquitous “test” for “statistical significance” cannot, and will not, prove that a Higgs boson exists, any more than it can prove the reality of God, the existence of a good pain pill, or the validity of loose monetary policy.
A statistically significant departure from an assumed-to-be-true null hypothesis is by itself no proof of anything. Likewise, failure to achieve statistical significance at the .05 or other stipulated level is not proof that nothing of importance has been discovered.
It sounds too simple to be true, but in fact the two most fundamental problems with the test of statistical significance stem from bits of faulty logic.
The test of significance customarily begins with the stipulation of a “null hypothesis.” Expressed in algebraic terms, a new object A is on average assumed to be no different from a familiar object B, where the objects could be types of weight loss pills, tax schemes, or differently named physical particles. Data are collected, experimentally or otherwise, and then a calculation is made to determine the likelihood that data greater than that which we see on average could have occurred if in fact there is no observable difference between the objects under study.
Statistical significance is junk science, and its big piles of nonsense are spoiling the research of more than particle physicists
The formal name for this odd calculation is “p value”. If the p value has a low number – in social sciences and business, if p falls below .05, or a 1-in-20 chance – the result of the experiment is said to be “statistically significant.” The claim is that the new object A, for example, is statistically significantly different from the old object B, because the chance of seeing a bigger difference between A and B – bigger than the difference you have seen – is small. It’s a strange standard.
Likewise, if p exceeds .05 (or whatever arbitrary line the scientists have drawn – lower in physics, higher in business), the result of the experiment is said to be inconclusive, and thus ignorable.
The null hypothesis test procedure is not the only test of significance but it is the most commonly used and abused of all the tests. From the get go, the test of statistical significance asks the wrong question. The test asks: “Assuming that the null hypothesis is true – that the Higgs boson (or whatever) does not exist – what is the probability of seeing a result at least as large as the one we have seen in the data?” This probability calculation is the p-value.
In framing the quantitative question the way they do, the significance-testing scientists have unknowingly reversed the fundamental equation of statistics. Believe it or not, they have transposed their hypothesis and data, forcing them to grossly distort the magnitudes of probable events – and here’s why.
They have fallen for a mistaken logic called in statistics the “fallacy of the transposed conditional.” If Mrs. Smith gets a cramp this week, and dies, one could not simply conclude that Mrs. Smith probably died from a cramp. This is because the probability of having a cramp, given that you are dead, is not equal to the probability that you are dead, given that you had a cramp. Mrs. Smith might have died for any number of other reasons. But that is precisely the reversal of hypothesis and data that, however illogical, particle physicists in Geneva – and most other scientists in fields from economics to medicine – continue to make.
I show in a book I wrote on the subject with Deirdre N. McCloskey, The Cult of Statistical Significance (2008), that the null hypothesis test procedure – another name for statistical significance testing – produces many such errors, with tragic results for real world economies, law, medicine, and even human life.
In a decades-long survey of leading journals, from the American Economic Review to the New England Journal of Medicine, we find that eight or nine of every 10 articles assumes that statistical significance demonstrates scientific, economic, or other human importance, and that a lack of statistical significance – statistical “insignificance,” or a p-value greater than .05 – indicates a lack of importance. Statistical significance is neither necessary nor sufficient for proving a physical, economic, or medical result. But the bureaucracies of science –from government grantors to journal referees – continue to insist on demonstrations of statistical significance, regardless of the real economic, medical, physical, or other effects revealed by the total evidence.
Consider again the illogic of the physicists’ procedure. The signal in the data which has been observed over and above background noise (denoted as being at 5 sigma) is possibly a Higgs boson – that is true. But in sober moments – when flash-bulbs fade and fizzy drinks fall flat – those same particle physicists admit that the jury is still out – that the statistically significant bump could be “consistent with” other plausible hypotheses, not specified by their models – just like Mrs. Smith could have died of something other than cramp, and probably did.
This is self-evident. A statistically significant result might be evidence of some other particle or field – a Jove or Zeus or Prometheus just like the anticipated Higgs. But because the models used by the physicists do not assign probabilistic weights to Higgs and its competing hypotheses, prior and posterior beliefs about all hypotheses remain static, neglected, and unknown.
Thus the reported chance of finding a Higgs boson – measured, the physicists illogically claim, by their super-small p-value – is incorrect. The p-value merely shows the likelihood that data that were not observed, did not occur – that is, particles heavier than 125 GeV were with high probability not explained by the null hypothesis of “no Higgs boson.” But it’s impossible to go from “I see something different from the null hypothesis” to “I see my favourite hypothesis” without adding in some new assumptions, taking us from the fallacy of the transposed conditional to clear statements about the probability of the favoyred hypothesis, such as Higgs.
Significance testers won’t say what those additional assumptions are, but they seem content to make additional inferences based upon those faulty assumptions.
There is a second, equally fundamental problem with the theory and practice of significance testing. The test does not tell us how big or small the effect size is; it doesn’t tell us how important (or useful, or dangerous, or surprising) the effect is, in a metric of big and small – what McCloskey and I call the “oomph.” As the eminent statistician, Leonard “Jimmie” Savage noted in Foundations of Statistics (1954), statistical significance tells us what to say but not what to do. Think of the he-said, she-said quality of the debate about “age” as a “significant” factor in mammogram testing. Young women could be forgiven for thinking the need to test is a real coin flip.
Take another random example: whale hunting. In June 2005 the Japanese government increased the limit for the number of whales which may be killed in Antarctica – from 440 whales annually to over 1,000. In the face of international opposition Deputy Commissioner Akira Nakamae told BBC World News: “We will implement JARPA-2 [the plan for additional killings] according to the schedule, because the sample size is determined in order to get statistically significant results.” Using the methods of statistically significance, the Japanese government raised the sample size to reach a conclusion that killing more whales was acceptable. To “kill more whales” is “to be more significant”—by raising sample size – as if more precise. Sadly, this backward logic is common in marine biology as much as in field experiments in economics at the World Bank.
Or consider the “significance” of damage done in a case involving thousands of humans, some of them dead. In the early 2000s quite a few Vioxx takers experienced the wrath of the so-called 5% rule of statistical significance.
The clinical trial was conducted in 2000 and published in the Annals of Internal Medicine (2003). The sponsoring company, Merck, reported that five patients taking Vioxx suffered heart troubles – fatal and not – during the clinical phase. That compared with only one bad result in the control group, “a difference [in bad outcomes] that did not reach statistical significance.” The erroneous belief among junk scientists is that failing to reach statistical significance is the same as finding no important difference between the two outcomes. On top of that investigators discovered they did not report three of eight total bad outcomes – to achieve an insignificant difference, it seems – the error opposite of the one committed by whalers.
Banishing this significance junk seems possible. Even the U.S. Supreme Court agrees. On March 22nd, 2011, in Matrixx Initiatives, Inc. v Siracusano, an important case of securities law, the Supreme Court unanimously rejected use of bright-line rules of statistical significance as a way of hiding adverse information from investors.
The case involved a homeopathic medicine called Zicam, a zinc-based common cold remedy produced by Matrixx Initiatives, Inc. When applied through the nose the drug causes some users to experience burning sensations and others to suffer anosmia, the permanent loss of smell. Matrixx ignored a number of adverse reports it had received from doctors and users since 1999. One doctor told the company that zinc toxicity was discovered by biologists back in the 1930s. When a doctor appeared on Good Morning America in 2004 and spilled the beans on the company, Matrixx stock price plummeted. Again the company hid, this time behind the argument that the adverse effects from taking Zicam up the nose were –¬ wait for it –¬ “not statistically significant.”
The Supreme Court rejected the conventional argument unanimously. The Court concluded: “Matrixx¹s argument [about the adverse effects of Zicam] rests on the premise that statistical significance is the only reliable indication of causation. This premise is flawed.”
Said Justice Breyer sarcastically during oral arguments: “This statistical significance always works and always doesn’t work.” In other words, the high Court agrees: hard-and-fast significance rules are junk science and need to be scrapped.
Stephen T. Ziliak is Professor of Economics at Roosevelt University in Chicago.