It’s been said a million times and in a million places that a p-value is not the probability of given the data.
But there is a different type of confusion about p-values. This issue arose in a discussion on Andrew’s blog.
Andrew criticizes the New York times for giving a poor description of the meaning of p-values. Of course, I agree with him that being precise about these things is important. But, in reading the comments on Andrew’s blog, it occurred to me that there is often a double misunderstanding.
First, let me way that I am neither defending nor criticizing p-values in this post. I am just going to point out that there are really two misunderstandings floating around.
(1) The p-value is not the probability of given the data.
(2) But neither is the p-value the probability of something conditional on .
Deborah Mayo pointed this fact out in the discussion on Andrew’s blog (as did a few other people).
When we use p-values we are in frequentist-land. (the null hypothesis) is not a random variable. It makes no sense to talk about the posterior probability of . But it also makes no sense to talk about conditioning on . You can only condition on things that were random in the first place.
Let me get more specific. Let be a test statistic and let be the realized valued of . The p-value (in a two-sided test) is
where is the null distribution. It is not equal to . This makes no sense. is not a random variable. In case the null consists of a set of distributions, the p-value is
You could accuse me of being pedantic here or of being obsessed with notation. But given the amount of confusion about p-values, I think it is important to get it right.
The same problem occurs when people write . When I teach Bayes, I do write the model as . When I teach frequentist statistics, I write this either as or . There is no conditioning going on. To condition on would require a joint distribution for . There is no such joint distribution in frequentist-land.
The coverage of a confidence interval is not the probability that traps conditional on . The frequentist coverage is
Again, there is no conditioning going on.
I understand that people often say “conditional on ” to mean “treating as fixed.” But if we want to eradicate misunderstandings about statistics, I think it would help if we were more careful about how we choose our words.