What a p-value Actually Says
The p-value is the most cited and most misunderstood number in science. It appears in thousands of papers every day, gets reported in news articles about medical breakthroughs, and is used as the primary gatekeeper for whether a result gets published. It is also routinely interpreted incorrectly, even by the researchers who compute it.
Let's be precise about what it actually means — and what it doesn't.
The definition
A p-value is the probability of observing data at least as extreme as what you collected, assuming the null hypothesis is true.
That's it. That one sentence contains the whole idea, but it's worth unpacking.
Suppose you're testing whether a coin is fair. You flip it 20 times and get 15 heads. The null hypothesis is that the coin is fair (). You ask: if the coin really were fair, how often would I get 15 or more heads in 20 flips just by chance?
That probability — about 2% in this case — is the p-value.
What it doesn't mean
The p-value is not the probability that the null hypothesis is true. It's not the probability that you got a false positive. It's not the probability that your result will replicate.
This confusion is so common it has a name: transposing the conditional. The p-value is . What people want is — the probability the null is true given what you observed. These are related by Bayes' theorem, and they are not the same.
Imagine 1,000 researchers each test a different hypothesis. Only 100 of those hypotheses are actually true. Each test has 80% power (probability of detecting a true effect) and uses .
True effects detected: significant results.
False positives from the 900 null hypotheses: significant results.
Total significant results: .
Of those 125 "significant" findings, 45 are false positives — a false discovery rate of 36%. This is before any publication bias, p-hacking, or other problems. Just from the mathematics of testing many hypotheses.
If only significant results get published, the literature will be 36% wrong even with perfectly conducted experiments.
The significance threshold
The convention of was proposed by Ronald Fisher in the 1920s as a convenient rule of thumb. He explicitly said it should not be used mechanically. It was not designed as a universal truth threshold.
A p-value of 0.049 and 0.051 are essentially the same quantity — nearly identical evidence — but one gets published and the other doesn't, under mechanical significance testing. This is incoherent. The line is arbitrary.
A large enough sample will find a statistically significant difference between almost any two groups, because even tiny real differences will eventually produce small p-values. A study of a million people might find that left-handed people are 0.003% more likely to prefer tea, with .
This is "significant" in the statistical sense. In any useful sense, it is not significant at all.
Always ask: even if this effect is real, is it large enough to matter? Report effect sizes alongside p-values. The p-value tells you the signal exists; it says nothing about whether the signal is worth hearing.
What p-values are good for
Despite all this, p-values are genuinely useful when interpreted correctly. They quantify how consistent data are with a null hypothesis. They provide a common scale for comparing evidence across studies. They are a reasonable guard against announcing effects that might just be noise.
The problem is not the p-value — it's treating 0.05 as a universal truth machine, ignoring effect sizes, and only publishing significant results. Those are problems with practice, not with the tool.
The fix is not to abandon p-values but to use them alongside confidence intervals, effect sizes, replication, and domain knowledge. A p-value of 0.03 with a tiny effect, one study, and a surprising claim should be treated very differently from a p-value of 0.03 with a large effect, multiple replications, and a mechanistic explanation.
See Hypothesis Testing for the formal framework, and Confidence Intervals for the complementary approach.