What a P-Value Actually Means

Why does “p < 0.05” show up everywhere?

You’ve seen it in every paper, every abstract, every poster session: p < 0.05. But if someone asked you to explain, in plain terms, what that number actually means — could you? In this blog, we’ll break down the p-value without the math, using what we just learned about the normal distribution and the Central Limit Theorem.

What a P-Value Really Is

A p-value answers one specific question: if there were truly no effect or no difference (this is called the null hypothesis), how likely would it be to see results as extreme as the ones you got, just by chance?

That’s it. A p-value is not a verdict on your hypothesis. It’s a measure of surprise: how unusual your data would look if nothing were actually going on.

This is exactly why the bell curve from our last blog matters so much. Most statistical tests compare your result to the distribution of results you’d expect from random chance alone, and that distribution is usually a normal one, thanks to the Central Limit Theorem.

A Simple Way to Picture It

Imagine flipping a coin 100 times and getting 65 heads. Is the coin rigged, or could that happen with a fair coin just by luck?

The p-value tells you: if the coin were perfectly fair, how often would you expect to see a result this lopsided (or more) purely by chance? If the answer is “rarely,” that’s evidence something’s off. If the answer is “pretty often,” you don’t have much to go on.

A small p-value means your result would be unusual if nothing were really happening. It does not mean your result is important, large, or even true.

What a P-Value Is NOT

This is where most confusion starts. A p-value is not:

• The probability that your hypothesis is true

• The probability that the result happened by chance (a common but incorrect shortcut)

• A measure of how big or important an effect is

• Proof of anything — it’s evidence, not a verdict

A p-value only tells you how surprising your data is under one specific assumption (the null hypothesis). Nothing more.

Statistical Significance vs. Clinical Significance

Here’s the part that trips up even experienced researchers: with a large enough sample size, even a tiny, meaningless difference can produce a p-value below 0.05.

Imagine a drug that lowers blood pressure by 0.5 mmHg, clinically irrelevant, but the study enrolled 50,000 patients. That tiny effect can easily become “statistically significant.”

That’s why a p-value should never be read alone. Always ask two questions:

• Is this result statistically significant? (the p-value)

• Is this result clinically meaningful? (the actual size of the effect)

Key Takeaways

• A p-value tells you how surprising your data would be if the null hypothesis were true, not whether your hypothesis is correct.

• p < 0.05 is a convention, not a law of nature — it means “this would happen less than 5% of the time by chance alone.”

• A p-value says nothing about effect size or clinical importance.

• With large enough samples, even trivial differences can become “statistically significant.”

Up Next:

Next, we’ll unpack confidence intervals which is a tool that tells you not just whether an effect is real, but how precise your estimate of it actually is.