The p-value remains one of the most widely used yet frequently misunderstood statistics in medical research. This article provides a clear, practical guide to interpreting p-values correctly while highlighting the most common statistical errors that lead to false conclusions. You will learn what a p-value actually measures, why it is not the "probability that the null hypothesis is true," and how to avoid common mistakes like p-hacking, misinterpreting non-significance, and treating p-values as effect sizes. Real medical examples and actionable tips are included throughout.
A p-value is the probability of observing your data (or something more extreme) if the null hypothesis is true. In medical research, the null hypothesis usually states there is no effect or no difference between groups.
For example, if you test a new drug versus a placebo, a p-value of 0.03 means that, assuming the drug has no real effect, there is a 3% chance you would see a difference as large as the one observed purely by random chance.
This definition is precise, but it is easily twisted. The p-value does not tell you the probability that the null hypothesis is true. It does not tell you the size of the effect. It is a tool for assessing compatibility between your data and the null hypothesis.
Correct interpretation requires context. A p-value alone is never enough to draw a conclusion.
"A p-value does not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone." — American Statistical Association Statement on P-Values
Even experienced researchers fall into these traps. Being aware of them helps you read medical literature more critically.
A very small p-value does not mean a large effect. In a large trial, a tiny improvement in blood pressure can yield p < 0.001. The clinical relevance depends on the magnitude of the difference, not the p-value alone.
A statistically significant result may be clinically irrelevant. A drug that lowers cholesterol by 1 mg/dL with p = 0.01 might be statistically significant but offers no meaningful benefit to patients.
Running dozens of tests and only reporting the ones with p < 0.05 drastically inflates false-positive rates. Preregistering analyses and correcting for multiple comparisons is essential.
A non-significant p-value (e.g., 0.20) does not prove the null hypothesis. It simply means the data are not strong enough to reject it. The true effect could be small, or the study underpowered.
Every p-value calculation depends on assumptions like normality, independence, and equal variance. Violating these can make p-values misleading.
Sample size directly affects p-values. In small studies, even large effects may not reach significance. In very large studies, trivial effects can produce tiny p-values.
| Scenario | True Effect | Sample Size | P-Value (approximate) | Interpretation |
|---|---|---|---|---|
| Small effect, large study | Very small | 10,000 | < 0.001 | Statistically significant but clinically trivial |
| Large effect, small study | Large | 20 | 0.08 | Not significant, but effect may be real — underpowered |
| Moderate effect, adequate study | Moderate | 200 | 0.03 | Statistically significant and likely clinically relevant |
| No effect, any study | Zero | Any | ~0.05 (5% of the time) | False positive if p < 0.05 |
A randomized trial tests a new blood pressure medication. The primary endpoint is systolic blood pressure change after 12 weeks. The p-value is 0.002, and the mean difference is 8 mmHg with a 95% confidence interval of 3 to 13 mmHg. The p-value indicates strong evidence against no effect. The effect size is clinically meaningful. This is a well-interpreted use of the p-value.
An observational study reports that patients taking a certain supplement have a 20% lower risk of heart disease with p = 0.04. The researchers conclude the supplement protects the heart. But the effect size is modest, the confidence interval is wide, and the study is observational — confounding is likely. The p-value alone cannot establish causation.
"It is easier to produce a statistically significant result than a scientifically meaningful one." — Modern medical statistician
Many journals now encourage reporting beyond p-values. These tools help you interpret results more completely.
Despite their flaws, p-values remain useful in medical research when interpreted correctly alongside other statistical measures. They provide a standardized way to quantify evidence against a null hypothesis. The key is to avoid treating them as automatic decision rules.
Think of a p-value as one piece of a puzzle — never the whole picture. When you combine it with effect sizes, confidence intervals, study design, and biological plausibility, you get a much clearer view of the truth.
P-values are not the enemy of good science, but their misinterpretation is a real problem. In medical research, where decisions affect patient lives, getting the interpretation right matters. Always ask: How big is the effect? How precise is the estimate? Could the result be due to bias or chance? By treating p-values as continuous measures of evidence rather than binary pass-fail tests, you will become a more critical reader and a better researcher. Keep the common mistakes in mind, use confidence intervals and effect sizes alongside p-values, and remember that no single number can replace thoughtful scientific judgment.
No. This is a common misconception. The p-value is not the probability that either hypothesis is true. It only tells you how likely the observed data (or more extreme) would be if the null hypothesis were true. To calculate the probability that a hypothesis is true, you need Bayesian methods that incorporate prior information.
No. A p-value does not measure the size of the effect, the clinical relevance, or the quality of the study design. You need confidence intervals, effect sizes, and an assessment of bias and confounding before making decisions about treatment effectiveness.
The significance level (often alpha = 0.05) is the threshold you set before the study to decide whether to reject the null hypothesis. The p-value is the actual probability calculated from your data. If the p-value is less than alpha, the result is called statistically significant.
Larger sample sizes increase statistical power, meaning even tiny true effects become detectable. A very small p-value in a massive study may reflect a trivial effect that has no clinical importance. Always check the effect size.
No. The difference is negligible. P-values are continuous, and small fluctuations around the threshold are common. Dichotomizing results based on an arbitrary cutoff (0.05) is a well-known statistical error. Focus on the confidence interval and effect size instead.
P-hacking involves running multiple analyses, selectively reporting significant results, or adding participants until the p-value falls below 0.05. To avoid it, pre-register your analysis plan, correct for multiple comparisons, and report all tests you performed, not just the significant ones.
Not directly. P-values depend on sample size, effect size, and variability. A study with a p-value of 0.01 is not necessarily more important than one with p = 0.04. Comparing confidence intervals or effect sizes across studies is far more informative.
It often means the study was underpowered to detect a clinically meaningful effect. A non-significant result does not prove the treatment is ineffective. You should examine the confidence interval to see if a meaningful effect remains plausible.
Yes, but most journals now require p-values to be reported alongside confidence intervals and effect sizes. Some journals actively discourage overreliance on p-values and encourage Bayesian or other approaches. Always follow the specific reporting guidelines of the journal.
Start by reading the abstract and then the results section carefully. Look for confidence intervals, effect sizes, and sample sizes. Ask yourself: Is the effect large enough to matter clinically? Could bias explain the result? Practice with real medical papers and discuss with colleagues or statisticians.
