Evidence appraisal
Translate results into absolute patient benefit, then assess whether the study can support the claim. A large relative effect or a small p-value is not sufficient by itself.
Quick revision summary
Precision and significance
A confidence interval describes uncertainty and precision under the statistical model. For a ratio measure, the null is 1; for a risk difference, it is 0.
Read effect size in the patient’s context
Relative measures describe proportional change; absolute measures depend on baseline risk. If an event falls from 10% to 6%, the absolute reduction is 4 percentage points and the relative reduction is 40%.
Assess bias and applicability before adopting the conclusion
Ask whether the study design can answer the question. Randomisation can reduce confounding, but allocation, missing data, deviations from treatment and selective reporting still matter.
Common mistake
A statistically significant result may be too small, uncertain or inapplicable to justify a clinical change.
Treatment effects, confidence intervals and study quality
Translate results into absolute patient benefit, then assess whether the study can support the claim. A large relative effect or a small p-value is not sufficient by itself.
Worked effect measures
| Measure | Example: risk falls from 10% to 5% | Interpretation |
|---|---|---|
| Absolute risk reduction | 0.10 − 0.05 = 0.05, or 5 percentage points | Five fewer events per 100 treated over the stated period. |
| Relative risk | 0.05 / 0.10 = 0.5 | Treatment risk is half the comparator risk. |
| Relative risk reduction | 1 − 0.5 = 0.5, or 50% | Must be interpreted with baseline risk. |
| NNT | 1 / 0.05 = 20 | Treat 20 people to prevent one additional event over that period. |
| Number needed to harm | 1 / absolute excess adverse-event risk | A separate outcome that belongs in the benefit–harm discussion. |
| Odds ratio | Odds of an event in one group divided by the other | Not identical to relative risk; approximation is better when events are rare. |
Precision and significance
A confidence interval describes uncertainty and precision under the statistical model. For a ratio measure, the null is 1; for a risk difference, it is 0. A wide interval can include clinically important benefit, no effect and harm. Compatibility with the null is not proof that the treatments are identical.
A p-value addresses the compatibility of the data with a specified model and null hypothesis, not the probability that the treatment works. Statistical significance does not establish clinical importance, freedom from bias or applicability. NNT depends on outcome, baseline risk and follow-up period; omitting the period makes comparison misleading.
Study design and bias
Randomisation reduces confounding on average; allocation concealment prevents foreknowledge influencing recruitment; blinding limits performance or detection bias where feasible. Attrition, selective reporting, deviations and poorly defined outcomes can still weaken an RCT. Intention-to-treat preserves the assigned groups for the relevant analysis but does not magically repair missing data.
Cohort and case-control designs have important roles, especially for exposures, prognosis or rare outcomes, but association can reflect confounding or selection. A systematic review is only as informative as its included evidence, methods and heterogeneity. Ask whether participants, intervention, comparator and outcomes match the clinical decision you face.
Read effect size in the patient’s context
Relative measures describe proportional change; absolute measures depend on baseline risk. If an event falls from 10% to 6%, the absolute reduction is 4 percentage points and the relative reduction is 40%. The number needed to treat is 1 divided by 0.04, or 25, for the outcome and follow-up represented by those data. These are illustrative calculations, not treatment claims.
The same relative effect can produce a smaller absolute benefit in a lower-risk population. Explain the outcome, comparator, time horizon and uncertainty whenever quoting an NNT. Include adverse effects, treatment burden and patient preferences rather than assuming that a favourable effect estimate settles the decision.
Assess bias and applicability before adopting the conclusion
Ask whether the study design can answer the question. Randomisation can reduce confounding, but allocation, missing data, deviations from treatment and selective reporting still matter. Observational associations may reflect differences between groups rather than a causal treatment effect. Laboratory findings can explain mechanisms without directly predicting clinical survival.
Check whether the participants, intervention, setting and outcomes resemble the clinical decision. A surrogate improvement may not establish a patient-important benefit. Confidence intervals express statistical uncertainty but do not account for every source of bias. A systematic review is only as useful as its methods and the relevant evidence it includes; pooling weak or incompatible studies does not automatically create certainty.
A distinction to remember
A statistically significant result may be too small, uncertain or inapplicable to justify a clinical change.
What extra information is needed before comparing two published NNT values?
Reveal the reasoning
The outcome, population / baseline risk, intervention and comparator, follow-up period and uncertainty. NNT values without this context may describe very different clinical questions.
Guidance to read alongside this chapter
Test your recall
Does statistical significance prove clinical importance?
Show answer
No; the size and relevance of the effect also matter.
