Read a research claim critically: a worksheet and worked example

Use this guide when a headline, presentation, or paper claims that an intervention improves an outcome. It helps you write a defensible summary and identify what remains unknown. It does not replace a full methodological review.

1. Turn the headline into an answerable question

Write down the population, intervention or exposure, comparator, outcome, and time period. “A new teaching method works” leaves all five unclear. “Among first-year students, does optional weekly practice improve a four-week assessment score compared with usual study?” is something you can investigate.

Find the underlying report rather than relying on the headline. Record its version and date, whether it is a preprint or published article, and any correction or update. Peer review is useful context, not a guarantee that a result is correct.

2. Read the methods before adopting the conclusion

Ask how participants entered the study and how groups were formed. Random assignment and self-selection create different interpretive problems. In an observational comparison, motivation or prior attainment could explain part of a difference between groups.

Check what was measured. A satisfaction rating, test score, and long-term retention measure answer different questions. Look for a prespecified primary outcome and analysis plan; if these are unavailable, record that limitation rather than assuming they existed.

Then inspect missing data. Who dropped out? Did the analysis exclude people after assignment? Could the reason for missingness relate to the outcome? A larger sample does not automatically repair a biased comparison.

3. Translate the effect into understandable numbers

Here is a fictional study of a study-support programme:

Group	Students passing	Students assessed	Pass proportion
Programme	60	100	60%
Comparator	50	100	50%

The observed difference is 10 percentage points: 60% minus 50%. The ratio is 1.2: 60% divided by 50%. The relative increase is 20%, using the comparator's 50% as the baseline. “20% more passed” and “20 percentage points more passed” are different claims.

These arithmetic summaries do not establish causation, precision, or generalisability. The table alone cannot tell us whether assignment was random, whether all eligible students were assessed, or how well the result would transfer to another course. Cochrane's chapter on interpreting results explains relative and absolute effect measures in its evidence-synthesis context.

4. Keep size, uncertainty, and importance separate

Look for an interval estimate alongside the effect. Ask whether the range includes effects small enough to be unimportant or large enough to matter. Interpret it under the method's assumptions; a conventional frequentist confidence interval is not a probability distribution over the true effect.

The American Statistical Association's statement cautions that a p-value does not tell you the probability a hypothesis is true, and a threshold alone should not determine a conclusion. A statistically significant result can be practically small. A non-significant result can remain too imprecise to rule out a meaningful difference.

Set the practical question in advance where possible: what improvement would justify the programme's time and workload? That threshold is a judgement about the decision, not something a p-value supplies.

5. Place the study in a body of evidence

Look for other studies and systematic reviews addressing the same question. Differences in population, implementation, follow-up, and measurement can explain apparently conflicting results.

Cochrane's certainty-of-evidence chapter discusses bias, inconsistency, indirectness, imprecision, and publication bias. These are useful prompts for reading, but a quick checklist is not a formal GRADE assessment.

Copyable evidence note

Claim being checked:
Source, version, date, and correction status:
Population and setting:
Intervention/exposure and comparator:
Outcome and follow-up:
Study design and group allocation:
Sample size, exclusions, and missing data:
Observed effect in natural units:
Uncertainty interval and assumptions:
Main plausible sources of bias:
Related evidence and disagreements:
What this evidence supports:
What it does not establish:
What information would change the conclusion:

For the fictional example, an appropriate summary is: “The programme group had a higher observed pass proportion, 60% versus 50%. The supplied counts alone do not establish whether the programme caused the difference or how precise the estimate is.”

All sample figures and teaching examples here are invented. The worksheet is original practical guidance informed by the linked methodological sources.

Suggest an edit