How to read a medical study, from relative risk to sample size

Start with the question the study asked

Every useful study answers a narrow question, so find it first. Ask who took part, what was compared with what, which outcome was measured and over how long. A result in young men followed for a year says little about older women over a decade.

Well reported trials make this easy. The CONSORT 2010 statement, used by medical journals, sets out a checklist of 25 items and a flow diagram that show how many people were enrolled, allocated, followed up and analysed [1]. If a report does not say how many people dropped out, treat its result with care.

Why the design matters

In a randomised controlled trial, chance decides who receives which treatment, so the groups differ only by the treatment and by luck. Even so, randomised trials can give biased results if they are not designed and run with care [1].

Observational studies follow people who made their own choices, such as those who take a supplement and those who do not. They can show an association and cannot on their own show cause, because the people who choose differ in many other ways. Animal and laboratory studies come earlier still, and their results often do not carry over to people.

Relative risk and absolute risk

The same result can be told in two ways. The Cochrane Handbook gives the example of a treatment that cuts the risk of death from 4% to 3%, which can be described as a 25% relative reduction or as a reduction of 1 percentage point in absolute risk [2]. Put another way, about 100 people would need the treatment for one of them to avoid death, a figure called the number needed to treat [2].

Gigerenzer and colleagues showed how this plays out in screening, where a 25% reduction in the risk of dying from breast cancer with mammography means 1 fewer death for every 1,000 women screened [3]. A Cochrane review of 35 studies found that a relative risk reduction is perceived as larger and is more persuasive than the same result given in absolute terms [4], so ask for both figures.

Sample size, chance and confidence intervals

A study reports its best estimate with a confidence interval, the range within which the true effect probably lies. The width of that interval depends to a large extent on the sample size, and larger studies tend to give narrower, more precise intervals [2].

In the Handbook’s example, an odds ratio of 0.75 with an interval of 0.70 to 0.80 is a precise estimate, while an interval of 0.50 to 1.10 shows little knowledge about the effect [2]. An interval that crosses 1 includes the possibility of no effect at all. The Handbook advises reporting the interval with the exact P value, and against leaning on the label statistically significant [2].

From paper to press release to headline

Many health stories begin as press releases, and the release can say more than the paper. A study of 462 press releases from UK universities found that 40% contained exaggerated advice, 33% exaggerated causal claims and 36% exaggerated inference to humans from animal research [5]. When a release exaggerated, the news stories that followed usually did too [5].

Gigerenzer and colleagues recommend asking for absolute risks in place of relative risks, and mortality rates in place of survival rates, because a higher survival rate with screening does not mean a longer life [3]. When a headline surprises you, find the paper and read its methods and its limitations.

A checklist for reading a study

Take these questions to any study or health story before you change what you do.

• Who took part, and are they like me?
• Was it a randomised trial, an observational study or work in animals?
• What was the outcome, and was it one that matters, such as illness or death?
• What was the absolute change in risk, and how many people needed treating?
• How many people took part, and how wide was the confidence interval?
• How many dropped out, and were they counted?
• Does the headline say more than the paper?
• Who paid for the study?

How The Wellness can help

On our own pages, a statistic sits beside its source, and if we cannot point to the source the number does not go on the page [6]. If a study has made you wonder about a test, a supplement, a diet or a treatment, bring it to a GP appointment, which is £150, and a GMC registered doctor will go through what it found and what it means for you. A longevity consultation by video is free.