Cite as: Oxman AD, Chalmers I, Dahlgren A (2022). Key Concepts for Informed Health Choices: 3.1 Evidence should be relevant. James Lind Library (www.jameslindlibrary.org).

© Andy Oxman, Centre for Epidemic Interventions Research, Norwegian Institute of Public Health, Norway. Email: oxman@online.no

Make well-informed choices.

What to do depends on judgements about a problem, the relevance of the available evidence, and the balance of expected benefits, harms, and costs.

The first step in making decisions about treatments is being clear about what your problem or goal is and what the options are. Only then can you make judgements about the relevance of the available evidence from treatment comparisons, including the relevance of:

• the outcomes that were measured,
• the participants in the comparisons,
• the treatments that were compared, and
• the circumstances in which the treatments were compared.

In this essay, we explain those considerations.

Once the trustworthiness of the available evidence has been assessed (as explained in essays 2.1-2.4) and its relevance (as explained in this essay), well-informed decisions about what to do depend on judgements about whether the expected advantages outweigh the expected disadvantages, including consideration of:

• the benefits, savings, harms, and costs of acting or not,
• the baseline risk or severity of the symptoms,
• the importance of each advantage and disadvantage,
• the certainty of the evidence, and
• the need for further fair comparisons.

We explain these considerations in the next essay (3.2).

Be clear about what the problem or goal is and what the options are.

Good decisions depend on correctly identifying the problem and considering an appropriate set of options to address the problems. For personal health choices, this means starting with a correct diagnosis (or assessment of risk) and then identifying the treatments that are available. For public health and health system policy decisions, this means describing the problem correctly and identifying the policy options relevant for that problem. Changing how a problem is framed can lead to different options for addressing it.

The following passage from Archie Cochrane’s autobiography is a striking illustration of the importance of correctly identifying the problem and considering appropriate options [Cochrane 1989]. He recalls an event when he was a doctor in a German prisoner of war camp. “The Germans dumped a young Soviet prisoner in my ward late one night. The ward was full, so I put him in my room as he was moribund and screaming and I did not want to wake the ward. I examined him. He had obvious gross bilateral cavitation and a severe pleural rub. I thought the latter was the cause of the pain and the screaming. I had no morphia, just aspirin, which had no effect. I felt desperate. I knew very little Russian then and there was no one in the ward who did. I finally instinctively sat down on the bed and took him in my arms, and the screaming stopped almost at once. He died peacefully in my arms a few hours later. It was not the pleurisy that caused the screaming but loneliness. It was a wonderful education about the care of the dying. I was ashamed of my misdiagnosis and kept the story secret.”

Consider the relevance of the outcomes measured in the research.

A fair comparison may not include all outcomes – short- and long-term – that are important. Patients, professionals, and researchers may have different views about which outcomes are important. For example, studies often measure outcomes, such as heart rhythm irregularities, as surrogates for important outcomes, like death after heart attack. The effects of treatments on surrogate outcomes often do not provide a reliable indication of the effects on outcomes that are important. Similarly, short-term effects may not reflect long-term effects.

Despite dozens of randomized trials since the introduction of the first oral agent for treating type 2 diabetes, it has remained uncertain if any of those medicines favourably affects outcomes that are important to people, including morbidity, mortality, and quality of life [Montori 2007]. A key reason for this is that the trials have focused on glucose control measured with laboratory tests rather than on outcomes that are important to people with diabetes. Unfortunately, those laboratory tests (glycosylated haemoglobin, type A1C – HbA1C) are not a reliable indicator of outcomes that are important to people with type 2 diabetes.

It is sometimes important to consider outcomes that are important to other people besides the person being treated. For example, the use of antibiotics may increase antibiotic resistance, and not being vaccinated for Covid-19 or not avoiding contact with other people may increase the risk of infection for others. Similarly, when decisions are made for a group of people rather than for individuals, the outcomes that are important to anyone who is affected should be considered.

Consider the relevance of fair comparisons in laboratories, animals, or highly selected people.

Studies that only include animals, or only a selected minority of people, may not provide results that are relevant to most people.

Table 1 shows examples of misleading extrapolation from animals or a selected minority of people, found in news reports [Haneef 2015]:

Table 1. Examples of misleading extrapolation from animals or a selected minority of people found in news reports*

* Based on [Haneef 2015]

Consider the relevance of the treatments that were compared.

A fair comparison of the effects of a surgical procedure done in a specialised hospital or delivered by an experienced surgeon may not provide a reliable estimate of its effects and safety in other settings, or in the hands of less experienced surgeons.

For example, results from a large randomized trial showed that endarterectomy (surgical removal of part of the inner lining of an artery) for asymptomatic carotid stenosis (narrowing of the large arteries on each side of the neck) reduced the five-year absolute risk of stroke by about 5% [Rothwell 2005]. However, the trial only accepted surgeons with a good safety record, rejecting 40% of applicants and subsequently barring those who had adverse operative outcomes in the trial from further participation. The benefit from surgery was largely attributable to the low operative risk. Operative mortality was eight-fold higher outside of the trial and the risk of stroke and death was about three-fold higher.

Similarly, comparing a new drug to a drug or dose that is not commonly used (and which may be less effective or safe than those in common use) would not provide a relevant estimate of how the new drug compares to what is commonly done.

For example, in randomized trials of atypical antipsychotics for schizophrenia, haloperidol (one of the most frequently prescribed “typical” antipsychotics worldwide) was used as the comparison treatment [Hugenholtz 2006]. However, the trials used haloperidol in doses that were higher than that recommended. In a meta-analysis of 52 randomized trials that controlled for the higher-than-recommended dose of comparator drugs, differences in effectiveness and overall tolerability between typical and atypical antipsychotics disappeared, suggesting that the perceived benefits of atypical antipsychotics were due to excessive doses of the comparison treatments, such as haloperidol [Geddes 2000].

Consider the relevance of the circumstances in which the treatments were compared.

Some treatment comparisons are designed to find out if a treatment can work under ideal circumstances, for example, with people who are most likely to benefit and most likely to comply, and with highly trained practitioners who deliver the treatment exactly as intended. These comparisons, which are sometimes called ‘explanatory’ or ‘efficacy’ studies, may not reflect what happens under usual circumstances.

The North American Symptomatic Carotid Endarterectomy Trial is an example of an “explanatory randomized trial” [Barnett 1991, Thorpe 2009, Zwarenstein 2008]. It demonstrated that a lower risk of stroke was experienced by highly selected patients with severe narrowing of the large arteries on either side of the neck (carotid stenosis). Those patients who had a high risk of stroke were most likely to respond to surgical removal of the inner layer of those arteries (endarterectomy). Participating patients were closely followed and were operated on by skilled surgeons in academic and specialist hospitals with adherence to a strict protocol. The trial showed that endarterectomy reduced the risk of stroke under those ideal circumstances, but it did not provide a reliable estimate of the beneficial and harmful effects of endarterectomy under more typical circumstances.

Implications

• Make sure you are considering the correct diagnosis or problem, and appropriate options for addressing it.
• Always consider the possibility that important outcomes may not have been addressed in fair comparisons. Avoid being misled by surrogate outcomes.
• Results of systematic reviews of studies in animals, or highly selected groups of people, may be misleading.
• Be aware that treatments available to you may be sufficiently different from those in the available research studies that the results of those studies may not apply to you.
• Be aware that the results of studies with the aim of finding out if a treatment can work may overestimate the benefits of a treatment given under more usual circumstances.

This James Lind Library Essay has been republished in the Journal of the Royal Society of Medicine 2023;116:177-180. Print PDF

< Previous Essay | Next Essay >

References

Barnett HJM, Taylor DW, Haynes RB, Sackett DL, Peerless SJ, Ferguson GG, et al. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. N Engl J Med. 1991;325(7):445-53. https://doi.org/10.1056/nejm199108153250701

Cochrane AL, Blythe M. One Man’s Medicine. An autobiography of Professor Archie Cochrane. London: The British Medical Journal; 1989.

Geddes J, Freemantle N, Harrison P, Bebbington P. Atypical antipsychotics in the treatment of schizophrenia: systematic overview and meta-regression analysis. BMJ. 2000;321(7273):1371-6. https://doi.org/10.1136/bmj.321.7273.1371

Haneef R, Lazarus C, Ravaud P, Yavchitz A, Boutron I. Interpretation of results of studies evaluating an intervention highlighted in Google Health News: a cross-sectional study of news. PLoS One. 2015;10(10):e0140889. https://doi.org/10.1371/journal.pone.0140889

Hugenholtz GW, Heerdink ER, Stolker JJ, Meijer WE, Egberts AC, Nolen WA. Haloperidol dose when used as active comparator in randomized controlled trials with atypical antipsychotics in schizophrenia: comparison with officially recommended doses. J Clin Psychiatry. 2006;67(6):897-903. https://doi.org/10.4088/jcp.v67n0606

Montori VM, Gandhi GY, Guyatt GH. Patient-important outcomes in diabetes–time for consensus. Lancet. 2007;370(9593):1104-6. https://doi.org/10.1016/s0140-6736(07)61489-5

Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82-93. https://doi.org/10.1016/s0140-6736(04)17670-8

Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009;180(10):E47-57. https://doi.org/10.1503/cmaj.090523

Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390. https://doi.org/10.1136/bmj.a2390