Cite as: Oxman AD, Chalmers I, Dahlgren A (2022). Key Concepts for Informed Health Choices: 1.1 Assumptions that treatments are safe or effective can be misleading. James Lind Library (www.jameslindlibrary.org).

© Andy Oxman, Centre for Epidemic Interventions Research, Norwegian Institute of Public Health, Norway. Email: oxman@online.no

This is the first of four essays in this series explaining Key Concepts that can help you avoid being misled by untrustworthy treatment claims.

Claims about the effects of treatments are made all the time – in print, broadcasting, and digital media, and in personal communications by friends, family, colleagues, healthcare providers, shamans, and others. Often the basis for those claims is unclear. Sometimes they are based on assumptions that treatments are safe or effective.

Trust in claims about treatment effects may be based on seemingly logical assumptions about research or about treatments. Or it may be based on the assumption that the source of a claim alone is a sufficient basis for trust. These assumptions can be misleading and the consequences of being misled by these assumptions can be deadly.

For example, in many countries, parents and health professionals were led to believe that babies should be put to sleep on their tummies, so that they would not choke if they puked. The basis for this claim was the seemingly logical assumption that babies might choke if they puked while lying on their backs. Trust in this claim was often based on the source of the claim – health professionals. Benjamin Spock, an influential American child health expert, included this claim in his bestselling book Baby and Child Care, and advised his millions of readers to put babies to sleep on their tummies.

However, researchers looking into possible causes of unexplained deaths during infancy found that babies who had died were more likely to have been put to sleep on their tummies than babies who had survived infancy. Three times as many babies died suddenly and unexpectedly if they had been put to sleep on their tummies. Earlier recognition of the risks of putting babies to sleep on their tummies might have prevented such deaths. It has been estimated that over 10,000 infant deaths in the UK and at least 50,000 in Europe, the USA, and Australasia [Gilbert 2005] are likely to have been attributable to front sleeping.

This example illustrates that it can be dangerously misleading to simply assume that:

1. treatments are safe,
2. a plausible explanation of how or why a treatment works is a sufficient basis for a claim about treatment effects,
3. a treatment is helpful or safe based on how widely used it is or has been, and
4. the source of a claim alone is a sufficient basis for believing a claim about treatment effects.

In this and the next three essays in this series we explain concepts such as these and how they can help you to avoid believing and acting on unreliable claims.

Assumptions that treatments are safe or effective can be misleading.

In this essay, we explain five assumptions about the safety and effectiveness of treatments that can be misleading. These are assuming that:

• treatments are safe,
• treatments have large, dramatic effects,
• treatment effects are certain,
• it is possible to know who will benefit and who will be harmed, or
• comparisons are not needed.

The basis for these concepts is described elsewhere [Oxman 2022].

Do not assume that treatments are safe.

People often exaggerate the benefits of treatments and ignore or downplay potential harms. However, few effective treatments are 100% safe. Similarly, people in need or desperation hope that treatments will work, and they may ignore potential harms – especially when reliable evidence of treatment effects is lacking. As a result, they may waste time, money, or both on treatments that have never been shown to be useful and may cause harm. The harm that is caused may be minor, but treatments also sometimes cause serious, irreversible harms, including death.

Even simple advice can sometimes cause serious harm, such as advice to put babies to sleep on their tummies, which resulted in tens of thousands of deaths that could have been prevented [Gilbert 2005].

Do not assume that treatments have large, dramatic effects.

Large effects (where everyone or nearly everyone who is treated experiences a benefit or a harm) are easy to detect without fair comparisons. However, few treatments have effects that are so large that fair comparisons (designed to minimise the risk of being misled by systematic errors (biases) or the play of chance) are not needed. Treatments that do not have large, dramatic effects may be helpful, but fair comparisons are needed to determine how safe and helpful they are.

Some treatments have obvious effects. For example, if someone is bleeding and losing lots of blood, it is obvious that it is a good idea to stop the bleeding. However, most effective treatments do not have such obvious effects. For example, any effects of exercise or changes in diet on heart disease or cancer may occur only after many years. Some medicines and medical procedures have immediate and obvious effects, such as giving adrenaline to someone with a severe allergic reaction; transfusing blood to someone who has lost a lot of blood; or draining pus from a painful abscess. However, like changes in exercise or diet, any effects of most medicines and medical procedures do not have such easily observed or experienced effects by everyone who receives the medicine or procedure. This includes common medications used to prevent heart disease or strokes, such as medicines for high blood pressure or high cholesterol, which help some people but not everyone who takes them [Leucht 2015]. It also includes treatments for cancer and pain, and complementary and alternative medicines, such as herbal remedies, public health measures (such as closing schools to reduce the spread of Covid-19), and changes in the ways healthcare is delivered or financed.

Do not assume that treatment effects are certain.

Fair comparisons of treatments can provide a basis for confidence about the probability of beneficial and harmful effects of treatments. However, it is rarely, if ever, possible to be 100% certain about the size of treatment effects, or to predict exactly what will happen if a treatment is used. This is especially true for treatments that are intended to prevent adverse effects happening a long time in the future. Fair comparisons of such treatments are difficult because they entail following people up for a very long time and it is rarely possible to ensure that people adhere to the advice they are given. Consequently, claims about the effects of such treatments are often based on associations and belief in explanations of how the treatments work. Some people argue that there should be different standards for judgements about the trustworthiness of claims when fair comparisons are difficult. However, it can be lethal not to acknowledge and reduce important uncertainties, even when there is limited potential for doing so. It is also important to recognise that certainty about treatment effects can change as new information becomes available. This is especially true for new problems and treatments, such as treatments for Covid-19.

For example, at the start of the Covid-19 pandemic, little was known about the effects of measures to control it. However, in less than a year, over 2000 randomized trials addressing these uncertainties were registered [Dillman 2020]. Dexamethasone – an inexpensive and widely used medicine – was shown to reduce mortality among patients with severe Covid-19 disease [Sterne 2020]. On the other hand, no evidence was found to justify the use of another inexpensive and widely used medicine – hydroxychloroquine – and it was found to have harmful effects [Singh 2021]. At the same time, there have been very few reports of fair comparisons of measures to reduce the spread of Covid-19 (such as closing schools), and major uncertainties exist about the effects – wanted and unwanted – of these measures [Haber 2021].

Do not assume that it is possible to know who will benefit and who will be harmed.

For some kinds of health problems, fair treatment comparisons can be made by giving different treatments to a patient at different times, and then comparing the outcomes associated with each of the different treatment periods. These are called n-of-1 trials because they compare the effects of alternative treatments in one patient [Guyatt 1990]. For example, n-of-1 trials have compared paracetamol to non-steroidal anti-inflammatory drugs (NSAIDs) in patients with osteoarthritis of the hip or knee [Wegman 2003]. The results varied across patients. Most uncertainties about the effects of treatments cannot be compared in this way, however. For example, this person-specific approach cannot usually be used to compare a surgical treatment with a drug treatment.

Most treatment comparisons involve comparing similar groups of patients assigned to one of two or more alternative treatments. Fair comparisons of treatments usually tell us what happened, on average, in groups of similar people. Usually, in a group of people who have used a treatment, some people benefit, some do not, and some may even be harmed. For example, the proportion of people who benefit from common pharmacological treatments varies from 1.5% – for aspirin to prevent serious vascular events (myocardial infarction, stroke, or vascular death) in people at high risk – to 58% – for proton pump inhibitors for relief of reflux oesophagitis [Leucht 2015]. It is rarely possible to know in advance who will benefit from which treatment among alternatives, who will not benefit, or who will be harmed. Paradoxically, there is only way to know whether “personalised medicine” – customising treatment for individuals – works. This entails testing it in fair comparisons. Unless individual treatment customisation is 100% effective and 100% safe, it is still not possible to know in advance who will benefit from “personalised care” and who will not. Beyond n-of-1 trials, “personalised medicine” is not really personalised; it is simply an effort to identify subgroups of people who are most likely to benefit from specific treatments.

For example, HER2-positive breast cancer exists when breast cancer cells have a protein receptor called HER2 (human epidermal growth factor receptor 2). About 20% of breast cancers are HER2-positive. Trastuzumab (Herceptin) and other monoclonal antibodies that block HER2 receptors to keep cancer cells from growing are used to treat HER2-positive breast cancer. Accordingly, those medicines are given to women with HER2-positive breast cancer and not to other women with breast cancer. However, not all women with HER2-positive breast cancer benefit from the medicine and some will experience serious harmful effects, such as congestive heart failure (CHF). For example, for women with breast cancer detected at an early stage who have a moderate risk of cancer recurrence or death in the next three years (30%) and a moderate risk of CHF (2%), only about 10% more women who take the medicine will benefit (experience disease-free survival) and about 8% more will be harmed (experience CHF) [Moja 2012]. It is not possible to predict who among those women will benefit and which will be harmed.

Do not assume that comparisons are not needed.

Unless a treatment is compared to something else, it is not possible to know what would happen without the treatment. This makes it difficult to attribute outcomes to the treatment. Whenever comparative terms are used to describe a treatment – for example, “faster relief” or “better” – ask “compared to what?”. Sometimes people argue that a fair treatment comparison is impossible because the treatment is ‘holistic’, ‘individualised’, or ‘complex’. However, as with any other treatment, claims about the effects of such treatments depend on the results of comparing them with one or more alternatives. How trustworthy those claims are depends on how fair the comparisons are.

For example, a television commentator in the U.S. reported that “Between late-December of 2020 and last month [April 2021], a total of 3,362 people apparently died after getting the Covid vaccine in the United States.” He exclaimed: “That is an average of roughly 30 people every day,” and he went on to suggest that the vaccine was killing people [Qiu 2021]. There are many problems with that claim, including the lack of a comparison – how many similar people who had not been vaccinated died or would have been expected to die? Given that over 250 million doses of Covid-19 vaccines had been administered at that time [ACIP 2021], and that old people and others with a high chance of dying were prioritised for vaccination, it would be surprising if some of those people did not die after receiving the vaccine. That does not mean the vaccine caused them to die. The U.S. Centers for Disease Control and Prevention (CDC) Advisory Committee on Immunization Practices (ACIP) reported that there were 17 reported deaths per million vaccinated people (up to May 17, 2021) [ACIP 2021]. The proportion of Americans who died from any cause in 2019 was 8,697 per million [CDC 2020]. That corresponds to an average of 7,821 people dying every day. Most of them probably drank some water before dying. So, you could say that 1000s of Americans apparently died every day after drinking water. That does not mean that drinking water caused them to die.

Implications

• Always consider the possibility that a treatment may have harmful effects.
• Claims of large treatment effects are likely to be wrong. Expect treatments to have moderate, small, or trivial effects (wanted or unwanted), rather than dramatic effects. If estimates of treatment effects are not based on systematic reviews of fair comparisons of treatments, be sceptical about claims of small or moderate effects of treatments.
• It is important to recognise that there is some uncertainty about the effects of all treatments, and that there is likely to be more uncertainty about some types of treatments. Choices are still required but it is preferable to acknowledge, accept, and take account of uncertainty than to deny it and make misinformed and potentially dangerous decisions.
• Fair treatment comparisons provide the best basis for making well-informed decisions about treatments, but there is almost always some uncertainty about who will benefit, who will not, and who will be harmed.
• Always ask which comparisons provide the basis for claims about the effects of treatments. Claims that are not based on fair comparisons are not reliable.

< Previous Essay | Next Essay >

This James Lind Library Essay has been republished in the Journal of the Royal Society of Medicine 2022;115:354-359. Print PDF

References

ACIP. COVID-19 Vaccine Safety Technical (VaST) Work Group Assessment. (May 12, 2021). Atlanta: Centers for Disease Control and Prevention, 2021. https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2021-05-12/08-COVID-Lee-508.pdf

CDC, National Center for Health Statistics (2020). Underlying Cause of Death 1999-2020 on Centers for Disease Control and Prevention WONDER Online Database. Atlanta: Centers for Disease Control and Prevention. http://wonder.cdc.gov/ucd-icd10.html

Dillman A, Zoratti MJ, Park JJH, Hsu G, Dron L, Smith G, Harari O, Rayner CR, Zannat NE, Gupta A, Mackay E, Arora P, Lee Z, Mills EJ (2020). The Landscape of Emerging Randomized Clinical Trial Evidence for COVID-19 Disease Stages: A Systematic Review of Global Trial Registries. Infect Drug Resist 13:4577-87. https://doi.org/10.2147/IDR.S288399

Gilbert R, Salanti G, Harden M, See S (2005). Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. Int J Epidemiol 34:874-87. https://doi.org/10.1093/ije/dyi088

Guyatt GH, Keller JL, Jaeschke R, Rosenbloom D, Adachi JD, Newhouse MT (1990). The n-of-1 randomized controlled trial: clinical usefulness. Our three-year experience. Ann Intern Med 112:293-9. https://doi.org/10.7326/0003-4819-112-4-293

Haber N, Clarke-Deelder E, Salomon J, Feller A, Stuart E (2021). Covid-19 policy impact evaluation: a guide to common design issues. arXiv preprint.  https://arxiv.org/abs/2009.01940v6

Leucht S, Helfer B, Gartlehner G, Davis JM (2015). How effective are common medications: a perspective based on meta-analyses of major drugs. BMC Med. 13:253. https://doi.org/10.1186/s12916-015-0494-1

Moja L, Tagliabue L, Balduzzi S, Parmelli E, Pistotti V, Guarneri V, D’Amico R (2012). Trastuzumab containing regimens for early breast cancer. Cochrane Database Syst Rev 2012(4):Cd006243. https://doi.org/10.1002/14651858.cd006243.pub2

Oxman AD, Chalmers I, Dahlgren A (2022). Informed Health Choices Group. Key Concepts for Informed Health Choices: a framework for enabling people to think critically about health claims (Version 2022). IHC Working Paper. http://doi.org/10.5281/zenodo.6611932

Qiu L. No, Covid-19 vaccines are not killing more people than the virus itself. New York Times. May 7, 2021. https://www.nytimes.com/live/2020/2020-election-misinformation-distortions

Singh B, Ryan H, Kredo T, Chaplin M, Fletcher T (2021). Chloroquine or hydroxychloroquine for prevention and treatment of Covid‐19. Cochrane Database Syst Rev 2021(2): CD013587. https://doi.org//10.1002/14651858.CD013587.pub2

Sterne JAC, Murthy S, Diaz JV, Slutsky AS, Villar J, Angus  DC, et al (2020). Association between administration of systemic corticosteroids and mortality among critically ill patients with Covid-19: a meta-analysis. JAMA 324:1330-41. https://doi.org/10.1001/jama.2020.17023

Wegman AC, van der Windt DA, de Haan M, Devillé WL, Fo CT, de Vries TP (2003). Switching from NSAIDs to paracetamol: a series of n of 1 trials for individual patients with osteoarthritis. Ann Rheum Dis 62:1156-61. https://doi.org/10.1136/ard.2002.002865