Comparator Bias

Mann H, Djulbegovic B. Why comparisons must address genuine uncertainties

Cite as: Mann H, Djulbegovic B (2004). Why comparisons must address genuine uncertainties. James Lind Library (www.jameslindlibrary.org).
Accessed
© Howard Mann, 2004.

Author contact details: Howard Mann, Department of Radiology, 1A71 University Hospital, 50 North Medical Drive, Salt Lake City, UT 84132, USA. E-mail: howard.mann@hsc.utah.edu

Controlled clinical trials are done to reduce uncertainties about the relative merits of different treatments
Researchers may believe - and patients and physicians may hope - that a particular treatment (perhaps because it is new) is better than other available treatments; but it may often turn out to be worse (Silverman, cautionary tales, JLL). When the British Medical Research Council’s controlled trial of streptomycin for pulmonary tuberculosis was conceived in 1946 (MRC 1948), none of the therapies used to treat the disease had been shown in controlled clinical trials to be useful; indeed, one controlled trial had shown gold salt therapy to do more harm than good (Amberson et al. 1931). Although streptomycin was known to be useful in forms of tuberculosis which had previously always been fatal (MRC 1948), there was uncertainty about how useful the new drug would be in pulmonary tuberculosis, from which patients often recovered after treatment with bed rest alone. Patients in the MRC trial were accordingly randomized either to bed rest alone, or to bed-rest and streptomycin.

The same reasoning is applicable when controlled trials are designed today. After considering systematic reviews of the relevant existing evidence, patients and their doctors must be substantially uncertain about which among the treatment options - including no active treatment - is preferable. This implies ensuring that no patient who agrees to participate in the trial will knowingly be disadvantaged, whichever one of the comparison treatments the patient is assigned to receive.

Comparator bias results from violating the principle that clinical trials should only be done when there is uncertainty about the relative merits of treatment alternatives
Clinical trials are done to reduce uncertainty, and they should only be done if physicians and the patients are uncertain which of existing alternatives is better (Hill 1963; Djulbegovic 2001). This requirement is sometimes referred to as “the uncertainty principle” (Peto and Baigent 1998) or “equipoise” (Freedman 1987; Edwards et al. 1998).

If one or more of the treatments selected for the comparison is known to be worse than others, not only will patients be denied effective treatment, but this ‘comparator bias’ will result in unfair tests of treatments. Even if other sources of bias have been well controlled in such studies, their results will mislead patients and their doctors. Unfortunately, comparator bias is sometimes deliberately introduced for just this purpose, usually with a view to showing that new treatments are preferable to existing alternatives (Sackett and Oxman 2003).

Comparator bias can be introduced by:

(i) withholding a treatment known to be beneficial
Comparator bias is introduced when treatments known to be beneficial are withheld from patients participating in controlled trials. The reason that bed rest alone was an acceptable treatment for half the patients in the MRC trial of streptomycin for pulmonary tuberculosis was that there was no known effective treatment for the condition. When systematic reviews of existing evidence show that existing treatments are more helpful than doing nothing, or than using placebos, comparator biases will be introduced when patients are denied effective treatments, and the active treatments studied in the trial will be given an unfair advantage.

For example, even though the effectiveness of erythropoietin in preventing anemia in cancer patients had been convincingly demonstrated by a number of controlled trials, some researchers continued to compare the drug with placebos (Clark et al. 2002), presumably because they failed to perform a systematic review of previous trials before embarking on further research.

The use of no-treatment or placebo comparison groups in clinical trials may also reflect who has sponsored the trial. For example, a systematic review of treatments used in a blood disorder, multiple myeloma, showed that placebo or no-therapy controls were used in 60 per cent of commercially sponsored trials, but in only 21 per cent of studies supported with public funds (Djulbegovic et al. 2000).

(ii) giving an inappropriately low dose of a treatment
Comparator bias can also result when a treatment is compared with an inappropriately low dose of another treatment. This has occurred in comparisons of new non-steroidal anti-inflammatory agents used for arthritis with older drugs in the same class (Rochon et al. 1994). Inappropriately low doses can also result from giving a treatment by an inappropriate route, for example, by comparing intravenous administration of a drug with oral administration of a drug that is poorly absorbed from the gastro-intestinal tract (Johanson and Gøtzsche 1999).

(iii) giving an inappropriately high dose of a treatment
The net usefulness of treatments often requires trade-offs between wanted and unwanted effects. Treatments may be of real value if, although their beneficial effects are no better than alternatives, they have fewer adverse effects. Some of the newer drugs for treating schizophrenia, for example, may be preferable to established drugs for this reason. However, this apparent advantage may be because the newer agents have been compared with inappropriately high doses of the older, comparator drug (Waraich et al. 2004).

How can comparator bias be reduced?
Reducing some forms of bias is straightforward: allocation bias, for example, is controlled by strict random allocation of patients to treatment comparison groups. Although comparator bias cannot be dealt with quite so straightforwardly, there is nevertheless great scope for reducing it. In particular, the choice of comparison groups in controlled trials should be informed by systematic reviews of relevant existing evidence. When, in the light of this evidence and other considerations, patients and doctors are uncertain which among treatment options is preferable (Djulbegovic 2001), the preconditions for avoiding comparator bias exist (Mann and Djulbegovic 2003).

References

Clark O, Adams JR, Bennett CL, Djulbegovic B (2002). Erythropoietin, uncertainty principle and cancer related anaemia. BMC Cancer 2(2):23.

Edwards SJL, Lilford RJ, Braunholtz DA, Jackson JC, Hewison J, Thornton J (1998). Ethical issues in the design and conduct of randomized controlled trials. Health Technol Assessment 2(15):1-130.

Djulbegovic B (2001). Acknowledgment about uncertainty: a fundamental means to ensure scientific and ethical validity in research. Current Oncology Reports 3:389-395.

Djulbegovic B, Lacevic M, Cantor A, Fields KK, Bennett CL, Adams JR, Kuderer NM, Lyman GH (2000). The uncertainty principle and industry-sponsored research. Lancet 356:635-638.

Freedman B (1987). Equipoise and the ethics of clinical research. N Engl J Med317:141-5.

Hill AB (1963). Medical ethics and controlled trials. BMJ 1:1043-1049.

Johansen HK, Gotzsche PC (1999). Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis. JAMA 282:1752-9.

Mann H, Djulbegovic B (2003). Choosing a control intervention for a randomised clinical trial. BMC Med Res Methodol 3:7.

Peto R, Baigent C (1998). Trials: the next 50 years. BMJ 317:1170-1171.

Sackett DL, Oxman AD (2003). HARLOT plc: an amalgamation of the world's two oldest professions. BMJ 327:1442-1445.

Rochon PA, Gurwitz JH, Simms RW, et al (1994). A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch Intern Med 154:157-63.

Waraich PS, Adams CE, Roque M, Hamill KM, Marti J (2004). Haloperidol dose for acute phase of schizophrenia (Cochrane Review). In : The Cochrane Library, Issue 1. Oxford: Update Software 2003.