Avoiding biased comparisons

Sometimes treatments have dramatic effects (click here to list relevant records). These may be unintended and specific, for example, when a person has an allergic reaction to an antibiotic drug. Treatments can also have striking beneficial effects, like adrenaline for life-threatening allergic reactions (McLean-Tooke et al. 2003). Such striking effects are rare, however. Usually, treatment effects are more modest, but nevertheless well worth knowing about, for example, using aspirin to reduce the risk of heart attack (Elwood 2004).

For example, aspirin doesn’t prevent all premature deaths after a heart attack, but it does reduce the likelihood of death by about twenty per cent, which is important in such a common condition. If these moderate but important effects of most treatments are to be detected reliably, care must be taken to ensure that biased comparisons don’t lead us to believe that treatments are useful when they are useless or harmful, or useless when they can actually be helpful.

Biases in tests of treatment are those influences and factors that can lead to conclusions about treatment effects that are systematically different from the truth. Although many kinds of biases can distort the results of health research (Sackett 1979), we have concentrated in The James Lind Library on those biases that must be minimised in fair tests of treatments. These are:

• biases due to differences in people compared;
• biases due to differences in the way treatment outcomes are assessed;
• biased reporting of the available evidence; and
• biased selection from the available evidence.

Ignoring these biases (or sometimes unscrupulously taking advantage of them), may lead people to believe that a new treatment is better than an existing treatment, when it is not. This could result from basing conclusions on:

• studies that compare the progress of relatively well people given a new treatment with the progress of relatively ill people given a standard treatment (allocation bias).
• studies in which assessments of treatment outcomes are likely to be biased in favour of a new treatment, for example, by comparing the opinions of people who know that they have used an expensive new treatment with the opinions of those who may be disappointed that they were continuing to use an unexciting standard treatment (observer or measurement bias).
• only studies that show a new treatment in a favourable light, and not those that suggest that it may be harmful, which are often not reported (reporting bias).
• biased selection from and interpretation of the available evidence to support a particular viewpoint (reviewer bias).

Usually, the unfair tests of treatment resulting from these biases are not recognised for what they are. However, people with vested interests sometimes exploit these biases so that treatments are presented as if they are better than they really are (Sackett and Oxman 2003).

Whether biases are inadvertent or deliberate, the consequences are the same: unless tests of treatment are fair, some useless or harmful treatments will seem to be useful, while some useful treatments will seem useless or harmful.

References

Elwood P (2004). The first randomised trial of aspirin for heart attack and the advent of systematic overviews of trials. The James Lind Library (www.jameslindlibrary.org).

McLean-Tooke APC, Bethune CA, Fay AC, Spickett GP (2003). Adrenaline in the treatment of anaphylaxis: what is the evidence? BMJ 327:1332-1335.

Sackett DL (1979). Bias in analytic research. Journal of Chronic Diseases 32:51-63.

Sackett DL, Oxman AD (2003). HARLOT plc: an amalgamation of the world's two oldest professions. BMJ 2003;327:1442-1445.