Systematic reviews of all the relevant evidence:

Avoiding biased selection from the available evidence

Biases can distort tests of medical treatments and lead to erroneous conclusions. They can also distort reviews of evidence. Plans for systematic reviews should be set out in protocols, such as those published by The Cochrane Collaboration, making clear what measures will be taken to reduce biases.

These include specifying clearly:

• which question about treatments will be addressed in the review;
• the criteria that will make a study eligible for inclusion;
• the strategies that will be used to search for potentially eligible studies; and
• the steps that will be taken to minimise biases in selecting studies and data for inclusion in the review (Berlin 1997).

Different systematic reviews addressing what appears to be the same question about the effects of medical treatments quite often reach different conclusions. Sometimes this is because the questions addressed are subtly different. Sometimes it reflects differences in the materials and methods used by the reviewers, and in these circumstances it is important to judge which of the reviews are most likely to have reduced allocation biases most successfully.

It is also worth considering whether the reviewers have other interests that might affect the conduct or interpretation of their review. For example, people associated with the manufacturers of evening primrose oil reviewed the drug’s effects on eczema (Morse et al. 1989). They reached a far more enthusiastic conclusion about the value of the drug than a review done by investigators with no commercial interest, who included the results of unpublished studies in their assessment (Williams 2003).

It is not only commercial interests that can lead to biased selection from the available evidence for inclusion in reviews. We all have prejudices that can lead to biased selection of evidence, and researchers, health professionals, patients and others assessing the effects of treatments are not immune. Conflicts of interest have been recognized as important, and some measures are being taken to take account of them.

References

Berlin JA (1997). Does blinding of readers affect the results of meta-analyses? University of Pennsylvania Meta-analysis Blinding Study Group. Lancet 350:185-186.

Morse PF, Horrobin DF, Manku MS, Stewart JC, Allen R, Littlewood S, Wright S, Burton J, Gould DJ, Holt PJ, et al (1989). Meta-analysis of placebo-controlled studies of the efficacy of Epogam in the treatment of atopic eczema. Relationship between plasma essential fatty acid changes and clinical response. British Journal of Dermatology 121:75-90.

Sackett DL, Oxman AD (2003). HARLOT plc: an amalgamation of the world's two oldest professions BMJ 327:1442-1445.

Williams HC (2003). Evening primrose oil for atopic dermatitis. BMJ 327:1358-1359.