Because single tests of treatments can be misleading, systematic reviews are used to identify, evaluate and summarize all the evidence relevant to addressing a particular question.

Biases can distort individual tests of medical treatments and lead to erroneous conclusions. They can also distort reviews of evidence. Plans for systematic reviews should be set out in protocols, such as those published by  Cochrane (formerly, The Cochrane Collaboration), making clear what measures will be taken to reduce biases.

These include specifying clearly:

• which questions about treatments will be addressed in the review;
• the criteria that will make a study eligible for inclusion;
• the strategies that will be used to search for potentially eligible studies; and
• the steps that will be taken to minimise biases in selecting studies and data for use in the review (Berlin 1997).

Different systematic reviews addressing what appears to be the same question about the effects of  treatments quite often reach different conclusions. Sometimes this is because the questions addressed are subtly different. Sometimes it reflects differences in the materials and methods used by the reviewers. In these circumstances it is important to judge which of the reviews are most likely to have been most successful in reducing biases.

It is also worth considering whether the reviewers have other interests that might affect the conduct or interpretation of their review. For example, people associated with the manufacturers of evening primrose oil reviewed the drug’s effects on eczema (Morse et al. 1989). They reached a far more enthusiastic conclusion about the value of the drug than a review done by investigators with no commercial interest, who included the results of unpublished studies in their assessment (Williams 2003).

It is not only commercial interests that can lead to biased selection from the available evidence for inclusion in reviews. We all have prejudices that can lead to biased selection of evidence, and we should not expect researchers, health professionals, patients and others assessing the effects of treatments are not immune.

< Previous Essay | Next Essay >

The text in these essays may be copied and used for non-commercial purposes on condition that explicit acknowledgement is made to The James Lind Library (www.jameslindlibrary.org).