Up-to-date, systematic reviews of all relevant, reliable evidence

Fair tests of treatments in health care
The realisation that systematic reviews are needed to provide fair tests of treatments has been reflected in a rapid increase in the numbers of reports of systematic reviews being published on paper and electronically (The Cochrane Collaboration). These are being used (i) to inform clinical practice, for example, the Scottish Intercollegiate Guidelines Network; (ii) to assess which medical treatments are cost-effective, for example, by the National Institute for Health and Clinical Excellence; and (iii) to meet the needs of patients for reliable information about the effects of treatments, for example, through Informed Health Online.

The results of individual fair tests of medical treatments are only very rarely set systematically in the context of other similar studies, using methods to reduce biases and the play of chance. This failure to do systematic reviews of research on the effects of treatments has resulted in a great deal of avoidable suffering. Fair tests of treatments in health care also entails unbiased preparation of systematic reviews of all the relevant, reliable research studies of the treatments being assessed.

There are some examples of this process going back more than 200 years. In 1753, for example, in his review of the large number of reports about the prevention and treatment of scurvy, James Lind noted:

"As it is no easy matter to root out prejudices,… it became requisite to exhibit a full and impartial view of what had hitherto been published on the scurvy… Indeed, before the subject could be set in a clear and proper light, it was necessary to remove a great deal of rubbish." (Lind 1753)

Systematic reviews of all relevant research addressing questions about the effects of treatments are increasingly seen as providing the most reliable basis for conclusions about treatment effects. Sometimes systematic reviews will show that no reliable evidence exists, and this is one of their most important functions. Similarly, systematic reviews may sometimes confirm that reliable evidence is limited to a single study, and here, too, it is important to make this situation explicit.

Unfinished business
These and similar developments show that the importance of systematic reviews has been accepted by those who are trying to improve access to the evidence needed to inform choices in health care. However, there is still a long way to go: it has been estimated that the Cochrane Collaboration’s current output of several thousand systematic reviews will need to be increased to well over 10,000 to cover existing evidence (Mallett and Clarke 2002), and then kept up to date as new evidence emerges. Indeed, one journal editor has suggested that there should be a moratorium on all new research until we’ve caught up with what existing evidence can tell us (Bausell 1993).

Those responsible for disbursing funds for research must ensure that resources are provided to cope with this backlog, and that new studies are only supported if systematic reviews of existing evidence have shown that additional studies are necessary, and that they have been designed to take account of the lessons from previous research. If journal editors are to serve their readers better, they must follow the lead of The Lancet and ensure that reports of new studies make clear what contribution new evidence has made to an up-to-date systematic review of all the relevant evidence (Young and Horton 2005).

The increased availability of up-to-date, systematic reviews is improving the quality of information about the effects of treatments, but the conclusions of systematic reviews should not be accepted uncritically. Different reviews purportedly addressing the same question about treatments sometimes arrive at different conclusions. Their authors are human and we need to be aware that they may select, analyse and present evidence in ways that support their prejudices and interests. The continuing evolution of reliable methods for preparing and maintaining systematic reviews will help to address this problem, but they cannot be expected to abolish it.

Although growth in the numbers of systematic reviews has increased the availability of the primary fair tests of treatments in health care, these reviews often also reveal the poor quality and irrelevance of much research on the effects of treatments. As one editorialist commenting on “the scandal of poor medical research” put it, we need less research, better research and research done for the right reasons (Altman 1994). It seems unlikely that this will be achieved without greater public understanding of the rationale for and characteristics of fair tests of treatments, and greater public influence on and involvement in all phases of fair testing of treatments. Promotion of this agenda depends on uncertainties about the effects of treatments being confronted by new alliances of patients and clinicians (Chalmers 2004; James Lind Alliance).

The public and health professionals will be well served when they have readier access to up-to-date, systematic reviews of all relevant, reliable evidence addressing important uncertainties about the effects of treatments, and to information about ongoing research addressing these uncertainties (Smith and Chalmers 2001).

Cite As:

Editorial commentary (2007). Up-to-date, systematic reviews of all relevant, reliable evidence. The James Lind Library (www.jameslindlibrary.org).