Systematic reviews of all the relevant evidence

One of the twentieth century pioneers of fair tests of treatments, Austin Bradford Hill, noted that readers of reports of research want answers to four questions: ‘Why did you start?’, ‘What did you do?’, ‘What did you find?’, and ‘What does it mean anyway?’ (Hill 1965). The quality of the answer to Hill's last question is particularly important because this is the element of a research report which is most likely to influence actual choices and decisions about treatments.

Only very rarely will a single fair test of a treatment yield sufficiently strong evidence to provide a confident answer to the question ‘What does it mean?’. A fair test of a treatment is usually one of a number of tests addressing the same question. For a reliable answer to the question ‘What does it mean?’, then, it is important to interpret the evidence from a particular fair test in the context of a careful assessment of all the evidence from fair tests that have addressed the question concerned.

The president of the British Association for the Advancement of Science expressed the need to observe this principle more than a century ago:

"If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight.... Two processes are thus at work side by side, the reception of new material and the digestion and assimilation of the old…The work which deserves, but I am afraid does not always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out." (Rayleigh 1885)

Perhaps it is because application of this principle in practice still attracts little credit within academia that very few reports of fair tests of treatments discuss their results in the context of a systematic assessment of all the other relevant evidence (Clarke et al. 2002). As a result, it is usually difficult for readers to obtain a reliable answer to the question ‘What does it mean?’ from reports of new research.

As noted in an earlier explanatory essay, embarking on new tests of medical treatments without first reviewing systematically what can be learnt from existing research is dangerous, wasteful and unethical (see Why comparisons must address genuine uncertainties). Reporting the results of new tests without interpreting new evidence in the light of systematic assessments of other relevant evidence is also dangerous because it results in delays in the identification of both useful and harmful treatments (Antman et al. 1992). For example, between the 1960s and the early 1990s, over 50 fair tests of drugs to reduce heart rhythm abnormalities in people having heart attacks were done before it was realised that these drugs were killing people. Had each report assessed the results of new tests in the context of all the relevant evidence, the lethal effects of the drugs could have been identified a decade earlier, and many unnecessarily premature deaths could have been avoided.

In an age of electronic publishing it should be possible to deal with the limitations found in most reports of new research (Chalmers and Altman 1999; Smith and Chalmers 2001). However, rather than basing conclusions about the treatments on one or a few individual studies, users of research evidence are increasingly turning for reliable information to up-to-date, systematic reviews of all relevant, reliable evidence, because these are increasingly recognised as providing the best basis for conclusions about the effects of medical treatments.

Just as it is important to take steps to avoid being misled by biases and the play of chance in planning, conducting, analysing and interpreting individual fair tests of treatments, so also must similar steps be taken in planning, conducting, analysing and interpreting systematic reviews. This entails:

• specifying the question to be addressed by the systematic review
• defining eligibility criteria for studies to be included
• identifying (all) potentially eligible studies
• applying eligibility criteria in ways that limit bias
• assembling as high a proportion as possible of the relevant information from the studies
• analysing this information, if appropriate and possible, using meta-analysis and a variety of analyses
• preparing a structured report

One manifestation of the increasing recognition of the crucial importance of systematic reviews for assessing the effects of treatments is the rapid evolution of methods to improve the reliability of reviews themselves. The first edition of a book entitled Systematic Reviews was less than 100 pages long (Chalmers and Altman 1995): only six years later, the second edition weighed in at nearly 500 pages and included rapidly evolving strategies for increasing the information obtained from research (Egger et al. 2001).

There are currently important developments in the methods used for preparing systematic reviews, including those needed to identify unanticpated effects of treatments (Glasziou et al. 2004) and for incorporating the results of research describing and analysing the experiences of people giving and receiving treatments (Thomas 2004). Relevant material will be added to The James Lind Library as it emerges.

References

Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC (1992). A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. JAMA 268:240-48.

Chalmers I, Altman DG (1995). Systematic Reviews. London: BMJ Publications.

Chalmers I, Altman DG (1999). How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet 353:490-493.

Egger M, Davey Smith G, Altman D (2001). Systematic Reviews in Health Care: meta-analysis in context. 2nd Edition of Systematic Reviews. London: BMJ Books.

Glasziou P, Vandenbroucke J, Chalmers I (2004). Assessing the quality of research BMJ 328:39-41.

Hill AB (1965). Cited in 'The reasons for writing'. BMJ 4:870.

Rayleigh (1885). Address by the Rt. Hon. Lord Rayleigh. In: Report of the fifty-fourth meeting of the British Association for the Advancement of Science; held at Montreal in August and September 1884, London: John Murray.

Smith R, Chalmers I (2001). Britain's gift: a ‘Medline’ of synthesized evidence. BMJ 323:1437-1438.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004). Integrating qualitative research with trials in systematic reviews BMJ 328:1010-1012.