Fox DM (2011). Systematic reviews and health policy: the influence of a project on perinatal care since 1988.

© Daniel Fox, 100 West 12th Street, 3T, New York, NY 10011, USA.

Cite as: Fox DM (2011). Systematic reviews and health policy: the influence of a project on perinatal care since 1988. JLL Bulletin: Commentaries on the history of treatment evaluation (


Context: Interrelated publications between 1988 and 1992 have influenced health policy and clinical practice: The Oxford Database of Perinatal Trials (ODPT), Effective Care in Pregnancy and Childbirth (ECPC), A Guide to Effective Care in Pregnancy and Childbirth (GECPC), and Effective Care of the Newborn Infant (ECNI). These publications applied and advanced methods that had a substantial history in the medical, biological, physical, and social sciences. Their unique contribution was to demonstrate the feasibility of organizing and sustaining programs to conduct systematic reviews across an entire field of health care. The publications also influenced subsequent advances in the methodology of systematic reviews and contributed to their proliferation; in large measure, but not entirely, because their editors and many of the authors participated in organizing and developing the Cochrane Collaboration. This article describes how and why these publications attracted favorable attention and resources from policymakers in numerous countries.

Methods:This article applies historical methods to the analysis of primary sources that help explain the influence of systematic reviews, mainly on health policy. These methods guide an analysis of the rhetoric of the two volumes of ECPC and of primary sources generated as systematic reviews influenced health policy. The analysis of rhetoric employs the methods of intellectual history and social studies of science. The analysis of policymaking uses the methods of political and policy history, political science, and public administration. Because the focus of this article is how science influenced policy it alludes to but does not describe in detail the literature on the methods, production, and publication of systematic reviews.

Findings:The influence of the four publications on policy was mainly a result of (1) their powerful blending of the rhetoric of scientific and polemical discourse, especially but not exclusively in ECPC; (2) a growing constituency for systematic reviews as a source of ‘evidence-based’ health care among clinicians, journalists, and consumers in many countries; and (3) recognition by significant policymakers who allocate resources to and within the health sector that systematic reviews could contribute to making health care more effective and to containing the growth of costs.

Conclusions:Analysis of this aspect of the history of producing and applying systematic reviews informs understanding of how knowledge derived from research informs policy.


Systematic reviews, articles, and reports that use rigorous methods to identify relevant studies, appraise their quality, and summarize their results have proliferated in the literature of health services as well as in the literatures of criminal justice, education, social and addiction services, and, recently, international development. Four interrelated publications between 1988 and 1992 contributed to the developing methodology of systematic reviews and, because they applied this methodology to interventions across an entire field of health services, attracted attention from many clinicians and policymakers. These publications are The Oxford Database of Perinatal Trials (ODPT), Effective Care in Pregnancy and Childbirth (ECPC), A Guide to Effective Care in Pregnancy and Childbirth (GECPC), and Effective Care of the Newborn Infant (ECNI) (Chalmers 1988–1992; Chalmers, Enkin, and Keirse 1989; Enkin, Keirse, and Chalmers 1989; Sinclair and Bracken 1992). This article describes the four publications as the ECPC project, emulating the usage of Frederick Mosteller, a distinguished statistician who wrote an important early article about their significance (Mosteller 1993).

A growing literature grounds the history of systematic reviews in the history of science and medicine. Although the ‘need to synthesize research evidence has been recognized for well over two centuries,’ explicit methods for such research emerged only in the twentieth century (Chalmers et al. 2002). In the 1970s and 1980s, ‘systematic reviews and meta-analyses ... began to appear in a variety of health fields’ (Bastian et al. 2010). The methodology of systematic reviewing and the research infrastructure that produced them continue to evolve (Starr et al. 2009).

Other literature describes the production of systematic reviews since the publication of the ECPC project. Most of this literature describes the history of the Cochrane Collaboration, organized in 1993 by Iain Chalmers and other participants in ECPC (Chalmers et al. 1993; Chalmers 1993, 2003; Dickersin and Manheimer 1998; Starr et al. 2009). I have described elsewhere the influence of the Cochrane Collaboration and other sources of systematic reviews on a consortium of American states that began to commission reviews in 2003 (Fox 2010a).

This article links this history of science and its sponsorship to the contemporary history of health policy. Mosteller described the ECPC project as the ‘most advanced current example of a basis of practicing medicine founded on both empirical evidence and theory, ... and it illustrates what can be done’ (Mosteller 1993, 524). I focus on another aspect of the project’s impact: why and how systematic reviews have influenced policy. I also briefly examine aspects of its influence on clinical practice that relate to its effects on policy.

I organize the article in six sections. First, I discuss the rhetoric of the two volumes of ECPC, what I call its ‘two voices.’ Next I describe initial responses to that rhetoric within the medical profession. The following two sections examine responses to the ECPC project and subsequent systematic reviews that it influenced among significant groups in the health sector: first, antagonism to these reviews and, then, support for conducting and using them. Then I explain why the ECPC project and subsequent reviews that it influenced attracted the interest of some policymakers who allocate resources for health services. Finally, I summarize what these policymakers have done to subsidize the production and en- courage the application of systematic reviews since the publication of the ECPC project.

Although the focus of this article is the influence of the ECPC project on policy, several reviewers asked me to summarize some other notable effects. ODPT, one reviewer wrote, ‘pointed out the research foundation needed for reviews (i.e., problems with reporting biases, searching and the need for trials registers) and served as the model for Cochrane’s Central Database of Controlled Trials.’ GECPC was a ‘model for extending the reach of the work to non-doctors (e.g., midwives) and consumers.’ And ECNI, another reviewer observed, ‘could be credited with extending the approach [to a new field of care and was] a prelude to the Cochrane Collaboration.’ These achievements contributed to the influence of systematic reviews in health affairs. I will now describe the politics through which that influence occurred.

The rhetoric of ECPC

The editors and their coauthors of many of the chapters in ECPC’s two volumes wrote in two rhetorical styles, or voices, both of which have helped shape the politics of conducting and using systematic reviews. These voices enhanced the resonance of the volumes with particular audiences in the health sector, in the media, among consumers, and in government. In one of their voices, the editors and their colleagues described the methods and findings of systematic reviews in the conventional uninflected and impersonal language of biomedical science. In the other, they spoke as eloquent polemicists against authoritarianism in clinical practice and on behalf of more effective perinatal care.

When I began to write this article, I lacked evidence about the extent to which the editors consciously wrote in two voices. I knew from reading, personal observation, conversations, and correspondence that the editors were deeply committed to social justice and especially to the rights of patients. In early drafts of this article, I conjectured that they may have been stimulated by contemporaneous scholarship on the rhetoric of science, much of it by scholars who shared their commitment to social justice (Gross 2006). Then Murray Enkin, an editor of ECPC (as well as GECPC), commented on one of my drafts: ‘I don’t think Iain [Chalmers], Marc [Keirse], and I consciously chose to use the two voices of scientific ‘objectivity’ and polemic for more patient-centered care, but when you pointed this out I wanted to shout, ‘Right on, that was the way it was.’’ Oral history is the history of memory, even in email format. Nevertheless, Enkin’s recollection of the editors’ motives must suffice as an explanation until additional primary sources become available.

Archie Cochrane was polemical in his Foreword, as he had been many times during his distinguished scientific career. He repeated an accusation he had made in 1979, that obstetricians made the ‘worst’ use of randomized controlled trials among medical specialists. Then he pro- claimed that the systematic reviews were a ‘new achievement’ and a ‘milestone’ (Chalmers, Enkin, and Keirse 1989, vii). The three editors used more conciliatory words in their preface. Chalmers, Enkin, and Keirse called attention to the variation in clinicians’ opinions about the ‘best means’ for ‘achieving the objectives of care.’ Such variation had multiple causes, they wrote. They emphasized the variation resulting from clinicians’ ‘collective uncertainty’ about the ‘effectiveness and safety of many of the elements of care’ (1989, ix–x). They apparently thought it politic to introduce the volumes with the reassuring assumption that as men and women of science, clinicians would reduce this variation when persuasive evidence of effectiveness became available. Their American contemporary, John E Wennberg, had recently made the same assumption when he described how his studies of variation in practice within geographic areas would influence clinicians (Wennberg 1984).

The editors used their two voices in the opening paragraph of the acknowledgments. They began by according equal esteem to participants in research and to scientists: ‘Primary credit for this book must go to the women and investigators who participated in and conducted the controlled trials of care in pregnancy and childbirth on which so much of the evidence in this book is based.’ They concluded the paragraph by thanking Archie Cochrane for both his passion and his commitment to experimental methods. Cochrane, they wrote, ‘first drew our attention to the particular importance’ of these investigators and the women they studied for an ‘understanding of the effects of care during pregnancy and childbirth’ (1989, xi).

Next the editors listed the contributors to ECPC and their colleagues who had provided information about unpublished trials (xi–xiii). In the context created by Cochrane’s provocative rhetoric, the conciliatory opening paragraph of the preface, and the two voices of the previous paragraph, acknowledging these colleagues was political as well as conventional. The list was evidence that the potential constituents of ECPC, and of systematic reviews more generally, were numerous and international.

The table of contents introduced the unconventional scope and broad evidentiary base of the volumes, the first on pregnancy and the second on childbirth. Methodology was the subject of the first five chapters of volume 1. Three of these chapters described randomized clinical trials (RCTs) and other study designs, as well as the methodology of systematic reviews. The fourth chapter discussed relevant methods in the social sciences, and the fifth, economic evaluation. The next eight chapters addressed the ‘social context of care during pregnancy and childbirth.’ The chapters that followed discussed the ‘needs of childbearing families,’ ‘social policies and the organization of health care,’ ‘social and psychological support during pregnancy,’ and ‘advice for pregnancy.’ Not until chapters 14 through 47 of the volume would clinicians find discussion of evidence about the effectiveness of the interventions that comprised most of their work with pregnant women and newborn children (xv–xvi). Moreover, the editors placed the ECPC’s only chapter on biology at the beginning of the second volume, perhaps signaling that unlike most of their peers, they did not regard the biological sciences as the only, or necessarily the most important, disciplines underlying clinical practice.

After the chapter on biology, however, volume 2 resembled its predecessor. Three chapters addressed, respectively, ‘social and professional support during childbirth,’ ‘hospital policies for labor and delivery,’ and ‘hospital admission practices.’ The next thirty-two chapters discussed aspects of care during labor and delivery and care of mothers and newborn infants, followed by chapters on ‘care of the bereaved after perinatal death’ and ‘unhappiness after childbirth.’ Two of three concluding chapters examined strategy and tactics for an international movement to promote effective care during pregnancy and childbirth, including possible actions by policymakers outside the health sector. In the last chapter the editors restated the book’s thesis—that reliable evidence, synthesized in rigorous systematic reviews, could be used to reduce unwarranted variation—and then summarized ways in which such reviews could inform policy and practice (xvii–xviii).

The editors challenged conventional assumptions within medicine at key points in the volumes. At the end of the first chapter, for example, Chalmers emphasized the antipathy of the ECPC project to assertions of clinical authority that were not informed by experimental evidence: ‘A characteristic of a well-designed enquiry into the effects of clinical practice,’ he wrote, ‘is that it is basically anti-authoritarian.’ Then he identified the ‘variety of strategies’ persons who derived authority from power rather than knowledge could use to ‘discredit studies that yield results that challenge their certainties.’ He concluded the paragraph with an example of the transformative power of systematic reviews: the ‘evidence used to prepare this book’ had ‘helped to shatter … fondly cherished certainties’ of its editors and authors (31).

The editors and two other authors (Jini Hetherington and Diana Elbourne) began the second chapter of volume 1 by criticizing conventional standards for evaluating evidence regarding the effectiveness of interventions. ‘Some scientists,’ they wrote, ‘have a remarkably casual attitude to the process of synthesizing research evidence.’ This attitude often led them to disguise their bias and self-interest in the rhetoric of science. Such researchers draw ‘conclusions … based more on factors such as their gender and how they make a living then on the available evidence.’ Then the editors and their colleagues accused many medical scientists of neglecting three decades of work on research synthesis by statisticians and investigators in education and psychology (39–42). Now, however, they switched to their scientific voice for a detailed summary of the methodology of systematic reviews. But they began the concluding section with an ironic quotation from Charlie Brown of Peanuts: ‘I am always certain about things that are a matter of opinion.’ Chalmers recently recalled that this chapter was a ‘very rare example of a medical textbook with a Materials and Methods section’ (Chalmers, email, December 24, 2010). It was also a rare example of a textbook with two voices.

The editors began the concluding chapter of the second volume by restating their ‘underlying thesis’ in language that was less confrontational than what they had used in the first chapters of volume 1. ‘Evidence from well-controlled comparisons,’ they wrote, ‘provides the best basis for choosing among alternative forms of care.’ Still speaking in the voice of science, they reiterated points they had made in the first volume about the relevance of their work to policy and practice. Research ‘based on the study of groups’ was ‘relevant to guiding … broad policy for care’ because it ‘generates evidence about how people respond to particular forms of care on average (1465).

They even invoked clinicians’ experience of care rather than repeating their challenge to authority that was based on power and prestige rather than evidence. ‘Tailoring care to meet the specific needs of individuals,’ they wrote, ‘will continue to be more of an art than a science.’ As a result, the ‘differing circumstances and values of different individuals may provoke different reactions to the same quality of evidence’ (1466). One reviewer of a draft of my article complained that this rhetoric contradicted the goals of the ECPC project: ‘By conceding [an art of medicine] … to clinicians,’ he wrote, ‘they were regressing into a pre-modern concept of medicine’ when instead their ‘project [was] based on the premise that some therapeutic questions can only be answered at the population level’ (Morabia, comments on draft of this article, February 23, 2011).

The authors promptly revealed, however, that they had harnessed the cliché of the art of medicine to their polemical intent. They redefined the art of medicine to make it compatible with population-level evidence. This art ‘can be improved,’ they concluded, ‘by listening more carefully to what women have to say, and by involving them to a greater extent in decisions about their care.’

Despite their apparent embrace of traditional rhetoric regarding the art and science of medicine, the editors concluded the chapter with four appendices that, by implication, summarized the challenge of the ECPC project to clinicians and policymakers. These appendices listed four ‘forms of care’: those that ‘reduce negative outcomes of pregnancy and childbirth,’ that ‘appear promising but require further evaluation,’ that have ‘unknown effects which require further evaluation,’ and that ‘should be abandoned’ (1467–77). Because these appendices summarized what clinicians should do, stop doing, and study, they soon attracted attention both inside and outside medicine.

Initial responses to the rhetoric of ECPC within medicine

The blending of scientific and polemical rhetoric in ECPC evoked strong reactions. The author of a commentary in the British Journal of Obstetrics and Gynecology hailed ECPC as ‘probably the most important book on obstetrics to appear in this century’ (Paintin 1990, 967). But a reviewer in another obstetrical journal pointed out, ‘The price of 225 [pounds sterling] should protect aspiring registrars [residents] from acquiring too many confused ideas from its pages (Hawkins 1990, p 344). Another senior obstetrician, in a conversation reported to Chalmers, compared the authors with terrorists: ‘an obstetrical Baader-Meinhof [i.e., anarchist] gang’ (Chalmers, email, December 24, 2010). The rhetoric of ECPC may have encouraged these reviewers to write their own polemics.

Most reviewers commented only indirectly on the rhetoric of ECPC and its companion publications. Their reviews emphasized the explanatory power of systematic reviews of randomized evidence while ignoring the volumes’ explicit and implicit polemic. One called the Oxford Database of Perinatal Trials (ODPT) a ‘great visionary achievement’ and commended it as the ‘technology of the future’ to ‘those who enjoyed’ ECPC (Neilson 1991, 175). Another praised the potential of ECPC and GECPC to ‘make a major contribution towards improving the quality of pregnancy care’ but was silent about the authors’ suggestions about how to facilitate that contribution (Zander 1990, 396). A reviewer in the Journal of the American Medical Association characterized ECPC as ‘innovative,’ ‘important,’ and ‘monumental.’ He especially praised the chapter ‘Materials and Methods’ and the appendices to the concluding chapter (Gold 1990).

I have found only two authors who acknowledged the two voices of the volumes’ prose. Michael S Kramer began his review in Science by noting that ‘ECPC is very different from the traditional clinical textbook.’ Then he praised its ‘democratic and ‘caring’ flavor that lends a refreshingly human backdrop to its scientific rigor’ (Kramer 1991, 816). Similarly, Fred Mosteller described the authors’ ‘consistent concern … for the personal side of care’ as an ‘endearing feature’ (Mosteller 1993, 528).

Most of the authors of reviews and early articles about the project did not recount either the history of systematic reviews or contemporary international activity to improve and apply their methods. A news article in Science, ‘Meta-Analysis in the Breech,’ is a partial exception. The author, a freelance journalist, listed contemporary contributors to the methods of systematic reviews, for instance, Gene Glass in psychology, Richard Light in education, and Thomas Chalmers, Rory Collins, Kay Dickersin, Joseph Lau, and Richard Peto in medicine. But the journalist’s reason for writing the article was that ECPC was controversial. He characterized it as a ‘remarkable report [that] offers a window onto the possibilities raised by meta-analysis and the vitriol it evokes’ (Mann 1990, 476). In the history of science ECPC was the result of methodological developments that occurred over many years. The project was an important event in the history of health policy because it stimulated both controversy and new policy.

Antagonism to systematic reviews in the health sector

The methods and findings of ECPC and many subsequent systematic reviews antagonized some influential persons in the health sector. Many clinicians became uncomfortable, even angry, when findings of particular reviews contradicted their experience of treating patients. In countries with fee-for-service payment systems, these systematic reviews could re- duce clinicians’ incomes if they contributed to limiting or eliminating reimbursement for particular interventions. Examples of clinicians’ antagonism to systematic reviews since the mid-1990s include controversies over autologous bone marrow transplants, fertility services, and treatment of back pain (Chalkidou et al. 2009; Fox 2010a; Moynihan 2008; Rettig et al. 2007; Steinbrook 2008; Wall Street Journal 2009). William Silverman, a pioneer of American neonatal medicine and a friend and mentor of Iain Chalmers, suggested that such controversies revealed that the ‘statistical approach was anathema to free-wheeling doctors who resented any doubts about the effectiveness of their untested treatments’ (Watts 2005, 257).

Executives of both the firms that manufacture pharmaceutical drugs and medical devices and their trade associations have attacked systematic reviews that threatened to reduce their companies’ market share of particular products. Statins, atypical antipsychotic drugs, and implantable devices are recent examples (Fox 2010a; Gibson 2006; Moynihan 2010). Beginning in the early 1990s, trade associations and their member firms devised strategies to blunt or prevent the influence of independent systematic reviews (i.e., of studies they did not sponsor). They have, for instance, dedicated staff and engaged consultants to challenge reviews that threatened their profits and the methodology of research synthesis more generally, falsified or hidden data from clinical trials, and con- ducted media campaigns to promote evidence that contradicted the findings of reviews (Fox 2010a; Jorgensen, Hilden, and Goetzsche 2006; McCloskey and Ziliak 2008; Murray 2004; Pear and Dao 2004).

The firms also directly pressured policymakers to thwart systematic reviews they opposed. For example, an international pharmaceutical firm threatened to move a factory to another state if policymakers for Medicaid in Michigan used findings from reviews to deny coverage for any of its products (Haveman 2001). After a systematic review of a drug for influenza, the chairman of Glaxo Wellcome, in a conversation at Downing Street, ‘threatened to take his research out of ‘ Britain (Timmins 2009). And many companies made campaign contributions to elected officials and political parties (Fox 2010a).

The constituency for systematic reviews in the health sector

Despite this persistent antagonism to systematic reviews, a constituency for them among clinicians and health care managers has grown since publication of the ECPC project. A steadily growing number of international journals have published articles based on systematic reviews and on “evidence-based medicine” more generally. The authors of clinical practice guidelines issued by specialty societies, public agencies, and hybrid organizations (e.g., the Scottish Intercollegiate Guidelines Network) have increasingly based their recommendations on systematic reviews (Agency for Healthcare Research and Quality 2011). Medical students and house staff have learned the skills of critically appraising clinical literature, which are fundamental to understanding the methods and uses of systematic reviews (Choudhry et al. 2005; Montori and Guyatt 2008). Examiners for certification and recertification in some medical specialties now expect knowledge of the findings of systematic reviews.

This growing constituency was, as Mosteller predicted in 1992, in part a response to the example set by the ECPC project and to the persuasiveness of its two voices. No less important, however, new international organizations generalized the example, recruited participants, and organized them in international networks. Like many other scientific associations, the Cochrane and Campbell Collaborations set methodological standards and publish research that meets them.

Unlike other scientific organizations, however, the Collaborations have, since their inception (Cochrane in 1993, Campbell in 1999), reinforced and projected the two voices of the ECPC project. They have defined themselves as international movements composed of thousands of volunteers who share values about research and its application to patient care (even though most participants are employed by universities, freestanding research organizations, or organizations that provide health services and reward researchers with income and/or prestige). The officers and governing bodies of the Collaborations reinforce their self-image as movements, especially during annual international colloquia.

Chalmers, with others, led the organizing work for the Cochrane Collaboration, which by 2010 comprised 28,000 people in more than one hundred countries. The Cochrane Database of Systematic Reviews, the Collaboration’s electronic journal, attracted increasing numbers of readers and became a significant source of citations in other journals (Bastian et al. 2010; Starr et al. 2009; Tovey 2010).

The Campbell Collaboration (C2) and its Library have organized and disseminated systematic reviews in the fields of criminal justice, education, and social services (Campbell Collaboration 2011). The founders of C2 based its identity on Cochrane, as a movement ‘based on voluntary cooperation among researchers.’ A partial indicator of both the persuasiveness of findings from reviews and the success of their promoters is that the number of published reviews rose each year, from 87 in 1987 to 2,500 in 2005 (Moher et al. 2007).

Support from influential advocacy groups for women’s health and from journalists reinforced the constituency within the health sector for the ECPC project and the subsequent systematic reviews it influenced. In the mid-1990s, leading advocacy groups for women’s health financed a review that challenged widely accepted medical opinion about the benefits of autologous bone marrow transplants (Lerner et al. 2010; Rettig et al. 2007). The authors of the pathbreaking book Our Bodies Ourselves began to incorporate the findings of systematic reviews in new editions after publication of the ECPC project (Boston Women’s Health Book Collective 2011; Judith Norsegian, founder and leader of Our Bodies Ourselves project, telephone conversation, March 2011). In the early

1990s, journalists began to publish articles about the potential benefits of systematic reviews. For example, a noted science reporter wrote a cover article about ECPC and GECPC for Parade, the Sunday newspaper supplement with the highest circulation in the United States (Ubell 1993). Ray Moynihan, who began to cover systematic reviews for Australian newspapers and television at about the same time, subsequently earned an international reputation (Moynihan 2011). By 2000, reporters and columnists for the Wall Street Journal and other major newspapers were reporting regularly on systematic reviews (Fox 2010a).

Systematic reviews inform policymakers for health

The potential importance of systematic reviews for policy to allocate resources to and within the health sector emerged gradually. Most experts on policy who wrote about the ECPC project soon after its publication assumed that proponents of systematic reviews would accord priority to influencing clinicians’ work, for example, by cultivating champions of such reviews among them (Cleary and Fox 1993; Lomas 1993; Sisk 1993; Stocking 1993).

Nevertheless, public officials in a few countries began to consider how systematic reviews could inform policy for allocating resources. They speculated that the findings of systematic reviews could contribute to care that was more effective and efficient, mainly by identifying interventions that were unnecessary, ineffective, or harmful. Some also wondered whether reviews that met rigorous international standards could also discourage lobbying, by professional and commercial interest groups and advocacy organizations, for more spending for particular interventions (Fox 2005, 2010a; Fox and Greenfield 2006).

Most important, a small number of policymakers hoped that systematic reviews might help them contain the rate of increase in spending for health care, to spend less as well as to spend better. Since the 1970s, annual increases in health costs have usually exceeded general inflation in industrial countries and, as a result, constrained spending for other public purposes. In the United States, where health insurance for most people of working age is linked to employment, increasing health care costs also have restricted growth in wages and reduced firms’ profits by making them less competitive in the global economy.

Policymakers who paid attention to systematic reviews usually learned about them from persons who had earned their trust as advisers. Most of these advisers were academic researchers or staff of philanthropic foundations. A few were policymakers themselves. For example, Lee Greenfield, who began in 1993 to teach policymakers about the methods and uses of systematic reviews, chaired the committee of the Minnesota House of Representatives that appropriated funds for Medicaid and other health programs (Fox 2010a; Fox and Greenfield 2006).

The growing interest in systematic reviews among policymakers has had two principal results. The first is a gradual increase in public funding to conduct reviews and train reviewers, with the timing and amounts varying widely among countries. The second result is the establishment of public and quasi-public organizations to conduct, commission, and assess systematic reviews and to apply, or advise about applying, their findings. These new organizations are variously called institutes, committees, schemes, and commissions (Fox 2010a, 2010b). The oldest of them is the Australian Pharmaceutical Benefits Advisory Committee (PBAC), established many years earlier but an early adopter of systematic reviews.

Systematic reviews began to inform policy at different times in different countries. The first endorsement of systematic reviews by policymakers for health services seems to have occurred in Britain in 1991, when the new Research and Development (R&D) Programme of the National Health Service (NHS) agreed to finance, for three years, Chalmers’ proposal for ‘a Cochrane Centre to facilitate the preparation, maintenance and dissemination of systematic reviews of randomized controlled trials of health care’ (Chalmers 2003, 251).

The founding director of the R&D Programme, Michael Peckham, who had been a prominent medical academic, endorsed systematic reviews because they contributed to achieving the priorities of the NHS R&D Programme. The new Programme, he wrote, ‘is an essential activity —indeed, a prerequisite for achieving a cost-effective health service response to changes in needs as well as to innovation’ (Peckham 1991, 368).

Systematic reviews have informed policy to pay for pharmaceutical drugs in many countries. The Australian PBAC began in 1993 to require that systematic reviews be conducted whenever sufficient evidence from primary studies was available (Ruth Lopert, Australian civil servant with experience in policymaking for pharmaceutical drugs, interview, July 8, 2008). By 1998 such reviews had informed 28 percent of the decisions by PBAC about insurance coverage (David Henry, former PBAC research scientist, email, February 18, 2008). The New Zealand Pharmaceutical Agency has used findings from systematic reviews from its inception in 1993, and an intergovernmental agency that assesses health technology in Canada has used methods of systematic reviewing since the mid-1990s (Morgan et al. 2006). The first Reference Drug Program in that country, established in British Columbia in 1995, relied on systematic reviews, its founding director reports (Bob Nakagawa, British Columbia’s assistant deputy minister of health for pharmaceutical services, email, February 18, 2008). In both Australia and Canada, senior policymakers have intervened, not always successfully, to defend systematic reviewers, as well as the organizations that commissioned them, against negative criticism and lobbying by interest groups and advocates (Fox 2010a; Henry, email, February 18, 2008; Lenzer 2010, Lopert, interview, July 8, 2008; Moynihan 2010; Nakagawa, email, February 18, 2008).

Policymakers in the United States began to apply the findings of systematic reviews in the late 1990s. Soon after the publication of ECPC and GECPC, legislative leaders and executive branch officials in state government responded enthusiastically to a talk in which I described the four appendices at the end of the second volume of ECPC and the methodology that produced them (Fox and Greenfield 2006). A year later the senior health official in President George HW Bush’s administration endorsed the ECPC project at a conference in Washington, whose sponsors included the American College of Obstetricians and Gynecologists. Officials of federal research agencies did not act on this endorsement until 1996, when the Centers for Disease Control and Prevention began to commission reviews for its Guide to Community Preventive Services (Benedict et al. 2000). A year later, the predecessor of the Agency for Healthcare Research and Quality (AHRQ) designated the first “evidence-based practice centers” (EPCs) to conduct evidence reviews, including systematic reviews. The US Preventive Services Task Force, established in 1984 and administered by the AHRQ beginning in 1998, commissioned systematic reviews from EPCs as the basis for its recommendations about the effectiveness of interventions (Fox 2010a).

Conclusion: rhetoric and policy

Despite this paradox of the relationship between science and policy, the ECPC project has had substantial influence. The research on which it reported built on advances in methodology that occurred over many years in a variety of disciplines. The editors’ two voices, their blending of scientific and polemical rhetoric, helped communicate the potential significance of systematic reviews to diverse audiences, which included clinicians, consumers, journalists, and managers of health care organizations as well as policymakers.

The ECPC project was, at first, persuasive to a relatively small number of researchers, clinicians, and advocates for women’s health. These people became the core of a diverse and expanding international constituency. The editors and authors of the project became leaders in organizing, focusing, and expanding the constituency for systematic reviews. But the ECPC project also sparked antagonism within the medical profession and among commercial and advocacy groups.

This JLL Bulletin article has been co-published in The Milbank Quarterly 2011;89:425–449.


Iain Chalmers invited me to write this article for the James Lind Library and approved its co-publication by the Library and The Milbank Quarterly. I benefited from comments on successive drafts by Chalmers, Ulrich Tröhler, Alfredo Morabia, Murray Enkin, Andrew D Oxman, Jack Sinclair, Michael Bracken, Bradford H Gray, and anonymous reviewers for the Quarterly. The article is based on conversations over two decades with many researchers and policymakers as well as on conventional methods of research in contemporary history. Since 1989, the directors of the Milbank Memorial Fund have supported, morally and financially, my participation in some of the activities I report on in this article.


Agency for Healthcare Research and Quality (2011). National Guideline Clearing House (search term: systematic reviews). Available at .

Bastian H, Glasziou P, Chalmers I (2010). Seventy-Five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Medicine 7 September

Benedict I, Truman C, Smith-Akin K, Hinman A (2000). Developing the guide to community preventive services. American Journal of Preventive Medicine 18:18–26.

Boston Women’s Health Book Collective (2011). Our Bodies Ourselves. Available at .

Campbell Collaboration (2011). About Us. Available at .

Chalkidou K, Tunis S, Lopert R, Rochaix L, Sawicki P, Nasser M, Xerri B (2009). Comparative effectiveness research and evidence-based health policy: experience from four countries. Milbank Quarterly 87:339–67. Available at 

Chalmers I, ed. (1988–1992). The Oxford Database of Perinatal Trials. Oxford: Oxford University Press.

Chalmers I (1993). The Cochrane Collaboration: preparing, maintaining and disseminating systematic reviews of the effects of health care. In Warren KS, Mosteller F, eds. Doing More Good Than Harm: the evaluation of health care interventions. Annals of the New York Academy of Sciences 703:156–63.

Chalmers I (2003). The prehistory of the first Cochrane Centre. In Bosch X, ed. Back to the Front. Barcelona, privately printed, p 242–53.

Chalmers I, Enkin M, Keirse MJNC, eds. (1989). Effective Care in Pregnancy and Childbirth. Oxford: Oxford University Press.

Chalmers I, Enkin M, Keirse MJNC. Preparing and updating systematic reviews of randomized controlled trials of health care. Milbank Quarterly 1993;71:411-437.

Chalmers I, Hedges LV, Cooper H (2002). A brief history of research synthesis. Evaluation and the Health Professions 25:12–37.

Choudhry N, Fletcher R, Soumerai S (2005). Systematic review: the relationship between clinical experience and the quality of health care. Annals of Internal Medicine 142:260–73.

Cleary P, Fox DM (1993). In This Issue. The Milbank Quarterly 71:345–48.

Dickersin K, Mannheimer E (1998). The Cochrane Collaboration: evaluation of health care and services using systematic reviews of the results of randomized clinical trials. Clinical Obstetrics and Gynecology 41:315–31.

Enkin M, Keirse MJNC, Chalmers I (1989). A Guide to Effective Care in Pregnancy and Childbirth. Oxford: Oxford University Press.

Fox DM (2005). Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions. Health Affairs 24:114–22.

Fox DM (2010a). The convergence of science and governance: research, health policy and American States. Berkeley: University of California Press.

Fox DM (2010b). Realizing and allocating savings from improving health care quality and efficiency. Preventing chronic disease: public health research, practice, and policy 7. Available at .

Fox DM, Greenfield L (2006). Helping public officials use research evaluating health care. Journal of Law and Policy 14:531–50.

Gibson M (2006). Attacks on systematic reviews. US Cochrane Center Conference, Baltimore, July 14.

Gold E (1990). Review of ECPC, vols. 1–2. JAMA 264:2577–78.

Gross A (2006). The place of rhetoric in science studies. Carbondale: Southern Illinois University Press.

Haveman J (2001). Remarks of the then Commissioner of Community Health Services in Michigan at an off-the-record meeting of the Reforming States Group, November (with permission).

Hawkins D (1990). Review of ECPC. Journal of Obstetrics and Gynecology(UK). 10:344–47.

Jorgensen A, Hilden J,Gøtzsche P (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333:1–5. Available at .

Kramer M (1991). Review of Effective Care in Pregnancy and Childbirth and A Guide to Effective Care in Pregnancy and Childbirth. Science 251:815–16.

Lenzer J (2010). Independent drug review group in Canada is squeezed out. BMJ 341:c7009. Available at .

Lerner J, Fox DM, Ruzek S, Shearer G (2010). The case for a national patient library. Health Affairs 29:1914–19.

Lomas J (1993). Retailing research: increasing the role of evidence in clinical services for childbirth [in Canada]. The Milbank Quarterly 72:439–75.

Mann C (1990). Meta-analysis in the breech. Science 249:476–80.

McCloskey D, Ziliak S (2008). The cult of statistical significance: how the standard error costs us jobs, justice and lives. Ann Arbor: University of Michigan Press.

Moher DA, Tetzlaff J, Tricco A, Sampson M, Altman D (2007). Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4. Available at .

Montori V, Guyatt G (2008). Progress in evidence-based medicine. JAMA 300:1874–76.

Morgan S, McMahon M, Mitton C, Roughead E, Kirk R, Kanavos P, Devidas M (2006). Centralized drug review processes in Australia, Canada, New Zealand and the United Kingdom. Health Affairs 25:337–47.

Mosteller F (1993). The prospect of data-based medicine in the light of ECPC. The Milbank Quarterly 71:523–32.

Moynihan R (2008). Independent drug watchdog in Canada under funding threat. BMJ 336:1270.

Moynihan R (2010). Drug firm’s legal action against government advisers is described as “intimidation.” BMJ 341: c6527.

Moynihan R (2011). Biography. Wikipedia. Available at .

Murray A (2004). Political capital: trade group’s fight against drug review is self-defeating. Wall Street Journal, November 24.

Neilson JP (1991). Review of Oxford Database of Perinatal Trials. Journal of Epidemiology and Community Health 45:175.

Paintin DB (1990). Commentary: Effective Care in Pregnancy and Childbirth. British Journal of Obstetrics and Gynecology 97:967–73.

Pear R, Dao J (2004). States’ tactics aim to reduce drug spending. New York Times, November 21.

Peckham M (1991). Research and Development for the National Health Service. Lancet 338:367–71.

Rettig R, Jacobson P, Farquhar C, Aubry W (2007). False hope: bone marrow transplantation for breast cancer. New York: Oxford University Press.

Sinclair JC, Bracken MB, eds (1992). Effective Care of the Newborn Infant. Oxford: Oxford University Press.

Sisk JE (1993). Improving the use of research-based evidence in policy making: Effective Care in Pregnancy and Childbirth in the United States. The Milbank Quarterly 71:497–521.

Starr M, Chalmers I, Clarke M, Oxman AD (2009). The origins, evolution and future of the Cochrane Database of Systematic Reviews. International Journal of Technology Assessment in Health Care 25(suppl. 1):182–95.

Steinbrook R (2008). Saying No isn’t NICE — the travails of Britain’s National Institute for Health and Clinical Excellence. New England Journal of Medicine 359:1977–80.

Stocking BC (1993). Implementing the findings of Effective Care in Pregnancy and Childbirth in the United Kingdom. The Milbank Quarterly 71:497–521.

Timmins N (2009). The NICE way of influencing health spending: a conversation with Sir Michael Rawlins. Health Affairs 28:1360–65.

Tovey D (2010). The impact of Cochrane Reviews. Cochrane Library, July7. Available at

Ubell E (1993). Are births as safe as they could be? Parade, February 7, p. 1.

Wall Street Journal (2009). Of NICE and men. Editorial, July 7. Available at .

Watts G (2005). William Silverman: one of the founders of American neonatal medicine. BMJ 330:257.

Wennberg JE (1984). Dealing with medical practice variations: a proposal for action. Health Affairs 3:6–32.

Zander L (1990). Review of Effective Care in Pregnancy and Childbirth and A Guide to Effective Care in Pregnancy and Childbirth. British Journal of General Practice 40:396.