Summary

Those promoting evaluation of the effects of health interventions to inform health choices face numerous challenges. The list includes: uncritical reliance on inadequately informed opinions, inadequate consideration of harmful effects of interventions, biased evaluations of effects, conflicts of interest, biased reporting of research, deficient reporting of research, deficient reviews, ineffective and inefficient peer review, not keeping up to date, unnecessary duplication of effort, misinformation, limited access to trustworthy information about the effects of policies and practices, gaps between evidence-based recommendations and professional practice, inadequately informed policymaking, and lack of patient and public participation in decision-making.

Over the past five decades, much progress has been made towards addressing these deficiencies, but improvement has been complex and non-linear, and many challenges remain. Research funders, publishers, academic institutions, healthcare providers, and governments each have primary responsibility for addressing different elements of these challenges and should be held accountable. To do and sustain the necessary work, collaborations of researchers, editors, health professionals, policymakers, patients, and the public are needed, and they should hold research funders, publishers, academic institutions, healthcare providers, and governments accountable.

Introduction

When health professionals intervene in the lives of others their actions sometimes inadvertently do more harm than good. The same risk is true for legislators or individuals making health choices. Informed health choices can increase the probability that patients benefit, that is that wanted intervention effects will outweigh unwanted effects. For this reason, all health policies and practices should be informed by the best available evidence of intervention effects.

In this commentary, we consider some of the challenges that confront those promoting evaluation of the effects of health interventions to inform health choices. By ‘health choice’ we mean any action that individuals or groups can choose to take in the belief that it will protect or improve their health or the health of others. ‘Health interventions’ include everything from dietary, device, surgical or pharmaceutical interventions for individuals to policies for population health.

We are not trying to cover everything that goes into making informed choices in this commentary. We focus particularly on evaluation of the effects of health interventions. People make health choices based largely on what they believe will happen because of their choices (the expected effects). For informed health choices, reliable information about the probability of those effects is essential, but not sufficient. Other types of research are also important, as is efficiency, equity, and inclusivity. However, it is not possible to make informed judgements about efficiency and equity without reliable information about the effects of a health choice.

Our interest in informed health choices began in the 1970’s. It started with raising questions about the opinions of authorities. This led to collaborating to help address some of the challenges considered in this commentary. Brief accounts of how we came to question authority and to recognise the need to collaborate can be found in Appendix 1.

This commentary is based on discussions we had over several months, our experience, and feedback from colleagues. In Appendix 1, we address how our experience may have affected this commentary. This, together with the references we cite, may help those bothering to reflect on our opinions to decide what to believe or do.

People have recognised for hundreds of years some of the problems we address. Some illustrative examples are outlined in Figure 1. However, over the past five decades awareness of these problems and efforts to address them have increased dramatically. Illustrative examples of this can be found in Appendix 2. There has been astonishing progress towards making informed health choices a reality.

Figure 1. Illustrative examples of promoting evaluation of the effects of health interventions to inform health choices dating back to the fifth century BCE.*

As early as 400 BCE Hippocrates stressed that a competent researcher must ensure that his starting point is knowledge of what has already been discovered, and Socrates questioned authority, including his own (Plato, Apology 21c – 21d).

More than a millennium ago, in the 9th century CE, Al-Razi (Rhazes) recognized that comparisons are needed to evaluate the effects of interventions, and for centuries people have recognised that in unbiased comparisons of interventions, like must be compared with like. For example, in 1364, Francisco Petrarca proposed an experiment to assess whether people would be better off avoiding rather than seeking medical treatment. In 1648, Jean-Baptiste van Helmont proposed casting lots to decide which patients should be treated by orthodox physicians with bloodletting and purging, and which by him without these treatments. The James Lind Library contains hundreds of reports of using prospective allocation to reduce the risk of comparison groups being biased, published as early as 1575 (by Ambroise Paré) and during the 1800s and early 1900s. In 1812 Frances Burney described in excruciating detail her experience undergoing mastectomy for breast cancer without anaesthesia, highlighting adverse consequences of a poorly informed decision by physicians and the lack of patient participation in decision-making.

By the end of the 1950s there was growing recognition of the use of random allocation to reduce the risk of comparison groups being biased with respect to prognosis and responsiveness to the interventions.

1941	Joseph Bell, in an exceptionally clearly written report of his trial of whooping cough vaccine, reported his use of random sampling numbers to generate unbiased comparison groups.
1948	The UK Medical Research Council’s report of a controlled trial of streptomycin for pulmonary tuberculosis was a methodological landmark because it provided detailed information about the trial, and in particular, the steps taken to prevent foreknowledge of treatment assignments.
1952	William Silverman published the first of many randomized trials in neonatology. “He played a role in North America that was equal to or even beyond that of Sir Austin Bradford Hill in the UK in terms of introducing randomised trials. Silverman carried out trials in areas where the overwhelming opinion was what was going on was correct. It took an enormous amount of courage to carry out these trials.” (Oransky, 2005)
1955	Thomas Chalmers , one of the most vocal promoters of randomized trials, published a detailed report of a trial that found no evidence that the prolonged bed rest that was commonly prescribed for hepatitis at the time promoted recovery. (Chalmers et al., 1955) He suggested that inadequately evaluated new treatments should only be used in randomised trials until their effects became known.
1959	By the end of the 1950s, the UK Medical Research Council had accumulated substantial experience of designing, running, analysing and reporting controlled trials. Austin Bradford Hill was invited to plan and chair a meeting in Vienna in 1959 to report this experience.

*Additional important events in the history of evaluations of interventions can be found in A Timeline of Fair Tests of Treatments in the James Lind Library.

To get to where we are today has been a long and winding road and still with a long way to go. For example, in 1987 Cynthia Mulrow showed that review articles in major medical journals were not using systematic, explicit methods to reduce the risk of being misled by bias or the play of chance.(Mulrow, 1987) There has since been an explosion in the production and publication of systematic reviews, but there is unnecessary and confusing duplication of effort.(Hoffmann et al., 2021, Page et al., 2016, Ioannidis, 2016, Bastian et al., 2010) Access to high-quality reviews is also limited, partly because of poor communication and pay walls.(Oxman et al., 2020, Oxman and Paulsen, 2019) For these and other reasons, many health choices still are not informed by high quality, up-to-date systematic reviews.(Ioannidis, 2016, Bastian et al., 2010, Clarke et al., 2024)

The fundamental problem underlying poorly informed health choices is uncritical reliance on poorly informed opinions. Often, authorities (and others) express strong opinions about the effects of interventions without being explicit about the basis for their opinions, or cherry-picking evidence.  Those opinions influence choices without people questioning their basis. In the current political environment, with increasing authoritarianism and massive amounts of misinformation, this is particularly important for all kinds of interventions, not just healthcare interventions.(Nord et al., 2024, Swire-Thompson and Lazer, 2020, Aronson et al., 2019)

Other problems using evaluations of the effects of health interventions to inform policies and practice include:

• inadequate consideration of the harmful effects of intervention
• biased evaluations of intervention effects
• conflicts of interest
• biased reporting of research
• deficient reporting of research
• deficient reviews
• ineffective and inefficient peer review
• failure to keep reviews up to date
• unnecessary duplication of effort
• misinformation
• limited access to trustworthy information about the effects of policies and practices
• gaps between evidence-based recommendations and professional practice
• inadequately informed policymaking
• lack of patient and public participation in decision-making

Collaboration is needed to address all these challenges. Like the importance of thinking critically about opinions, collaboration and collective action are more important than ever, given the many challenges facing humanity. These include massive inequities, pandemics, antimicrobial resistance, unsustainable use of resources, and climate change.

Uncritical reliance on inadequately informed opinions

Researchers, doctors, pundits, influencers, and other authorities often disagree about the effects of interventions. This may be because their opinions are not routinely informed by systematic reviews. People frequently consider: Who expressed the opinion? How strong is their opinion? And how much experience do they have? The answers to these questions do not always provide a trustworthy basis for judging the reliability of opinions.

This does not mean that all opinions should be given equal weight – or that the existence of conflicting opinions means that no useful conclusion can be reached. How much weight to give an opinion about the effects of a health intervention should be based on the strength of the supporting evidence. Experts, like everyone else, do not always base what they say on the results of systematic reviews. For example, experts did not begin to recommend aspirin after a heart attack until years after there was strong evidence supporting its use. (Antman et al., 1992) Conversely, experts continued to recommend medicines to reduce heart rhythm abnormalities for years after there was strong evidence that these medicines increased the risk of early death after a heart attack.

People often rely on the opinions of experts or other authorities. That is reasonable, but people frequently do not handle those opinions with due caution. Expert opinions are nearly always based on evidence of some kind. This can be, for example, a systematic review of randomised trials, anecdotal experience, or the results of laboratory studies. The problem is that unless an expert (or anyone else) is explicit about the basis for an opinion about the effects of an intervention, it is not possible to critically appraise that opinion. (Schunemann et al., 2019)

Recommendations for practice and healthcare advice are much more likely to be explicitly informed by evidence today than they were 50 years ago. But all too often experts (and others) are still not sufficiently explicit about the basis of their opinions about intervention effects. (Ponce et al., 2017, Morgan, 2014)

Informed decisions about what to believe, and what to do, depend on critical thinking. (Sharples et al., 2017) Critical thinking in turn depends on understanding and applying principles for assessing the trustworthiness of claims about intervention effects and for making informed choices. (Aronson et al., 2019) Over the past five decades, teaching evidence-based practice (EBP) to health professionals has spread around the world. However, a lack of EBP knowledge and skills is still one of the most frequently reported barriers to practising EBP. (Albarqouni et al., 2018) Similarly, critical thinking is widely considered to be an important 21^st century competence that should be taught in primary and secondary schools. (Voogt and Roblin, 2012) However, although there is broad support among the public and among teachers for teaching critical thinking, it is not being taught or practised adequately.(Dahlgren et al., 2021, Reboot Foundation, 2020)

Improving the extent to which health choices are informed requires enabling people to avoid uncritical reliance on inadequately informed opinions. This depends on developing, evaluating, and implementing effective interventions to enable people to question the basis for opinions about the effects of interventions.

Inadequate consideration of harmful effects of interventions

Almost all health interventions have unwanted effects that must be weighed against potential desired effects. These include individual treatments, but also public health interventions and educational interventions. (Stratil et al., 2024, Oxman et al., 2024) Poor reporting of harms makes this weighing up difficult. Findings that suggest benefits tend to be emphasized, while potential harms are downplayed or ignored. (Zorzela et al., 2016, Bonell et al., 2015) Moreover, reliable evidence of harmful effects, especially long-term harms, often lags behind evidence of beneficial effects. (Cuervo and Clarke, 2003, Chou et al., 2010, Zorzela et al., 2016)

Added to these deficits, press releases for research reports are often designed to attract favourable media attention, and news reports of those studies do the same. (Yavchitz et al., 2012) Most news reports about health interventions mention at least one benefit, but fewer than half mention or adequately discuss harms. (Oxman et al., 2022b) Patients and health professionals tend to overestimate the benefits and underestimate harms of health interventions. (Hoffmann and Del Mar, 2017, Hoffmann and Del Mar, 2015, Rejas Bueno et al., 2022)

Improving the chances that wanted intervention effects outweigh unwanted effects will depend on researchers, health professionals, policymakers, patients and the public ensuring that harmful effects are adequately measured, reported and considered.

Biased evaluations of intervention effects

The use of randomised trials to evaluate the effects of interventions has increased dramatically. (Bastian et al., 2010) But many evaluations (both randomised and non-randomised) do not take adequate steps to reduce the risk of bias. (Dechartres et al., 2016, Ioannidis et al., 2014)

Random allocation of people to comparison groups is unbiased with respect to prognosis and responsiveness to the compared interventions. No other way of creating comparison groups has these properties. This is because it cannot be assumed that all factors relevant to prognosis and responsiveness have been distributed in an unbiased way between comparison groups. (Kleijnen et al., 1997)

When only a few people are randomly allocated to comparison groups, important differences can occur by chance. Many randomised trials are too small to reliably detect even very large effects and can be misleading due to bias, as well as from the play of chance. (Oxman et al., 2023b, Pereira et al., 2012, Nagendran et al., 2016)

Randomised and non-randomised evaluations of the effects of health interventions can be misleading for other reasons. (Sterne et al., 2016, Guyatt et al., 2025b) These include failure to ensure that the people being compared were cared for similarly and were unaware of the intervention they had received. They also include ensuring that outcomes were measured consistently and reliably, and that there was little loss to follow-up. (Oxman et al., 2023b, Sterne et al., 2016, Guyatt et al., 2025b, Oxman et al., 2023a)

Analyses of data from large databases of routinely collected data are sometimes advocated as an alternative to randomised trials. However, claims based on “big data” or “real world data” can be misleading. (Oxman et al., 2022a) More data simply yield more statistically precise estimates of the effects of whatever biases there might be in a treatment comparison that uses routinely collected data. When using routinely collected data, it is only possible to control for known confounders that have been measured. Unfortunately, routinely collected data often do not include sufficient detail to confidently conclude that any association found between an intervention and an outcome means that the intervention caused the outcome.

When there are important uncertainties about the effects of health interventions, they should ideally be evaluated in well-designed randomised trials. The trials need to be large enough to yield reliable estimates of effects on important outcomes. As the Australian economist and politician Andrew Leigh writes at the end of his book describing randomised trials of all types of interventions: “If we let our curiosity roam free, we might be surprised how much we can learn about the world, one coin toss at a time.” (Leigh, 2018)

Because resources are limited, it is vital to identify important uncertainties about the effects of interventions then set priorities for evaluations that minimise the risk of bias. Collaboration among research funders, research institutions, researchers, policymakers, patients and the public can help do this. (The James Lind Alliance)

Conflicts of interest

In addition to wanting to help people, people may have other interests in promoting a particular intervention, for example, to make money. They may promote an intervention by exaggerating its benefits, ignoring potential harms, cherry picking which information is used, or making exaggerated or false claims. Conversely, people may object to an intervention for a range of reasons, such as cultural practices.

Financial conflicts of interests can lead to bias in several ways. (Dunn et al., 2016) Researchers with conflicts of interest are more likely to choose less effective control interventions, leading to more favourable results for a new drug. (Dunn et al., 2013, Hugenholtz et al., 2006, Lathyris et al., 2010, Lundh et al., 2017) They may be more likely to selectively report outcomes that favour the intervention and not to publish the results of a trial if it does not favour the intervention. (Dwan et al., 2013, Sandström and Hällsten, 2008) They may also be more likely to draw favourable conclusions and recommend the intervention when they have a financial conflict of interest. (Als-Nielsen et al., 2003, Chartres et al., 2016, Yank et al., 2007, Norris et al., 2011)

Financial conflicts of interest also can influence what research is funded by industry. (Fabbri et al., 2018) Review authors may also be more likely to interpret results favourably when they have financial conflicts of interest. (Barnes and Bero, 1998, Jørgensen et al., 2006, Bes-Rastrollo et al., 2013, Dunn et al., 2014, Mandrioli et al., 2016, Hansen et al., 2019) Cost-effectiveness studies funded by industry are more likely to present favourable results than other studies, (Bell et al., 2006) and authors of clinical practice guidelines may be more likely to recommend a treatment when they have a financial conflict of interest. (Norris et al., 2011, Cosgrove et al., 2013, Nejstgaard et al., 2020, Tabatabavakili et al., 2021)

Nonfinancial interests can also influence which research is funded, (Tamblyn et al., 2018, Sandström and Hällsten, 2008, Abdoul et al., 2012) and which interventions are recommended. (Akl et al., 2014, Norris et al., 2012) They may also influence the results of evaluations of the effects of interventions and systematic reviews, although evidence of this is limited. (Panagiotou and Ioannidis, 2012, Bero, 2014, Akl et al., 2014) Assessing whether an interest poses a conflict of interest requires the evaluation of whether it increases the risk of inappropriate judgment, decision, or action. Classifying different types of interest may facilitate reporting and managing conflicts of interest. (Akl et al., 2022)

It seems likely that prior beliefs and career advancement are associated with reporting practices that distort the interpretation of research results and the reproducibility of research. It has been argued that it is inappropriate to classify these as conflicts of interest. (Bero, 2014) But that does not mean they do not warrant concern.

A systematic review of spin in the biomedical literature did not find evidence of an association between author characteristics and spin. (Chiu et al., 2017) There is, however, a great deal of empirical evidence supports the notion that confirmation bias is extensive and influential. (Nickerson, 1998) The evidence also supports the view that once one has a belief, the primary motivation in seeking and evaluating information is to defend or justify that belief.

People tend to seek information that they consider supportive of their existing beliefs and to interpret information in ways that endorse those beliefs. Conversely, they tend not to seek and perhaps even to avoid information that contradicts their beliefs. (Nickerson, 1998) Citation bias (selective citation of scientific articles based on their results, rather than their methods) can be seen as a manifestation of confirmation bias. (Gøtzsche, 2022) Studies of citation bias have found that articles in which the authors explicitly concluded to have found support for their hypothesis were cited 2.7 times as often as articles that did not. (Duyx et al., 2017). Similarly, there is evidence that pressure to “publish or perish” increases researchers’ bias. (Fanelli, 2010, Cobey et al., 2024)

Most research on funding bias (conflicting interests of funders) and conflicts of interests of researchers and authors of reviews or guidelines is devoted to evaluating the prevalence, nature, and effects of disclosing conflicts of interest. (Graham et al., 2025, Korfitsen et al., 2025) While disclosure policies are ubiquitous, those policies are not consistently designed, implemented, or enforced. Disclosure alone is insufficient. It is not particularly effective in mitigating undesirable consequences of conflicts of interest. (Graham et al., 2025) Effective strategies are needed to assess whether an interest constitutes a conflict of interest and to better manage conflicts of interest. (Xun et al., 2024) There is also a need to identify and reduce incentives that contribute to distorting the conduct, reporting, and interpretation of research. (Trueblood et al., 2025, Allen, 2025)

Biased reporting of research

Many evaluations of the effects of interventions are not published, (Chapman et al., 2014, Hopewell et al., 2009, Dwan et al., 2013) and outcomes are sometimes selectively reported in research reports. (Dwan et al., 2013, Fleming et al., 2015, Page et al., 2014, Jones et al., 2015, Riveros et al., 2013) Those outcomes that are published are more likely to report favourable results. Consequently, relying only on published reports sometimes results in overestimating wanted effects of interventions and underestimating unwanted effects.

The International Committee of Medical Journal Editors (ICMJE) adopted a policy in 2004 to mitigate bias resulting from selective reporting of research. (De Angelis et al., 2004) The policy requires, as a condition of consideration for publication, registration of trials in a trial registry. In 2006, the World Health Organization established the International Clinical Trials Registry Platform to link trial registries to ensure a single point of access and unambiguous identification of trials, and to establish standards for trial registries. (World Health Organization, 2018) These and other initiatives have helped accurately document non-publication, but biased reporting of evaluations of interventions remains a problem. Better understanding and evaluation of solutions is needed. This will require the collaboration of research funders, (Goldacre, 2017) industry and regulators, (Goldacre et al., 2017, Baudard et al., 2017, Manheimer and Anderson, 2002) trial registries, journal editors, (De Angelis et al., 2004) research institutions, (DeVito et al., 2024) systematic review authors, (Page et al., 2023, Guyatt et al., 2025b, Baudard et al., 2017, Kirkham et al., 2018) and researchers. (Hohlfeld et al., 2024)

Deficient reporting of research

For people to be able to critically appraise research, the research must be clearly and completely reported. Recognition of inadequacies in the reporting of research dates back at least 50 years. (Begg et al., 1996) In 1996, two groups working independently to address this problem collaborated to create the first Consolidated Standards of Reporting Trials (CONSORT) statement in 1996. The statement is a checklist of items that should be included in reports of randomised trials, which has been updated three times since 1996. (Hopewell et al., 2025) The original Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement for reporting protocols for randomised trials was published in 2013. (Hróbjartsson et al., 2025) Major updates of the CONSORT and SPIRIT guidelines were published in five major medical journals in 2025. (Hopewell et al., 2025, Hróbjartsson et al., 2025)

Other deficiencies in reporting research have been recognised, and many other checklists for reporting research have been developed subsequently. This includes checklists to improve the reporting of descriptions of interventions, (Hoffmann et al., 2014, Hoffmann et al., 2017) systematic reviews, (Page et al., 2021) and guidelines.(Chen et al., 2017) The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network was officially launched in 2008 as a global collaborative initiative. Its objective is to improve the value and reliability of published health research by promoting transparent and accurate reporting and wider use of evidence-based reporting guidelines. (Catalá-López et al., 2023) The EQUATOR Network’s website contains a database of reporting guidelines for health research. (EQUATOR Network)

The first formal Outcome Measures in Rheumatology (OMERACT) conference was convened in 1992 to address deficiencies in the measurement and reporting of outcomes in randomised trials of interventions for people with autoimmune and musculoskeletal diseases. (Boers et al., 2014, de Wit et al., 2017) In 2010 the Core Outcome Measures in Effectiveness Trials (COMET) initiative was launched. (Kirkham et al., 2017, Kirkham et al., 2019) It promotes the development and use of core outcome sets to improve the measurement and reporting of outcomes for trials in all specific areas of health or health care.

Reporting guidelines and core outcome sets have been widely disseminated and endorsed, but there remain important inadequacies in reporting research. (Turner et al., 2012, Samaan et al., 2013, Glasziou et al., 2014) There is a wide range of strategies that research funders, editors and peer reviewers, academic and research institutions could currently use to improve adherence to reporting guidelines, but little evidence of the effects of those strategies. (Blanco et al., 2019, Blanco et al., 2020, Nasser et al., 2017)

Deficient reviews

Reviews that fail to use systematic methods may yield biased or imprecise estimates of the effects of interventions. Unsystematic searches mean good studies may not be found, and poor selection or appraisal of identified studies may lead to bias. Finally, the synthesis of the results of the included studies may be inadequate or inappropriate. (Whiting et al., 2016)

Systematic reviews and meta-analyses of the effects of health interventions began to appear in growing numbers in the 1970’s and 80’s. (Bastian et al., 2010) In 1993 when the Cochrane Collaboration was launched there still were not many systematic reviews. The Oxford Database of Perinatal Trials was first published on floppy discs in 1992.(Starr et al., 2009) It became the Cochrane Pregnancy and Childbirth Database in 1993 and was used as a pilot for the Cochrane Database of Systematic Reviews. Issue 2, published in 1994 included 615 reviews. Hundreds of systematic reviews are published every week now. (Hoffmann et al., 2021, Bastian et al., 2010) There are about 9500 Cochrane reviews, and over 500,000 systematic reviews in the Epistemonikos database. (Epistemonikos, 2023)

However, most reviews are not systematic, and many systematic reviews have serious flaws. (Page et al., 2016, Ioannidis, 2016) While there have been slow improvements, there is a need for ongoing collaboration among organisations such as the Cochrane Collaboration, the Campbell Collaboration, and International Collaboration for the Automation of Systematic Reviews to improve the efficient and effective production of systematic reviews.

Automation tools have the potential to improve the speed and efficiency of systematic review production, their accuracy, and to keep them up to date. (Clark et al., 2020) However, caution is needed as evaluations show mixed results with the potential to decrease, as well as increase, accuracy and only for some review steps. (Clark et al., 2025) Those tools that have been shown to be helpful remain underused. Improved and automated review methods can also help to improve the quality of reviews. But the methods used in systematic reviews have become increasingly complex – sometimes too complex. (Guyatt et al., 2025a) To paraphrase a quote attributed to Albert Einstein: “The methods used in systematic reviews should be made as simple as possible, but not simpler.”

Ineffective and inefficient peer review

Journals rely on peer review to ensure the quality of the research they publish, and decisions about which research proposals are funded rely heavily on peer review. (Demicheli and Di Pietrantonj, 2007, Guthrie et al., 2018, Shepherd et al., 2018) However, peer review is highly variable, inconsistent, and often flawed. (Smith, 2006, Smith, 2010, Guthrie et al., 2018, Tamblyn et al., 2018, Abdoul et al., 2012) There is little evidence of the effects of peer review on the quality of published research evidence, (Jefferson et al., 2007, Bruce et al., 2016) and very little evidence on the effects of peer review of proposals for research funding. (Demicheli and Di Pietrantonj, 2007, Guthrie et al., 2018) For the most part it is done by volunteers, few of whom have formal training, and they commonly miss major errors. For example, the British Medical Journal (BMJ) sent three papers, each of which had nine major methodological errors inserted, to about 600 peer reviewers. (Schroter et al., 2008, Bruce et al., 2016) On average, the peer reviewers detected about one-third of the errors in each paper. In addition, peer reviewers fail to detect or comment on spin. (Lazarus et al., 2015)

Peer-reviewed journals generally do not have explicit or standardized criteria for review of manuscripts; instead, they rely on the judgment of editors and reviewers. This leaves the process open both to arbitrariness and to systematic bias. Reviewers’ agreement on the merit of submitted manuscripts is poor, and manuscripts that report positive (“statistically significant”) findings are more likely to be accepted. (Cook et al., 1993, Thornton and Lee, 2000, Hopewell et al., 2009)

A systematic review published in 2007 found only 28 comparative studies of the effects of processes in editorial peer review. (Jefferson et al., 2007) The same year another systematic review found only 10 comparative studies of the effects of grant giving peer review processes. (Demicheli and Di Pietrantonj, 2007) A more recent review found 83 studies of innovations to improve the effectiveness and efficiency of peer review of health research funding. The studies had important limitations, but many innovations appear promising and warrant further evaluation.(Shepherd et al., 2018)

A systematic review of peer reviewer training to improve grant or journal peer review found 10 randomised trials. (Hesselberg et al., 2023) The results suggest training peer reviewers may lead to little or no improvement in the quality of peer review. Another review of the use of blinding to reduce bias in peer review of grant applications found only three comparative studies. (Qussini et al., 2023)

Strategies that work are needed to improve the effectiveness and efficiency of peer review.

Failure to keep reviews up to date

A landmark study by Eliot Antman and colleagues, published in 1992, compared the results of cumulative meta-analyses of treatments for myocardial infarction with the recommendations of clinical experts writing (unsystematic) review articles and textbook chapters. (Antman et al., 1992) They showed that research had continued long after robust estimates of treatment effects had accumulated, and that recommendations had overlooked strong, existing evidence from randomized trials, both of beneficial and of lethal effects of treatments.

An analysis of 50 reports including over 1500 cumulative meta-analyses of health intervention studies was published 22 years later, in 2014. (Clarke et al., 2014) This analysis showed “that, had researchers assessed systematically what was already known, some beneficial and harmful effects of treatments could have been identified earlier and might have prevented the conduct of the new trials. This would have led to the earlier uptake of effective health and social care interventions in practice, less exposure of trial participants to less effective treatments, and reduced waste resulting from unjustified research.”

In addition to updating systematic reviews to take account of new evidence, reviews should be updated to respond to valid criticisms, to correct mistakes, and to take account of improvements in the methods used to summarise what is known based on the best available evidence.

Keeping reviews up to date is challenging. Rabia Bashir and colleagues examined the timing of updates of Cochrane reviews published in 2010 relative to the availability of new evidence. (Bashir et al., 2018) They found that only 43% of those reviews were updated before June 2017. Very few updates led to a changed conclusion.

In another study, Hilda Bastian and colleagues charted updating of a cohort of Cochrane reviews from 2003 to 2018. (Bastian et al., 2019) They found that most Cochrane reviews were updated, but they were becoming more out of date over time. Updates led to more included studies, although they rarely led to major changes in conclusions.

Prior to those two studies, the Cochrane Collaboration modified its policy from updating when new evidence becomes available, to updating every two years, to updating based on need. (Garner et al., 2016) Earlier studies examining the timing of systematic review updates had found that around a third were updated within two years and that the median update time was more than five years. (Bashir et al., 2018)

Tanja Rombey and colleagues charted characteristics of a random sample of 100 non-Cochrane updates of systematic reviews published in 2016-2017. (Rombey et al., 2020) The updates were published a median of five years after the previous systematic review. 31/100 updates reported that the conclusion had changed. Kaveh Shojania and colleagues assessed when 100 high-quality systematic reviews published between 1995 and 2005 went out of date. (Shojania et al., 2007) The included reviews had been assessed by ACP Journal Club to be of high quality and directly relevant to clinical practice. This study found that signals for updating occurred frequently and within a relatively short time for these systematic reviews.

“Living systematic reviews” are continually updated analyses, incorporating relevant new evidence as it becomes available. (Elliott et al., 2017) Although this has been described as a novel approach to updating systematic reviews, it is like the approach that was used by The Oxford Database of Perinatal Trials, which became the Cochrane Pregnancy and Childbirth Database and was the pilot for the Cochrane Database of Systematic Reviews.(Chalmers et al., 1986, Chalmers, 1993, Starr et al., 2009) What’s more recent is identifying a subset of reviews for which this approach is appropriate and the use of automation to assist with some systematic review tasks. These include searching, eligibility assessment, identification and retrieval of full-text reports, extraction of data, and risk of bias assessment. (Thomas et al., 2017)

The evolution of “living guideline recommendations” is linked to that of “living systematic reviews”. Studies have documented how quickly practice recommendations need updating. (Shekelle et al., 2001, Neuman et al., 2014, Martínez García et al., 2014) Collaboration between “living systematic reviews” and “living guideline” teams is needed, (Akl et al., 2017) and digital tools can facilitate keeping practice recommendations up to date. (Martínez García et al., 2017, Vandvik et al., 2013, Tendal et al., 2021)

Unnecessary duplication of effort

Plans for randomised trials should use systematic reviews to avoid conducting new trials that are unnecessary (and thus probably unethical). (Chalmers et al., 2014, Lund et al., 2016) Cumulative meta-analyses have shown that new trials might have been recognised as unjustified had a systematic review informed plans for new trials. (Antman et al., 1992, Clarke et al., 2014)

A review of reports of 1,523 trials published between 1963 and 2004 found that fewer than 25% of preceding trials were cited. (Robinson and Goodman, 2011) Another review of 175 reports of trials in five high-profile general medical journals between 1997 and 2022 investigated the use of a systematic review of preceding trials to inform the design of the trial and an updated systematic review to put the new findings in context. (Clarke et al., 2024) The Introduction sections of only 3% (5/175) of these reports contained references to up-to-date systematic reviews to inform the design of the new trial.

Many published systematic reviews duplicate other reviews on the same topic without adding anything important. (Siontis et al., 2013, Naudet et al., 2017, Ioannidis, 2016) In addition, some duplicate reviews are discordant and confusing. (Jadad et al., 1997) Some intentional replication of systematic reviews by different teams might be useful. However, for many topics, duplication is a waste of resources.

Pressure on academics to publish contributes to unnecessary duplication. Many institutions base promotion and tenure on publication quantity, rather than quality. When income and professional advancement depends on publication output, systematic review authors may choose to publish a review, even if it duplicates another review unnecessarily.

To help avoid unintended and unnecessary duplication of systematic reviews, review authors should register protocols for their reviews, and they should search for other systematic reviews and review protocols on the same topic before undertaking a new review. (Stewart et al., 2012)

Misinformation

Mass media are a source of health information for many people. Researchers have studied and criticised the quality of health information in the mass media for at least five decades. A systematic review of 44 studies of the quality of health news found that many news reports gave an unbalanced and oversimplified picture of the potential consequences of health interventions. (Oxman et al., 2022b)

Over the past three decades, people have been increasingly using the internet and social media to seek and share health information. (Swire-Thompson and Lazer, 2020, Suarez-Lledo and Alvarez-Galvez, 2021) Several systematic reviews have found high prevalences of poor-quality online health information and misinformation. (Swire-Thompson and Lazer, 2020, Suarez-Lledo and Alvarez-Galvez, 2021, Eysenbach et al., 2002, Roundtree et al., 2009, Borges do Nascimento et al., 2022)

The creation and dissemination of trustworthy information about the effects of health interventions can help to mitigate the adverse effects of misinformation. (Oxman et al., 2020, Oxman and Paulsen, 2019, Swire-Thompson and Lazer, 2020) Fact checking and automated detection of health misinformation could also help. However, there is limited evidence of the effects of these and other interventions. (Swire-Thompson and Lazer, 2020, Smith et al., 2023, Zhang et al., 2024, Schlicht et al., 2023) Artificial Intelligence (AI) could potentially help to reduce inequalities in access to evidence-based health information by facilitating equitable access to trustworthy information. (Meyrowitsch et al., 2023, Beam and Darmono, 2024) However, AI can also generate and worsen the spread of misinformation.

Eliminating misinformation is a worthy but Sisyphean task. In addition, teaching people to think critically about health claims and choices (including when and where to find trustworthy information) is essential.

Limited access to trustworthy information about the effects of policies and practices

To make informed decisions about interventions, patients and the public, health professionals and policymakers need information about effects based on the best available research evidence. (Oxman et al., 2020)  Those communicating evidence-based information about the effects of health interventions should make it easy for readers to quickly assess the relevance of the information. For each important outcome, they should help their target audience to understand the size of the beneficial and harmful effects and how sure we can be about those; presented in ways that avoid misleading. They should help their target audience to put information about the effects of interventions in context and to understand why the information is trustworthy.

There is an abundance of health information on the internet, but it is hard to find trustworthy information that is explicitly based on systematic reviews. (Oxman and Paulsen, 2019, Glenton et al., 2005) Patients and the public are unlikely to critically appraise the information that they find, and most are unlikely to understand key concepts people need to understand to assess claims about the effects of health care. (Dahlgren et al., 2021)

Clinicians frequently have questions about the care of patients in their practice. (Del Fiol et al., 2014) Roughly half of the questions are never pursued. This picture has been stable over time despite the broad availability of online evidence resources that can answer these questions. It may be difficult for health professionals to find the best available evidence due to time constraints, lack of access to user-friendly, up-to-date, evidence-based resources, or lack the skills needed to find and appraise up-to-date, evidence-based recommendations or systematic reviews.

Evidence-based textbooks that are kept up to date can help. However, the costs of these resources are passed along to the consumer, limiting access for many health professionals. In addition, a plethora of cheap, low-quality imitations may further limit access. The label “evidence-based” is sometimes applied to resources that simply reference the medical literature (and may not even do that), regardless how old or unsystematic they may be.

Gaps between evidence-based recommendations and professional practice

Underuse of effective interventions and overuse of interventions that are more likely to cause harm than good are common. (Brownlee et al., 2017, Glasziou et al., 2017) The size of gaps between evidence-based recommendations and clinical practice varies widely. (Squires et al., 2022, Schuster et al., 2005, Seddon et al., 2001, McGlynn et al., 2003, Grol, 2001) For example, a systematic review of inappropriate practice in Canada found a median proportion of inappropriate care of 30%. Underuse was more frequent (median 44%) than overuse (median 14%). (Squires et al., 2022) A review of studies in the U.S. found that about 50% of people received recommended preventive care. (Schuster et al., 2005) None of the studies reported a percentage of people receiving contraindicated preventive care. An average of 70% of patients received recommended acute care, and 30% received contraindicated acute care. For chronic conditions, 60% received recommended care and 20% received contraindicated care. A review of studies of quality of care in general practice in the UK, Australia, and New Zealand found that in almost all the included studies care did not attain the standards set out in national guidelines or by researchers. (Seddon et al., 2001)

A systematic review of frameworks and taxonomies of factors that prevent or enable improvements in professional practice found 57 potential determinants of practice grouped in seven domains: guideline factors, individual health professional factors, patient factors, professional interactions, incentives and resources, capacity for organisational change, and social, political, and legal factors. (Flottorp et al., 2013)

An overview of systematic reviews of the effectiveness of interventions for implementing evidence-based clinical practice guidelines found that: (Fretheim et al., 2015)

• Implementation interventions such as electronic decision-support, educational meetings, outreach visits, audit and feedback, and tailored interventions are probably effective, but:
  • The size of the effect varies.
  • The effect on clinical practice is most often moderate.
  • The expected effect on health outcomes is modest.
• For other interventions, the size of the effect varied considerably across studies. It is difficult to explain this variation. Consequently, it is uncertain how much these interventions will improve adherence to clinical guidelines.
• For some measures, such as financial incentives and public release of performance data, evidence is lacking or scarce. We therefore cannot say how effective these types of interventions are.

Over the past three decades, increased attention has been given to the need to narrow the gap between research and practice, (Balas and Boren, 2000, Morris et al., 2011, Djulbegovic and Guyatt, 2017) but important gaps remain. Many initiatives to close gaps are narrowly focused and short term. Hence, ongoing collaborative efforts are needed to continually reduce those gaps. This includes addressing barriers to improvements and evaluating the effects of implementation interventions.

Inadequately informed policymaking

Substantial sums of money are invested each year in public programmes and policies. For most of these, little is known about their effects, including whether public programmes and policies can fulfil their primary objectives. Furthermore, what is known is often not used to inform policy decisions. (Kebebe et al., 2012) Because public resources are limited, it is important to use them effectively, efficiently, and equitably.

When making decisions about public programmes, good intentions and plausible theories are not enough. Research evidence, values, political considerations, and judgements are all needed for well-informed decisions. However, decisions are often made without systematic or transparent access and appraisal of relevant research evidence and without an adequate evaluation of intended and unintended effects of policies or programmes.

Evidence-informed policymaking aims to ensure that decision making is well-informed by the best available research evidence. (Oxman et al., 2009) It is characterised by systematic and transparent access to, and appraisal of, evidence as an input into the policymaking process. While the overall process of policymaking may not be systematic and transparent, within  that, systematic processes should be used to ensure that relevant research is identified, appraised, and used appropriately. To ensure that others can examine what research evidence was used to inform policy decisions, as well as the judgements made about the evidence and its implications, the processes should be transparent.

In 2005, John Lavis and colleagues conducted a survey of organisations engaged in supporting evidence-informed policy making. (Lavis et al., 2008) They identified many health technology assessment agencies and clinical practice guideline developers. However, the survey noted few examples of organisations that support the use of research evidence for decisions about health policies or programmes. There have been several initiatives to promote evaluation of the impacts of policies. (Oxman et al., 2010) However, a review of laws and policies requiring routine evaluation of public policies or programmes found only five examples. (Kebebe et al., 2012) None required routine use of systematic reviews of relevant research before launching new programmes or focused specifically on impact evaluations.

Evidence-based policy briefs and policy dialogues and decision frameworks can help to ensure appropriate use of research evidence. (Moberg et al., 2011, Moberg et al., 2018) But these are rarely used in practice. Fair processes for making policy decisions can ensure that decisions are made in a way that everyone affected can see as fair, even if they don’t agree with the results. Four key principles underly fair processes: reporting, reasonableness, revision, and regulation. (Daniels and Sabin, 1997) In a fair process, the reasons for a decision are transparent and clearly reported. There are no secret deals or hidden motives. The reasons and evidence used to justify a decision must be reasonable. That is, they must adhere to agreed-upon principles. If someone feels a decision is mistaken or unfair, there should be a way to challenge it, and the process should allow for revision if better reasons or evidence come up. To ensure that the process is fair, it must be regulated, and the regulations must be enforced.

Increased attention has been given to the need to evaluate the impacts of health policies and for using systematic reviews to inform health policy. (World Health Organization, 2005, Hanney and González-Block, 2009, World Health Organization, 2021, World Health Organization, 2022a, World Health Organization, 2022b, Beane et al., 24 October 2008) However, few governments have in place fair processes that ensure that policy decisions are informed by systematic reviews of relevant research. And few governments have processes that ensure that the effects of implementing policies are evaluated when there are important uncertainties.

Lack of patient and public participation in decision-making

Funders’ decisions about research priorities for evaluating the effects of interventions should reflect the needs of the end users: patients, carers, and clinicians. There are marked differences between the types of treatments prioritised for evaluation by patients, carers, and clinicians and those currently being evaluated by researchers. (Crowe et al., 2015) Inclusive participation of patients and clinicians in priority setting processes could help to address this. (The James Lind Alliance)

Patients’ decisions about their care should allow involvement to the extent they want. A reason for this is that people vary greatly in the importance they attribute to outcomes, (MacLean et al., 2012, Oxman et al.) and patients may value the pros and cons of an intervention differently than their physician. (Alonso-Coello et al., 2015) Most patients prefer sharing decisions with their doctors. (Chewning et al., 2012) Yet, few health-care providers consistently attempt to facilitate patient involvement, and even fewer adjust care to patient preferences. (Couët et al., 2015)

Decision aids can help, but few decision aids are currently used in clinical practice. (Stacey et al., 2024)

Involving patients and the public in the development of clinical guidance has increased in recent decades. But there is little research to inform decisions about how to do this. (Schunemann et al., 2006, Rashid et al., 2017, Bryant et al., 2022)

The International Conference on Primary Health Care, meeting in Alma-Ata in 1978, declared that people have the right and duty to participate individually and collectively in the planning and implementation of their health care. (World Health Organization, 1978) In addition to being a democratic right, inclusive participation in deliberative and decision-making processes has the potential to improve the quality of the judgements and decisions that are made, build trust, improve adherence, and help to ensure transparency and accountability. (Norheim et al., 2021) Engaging members of the public in policymaking can help to ensure that: (Moberg et al., 2011)

• Their concerns are heard and considered
• Problems are analysed, described, and perceived correctly
• Appropriate solutions are identified
• Important barriers to implementing solutions are considered
• Effective implementation strategies are identified
• Appropriate values are used when weighing the pros and cons of options
• Policy decisions are appropriate, understood, and acceptable
• Marginalised communities are included in decision making

Public engagement may also help to ensure that important uncertainties are identified and addressed. (Tan et al., 2022) However, participation without clear objectives may anger participants and fail to benefit the policymaking process or outcomes. Poorly planned and implemented participation can create mistrust, waste people’s time, and undermine future attempts at to engage the public. (Involve, 2005)

Recognition of the importance of patient and public participation in research, systematic reviews, clinical practice guidelines, clinical decision-making, and health policymaking has grown in recent decades. However, participation is limited in practice and policy and there is little evidence of the effects of strategies to facilitate participation. (Brett et al., 2014, Concannon et al., 2014, Domecq et al., 2014, Pollock et al., 2018, Nilsen et al., 2006, Wiles et al., 2022, Abelson et al., 2003, Conklin et al., 2015, Lloyd et al., 2021, Baumann et al., 2022, Aguilera et al., 2024, Munthe-Kaas et al., 2024)

The way forward

Addressing the multifaceted challenges in medicine, public health, research, and science increasingly demands the pooling of diverse expertise, resources, and perspectives. There has been increasing recognition of the power of focused collaborative endeavours, such as the Human Genome Project, the Cochrane and Campbell Collaborations, collaborations fostered by the Oxford Clinical Trials Service Unit, and other collaborations referred to in this commentary. (Wang and Barabási, 2021) An analysis of nearly 20 million research articles and over two million patents found a clear global trend towards team science across all scientific disciplines. (Wuchty et al., 2007) Teams now also produce the exceptionally high-impact research, even where that distinction was once the domain of solo authors. Since the foundation of the World Health Organization in 1948, the world has experienced public health challenges that have required international collaborations to improve health around the world. (World Health Organization, 2023)

Although there have been dramatic improvements in the design, implementation, and use of health research evidence to inform policy and practice, there are still problems that need to be addressed. In Table 1, we suggest key challenges that need to be addressed by research funders, publishers, academic institutions, healthcare providers, governments, and collaborations that aim to promote informed health choices. More specific recommendations that overlap with these suggestions have been made elsewhere. (Chalmers and Glasziou, 2009, Chalmers et al., 2014, Ioannidis et al., 2014, Al-Shahi Salman et al., 2014, Chan et al., 2014, Glasziou et al., 2014, Elshaug et al., 2017, Bradley et al., 2020)

Table 1. Key challenges for research funders, publishers, academic institutions, healthcare providers, governments, and collaborations that aim to promote informed health choices

In Figure 2, we suggest a framework, linking together the way forward and the challenges we have discussed in this commentary with using evaluations of the effects of interventions to inform health choices.

Conclusion

We have come a long way over the past five decades, but we still have a long way to go. Ongoing collaborative efforts are needed to ensure that the effects of interventions are evaluated when there are important uncertainties about their effects. Efforts are also needed to ensure that systematic reviews of the effects of interventions are used to inform policy and practice. Questioning authority and thinking critically about the basis for opinions has been at the heart of the progress we have made and is essential for continued progress. This includes questioning the basis for the opinions we have expressed in this commentary.

There is joy as well as power in collaboration. To paraphrase what Dave Sackett wrote in the preface to the first edition of the Cochrane Collaboration Handbook, (Sackett et al., 1994) the emphasis on collaboration is not simply a sentimental comment on the ‘generosity of spirit’ of those who become involved (although this spirit certainly makes collaboration a pleasure). The shared will to collaborate is a precondition for efficient improvement in the use of research to inform policy and practice and ensuring that interventions are likely to do more good than harm.

In summary, “good intentions and plausible theories are insufficient for selecting policies and practices for protecting, promoting and restoring health. Humility and uncertainty are preconditions for unbiased assessments of the effects of the prescriptions and proscriptions of policy makers and practitioners for other people. The interests of the public will be served more responsibly and ethically when research designed to reduce the likelihood that we will be misled by bias and the play of chance has become an expected element of professional and policy making practice, not an optional add-on.” (Chalmers, 2003b)

Acknowledgements

We want to thank Curt Furberg for permission to use the cartoons from [All that glitters is not gold – II: How to evaluate clinical studies], (Furberg and Furberg, 2001) and Google Gemini for assistance preparing the text for the cartoons. We are grateful to the following people for their comments on an earlier (and much worse) draft of this commentary: Lisa Bero, Alan Cassels, Mike Clarke, Kay Dickersin, Jeremy Grimshaw, Gordon Guyatt, Sally Hopewell, France Légaré, David Naylor, and Jimmy Volmink. None of them should be held responsible for whatever we got wrong or left out after taking account of their feedback. We are especially grateful to Jan Chalmers for her assistance and tolerance.

 

Appendix 1. How we came to question authority and celebrate collaboration and how this may have affected this commentary

After working for a couple of years (1969-1970) in a Palestinian refugee camp in Gaza, Iain Chalmers returned to Britain, to train in obstetrics, in Cardiff. As a junior obstetrician he was confused by the conflicting opinions of senior doctors about when and how to intervene in pregnancy and childbirth. In 1972, a very readable little book came to his rescue. Effectiveness and efficiency: Random Reflections on Health Services had been written by Archie Cochrane. (Chalmers, 2003a) Archie’s book and his subsequent friendship made Iain a lifelong sceptic about therapeutic claims unsupported by reliable evidence. It helped him to understand why some forms of research – particularly randomised trials – were likely to generate more reliable information than others. In addition, Iain could identify strongly with Archie’s commitment to the decent principle of equitable access to (effective) health care, and his emphasis on the need to provide humane and dignified (and thus effective) care when no effective cure was available.

After discussions with one of Archie’s young colleagues – David Bainton – Iain decided to go with David to the London School of Hygiene and Tropical Medicine and the London School of Economics to learn more about evaluating health care. Iain and David attended a master’s degree course in social medicine. The first year of the course, which was the creation of Jerry Morris, was the most stimulating twelve months of formal education Iain had experienced, partly because students were actively encouraged to challenge authority!

In 1974, Iain started to search systematically for reports of controlled trials, with encouragement from Archie Cochrane and the help of a few generous volunteers. A few years later, the Maternal and Child Health Division of WHO provided funds to support this work. (McPherson et al., 1979) During this time, while working on a chapter on causal inference in obstetric practice, Iain outlined a plan to address uncertainties about the effects of interventions in systematic reviews of controlled trials in pregnancy, childbirth, and early infancy. The plan suggested that it would be worth exploring whether the results of similar trials could be combined to yield statistically more robust estimates of treatment effects. In 1978, an institutional base to develop these ideas became available when Iain was asked by the Department of Health to establish the National Perinatal Epidemiology Unit in Oxford.

At that time, there was disagreement about which aspects of the management of labour might predispose to worrying fetal distress. Using the register of trials, Iain and his colleagues identified and analysed together just over 2000 infant-mother pairs who had participated in one of four RCTs. The comparison groups were (i) intermittent auscultation, (ii) continuous heart rate monitoring, and (iii) continuous heart rate monitoring with fetal scalp blood sampling. None of the babies assigned to the third, most intensive form of intrapartum care developed neonatal seizures. None of the 13 babies who had had seizures had been assigned to the acid-base assessment group – a pattern that was unlikely to reflect chance (P<.05). This finding could not have been made without the trialists agreeing to pool their data in a systematic review aggregating and analysing the four data sets together.

The published report of this systematic review (Chalmers, 1979) concluded that there were hints that continuous fetal heart rate monitoring, when supplemented with adequate facilities for assessing fetal acid-base status and applied to high-risk cases, may be associated with a reduced risk both of caesarean section and of neonatal convulsions and cerebral irritability. Encouraged by this tentative evidence a much larger replication to which over 10,000 mother-baby pairs at the National Maternity Hospital, Dublin, had contributed. This confirmed the protective effect of fetal heart rate monitoring accompanied by acid-base assessments.

Iain’s experience producing that first systematic review drove home for him the necessity of collaboration to determine when interventions do more good than harm.

Andy Oxman is nine years younger than Iain but learned the importance of questioning authority and the necessity of collaboration at a younger age. He turned 18 in 1970. At that time the U.S. was waging war in Vietnam, and 18-year-old young men had to register in the military draft. Andy protested the war and chose not to register. When he graduated from high school, he joined Institute Mountain West in Golden, Colorado in 1970 (a branch of the Institute for the Study of Nonviolence in Palo Alto, California). The Institute organised workshops on various topics with readings and discussion, organised protests, and worked in the community. This experience drove home for Andy the necessity of collaboration, but he was naïve with respect to determining when interventions do more good than harm.

In 1973, Andy and two friends from the Institute planned to travel and learn about how to change the world. In deciding where to go, they would ask themselves what they could learn from going there, and what did they have to offer. The latter question led all three to seek professional training when they returned to the U.S after being in Chile during the coup in September 1973. Before going to Chile, they had visited a rural community health centre in New Mexico, which was making improvements in the community it served that went beyond providing health care. That impressed Andy, who became interested in community oriented primary care (COPC). COPC focuses on addressing the health needs of a community by integrating public health principles with primary care, emphasizing community participation and collaboration to improve overall health outcomes.

After graduating from medical school in 1979, Andy moved to Norway, where Trine, his wife, was a medical student, and he worked as a doctor there until 1984. The last two years he worked as a general practitioner (GP) in a small town in northern Norway. He was frequently uncertain about when and how to intervene, and whether he was doing more good than harm. He still was interested in COPC and wanted to learn more epidemiology. After exploring some options, Andy and Trine moved to Canada, where he trained in community medicine at McMaster University. The community medicine training included a master’s in design, measurement, and evaluation (DME). The DME programme was started by David Sackett in 1972.  It was targeted at health professionals and empowered them to critically appraise research evidence and to conduct methodologically sound studies. Andy’s experience of the DME program was like Iain’s experience of the master’s degree course in social medicine.

When Andy chose a thesis topic, he was inspired by a book written by two social scientists, Richard Light and David Pillemer. (Light and Pillemer, 1984) In their book they eloquently summarised the state of the art of “the science of reviewing research”. His thesis (Oxman, 1987) was the basis for “guidelines for reading literature reviews”, which Andy and Gord Guyatt published in 1988. (Oxman and Guyatt, 1988) At that time the term “systematic review” was not being used and there were not many meta-analyses published in health care. Andy’s thesis was also the basis for an elective DME course on how to conduct systematic reviews. Andy and Gord first offered the course in 1987, and Iain Chalmers and Murray Enkin were students. At that time, they had already prepared (but not yet published) around 600 meta-analyses based on more than 3500 reports of controlled trials included in the register of trials in perinatal medicine that they had systematically identified and assembled. (Antes and Oxman, 2001, Starr et al., 2009)

Five years later, in 1992, with funding from the newly established British National Health Service’s Research and Development Programme, Iain established a ‘Cochrane Centre’ to facilitate the preparation and maintenance of systematic reviews of randomised trials of health care. This was based on Cochrane’s call for systematic, up-to-date reviews of all relevant randomised trials of health care. The collection of systematic reviews of randomised trials of care during pregnancy and childbirth that Iain and his colleagues produced and maintained was a model that people around the world wanted to emulate for other areas of health care.

A year later, in 1993, the UK Cochrane Centre organised the first ‘Cochrane Colloquium’ and the Cochrane Collaboration was established. (Chalmers, 1993, Chalmers, 2003a, Antes and Oxman, 2001, Cassels, 2015) Dave Sackett was the first chair of the Steering Group and, together with Andy, edited the first edition of the Cochrane Handbook, which provided guidance for conducting systematic reviews of the effects of interventions. (Oxman et al., 1994)

Our experience, especially our involvement in the Cochrane Collaboration, has influenced which challenges we have considered in this commentary and our reflections on the progress that has been made over the past five decades. We have also been involved in some other collaborations that we have mentioned or referenced. It has been more than a decade since we both retired from leadership roles in the Cochrane Collaboration, we are fully retired now, and we are losing our marbles. So, we are not on top of what has happened more recently.

Paul Glasziou retired while we were writing this commentary. He is still pursuing his research priorities and is more on top of recent developments than Iain and Andy. In 1979 as a young intern Paul was often puzzled about the medical decision-making process. He knew a lot of pathophysiology and lists of differential diagnosis, but beyond a few aphorisms had learned a minimal framework for decision making. Then at the local medical book shop’s “for sale” bin he stumbled on a copy of Rational Diagnosis and Treatment by Henrik Wulff – a Danish physician. The book discussed the – for him – completely foreign concepts of probability, chance, and Bayes theorem. It also had an Appendix on randomised trials. It was exactly the kind of medical textbook he’d been looking for.

Paul later commenced a PhD in medical decision making to learn more about decision-making frameworks. The concepts of “treatment and test thresholds” helped clarify the intrinsic uncertainties of many difficult medical decisions, such as when to operate on an acute abdomen or how often to screen. But this recognition of uncertainty also created a second puzzle: where do we get reliable probabilities from? Often published decision analyses chose a single study when there were several studies. Was there a good way to choose one (critical appraisal) or combine them (meta-analysis)?

As a full-time researcher, Paul addressed questions about decision making, clinical trials, and systematic reviews. After some years, he had the good fortune to have Dave Sackett – one of the founders of evidence-based medicine – visit the University of Sydney for a “clinical epidemiology” workshop in which Paul tutored. Dave inspired him to go back into clinical work. He initially did Friday evenings in an emergency department. A few years later he commenced training as a GP. His central goal was to work out how to better use existing research to make better decisions for and with patients.

That idea sounds simple but – given the massive and messy medical literature – turned out to be a giant tangle to be unpicked. In tackling that problem he was very fortunate to be able to collaborate with some of the leading thinkers in the world of evidence-based medicine and systematic reviews. At the time, colleagues in general practice were not that interested in evidence-based medicine. Rather, they usually depended upon specialists to sort out the medical literature and decision making for them. This was a perspective the anti-authoritarian Paul saw as misguided. Fortunately, over the decades GP views have changed.

In seven years as the director of the Centre for Evidence-Based Medicine in Oxford, he was very fortunate to be able to actively pursue challenges to evidence-based medicine with an outstanding and diverse group of colleagues in multiple disciplines. One of those colleagues was Iain Chalmers. Their regular discussions and projects led him to understand that many of the barriers in translating research into practice were deficiencies in the research not the practitioners. Those deficiencies had to be addressed as prelude to the practice of evidence-based medicine. For example, as a co-editor of the Journal of Evidence-Based Medicine, he found that the articles he considered “practice changing” sometimes didn’t change his practice! This was particularly true for non-drug interventions. One key problem was that the interventions often were so poorly described that it was impossible to use them in the clinic. Recognition of this led him to Co-develop the TIDieR statement – for completely reporting interventions, (Hoffmann et al., 2014) and to found the Handbook of Non Drug Interventions (HANDI). (Royal Australian College of General Practitioners, 2025)

Paul’s concern with the poor description of interventions and Iain Chalmers’ concern with questions being asked by research that were not relevant to clinicians and patients, plus their joint concerns about poor study design and non-publication led them to estimate that 85% of research was avoidably wasted. (Chalmers and Glasziou, 2009) Hence, to improve the translation of research into practice, addressing avoidable waste is a necessary prelude to addressing the research-practice gap. (Macleod et al., 2014) Many of the elements in this paper are about those concerns.

We hope that our reflections can provide some help to a few of the many people on the long and winding road towards promoting evaluation of the effects of health interventions to inform health choices.

Appendix 2. Illustrative examples of recognition of the problems we describe and potential solutions for those problems beginning in the 1970’s*

*These overlapping timelines are by no means complete but illustrate some key examples of recognition of problems and efforts to address them.

Evaluations of the effects of health interventions
1972	Archie Cochrane published Effectiveness and Efficiency: Random Reflections on Health Services, in which he set out clearly the importance of using randomised trials to inform decisions about which health interventions should be paid for by the UK National Health Service.
1978	Richard Peto showed that to be useful, trials might well need thousands of participants and that very large trials must be kept simple so that the workload per participant is kept to a minimum.
1980	Using drug research done during the late 1970s, Elina Hemminki showed that studies of new drugs submitted to licensing authorities were less likely to be published subsequently if they had looked for adverse effects.
1986	The International Study of Infarct Survival  (ISIS) Collaborative Group published the report of a large (16,027 patients) simple randomised trial evaluating the effects of giving intravenous atenolol to patients with an acute heart attack, and the report interpreted the results in the context of a systematic review.
1992	Kay Dickersin and colleagues published the first of several studies which provided the rationale for her lifelong commitment to reducing biased under-reporting of research.
1986	John Simes proposed international registration of all clinical trials after he showed that conclusions about treatments for ovarian cancer differed depending on whether the results of unpublished trials had been considered.
2000	The US National Institutes of Health launched ClinicalTrials.gov.
2003	Joel Lexchin and colleagues published a systematic review of the effects of pharmaceutical industry sponsorship on research results and quality. Subsequently, multiple systematic reviews have documented how conflicts of interest can lead to bias.
2004	By comparing trial protocols with trial reports, An-Wen Chan and his colleagues, demonstrated frequent, biased switching of primary outcomes based on whether differences in outcomes were statistically significant.
2004	The International Committee of Medical Journal Editors (ICMJE) adopted a policy requiring, as a condition of consideration for publication, registration of trials in a trial registry.
2006	WHO launched a global network of clinical trial registers as the first step towards establishing a web-based search platform where members of the public can obtain full and detailed information about evaluations of the effects of health interventions. There were at least 50 trial registers at that time.
2013	The original Standard Protocol Items: Recommendations for Interventional Trials  (SPIRIT) statement for reporting protocols for randomised trials was published.
2013	The AllTrials campaign was formed as an International initiative of Ben Goldacre, BMJ, Centre for Evidence-based Medicine, Cochrane Collaboration, James Lind Initiative, PLOS, and Sense about Science. The campaign aims to achieve a situation globally where all trials are registered and results reported.
2014	Malcolm Macleod and colleagues published a series of papers in The Lancet on avoidable waste in health research.
2018	Alice Fabbri and colleagues published a review of studies that explored the influence of industry sponsorship on research agendas, showing that industry tends to prioritize lines of inquiry that focus on products, processes, or activities that can be commercialized and away from questions that are the most relevant for public health.
Summaries of evaluations
1976	Gene Glass, in his presidential address to the American Educational Research Association, introduced the term ‘meta-analysis’ to denote statistical synthesis of the results of similar studies.
1978	Gregg Jackson identified major shortcomings in randomly selected review articles published in prominent social science journals.
1979	In a contribution to a book published by the UK Office of Health Economics, Archie Cochrane wrote: “It is surely a great criticism of our profession that we have not organised a critical summary, by speciality or subspeciality, adapted periodically, of all relevant [randomised trials].” (Cochrane, 1979)
1984	Richard Light and David Pillemer published Summing up: The Science of Reviewing Research, in which they described shortcomings of unsystematic ‘literature reviews’ and procedures that review authors should follow in preparing reviews.
1987	Cynthia Mulrow published an analysis of the quality of review articles in major general medical journals, showing that most review articles had ignored basic scientific principles.
1988	The first of several systematic reviews using individual patient data coordinated by the Clinical Trial Service Unit  were published by the Antiplatelet Trialists’ Collaboration and the Early Breast Cancer Trialists’ Collaborative Group.
1989	In a forward to Effective Care in Pregnancy and Childbirth, a textbook based on the Oxford Database of Perinatal Trials (ODPT), Archie Cochrane referred to ODPT, as “a real milestone in the history of randomised trials and in the evaluation of care” and expressed the hope that it would be widely copied by other medical specialties.
1992	Eliot Antman and colleagues compared the results of cumulative meta-analyses of treatments for myocardial infarction with the recommendations of clinical experts writing (unsystematic) review articles and textbook chapters. They showed that research had continued long after robust estimates of treatment effects had accumulated, and that recommendations had overlooked strong, existing evidence from randomised trials, both of beneficial and of lethal effects of treatments.
1993	Inspired by ODPT, the Cochrane Collaboration was established to address Archie Cochrane’s challenge by preparing, maintaining, and disseminating up-to-date systematic reviews of the effects of health interventions. As its name implied, the Cochrane Collaboration was based on the shared will to collaborate which existed among those who contributed to it. As Dave Sackett wrote in the first edition of the Cochrane Collaboration Handbook: The emphasis on collaboration was not simply a sentimental comment on the ‘generosity of spirit’ of those who became involved in building the Collaboration (although this spirit certainly made collaboration a pleasure). This shared will to collaborate was a precondition for meeting the Collaboration’s aims. (Sackett et al., 1994)
1995	David Naylor wrote in The Lancet “The Cochrane Collaboration is an enterprise that rivals the Human Genome Project in its potential implications for modern medicine.” (Naylor, 1995)
1998	Deborah Barnes and Lisa Bero showed that review articles on the health effects of passive smoking  were much more likely to conclude that passive smoking is not harmful if an author was affiliated with the tobacco industry.
1999	The Campbell Collaboration was established to produce systematic reviews of social interventions to inform policy and practice.
2014	Mike Clarke and colleagues published  an overview of more than 1,500 cumulative meta-analyses, showing how systematic reviews of existing evidence could have helped researchers, health professionals, patients, and funders make more informed decisions.
2016	John Ioannidis summarised evidence of massive production of systematic reviews, many of which are redundant or misleading.
Consideration of harmful effects
1978	Hershel Jick and Martin Vessey set out the basic principles of using case-control studies to evaluate possible adverse effects of drugs.
2001	Yoon Loke and Sheena Derry published a systematic review of reporting of adverse drug reactions in randomised trials, which found that trial reports often failed to provide details on how adverse drug reactions were defined or recorded.
2004	John Ioannidis and colleagues published a checklist for better reporting of harms in randomised trials as an extension of the Consolidated Standards of Reporting Trials (CONSORT) statement. The CONSORT harms statement was updated in 2022.
2007	The Cochrane Adverse Effects Methods Group was established to develop and promote methods for improving assessment and reporting of harmful effects in systematic reviews.
2013	Alex Hodkinson and colleagues published a systematic review that found poor reporting of harms in randomised trials across a range of clinical areas.
2015	Tammy Hoffmann and Chris Del Mar published a systematic review of patients’ expectations, which found that most participants overestimated intervention benefit and underestimated harm. In 2017 they published a systematic review of clinicians’ expectations, which found that clinicians more often underestimated harms and overestimated benefits.
2016	Liliane Zorzela and colleagues published the PRISMA harms checklist for improving reporting of harms in systematic reviews.
Reporting and peer review of research
1989	Drummond Rennie organised the first Peer Review Congress to promote research into the processes of selection and refinement of scientific manuscripts.
1992	The first Outcome Measures in Rheumatology (OMERACT) conference was convened with the aim of improving the measurement and reporting of outcomes in randomised trials of interventions for people with autoimmune and musculoskeletal diseases.
1996	The original Consolidated Standards of Reporting Trials (CONSORT) guideline for reporting randomised trials was published.
2006	The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network was established, with the aim of achieving accurate, complete, and transparent reporting of all health research studies to support research reproducibility and usefulness.
2007	Tom Jefferson and colleagues, in a Cochrane Methodology Review, found only 28 comparative studies of the effects of processes in editorial peer review; and Vittorio Demicheli and Carlo Di Pietrantonj, in a Cochrane Methodology Review, found only 10 comparative studies of the effects of grant giving peer review processes.
2008	The British Medical Journal (BMJ) sent three papers, each of which had nine major methodological errors inserted, to about 600 peer reviewers. On average, the peer reviewers detected about one-third of the errors in each paper.
2008	Paul Glasziou and colleagues documented that many trials and reviews omit crucial details about non-pharmacological interventions, making it difficult or impossible for clinicians to implement effective interventions. To address this problem, Glasziou and colleagues established an online formulary of non-drug interventions in 2013.
2009	The International Committee of Medical Journal Editors (ICMJE) adopted the “ICMJE Form for the Disclosure of Potential Conflicts of Interest” as a uniform mechanism for collecting and reporting authors’ relationships and activities that readers might consider relevant to a published work.
2009	The original Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guideline for reporting systematic reviews was published.
2010	The Core Outcome Measures in Effectiveness Trials (COMET) initiative was launched with the aim of promoting the development and use of core outcome sets to improve the measurement and reporting of outcomes.
2023	Jan-Ole Hesselberg and colleagues, in a Cochrane Methodology Review, found 10 randomised trials of reviewer training for improving grant or journal peer review that suggested training peer reviewers may lead to little or no improvement in the quality of peer review.
2025	Sally Hopewell and colleagues published major updates of the CONSORT and Standard Protocol Items: Recommendations for Interventional Trials (SPIRT) guidelines for reporting trial results and protocols.
Patient and public participation in decision making
1978	WHO’s Declaration of Alma Ata stated that people have the right and duty to participate individually and collectively in the planning and implementation of their health care.
1990	The International Association for Public Participation (IAP2) was founded to promote and advance public participation / community engagement globally through targeted initiatives.
2003	Annette O’Connor and colleagues published a Cochrane Review of the effects of decision aids for people facing health treatment or screening decisions. An updated version published in 2024 included 209 studies that showed that patient decision aids, compared to usual care, probably helped more adults reach informed values‐congruent choices.
2003	Involve, a public participation charity, was founded to work with governments, parliaments, civil society, academics and members of the public to create, advocate for and deliver new forms of public participation that re-vitalise democracy, improve decision-making, and enable people to shape the decisions that affect their lives.
2004	The James Lind Alliance was established in 2004 to promote and support involvement of patients and clinicians in setting research priorities.
2006	Imogen Evans and colleagues published the first edition of Testing Treatments to promote more critical public assessment of the effects of health interventions. The second edition was published and Testing Treatments international was launched in 2011 to promote critical thinking about treatment claims.
2008	Les Irwig and colleagues published Smart Health Choices to help people develop the skills to assess health advice – and hopefully to make decisions that will improve the quality of their care.
2008	Steven Woloshin and colleagues published Know Your Chances to help people see through hype and find credible information about health risks and the effects of interventions.
2009	Participedia was founded as a global network and crowdsourcing platform for researchers, educators, practitioners, policymakers, activists, and anyone interested in democratic innovations.
2012	Betty Chewning and colleagues published a systematic review showing that most patients prefer sharing decisions with physicians.
2013	Nicolas Couët and colleagues published a systematic review showing that few health-care providers consistently attempted to facilitate patient involvement, and even fewer adjusted care to patient preferences.
2015	The Informed Health Choices (IHC) Group published the first version of the Key Concepts for Informed Health Choices, a framework for enabling people to think critically about health claims. The international IHC Network was established to help people think critically about health information and how it informs their choices.
Evidence-informed policy and practice
1979	The Canadian Task Force on the Periodic Health Examination published recommendations that graded the level of evidence and classified the strength of its recommendations based on the level of the evidence. Subsequently, several other systems for grading the quality of evidence and the strength of recommendations were published. (Atkins et al., 2004)
1981	The Department of Clinical Epidemiology and Biostatistics at McMaster University published a series of articles on how to improve medical practice by reading medical journals critically.
1984	Brian Haynes and colleagues reviewed 248 evaluations of interventions to improve clinical practice. They updated this review in 1992 and 1995.
1987	The Annals of Internal Medicine started to publish structured abstracts, based on a proposal by Brian Haynes, making it easier for clinicians to read research reports critically.
1991	The Annals of Internal Medicine started publishing ACP Journal Club, which summarised the best new medical evidence from over 120 journals.
1992	The Evidence-Based Medicine (EBM) Working Group described a new approach to medical practice that de-emphasized uncritical reliance on unsystematic clinical experience and basic science as sufficient grounds for decision making and stressed critical use of evidence from evaluations of the effects of interventions and other applied research.
1992	UpToDate was founded by Burton Rose to solve the challenge of how to help physicians stay current and make better decisions for their patients. It first shipped on floppy disks by mail and was updated three times a year.
1993	Jeremy Grimshaw and colleagues published a systematic review of the effectiveness of guideline dissemination and implementation strategies, which they updated in 2004.
1993	The EBM Working Group published the first of a series of guides for using research to inform practice.
1994	The Cochrane Collaboration on Effective Professional Practice was registered as a collaborative review group. The name was changed to the Cochrane Effective Practice and Organization of Care Group (EPOC) in 1998.
2002	Simon Innvær and colleagues published a systematic review of facilitators of and barriers to the use of research evidence by health policymakers and updated the review in 2014.
2003	The Jameel Poverty Action Lab was founded to reduce poverty by ensuring that policy is informed by scientific evidence.
2004	The GRADE Working Group, a collaboration that included guideline developers using those various systems, published a systematic and explicit approach to making judgments about the quality of evidence and the strength of recommendations, which addressed shortcomings in other approaches. Subsequently, more than 120 organisations have endorsed or are using the GRADE approach, which has been continually developed.(Guyatt et al., 2025a)
2005	The World Health Assembly called on WHO member states to “establish or strengthen mechanisms to transfer knowledge in support of evidence-based public health and health-care delivery systems, and evidence-based health-related policies” and WHO launched the Evidence-informed Policy Network (EVIPNet) to strengthen the links between research and policy in low- and middle-income countries.
2008	The International Initiative for Impact Evaluation (3ie) was founded to improve the lives of poor people in low- and middle-income countries by providing, and summarising, evidence of what works, when, why, and for how much.
2015	The third edition of Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice recognised that only a few clinicians would become skilled at critically appraising original journal articles and the emphasis shifted to pre-appraised resources and particularly electronic publications that produce updated evidence summaries as the data appear and provide evidence-based recommendations for practice.
2017	Vikas Saini and colleagues published a series of articles about inappropriate use of health interventions. They found evidence of widespread overuse of health interventions that are more likely to cause harm than good and underuse of effective and affordable interventions.

References

Abdoul, H., Perrey, C., Tubach, F., Amiel, P., Durand-Zaleski, I. & Alberti, C.(2012). Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France. PLoS One, 7, e35247.

Abelson, J., Forest, P.-G., Eyles, J., Smith, P., Martin, E. & Gauvin, F.-P. (2003). Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Soc Sci Med, 57, 239-251.

Aguilera, B., Donya, R. S., Vélez, C. M., Kapiriri, L., Abelson, J., Nouvet, E., Danis, M., Goold, S., Williams, I. & Noorulhuda, M. (2024). Stakeholder participation in the COVID-19 pandemic preparedness and response plans: A synthesis of findings from 70 countries. Health Policy, 142, 105013.

Akl, E. A., El-Hachem, P., Abou-Haidar, H., Neumann, I., Schünemann, H. J. & Guyatt, G. H. (2014). Considering intellectual, in addition to financial, conflicts of interest proved important in a clinical practice guideline: a descriptive study. J Clin Epidemiol, 67, 1222-8.

Akl, E. A., Hakoum, M., Khamis, A., Khabsa, J., Vassar, M. & Guyatt, G. (2022). A framework is proposed for defining, categorizing, and assessing conflicts of interest in health research. J Clin Epidemiol, 149, 236-243.

Akl, E. A., Meerpohl, J. J., Elliott, J., Kahale, L. A. & Schünemann, H. J. (2017). Living systematic reviews: 4. Living guideline recommendations. J Clin Epidemiol, 91, 47-53.

Al-Shahi Salman, R., Beller, E., Kagan, J., Hemminki, E., Phillips, R. S., Savulescu, J., Macleod, M., Wisely, J. & Chalmers, I. (2014). Increasing value and reducing waste in biomedical research regulation and management. Lancet, 383, 176-85.

Albarqouni, L., Hoffmann, T., Straus, S., Olsen, N. R., Young, T., Ilic, D., Shaneyfelt, T., Haynes, R. B., Guyatt, G. & Glasziou, P. (2018). Core competencies in evidence-based practice for health professionals: consensus statement based on a systematic review and Delphi survey. JAMA Netw Open, 1, e180281.

Allen, K. A. (2025). Move beyond ‘publish or perish’ by measuring behaviours that benefit academia. Nature, 638, 861.

Alonso-Coello, P., Montori, V. M., Díaz, M. G., Devereaux, P. J., Mas, G., Diez, A. I., Solà, I., Roura, M., Souto, J. C., Oliver, S., Ruiz, R., Coll-Vinent, B., Gich, I., Schünemann, H. J. & Guyatt, G. (2015). Values and preferences for oral antithrombotic therapy in patients with atrial fibrillation: physician and patient perspectives. Health Expect, 18, 2318-27.

Als-Nielsen, B., Chen, W., Gluud, C. & Kjaergard, L. L. (2003). Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? JAMA, 290, 921-8.

Antes, G. & Oxman, A. D. (2001). The Cochrane Collaboration in the 20th Century. In: Egger, M., Davey Smith, G. & Altman, D. G. (eds.) Systematic Reviews in Health Care: Meta-Analysis in Context, Second Edition. London: BMJ Publishing Group.

Antman, E. M., Lau, J., Kupelnick, B., Mosteller, F. & Chalmers, T. C. (1992). A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA, 268, 240-8.

Aronson, J. K., Barends, E., Boruch, R., Brennan, M., Chalmers, I., Chislett, J., Cunliffe-Jones, P., Dahlgren, A., Gaarder, M., Haines, A., Heneghan, C., Matthews, R., Maynard, B., Oxman, A. D., Oxman, M., Pullin, A., Randall, N., Roddam, H., Schoonees, A., Sharples, J., Stewart, R., Stott, J., Tallis, R., Thomas, N. & Vale, L. (2019). Key concepts for making informed choices. Nature, 572, 303-306.

Atkins, D., Eccles, M., Flottorp, S., Guyatt, G. H., Henry, D., Hill, S., Liberati, A., O’connell, D., Oxman, A. D., Phillips, B., Schunemann, H., Edejer, T. T., Vist, G. E., Williams, J. W., Jr. & Grade Working Group (2004). Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group. BMC Health Serv Res, 4, 38.

Balas, E. A. & Boren, S. A. (2000). Managing Clinical Knowledge for Health Care Improvement. Yearb Med Inform, 65-70.

Barnes, D. E. & Bero, L. A. (1998). Why review articles on the health effects of passive smoking reach different conclusions. JAMA, 279, 1566-70.

Bashir, R., Surian, D. & Dunn, A. G. (2018). Time-to-update of systematic reviews relative to the availability of new evidence. Syst Rev, 7, 195.

Bastian, H., Doust, J., Clarke, M. & Glasziou, P. (2019). The epidemiology of systematic review updates: a longitudinal study of updating of Cochrane reviews, 2003 to 2018. medRxiv, 19014134.

Bastian, H., Glasziou, P. & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med, 7, e1000326.

Baudard, M., Yavchitz, A., Ravaud, P., Perrodeau, E. & Boutron, I. (2017). Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ, 356, j448.

Baumann, L. A., Reinhold, A. K. & Brütt, A. L. (2022). Public and patient involvement in health policy decision-making on the health system level – A scoping review. Health Policy, 126, 1023-1038.

Beam, A. & Darmono, L. (2024). Misinformation doesn’t have to get the last word: AI can support the truth, too. Harvard Public Health. Harvard Public Health.

Beane, B., Gingrich, N. & Kerry, J. 24 October 2008. How to take American health care from worst to first. New York Times. New York Times.

Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D., Olkin, I., Pitkin, R., Rennie, D., Schulz, K. F., Simel, D. & Stroup, D. F. (1996). Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA, 276, 637-9.

Bell, C. M., Urbach, D. R., Ray, J. G., Bayoumi, A., Rosen, A. B., Greenberg, D. & Neumann, P. J. (2006). Bias in published cost effectiveness studies: systematic review. BMJ, 332, 699-703.

Bero, L. (2014). What is in a name? Nonfinancial influences on the outcomes of systematic reviews and guidelines. J Clin Epidemiol, 67, 1239-41.

Bes-Rastrollo, M., Schulze, M. B., Ruiz-Canela, M. & Martinez-Gonzalez, M. A. (2013). Financial conflicts of interest and reporting bias regarding the association between sugar-sweetened beverages and weight gain: a systematic review of systematic reviews. PLoS Med, 10, e1001578.

Blanco, D., Altman, D., Moher, D., Boutron, I., Kirkham, J. J. & Cobo, E. (2019). Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open, 9, e026589.

Blanco, D., Schroter, S., Aldcroft, A., Moher, D., Boutron, I., Kirkham, J. J. & Cobo, E. (2020). Effect of an editorial intervention to improve the completeness of reporting of randomised trials: a randomised controlled trial. BMJ Open, 10, e036799.

Boers, M., Kirwan, J. R., Wells, G., Beaton, D., Gossec, L., D’agostino, M. A., Conaghan, P. G., Bingham, C. O., 3rd, Brooks, P., Landewé, R., March, L., Simon, L. S., Singh, J. A., Strand, V. & Tugwell, P. (2014). Developing core outcome measurement sets for clinical trials: OMERACT filter 2.0. J Clin Epidemiol, 67, 745-53.

Bolder Research Group 2016. Better Outcomes through Learning, Data, Engagement, and Research (BOLDER) – a system for improving evidence and clinical practice in low and middle income countries. F1000Res, 5, 693.

Bonell, C., Jamal, F., Melendez-Torres, G. J. & Cummins, S. (2015). ‘Dark logic’: theorising the harmful consequences of public health interventions. J Epidemiol Community Health, 69, 95-8.

Borges Do Nascimento, I. J., Pizarro, A. B., Almeida, J. M., Azzopardi-Muscat, N., Gonçalves, M. A., Björklund, M. & Novillo-Ortiz, D. (2022). Infodemics and health misinformation: a systematic review of reviews. Bull World Health Organ, 100, 544-561.

Bradley, S. H., Devito, N. J., Lloyd, K. E., Richards, G. C., Rombey, T., Wayant, C. & Gill, P. J. (2020). Reducing bias and improving transparency in medical research: a critical overview of the problems, progress and suggested next steps. J R Soc Med, 113, 433-443.

Brett, J., Staniszewska, S., Mockford, C., Herron-Marx, S., Hughes, J., Tysall, C. & Suleman, R. 2014. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect, 17, 637-50.

Brownlee, S., Chalkidou, K., Doust, J., Elshaug, A. G., Glasziou, P., Heath, I., Nagpal, S., Saini, V., Srivastava, D., Chalmers, K. & Korenstein, D. (2017). Evidence for overuse of medical services around the world. Lancet, 390, 156-168.

Bruce, R., Chauvin, A., Trinquart, L., Ravaud, P. & Boutron, I. (2016). Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med, 14, 85.

Bryant, E. A., Scott, A. M., Greenwood, H. & Thomas, R.( 2022). Patient and public involvement in the development of clinical practice guidelines: a scoping review. BMJ Open, 12, e055428.

Cassels, A. (2015). The Cochrane Collaboration: Medicine’s Best Kept Secret, Victoria, Canada, Agio Publishing House.

Catalá-López, F., Alonso-Arroyo, A., Page, M. J., Castelló-Cogollos, L., Hutton, B., Ridao, M., Tabarés-Seisdedos, R., Aleixandre-Benavent, R. & Moher, D. (2023). A cross-sectional analysis identified co-authorship networks and scientific collaboration on reporting guidelines for health research. J Clin Epidemiol, 157, 22-34.

Center For Open Science. Registered Reports: Peer review before results are known to align scientific values and practices [Online]. Available: https://www.cos.io/initiatives/registered-reports [Accessed 6 June 2025].

Chalmers, I. 1979. Randomized controlled trials of fetal monitoring, 1973-1977. In: Thalhammer, O., Baumgarten, K. & Pollack, A. (eds.) Perinatal Medicine. Stuttgart: George Thieme.

Chalmers, I. 1993. The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Ann N Y Acad Sci, 703, 156-63; discussion 163-5.

Chalmers, I. 2003a. The Prehistory of the First Cochrane Centre. In: BOSCH, F. X. & MOLAS, R. (eds.) Archie Cochrane: Back to the Front. Barcelona: Francesc Xavier Bosch.

Chalmers, I. 2003b. Trying to do more good than harm in policy and practice: the role of rigorous, transparent, up-to-date evaluations. The Annals of the American Academy of Political and Social Science, 589, 22-40.

Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. & Oliver, S. (2014). How to increase value and reduce waste when research priorities are set. Lancet, 383, 156-65.

Chalmers, I. & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. Lancet, 374, 86-9.

Chalmers, I., Hetherington, J., Newdick, M., Mutch, L., Grant, A., Enkin, M., Enkin, E. & Dickersin, K. (1986). The Oxford Database of Perinatal Trials: developing a register of published reports of controlled trials. Control Clin Trials, 7, 306-24.

Chalmers, T. C., Eckhardt, R. D., Reynolds, W. E., Cigarroa, J. G., Jr., Deane, N., Reifenstein, R. W., Smith, C. W. & Davidson, C. S. (1955). The treatment of acute infectious hepatitis. Controlled studies of the effects of diet, rest, and physical reconditioning on the acute course of the disease and on the incidence of relapses and residual abnormalities. J Clin Invest, 34, 1163-235.

Chan, A. W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D. & Van Der Worp, H. B. (2014). Increasing value and reducing waste: addressing inaccessible research. Lancet, 383, 257-66.

Chapman, S. J., Shelton, B., Mahmood, H., Fitzgerald, J. E., Harrison, E. M. & Bhangu, A. (2014). Discontinuation and non-publication of surgical randomised controlled trials: observational study. BMJ, 349, g6870.

Chartres, N., Fabbri, A. & Bero, L. A. 2016. Association of Industry Sponsorship With Outcomes of Nutrition Studies: A Systematic Review and Meta-analysis. JAMA Intern Med, 176, 1769-1777.

Chen, Y., Yang, K., Marušic, A., Qaseem, A., Meerpohl, J. J., Flottorp, S., Akl, E. A., Schünemann, H. J., Chan, E. S., Falck-Ytter, Y., Ahmed, F., Barber, S., Chen, C., Zhang, M., Xu, B., Tian, J., Song, F., Shang, H., Tang, K., Wang, Q. & Norris, S. L. (2017). A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement. Ann Intern Med, 166, 128-132.

Chewning, B., Bylund, C. L., Shah, B., Arora, N. K., Gueguen, J. A. & Makoul, G. (2012). Patient preferences for shared decisions: a systematic review. Patient Educ Couns, 86, 9-18.

Chiu, K., Grundy, Q. & Bero, L. (2017). ‘Spin’ in published biomedical literature: a methodological systematic review. PLoS Biol, 15, e2002173.

Chou, R., Aronson, N., Atkins, D., Ismaila, A. S., Santaguida, P., Smith, D. H., Whitlock, E., Wilt, T. J. & Moher, D. (2010). AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol, 63, 502-12.

Clark, J., Barton, B., Albarqouni, L., Byambasuren, O., Jowsey, T., Keogh, J., Liang, T., Moro, C., O’neill, H. & Jones, M. (2025). Generative artificial intelligence use in evidence synthesis: A systematic review. Res Synth Methods, 1-19.

Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P. & Scott, A. M. (2020). A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol, 121, 81-90.

Clarke, M., Brice, A. & Chalmers, I. (2014). Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One, 9, e102670.

Clarke, M., Chalmers, I., Alderson, P. & Hopewell, S. (2024). Reports of randomised control trials should begin and conclude with up-to-date systematic reviews of other relevant trials: a 25-year audit of the quality of trial reports. J R Soc Med, 117, 212-216.

Cobey, K. D., Ebrahimzadeh, S., Page, M. J., Thibault, R. T., Nguyen, P. Y., Abu-Dalfa, F. & Moher, D. (2024). Biomedical researchers’ perspectives on the reproducibility of research. PLoS Biol, 22, e3002870.

Cochrane, A. (1979). 1931-1971: A critical review with particular reference to the medical profession. Medicines for the year 2000. London: Office of Health Economics.

Concannon, T. W., Fuster, M., Saunders, T., Patel, K., Wong, J. B., Leslie, L. K. & Lau, J. (2014). A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med, 29, 1692-701.

Conklin, A., Morris, Z. & Nolte, E. (2015). What is the evidence base for public involvement in health-care policy?: results of a systematic scoping review. Health Expect, 18, 153-65.

Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., Mcilroy, W. & Oxman, A. D. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA, 269, 2749-53.

Cosgrove, L., Bursztajn, H. J., Erlich, D. R., Wheeler, E. E. & Shaughnessy, A. F. (2013). Conflicts of interest and the quality of recommendations in clinical guidelines. J Eval Clin Pract, 19, 674-81.

Couët, N., Desroches, S., Robitaille, H., Vaillancourt, H., Leblanc, A., Turcotte, S., Elwyn, G. & Légaré, F. (2015). Assessments of the extent to which health-care providers involve patients in decision making: a systematic review of studies using the OPTION instrument. Health Expect, 18, 542-61.

Crowe, S., Fenton, M., Hall, M., Cowan, K. & Chalmers, I. 2015. Patients’, clinicians’ and the research communities’ priorities for treatment research: there is an important mismatch. Res Involv Engagem, 1, 2.

Cuervo, L. G. & Clarke, M. (2003). Balancing benefits and harms in health care. BMJ, 327, 65-6.

Dahlgren, A., Furuseth-Olsen, K., Rose, C. J. & Oxman, A. D. (2021). The Norwegian public’s ability to assess treatment claims: results of a cross-sectional study of critical health literacy. F1000Research, 9, 179.

Daniels, N. & Sabin, J. (1997). Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philos Public Aff, 26, 303-50.

Davidson, M., Korfitsen, C. B., Riveros, C., Chaimani, A. & Boutron, I. (2025). Post-publication peer review and the identification of methodological and reporting issues in COVID-19 trials: a qualitative study. BMJ Evid Based Med.

De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J., Schroeder, T. V., Sox, H. C. & Van Der Weyden, M. B. (2004). Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med, 141, 477-8.

De Wit, M., Kirwan, J. R., Tugwell, P., Beaton, D., Boers, M., Brooks, P., Collins, S., Conaghan, P. G., D’agostino, M. A., Hofstetter, C., Hughes, R., Leong, A., Lyddiatt, A., March, L., May, J., Montie, P., Richards, P., Simon, L. S., Singh, J. A., Strand, V., Voshaar, M., Bingham, C. O., 3rd & Gossec, L. (2017). Successful Stepwise Development of Patient Research Partnership: 14 Years’ Experience of Actions and Consequences in Outcome Measures in Rheumatology (OMERACT). Patient, 10, 141-152.

Dechartres, A., Trinquart, L., Faber, T. & Ravaud, P. (2016). Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol, 77, 24-37.

Del Fiol, G., Workman, T. E. & Gorman, P. N. (2014). Clinical questions raised by clinicians at the point of care: a systematic review. JAMA Intern Med, 174, 710-8.

Demicheli, V. & Di Pietrantonj, C. (2007). Peer review for improving the quality of grant applications. Cochrane Database Syst Rev, 2007, Mr000003.

Devito, N. J., Morley, J. & Goldacre, B. (2024). Barriers and best practices to improving clinical trials transparency at UK public research institutions: A qualitative interview study. Health Policy, 142, 104991.

Djulbegovic, B. & Guyatt, G. H. (2017). Progress in evidence-based medicine: a quarter century on. Lancet, 390, 415-423.

Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., Brito, J. P., Boehmer, K., Hasan, R., Firwana, B., Erwin, P., Eton, D., Sloan, J., Montori, V., Asi, N., Dabrh, A. M. & Murad, M. H. (2014). Patient engagement in research: a systematic review. BMC Health Serv Res, 14, 89.

Dunn, A. G., Arachi, D., Hudgins, J., Tsafnat, G., Coiera, E. & Bourgeois, F. T. (2014). Financial conflicts of interest and conclusions about neuraminidase inhibitors for influenza: an analysis of systematic reviews. Ann Intern Med, 161, 513-8.

Dunn, A. G., Coiera, E., Mandl, K. D. & Bourgeois, F. T. (2016). Conflict of interest disclosure in biomedical research: A review of current practices, biases, and the role of public registries in improving transparency. Res Integr Peer Rev, 1.

Dunn, A. G., Mandl, K. D., Coiera, E. & Bourgeois, F. T. (2013). The effects of industry sponsorship on comparator selection in trial registrations for neuropsychiatric conditions in children. PLoS One, 8, e84951.

Duyx, B., Urlings, M. J. E., Swaen, G. M. H., Bouter, L. M. & Zeegers, M. P. (2017). Scientific citations favor positive results: a systematic review and meta-analysis. J Clin Epidemiol, 88, 92-101.

Dwan, K., Gamble, C., Williamson, P. R. & Kirkham, J. J. 2013. Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review. PLoS One, 8, e66844.

Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., Mcdonald, S., Salanti, G., Meerpohl, J., Maclehose, H., Hilton, J., Tovey, D., Shemilt, I. & Thomas, J. (2017). Living systematic review: 1. Introduction-the why, what, when, and how. J Clin Epidemiol, 91, 23-30.

Elshaug, A. G., Rosenthal, M. B., Lavis, J. N., Brownlee, S., Schmidt, H., Nagpal, S., Littlejohns, P., Srivastava, D., Tunis, S. & Saini, V. (2017). Levers for addressing medical underuse and overuse: achieving high-value health care. Lancet, 390, 191-202.

English, M., Irimu, G., Agweyu, A., Gathara, D., Oliwa, J., Ayieko, P., Were, F., Paton, C., Tunis, S. & Forrest, C. B. (2016). Building Learning Health Systems to Accelerate Research and Improve Outcomes of Clinical Care in Low- and Middle-Income Countries. PLoS Med, 13, e1001991.

Epistemonikos. (2023). Updated report – Strategy for systematic reviews [Online]. Available: https://www.epistemonikos.org/en/about_us/updated_report [Accessed].

Equator Network. Enhancing the Quality and Transparency Of health Research [Online]. Available: https://www.equator-network.org/ [Accessed 4 June 2025].

Eysenbach, G., Powell, J., Kuss, O. & Sa, E. R. (2002). Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA, 287, 2691-700.

Fabbri, A., Lai, A., Grundy, Q. & Bero, L. A. (2018). The Influence of Industry Sponsorship on the Research Agenda: A Scoping Review. Am J Public Health, 108, e9-e16.

Fanelli, D. (2010). Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PLoS One, 5, e10271.

Fleming, P. S., Koletsi, D., Dwan, K. & Pandis, N. (2015). Outcome discrepancies and selective reporting: impacting the leading journals? PLoS One, 10, e0127495.

Flottorp, S. A., Oxman, A. D., Krause, J., Musila, N. R., Wensing, M., Godycki-Cwirko, M., Baker, R. & Eccles, M. P. (2013). A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci, 8, 35.

Fretheim, A., Flottorp, S. & Oxman, A. (2015). [Effect of interventions for implementing clinical practice guidelines]. Oslo, Norway: Norwegian Knowledge Centre for the Health Services.

Furberg, B. & Furberg, C. (2001). [All that glitters is not gold – II: How to evaluate clinical studies], Lund, Sweden, Solutio AB, Bengt Furberg och Curt Furberg.

Garner, P., Hopewell, S., Chandler, J., Maclehose, H., Schünemann, H. J., Akl, E. A., Beyene, J., Chang, S., Churchill, R., Dearness, K., Guyatt, G., Lefebvre, C., Liles, B., Marshall, R., Martínez García, L., Mavergames, C., Nasser, M., Qaseem, A., Sampson, M., Soares-Weiser, K., Takwoingi, Y., Thabane, L., Trivella, M., Tugwell, P., Welsh, E., Wilson, E. C. & Schünemann, H. J. (2016). When and how to update systematic reviews: consensus and checklist. BMJ, 354, i3507.

Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D. & Wager, E. (2014). Reducing waste from incomplete or unusable reports of biomedical research. Lancet, 383, 267-76.

Glasziou, P., Straus, S., Brownlee, S., Trevena, L., Dans, L., Guyatt, G., Elshaug, A. G., Janett, R. & Saini, V. (2017). Evidence for underuse of effective medical services around the world. Lancet, 390, 169-177.

Glenton, C., Paulsen, E. J. & Oxman, A. D. (2005). Portals to Wonderland: health portals lead to confusing information about the effects of health care. BMC Med Inform Decis Mak, 5, 7.

Goldacre, B. (2017). The WHO joint statement from funders on trials transparency. BMJ, 357, j2816.

Goldacre, B., Lane, S., Mahtani, K. R., Heneghan, C., Onakpoya, I., Bushfield, I. & Smeeth, L. (2017). Pharmaceutical companies’ policies on access to trial data, results, and methods: audit study. BMJ, 358, j3334.

Gøtzsche, P. C. (2022). Citation bias: questionable research practice or scientific misconduct? J R Soc Med, 115, 31-35.

Graham, S. S., Grundy, Q., Sharma, N., Edward, J. S., Barbour, J. B., Rousseau, J. F., Majdik, Z. P. & Bero, L. (2025). Research on policy mechanisms to address funding bias and conflicts of interest in biomedical research: a scoping review. Res Integr Peer Rev, 10, 6.

Grol, R. (2001). Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care, 39, Ii46-54.

Guthrie, S., Ghiga, I. & Wooding, S. (2018). What do we know about grant peer review in the health sciences? F1000Res, 6, 1335.

Guyatt, G., Hultcrantz, M., Agoritsas, T., Iorio, A., Vandvik, P. O. & Montori, V. M. (2025a). Why Core GRADE is needed: introduction to a new series in The BMJ. BMJ, 389, e081902.

Guyatt, G., Wang, Y., Eachempati, P., Iorio, A., Murad, M. H., Hultcrantz, M., Chu, D. K., Florez, I. D., Hemkens, L. G., Agoritsas, T., Yao, L., Vandvik, P. O., Montori, V. M. & Brignardello-Petersen, R. (2025b). Core GRADE 4: rating certainty of evidence-risk of bias, publication bias, and reasons for rating up certainty. BMJ, 389, e083864.

Hanney, S. R. & González-Block, M. A. (2009). Evidence-informed health policy: are we beginning to get there at last? Health Res Policy Syst, 7, 30.

Hansen, C., Lundh, A., Rasmussen, K. & Hróbjartsson, A. (2019). Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev, 8, Mr000047.

Hesselberg, J. O., Dalsbø, T. K., Stromme, H., Svege, I. & Fretheim, A. (2023). Reviewer training for improving grant and journal peer review. Cochrane Database Syst Rev, 11, Mr000056.

Hoffmann, F., Allers, K., Rombey, T., Helbach, J., Hoffmann, A., Mathes, T. & Pieper, D. (2021). Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000-2019. J Clin Epidemiol, 138, 1-11.

Hoffmann, T. C. & Del Mar, C. (2015). Patients’ expectations of the benefits and harms of treatments, screening, and tests: a systematic review. JAMA Intern Med, 175, 274-86.

Hoffmann, T. C. & Del Mar, C. (2017). Clinicians’ Expectations of the Benefits and Harms of Treatments, Screening, and Tests: A Systematic Review. JAMA Intern Med, 177, 407-419.

Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., Lamb, S. E., Dixon-Woods, M., Mcculloch, P., Wyatt, J. C., Chan, A. W. & Michie, S. (2014). Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ, 348, g1687.

Hoffmann, T. C., Oxman, A. D., Ioannidis, J. P., Moher, D., Lasserson, T. J., Tovey, D. I., Stein, K., Sutcliffe, K., Ravaud, P., Altman, D. G., Perera, R. & Glasziou, P. (2017). Enhancing the usability of systematic reviews by improving the consideration and description of interventions. BMJ, 358, j2998.

Hohlfeld, A., Kredo, T. & Clarke, M. (2024). A scoping review of activities intended to reduce publication bias in randomised trials. Syst Rev, 13, 310.

Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., Bhandari, N., Butcher, N. J., Campbell, M. K., Chidebe, R. C. W., Elbourne, D., Farmer, A., Fergusson, D. A., Golub, R. M., Goodman, S. N., Hoffmann, T. C., Ioannidis, J. P. A., Kahan, B. C., Knowles, R. L., Lamb, S. E., Lewis, S., Loder, E., Offringa, M., Ravaud, P., Richards, D. P., Rockhold, F. W., Schriger, D. L., Siegfried, N. L., Staniszewska, S., Taylor, R. S., Thabane, L., Torgerson, D., Vohra, S., White, I. R. & Boutron, I. (2025). CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials. BMJ, 389, e081124.

Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D. & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev, MR000006.

Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., Bhandari, N., Butcher, N. J., Campbell, M. K., Chidebe, R. C. W., Elbourne, D. R., Farmer, A. J., Fergusson, D. A., Golub, R. M., Goodman, S. N., Hoffmann, T. C., Ioannidis, J. P. A., Kahan, B. C., Knowles, R. L., Lamb, S. E., Lewis, S., Loder, E., Offringa, M., Ravaud, P., Richards, D. P., Rockhold, F. W., Schriger, D. L., Siegfried, N. L., Staniszewska, S., Taylor, R. S., Thabane, L., Torgerson, D. J., Vohra, S., White, I. R. & Chan, A. W. (2025). SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials. BMJ, 389, e081660.

Hugenholtz, G. W., Heerdink, E. R., Stolker, J. J., Meijer, W. E., Egberts, A. C. & Nolen, W. A. (2006). Haloperidol dose when used as active comparator in randomized controlled trials with atypical antipsychotics in schizophrenia: comparison with officially recommended doses. J Clin Psychiatry, 67, 897-903.

INVOLVE (2005). People & Participation: How to put citizens at the heart of decision-making, London, Involve.

Ioannidis, J. P. (2016). The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q, 94, 485-514.

Ioannidis, J. P., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F. & Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. Lancet, 383, 166-75.

Jadad, A. R., Cook, D. J. & Browman, G. P. (1997). A guide to interpreting discordant systematic reviews. CMAJ, 156, 1411-6.

Jefferson, T., Rudin, M., Brodney Folse, S. & Davidoff, F. (2007). Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev.

Jones, C. W., Keil, L. G., Holland, W. C., Caughey, M. C. & Platts-Mills, T. F. (2015). Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med, 13, 282.

Jørgensen, A. W., Hilden, J. & Gøtzsche, P. C. (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ, 333, 782.

Kebebe, M., Oxman, A. D. & Glenton, C. (2012). Blueprints for informed policy decisions: a review of laws and policies requiring routine evaluation. Rapport fra Kunnskapssenteret. Oslo: Norwegian Knowledge Centre for the Health Services.

Kirkham, J. J., Altman, D. G., Chan, A. W., Gamble, C., Dwan, K. M. & Williamson, P. R. (2018). Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. BMJ, 362, k3802.

Kirkham, J. J., Davis, K., Altman, D. G., Blazeby, J. M., Clarke, M., Tunis, S. & Williamson, P. R. (2017). Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med, 14, e1002447.

Kirkham, J. J., Gorst, S., Altman, D. G., Blazeby, J. M., Clarke, M., Tunis, S. & Williamson, P. R. (2019). Core Outcome Set-Standardized Protocol Items: the COS-STAP Statement. Trials, 20, 116.

Kleijnen, J., Gøtzsche, P., Kunz, R. A., Oxman, A. D. & Chalmers, I. (1997). So what’s so special about randomisation? In: Maynard, A. & Chalmers, I. (eds.) Non-Random Reflections on Health Care Research: On the 25th Anniversary Of Archie Cochrane’s Effectiveness and Efficiency. London: BMJ Publishing Group.

Korfitsen, C. B., Nejstgaard, C. H., Hróbjartsson, A., Boutron, I., Bero, L. & Lundh, A. (2025). Peer reviewers’ conflicts of interest in biomedical research: scoping review. BMJ Evid Based Med.

Lathyris, D. N., Patsopoulos, N. A., Salanti, G. & Ioannidis, J. P. (2010). Industry sponsorship and selection of comparators in randomized clinical trials. Eur J Clin Invest, 40, 172-82.

Lavis, J. N., Paulsen, E. J., Oxman, A. D. & Moynihan, R. (2008). Evidence-informed health policy 2 – survey of organizations that support the use of research evidence. Implement Sci, 3, 54.

Lazarus, C., Haneef, R., Ravaud, P. & Boutron, I. (2015). Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention. BMC Med Res Methodol, 15, 85.

Leigh, A. (2018). Randomistas: How radical researchers are changing our world, New Haven, Yale University Press.

Light, R. J. & Pillemer, D. B. (1984). Summing Up: The Science of Reviewing Research, Boston, Harvard University Press.

Lloyd, N., Kenny, A. & Hyett, N. (2021). Evaluating health service outcomes of public involvement in health service design in high-income countries: a systematic review. BMC Health Serv Res, 21, 364.

Lund, H., Brunnhuber, K., Juhl, C., Robinson, K., Leenaars, M., Dorch, B. F., Jamtvedt, G., Nortvedt, M. W., Christensen, R. & Chalmers, I. (2016). Towards evidence based research. BMJ, 355, i5440.

Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B. & Bero, L. (2017). Industry sponsorship and research outcome. Cochrane Database Syst Rev.

Maclean, S., Mulla, S., Akl, E. A., Jankowski, M., Vandvik, P. O., Ebrahim, S., Mcleod, S., Bhatnagar, N. & Guyatt, G. H. (2012). Patient values and preferences in decision making for antithrombotic therapy: a systematic review: Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest, 141, e1S-e23S.

Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P., Al-Shahi Salman, R., Chan, A. W. & Glasziou, P. (2014). Biomedical research: increasing value, reducing waste. Lancet, 383, 101-4.

Mandrioli, D., Kearns, C. E. & Bero, L. A. (2016). Relationship between Research Outcomes and Risk of Bias, Study Sponsorship, and Author Financial Conflicts of Interest in Reviews of the Effects of Artificially Sweetened Beverages on Weight Outcomes: A Systematic Review of Reviews. PLoS One, 11, e0162198.

Manheimer, E. & Anderson, D. (2002). Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ, 325, 528-31.

Martínez García, L., Pardo-Hernández, H., Sanabria, A. J. & Alonso-Coello, P. (2017). Continuous surveillance of a pregnancy clinical guideline: an early experience. Syst Rev, 6, 143.

Martínez García, L., Sanabria, A. J., García Alvarez, E., Trujillo-Martín, M. M., Etxeandia-Ikobaltzeta, I., Kotzeva, A., Rigau, D., Louro-González, A., Barajas-Nava, L., Díaz Del Campo, P., Estrada, M. D., Solà, I., Gracia, J., Salcedo-Fernandez, F., Lawson, J., Haynes, R. B. & Alonso-Coello, P. (2014). The validity of recommendations from clinical guidelines: a survival analysis. Cmaj, 186, 1211-9.

Mcdonald, P. L., Foley, T. J., Verheij, R., Braithwaite, J., Rubin, J., Harwood, K., Phillips, J., Gilman, S. & Van Der Wees, P. J. (2024). Data to knowledge to improvement: creating the learning health system. BMJ, 384, e076175.

Mcglynn, E. A., Asch, S. M., Adams, J., Keesey, J., Hicks, J., Decristofaro, A. & Kerr, E. A. (2003). The quality of health care delivered to adults in the United States. N Engl J Med, 348, 2635-45.

Mcpherson, A., Richards, M. & Chalmers, I. (1979). Developing a bibliography of randomized controlled trials in perinatal medicine. Report submitted to the Family Health Division of the World Health Organization. Geneva.

Meyrowitsch, D. W., Jensen, A. K., Sørensen, J. B. & Varga, T. V. (2023). AI chatbots and (mis)information in public health: impact on vulnerable communities. Front Public Health, 11, 1226776.

Moberg, J., Lewin, S., Munabi-Babigumira, S., Oxman, A. D., Treweek, S. & Eds (2011). SURE Guides for Preparing and Using Evidence-Based Policy Briefs, Version 2.1, Geneva, World Health Organization.

Moberg, J., Oxman, A. D., Rosenbaum, S., Schunemann, H. J., Guyatt, G., Flottorp, S., Glenton, C., Lewin, S., Morelli, A., Rada, G., Alonso-Coello, P. & Group, G. W. (2018). The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst, 16, 45.

Morgan, M. G. (2014). Use (and abuse) of expert elicitation in support of decision making for public policy. Proc Natl Acad Sci U S A, 111, 7176-84.

Morris, Z. S., Wooding, S. & Grant, J. (2011). The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med, 104, 510-20.

Mulrow, C. D. (1987). The medical review article: state of the science. Ann Intern Med, 106, 485-8.

Munthe-Kaas, H. M., Oxman, A. D., Von Lieres, B., Gloppen, S. & Ohren, A. (2024). Public participation in decisions about measures to manage the COVID-19 pandemic: a systematic review. BMJ Glob Health, 9.

Nagendran, M., Pereira, T. V., Kiew, G., Altman, D. G., Maruthappu, M., Ioannidis, J. P. & Mcculloch, P. (2016). Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment. BMJ, 355, i5432.

Nasser, M., Clarke, M., Chalmers, I., Brurberg, K. G., Nykvist, H., Lund, H. & Glasziou, P. (2017). What are funders doing to minimise waste in research? Lancet, 389, 1006-1007.

Naudet, F., Schuit, E. & Ioannidis, J. P. A. (2017). Overlapping network meta-analyses on the same topic: survey of published studies. Int J Epidemiol, 46, 1999-2008.

Naylor, C. D. 1995. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet, 345, 840-2.

Nejstgaard, C. H., Bero, L., Hróbjartsson, A., Jørgensen, A. W., Jørgensen, K. J., Le, M. & Lundh, A. (2020). Association between conflicts of interest and favourable recommendations in clinical guidelines, advisory committee reports, opinion pieces, and narrative reviews: systematic review. BMJ, 371, m4234.

Neuman, M. D., Goldstein, J. N., Cirullo, M. A. & Schwartz, J. S. (2014). Durability of class I American College of Cardiology/American Heart Association clinical practice guideline recommendations. JAMA, 311, 2092-100.

Nickerson, R. S. (1998). Confirmation bias: a ubiquitous phenomenon in many guises. Rev Gen Psychol, 2, 175-220.

Nilsen, E. S., Myrhaug, H. T., Johansen, M., Oliver, S. & Oxman, A. D. (2006). Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev, CD004563.

Nord, M., Lundstedt, D., Altman, F., Angiolillo, D., Borella, T., Fernandes, L. & Gastaldi, A. (2024). Democracy Report 2024: Democracy Winning and Losing at the Ballot. University of Gothenburg: V-Dem Institute.

Norheim, O. F., Abi-Rached, J. M., Bright, L. K., Bærøe, K., Ferraz, O. L. M., Gloppen, S. & Voorhoeve, A. (2021). Difficult trade-offs in response to COVID-19: the case for open and inclusive decision making. Nat Med, 27, 10-13.

Norris, S. L., Burda, B. U., Holmer, H. K., Ogden, L. A., Fu, R., Bero, L., Schünemann, H. & Deyo, R. (2012). Author’s specialty and conflicts of interest contribute to conflicting guidelines for screening mammography. J Clin Epidemiol, 65, 725-33.

Norris, S. L., Holmer, H. K., Ogden, L. A. & Burda, B. U. (2011). Conflict of interest in clinical practice guideline development: a systematic review. PLoS One, 6, e25153.

Oransky, I. (2005). William Silverman. Lancet, 365, 116.

Oxman, A. (1987). A Methodological Framework for Research Overviews. MSc, McMaster University.

Oxman, A. D., Bjorndal, A., Becerra-Posada, F., Gibson, M., Block, M. A., Haines, A., Hamid, M., Odom, C. H., Lei, H., Levin, B., Lipsey, M. W., Littell, J. H., Mshinda, H., Ongolo-Zogo, P., Pang, T., Sewankambo, N., Songane, F., Soydan, H., Torgerson, C., Weisburd, D., Whitworth, J. & Wibulpolprasert, S. (2010). A framework for mandatory impact evaluation to ensure well informed public policy decisions. Lancet, 375, 427-31.

Oxman, A. D., Chalmers, I., Clarke, M., Enkin, M., Schulz, K., Starr, M., Dickersin, K., Herxheimer, A. & Silagy, C. (1994). Preparing and maintaining systematic reviews. In: Sackett, D. L. & Oxman, A. D. (eds.) Cochrane Collaboration Handbook. 1st ed. Oxford: The Cochrane Collaboration.

Oxman, A. D., Chalmers, I. & Dahlgren, A. (2022a). Key concepts for informed health choices. 1.2: Seemingly logical assumptions about research can be misleading. J R Soc Med, 115, 408-411.

Oxman, A. D., Chalmers, I. & Dahlgren, A. (2023a). Key concepts for informed health choices. 2.1: Comparisons of treatments should be fair. J R Soc Med, 116, 21-26.

Oxman, A. D., Chalmers, I. & Dahlgren, A. (2023b). Key concepts for informed health choices. 2.4: Descriptions of effects should reflect the risk of being misled by the play of chance. J R Soc Med, 116, 144-147.

Oxman, A. D., Chalmers, I., Dahlgren, A. & Informed Health Choices Group Consider how important each advantage and disadvantage is when weighing the pros and cons and making choices. In: Key Concepts for Informed Health Choices: a framework for enabling people to think critically about health claims (Version 2022).

Oxman, A. D., Glenton, C., Flottorp, S., Lewin, S., Rosenbaum, S. & Fretheim, A. (2020). Development of a checklist for people communicating evidence-based information about the effects of healthcare interventions: a mixed methods study. BMJ Open, 10, e036348.

Oxman, A. D. & Guyatt, G. H. (1988). Guidelines for reading literature reviews. CMAJ, 138, 697-703.

Oxman, A. D., Lavis, J. N., Lewin, S. & Fretheim, A. (2009). SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking? Health Res Policy Syst, 7 Suppl 1, S1.

Oxman, A. D. & Paulsen, E. J. (2019). Who can you trust? A review of free online sources of “trustworthy” information about treatment effects for patients and the public. BMC Med Inform Decis Mak, 19, 35.

Oxman, M., Chesire, F., Mugisha, M., Ssenyonga, R., Ngatia, B., Nsangi, A., Lewin, S., Moberg, J., Sewankambo, N., Kaseje, M., Melby-Lervåg, M., Fretheim, A., Oxman, A. D. & Rosenbaum, S. (2024). Development of a framework of potential adverse effects of interventions to improve critical thinking about health choices: A mixed methods study. F1000Res, 13, 1303.

Oxman, M., Larun, L., Pérez Gaxiola, G., Alsaid, D., Qasim, A., Rose, C. J., Bischoff, K. & Oxman, A. D. (2022b). Quality of information in news media reports about the effects of health interventions: Systematic review and meta-analyses. F1000Res, 10, 433.

Page, M. J., Mckenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S. & Forbes, A. *(2014). Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev.

Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., Mcdonald, S., Mcguinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P. & Mckenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 372, n160.

Page, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., Catalá-López, F., Li, L., Reid, E. K., Sarkis-Onofre, R. & Moher, D. (2016). Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med, 13, e1002028.

Page, M. J., Sterne, J. A. C., Boutron, I., Hróbjartsson, A., Kirkham, J. J., Li, T., Lundh, A., Mayo-Wilson, E., Mckenzie, J. E., Stewart, L. A., Sutton, A. J., Bero, L., Dunn, A. G., Dwan, K., Elbers, R. G., Kanukula, R., Meerpohl, J. J., Turner, E. H. & Higgins, J. P. T. (2023). ROB-ME: a tool for assessing risk of bias due to missing evidence in systematic reviews with meta-analysis. BMJ, 383, e076754.

Panagiotou, O. A. & Ioannidis, J. P. (2012). Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. J Clin Epidemiol, 65, 740-7.

Pereira, T. V., Horwitz, R. I. & Ioannidis, J. P. (2012). Empirical evaluation of very large treatment effects of medical interventions. JAMA, 308, 1676-84.

Pollock, A., Campbell, P., Struthers, C., Synnot, A., Nunn, J., Hill, S., Goodare, H., Morris, J., Watts, C. & Morley, R. (2018). Stakeholder involvement in systematic reviews: a scoping review. Syst Rev, 7, 208.

Ponce, O. J., Alvarez-Villalobos, N., Shah, R., Mohammed, K., Morgan, R. L., Sultan, S., Falck-Ytter, Y., Prokop, L. J., Dahm, P., Mustafa, R. A. & Murad, M. H. (2017). What does expert opinion in guidelines mean? a meta-epidemiological study. Evid Based Med, 22, 164-169.

Qussini, S., Macdonald, R. S., Shahbal, S. & Dierickx, K. (2023). Blinding Models for Scientific Peer-Review of Biomedical Research Proposals: A Systematic Review. J Empir Res Hum Res Ethics, 18, 250-262.

Rashid, A., Thomas, V., Shaw, T. & Leng, G. (2017). Patient and Public Involvement in the Development of Healthcare Guidance: An Overview of Current Methods and Future Challenges. Patient, 10, 277-282.

Reboot Foundation (2020). Critical Thinking Survey Report. Paris: Reboot Foundation.

Rejas Bueno, M., Bacaicoa López De Sabando, A. & Sánchez Robles, G. A. (2022). [Health professionals expectations’ about the benefit of regular primary care interventions]. Aten Primaria, 54, 102235.

Riveros, C., Dechartres, A., Perrodeau, E., Haneef, R., Boutron, I. & Ravaud, P. (2013). Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Med, 10, e1001566.

Robinson, K. A. & Goodman, S. N. (2011). A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med, 154, 50-5.

Rombey, T., Lochner, V., Puljak, L., Könsgen, N., Mathes, T. & Pieper, D. (2020). Epidemiology and reporting characteristics of non-Cochrane updates of systematic reviews: A cross-sectional study. Res Synth Methods, 11, 471-483.

Roundtree, A. K., Kallen, M. A., Lopez-Olivo, M. A., Kimmel, B., Skidmore, B., Ortiz, Z., Cox, V. & Suarez-Almazor, M. E. (2009). Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. J Clin Epidemiol, 62, 128-37.

Royal Australian College Of General Practitioners (2025). Handbook of Non-Drug Interventions (HANDI). Royal Australian College of General Practitioners,.

Sackett, D. L., Oxman, A. D. & Eds 1(994). Cochrane Collaboration Handbook, Oxford, The Cochrane Collaboration.

Samaan, Z., Mbuagbaw, L., Kosa, D., Borg Debono, V., Dillenburg, R., Zhang, S., Fruci, V., Dennis, B., Bawor, M. & Thabane, L. (2013). A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc, 6, 169-88.

Sandström, U. & Hällsten, M. (2008). Persistent nepotism in peer-review. Scientometrics, 74, 175-189.

Schlicht, I. B., Fernandez, E., Chulvi, B. & Rosso, P. (2023). Automatic detection of health misinformation: a systematic review. J Ambient Intell Humaniz Comput, 1-13.

Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L. & Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med, 101, 507-14.

Schunemann, H. J., Fretheim, A. & Oxman, A. D. (2006). Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement. Health Res Policy Syst, 4, 22.

Schunemann, H. J., Zhang, Y. & Oxman, A. D. (2019). Distinguishing opinion from evidence in guidelines. BMJ, 366, l4606.

Schuster, M. A., Mcglynn, E. A. & Brook, R. H.(2005). How good is the quality of health care in the United States? Milbank Q, 83, 843-95.

Seddon, M. E., Marshall, M. N., Campbell, S. M. & Roland, M. O. (2001). Systematic review of studies of quality of clinical care in general practice in the UK, Australia and New Zealand. Qual Health Care, 10, 152-8.

Sharples, J. M., Oxman, A. D., Mahtani, K. R., Chalmers, I., Oliver, S., Collins, K., Austvoll-Dahlgren, A. & Hoffmann, T. (2017). Critical thinking in healthcare and education. BMJ, 357, j2234.

Shekelle, P. G., Ortiz, E., Rhodes, S., Morton, S. C., Eccles, M. P., Grimshaw, J. M. & Woolf, S. H. (2001). Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA, 286, 1461-7.

Shepherd, J., Frampton, G. K., Pickett, K. & Wyatt, J. C. (2018). Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency. PLoS One, 13, e0196914.

Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S. & Moher, D. (2007). How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med, 147, 224-33.

Siontis, K. C., Hernandez-Boussard, T. & Ioannidis, J. P. (2013). Overlapping meta-analyses on the same topic: survey of published studies. BMJ, 347, f4501.

Smith, R. (2006). Peer review: a flawed process at the heart of science and journals. J R Soc Med, 99, 178-82.

Smith, R. (2010). Classical peer review: an empty gun. Breast Cancer Res, 12 Suppl 4, S13.

Smith, R., Chen, K., Winner, D., Friedhoff, S. & Wardle, C. (2023). A systematic review of Covid-19 misinformation interventions: Lessons learned. Health Aff, 42, 1738-1746.

Squires, J. E., Cho-Young, D., Aloisio, L. D., Bell, R., Bornstein, S., Brien, S. E., Decary, S., Varin, M. D., Dobrow, M., Estabrooks, C. A., Graham, I. D., Greenough, M., Grinspun, D., Hillmer, M., Horsley, T., Hu, J., Katz, A., Krause, C., Lavis, J., Levinson, W., Levy, A., Mancuso, M., Morgan, S., Nadalin-Penno, L., Neuner, A., Rader, T., Santos, W. J., Teare, G., Tepper, J., Vandyk, A., Wilson, M. & Grimshaw, J. M. (2022). Inappropriate use of clinical practices in Canada: a systematic review. CMAJ, 194, E279-e296.

Stacey, D., Lewis, K. B., Smith, M., Carley, M., Volk, R., Douglas, E. E., Pacheco-Brousseau, L., Finderup, J., Gunderson, J., Barry, M. J., Bennett, C. L., Bravo, P., Steffensen, K., Gogovor, A., Graham, I. D., Kelly, S. E., Légaré, F., Sondergaard, H., Thomson, R., Trenaman, L. & Trevena, L. (2024). Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev, 1, Cd001431.

Starr, M., Chalmers, I., Clarke, M. & Oxman, A. D. (2009). The origins, evolution, and future of The Cochrane Database of Systematic Reviews. Int J Technol Assess Health Care, 25 Suppl 1, 182-95.

Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., Carpenter, J. R., Chan, A. W., Churchill, R., Deeks, J. J., Hróbjartsson, A., Kirkham, J., Jüni, P., Loke, Y. K., Pigott, T. D., Ramsay, C. R., Regidor, D., Rothstein, H. R., Sandhu, L., Santaguida, P. L., Schünemann, H. J., Shea, B., Shrier, I., Tugwell, P., Turner, L., Valentine, J. C., Waddington, H., Waters, E., Wells, G. A., Whiting, P. F. & Higgins, J. P. (2016). ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 355, i4919.

Stewart, L., Moher, D. & Shekelle, P. (2012). Why prospective registration of systematic reviews makes sense. Syst Rev, 1, 7.

Stratil, J. M., Biallas, R. L., Movsisyan, A., Oliver, K. & Rehfuess, E. A. (2024). Development of an overarching framework for anticipating and assessing adverse and other unintended consequences of public health interventions (CONSEQUENT): a best-fit framework synthesis. BMJ Public Health, 2.

Suarez-Lledo, V. & Alvarez-Galvez, J. (2021). Prevalence of health misinformation on social media: systematic review. J Med Internet Res, 23, e17187.

Swire-Thompson, B. & Lazer, D. (2020). Public health and online misinformation: challenges and recommendations. Annu Rev Public Health, 41, 433-451.

Tabatabavakili, S., Khan, R., Scaffidi, M. A., Gimpaya, N., Lightfoot, D. & Grover, S. C. (2021). Financial Conflicts of Interest in Clinical Practice Guidelines: A Systematic Review. Mayo Clin Proc Innov Qual Outcomes, 5, 466-475.

Tamblyn, R., Girard, N., Qian, C. J. & Hanley, J. (2018). Assessment of potential bias in research grant peer review in Canada. CMAJ, 190, E489-e499.

Tan, A., Nagraj, S. K., Nasser, M., Sharma, T. & Kuchenmüller, T. (2022). What do we know about evidence-informed priority setting processes to set population-level health-research agendas: an overview of reviews. Bull Natl Res Cent, 46, 6.

Tendal, B., Vogel, J. P., Mcdonald, S., Norris, S., Cumpston, M., White, H., Leder, K., Navarro, D. F., Cheyne, S., Chakraborty, S., Murano, M., Millard, T., Callesen, H. E., Islam, R. M., Elliott, J. & Turner, T. (2021). Weekly updates of national living evidence-based guidelines: methods for the Australian living guidelines for care of people with COVID-19. J Clin Epidemiol, 131, 11-21.

The James Lind Alliance. James Lind Alliance Priority Setting Partnerships [Online]. Available: https://www.jla.nihr.ac.uk/ [Accessed 4 June 2025].

Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., Mcdonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T. & Elliott, J. (2017). Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol, 91, 31-37.

Thornton, A. & Lee, P. (2000). Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol, 53, 207-16.

Trueblood, J. S., Allison, D. B., Field, S. M., Fishbach, A., Gaillard, S. D. M., Gigerenzer, G., Holmes, W. R., Lewandowsky, S., Matzke, D., Murphy, M. C., Musslick, S., Popov, V., Roskies, A. L., Ter Schure, J. & Teodorescu, A. R. (2025). The misalignment of incentives in academic publishing and implications for journal reform. Proc Natl Acad Sci U S A, 122, e2401231121.

Turner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C. & Moher, D. (2012). Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev, 11, Mr000030.

Vandvik, P. O., Brandt, L., Alonso-Coello, P., Treweek, S., Akl, E. A., Kristiansen, A., Fog-Heen, A., Agoritsas, T., Montori, V. M. & Guyatt, G. (2013). Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest, 144, 381-389.

Voogt, J. & Roblin, N. P. (2012). A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies. J Curric Stud, 44, 299-321.

Wang, D. & Barabási, A.-L. (2021). The Science of Collaboration. The Science of Science. Cambridge: Cambridge University Press.

Whiting, P., Savović, J., Higgins, J. P., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J. & Churchill, R. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol, 69, 225-34.

Wiles, L. K., Kay, D., Luker, J. A., Worley, A., Austin, J., Ball, A., Bevan, A., Cousins, M., Dalton, S., Hodges, E., Horvat, L., Kerrins, E., Marker, J., Mckinnon, M., Mcmillan, P., Pinero De Plaza, M. A., Smith, J., Yeung, D. & Hillier, S. L. (2022). Consumer engagement in health care policy, research and services: A systematic review and meta-analysis of methods and effects. PLoS One, 17, e0261808.

World Health Organization (1978). Declaration of Alma-Ata. Geneva: World Health Organization.

World Health Organization (2005). Ministerial Summit on Health Research. In Fifty-eighth World Health Assembly: Resolutions and Decisions. Geneva: WHO.

World Health Organization (2018). International Standards for Clinical Trial Registries – Version 3.0. Geneva: World Health Organization.

World Health Organization (2021). Evidence, policy, impact. WHO guide for evidence-informed decision-making. Geneva: World Health Organization.

World Health Organization 2022a. Evidence as a catalyst for policy and societal change: towards more equitable, resilient and sustainable global health. Meeting report of the WHO Global Evidence-to-Policy Summit, 15–17 November 2021. Geneva: World Health Organization.

World Health Organization (2022b). Evidence. Knowledge Translation. Impact: Closing evidence-to-policy gaps. Geneva: World Health Organization.

World Health Organization (2023). Public health milestones through the years [Online]. Available: https://www.who.int/campaigns/75-years-of-improving-public-health/milestones#year-1945 [Accessed 6 June 2025].

Wuchty, S., Jones, B. F. & Uzzi, B. (2007). The increasing dominance of teams in production of knowledge. Science, 316, 1036-9.

Xun, Y., Estill, J., Khabsa, J., Florez, I. D., Guyatt, G. H., Norris, S. L., Lee, M. S., Ozaki, A., Qaseem, A., Schünemann, H. J., Shao, R., Chen, Y. & Akl, E. A. (2024). Reporting Conflicts of Interest and Funding in Health Care Guidelines: The RIGHT-COI&F Checklist. Ann Intern Med, 177, 782-790.

Yank, V., Rennie, D. & Bero, L. A. (2007). Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ, 335, 1202-5.

Yavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J. & Ravaud, P. (2012). Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med, 9, e1001308.

Zhang, S., Zhou, H. & Zhu, Y. (2024). Have we found a solution for health misinformation? A ten-year systematic review of health misinformation literature 2013-2022. Int J Med Inform, 188, 105478.

Zorzela, L., Loke, Y. K., Ioannidis, J. P., Golder, S., Santaguida, P., Altman, D. G., Moher, D. & Vohra, S.(2016). PRISMA harms checklist: improving harms reporting in systematic reviews. BMJ, 352, i157.