Introduction

A thousand years ago, the Persian physician Ibn Sina issued a strong warning about the pitfalls of using animals to develop medicines for humans: ‘Experiments should be carried out on the human body. If the experiment is carried out on the bodies of [other animals] it is possible that it might fail for two reasons: the medicine might be hot compared to the human body and be cold compared to the lion’s body or the horse’s body… The second reason is that the quality of the medicine might mean that it would affect the human body differently from the animal body … These are the rules that must be observed in finding out the potency of medicines through experimentation. Take note!’ (Ibn Sina 1012 CE).

While Ibn Sina’s advice to focus on humans was for the most part heeded for hundreds of years, the twentieth century saw an astounding increase in the number of animals used in research intended to have application to humans. Yet this has not borne fruit, and Ibn Sina’s advice remains relevant today, as Nasser et al. (2007) point out. A millennium after Ibn Sina wrote The Canon of Medicine, Dutch neurologists Janneke Horn and Martien Limburg published a systematic review of clinical trials of the neuroprotectant drug nimodipine (Horn and Limburg 2001; Horn 2001). It had been thought that nimodipine would reduce some of the adverse consequences of stroke, but the systematic review did not find this. The neurologists were bewildered; the clinical trials had only proceeded because studies in animals had indicated that the drug would be beneficial. Intrigued, they set out to retrieve all the relevant animal studies and examine them more closely. To their surprise, when these studies were reviewed systematically, the apparently beneficial effect in animals ‘disappeared’. Their systematic review was one of the first to be conducted in the field of animal research and clearly demonstrated the value of synthesising studies which, if considered individually, might be misleading. Ibn Sina had warned about the problems of extrapolating findings from one species to another, but systematic reviews were pointing to further problems – limitations in the way research was being conducted and interpreted.

Two decades earlier, in what has been described as a rallying call for evidence synthesis (Chalmers 2006; Clarke 2015), the Scottish doctor, Archie Cochrane had written: ‘It is surely a great criticism of our profession that we have not organised a critical summary, by speciality or subspeciality, adapted periodically, of all relevant randomised controlled trials.’ (Cochrane 1979). Collaborative reviews of certain clinical fields had been taking place sporadically (e.g. Chalmers et al 1989) but it was not until 1989 that the term ‘systematic review’ was first used. Not long after, in 1993, the international Cochrane Collaboration was established to generate robust evidence by systematically reviewing clinical trial data, so that informed decisions could be made about healthcare (Clarke 2015). By the early 2000s, systematic reviews were a familiar entity within clinical research and considered vital for addressing uncertainties about the effects of treatments (The James Lind Library 1.1). They provide a rigorous methodology for evaluating and synthesising a body of research so that, based on all the available relevant evidence, conclusions may be reached about an intervention’s effects.

UK

In 2002, clinical epidemiologists Peter Sandercock and Ian Roberts published a short commentary in the Lancet which made the case for conducting systematic reviews of animal studies on a more routine basis (Sandercock and Roberts 2002). The authors observed that about one in every 1000 MEDLINE records about human research was tagged as a meta-analysis compared with only one in every 10,000 records about animal research. Referring to the nimodipine research, they noted that, had the animal evidence been reviewed earlier, the 6400 or so patients who participated in the 22 clinical trials might have been spared the potential risk and inconvenience of taking part, never mind the costs of conducting the trials.

In 2002, Pandora Pound was working in the School of Social and Community Medicine (now Population Health Sciences) at the University of Bristol. With a background in the sociology of medicine, Pound was interested in the contribution of animal research to human medicine and had been discussing this with Susan Green, director of the patient safety charity SABRE, wondering whether systematic reviews of animal studies might  throw light on the issue. Pound and epidemiologist Shah Ebrahim, who was head of the School at the time, decided to investigate, and invited clinical epidemiologists Ian Roberts, Peter Sandercock and Michael Bracken to collaborate. The result was a 2004 paper published in the BMJ entitled ‘Where is the evidence that animal research benefits humans?’ (Pound et al. 2004). The authors argued that little data were available to support the use of animals in preclinical research and suggested that systematic reviews of animal studies might generate the evidence necessary to answer the question they posed.

Since animal research was particularly controversial at that time, Ebrahim, out of courtesy, gave advance notice of the paper’s publication to the Dean of Medicine and the Director of Research at the University of Bristol. Because the School was in receipt of Medical Research Council funding, Ebrahim also discussed it with Colin Blakemore, then Chief Executive of the MRC. All of those consulted strongly advised against publishing the paper, declaring that it would provide ammunition for the anti-vivisection movement and that it constituted an attack on essential biomedical research. The authors were also advised that they were not in a position to understand animal experiments. Disturbingly, Ebrahim was told that publishing the paper would not enhance his career prospects. In the early 2000s tensions were running high. Many laboratories, including Huntingdon Life Sciences in Cambridgeshire (now Harlan Laboratories), were the focus of regular anti-vivisection protests and a small number of activists had been imprisoned for criminal activity. But a Select Committee inquiry into the use of animals in scientific research (commissioned by the House of Lords) concluded that animal experimentation was ‘a valuable research method which has proved itself over time’ (Select Committee 2002). Within this febrile and polarised atmosphere few academics questioned animal research for fear of being branded anti-vivisectionist (the press having successfully associated this with criminality). All of this helped solidify the long-held view of preclinical animal research as a ‘sacred cow’ – a practice beyond criticism or challenge.

Unsurprisingly then, the paper, which was reported in the national media, met with some hostility. Mark Henderson, then science correspondent at The Times newspaper (now Director of Corporate Affairs at the Wellcome Trust) attempted to publicly discredit the authors by referring to them as ‘the anti-vivisection lobby, or at least its law-abiding element’ in an article for The Times (Henderson 2004), while the lobbying group ‘Coalition for Medical Progress’ (now Understanding Animal Research) attacked the paper and attempted to refute the claim that animal research lacked supporting evidence by referring to four disparate areas ‘where the advances are clear: polio, kidney dialysis, stomach ulcers and cystic fibrosis’ (Highfield 2004). In the ‘rapid responses’ to the paper published on the BMJ website, it was described by one critic as ‘spectacularly ill-judged’ and ‘scientifically invalid’ while another respondent stated that it ‘should never have been published in a peer-reviewed journal’ (Rapid Responses 2004). Nevertheless, many of the responses indicated that the paper – and the debate that it was provoking – was welcome.

In what seems unlikely to have been a coincidence, on the same day the paper was published the UK’s Royal Society published a ‘guide’, the opening lines of which claimed, ‘Humans have benefited immensely from scientific research involving animals, with virtually every medical achievement in the past century reliant on the use of animals in some way’ (Royal Society 2004). Blakemore publicly backed the Royal Society’s position, asserting ‘Animal research has contributed to virtually every area of medicine’ (BBC News website, 27th February 2004). Ironically, then, the paper’s publication was provoking exactly the sort of unsupported claims and assertions about the benefits of animal research that it highlighted as problematic. A few years later, Robert Matthews dissected claims such as those made in the Royal Society’s ‘guide’, and found that they were strikingly similar to an anonymous, one-page declaration by the US Public Health Service published in 1994, itself lacking any references or supporting evidence. After examining the statistical basis for such claims, Matthews argued that they should either be formally validated or replaced with statements capable of validation (Matthews 2008). Nevertheless, similar claims continue to be made.

It may have been that the title of the 2004 paper by Pound et al. was inflammatory, or perhaps it was simply because, for the first time in a highly respected medical journal, a group of scientists was raising doubts about the evidence underlying the practice of animal research. Nevertheless, despite its vilification within much of the animal research community, the paper kick-started a number of preclinical systematic reviews. Another positive outcome was that, as a result of Ebrahim’s discussions with Blakemore, the MRC provided a small pot of money for a comparison of systematic reviews of animal and human studies. A team led by clinical neurologist Malcolm Macleod successfully bid for this funding and went on in 2004 to form CAMARADES (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies) at Edinburgh University.

CAMARADES

The CAMARADES collaboration, based in Edinburgh, conducts preclinical systematic reviews and provides support for other researchers conducting systematic reviews and meta-analyses of animal studies. The collaboration was first established to address ‘translational failures’ in preclinical stroke research, but soon broadened its scope to include other diseases. CAMARADES currently has 5 national co-ordinating centres and many more global participating centres. It aims to identify potential sources of bias in animal research, generate recommendations for improving the design and reporting of animal studies, and develop meta-analysis methodology so that it better applies to animal studies. As well as acting as a repository for completed reviews, it provides guidance and tools such as SyRF (Systematic Review Facility), a free online systematic review platform.

In 2007, Malcolm Macleod’s team published the findings of their comparison of human and animal systematic reviews in the BMJ (Perel et al. 2007). The team identified 6 interventions for which there was unambiguous systematic review evidence of a treatment effect for humans: corticosteroids for brain injury, antenatal corticosteroids for neonatal respiratory distress, bisphosphonates for osteoporosis, antifibrinolytics for haemorrhage, thrombolytics for stroke and a neuroprotective agent (tirilazad) for stroke. They then searched for and systematically reviewed all published and unpublished controlled animal studies for the same 6 interventions. They assessed the methodological quality of the animal studies (based on measures taken to prevent bias) as ‘poor’ across all 6 interventions. Comparing the results from the systematic reviews of animal studies with the systematic reviews of clinical studies, the authors found that two interventions (bisphosphonates for osteoporosis, thrombolytics for stroke) were concordant, i.e. the findings from the animal studies agreed with the findings from the human studies, one intervention was partially concordant (antenatal corticosteroids for preterm delivery) and three were discordant (corticosteroids for brain injury, tirilazad for stroke and antifibrinolytics for haemorrhage). They concluded that discordance was likely to be due to bias within the animal studies, or to failure of the animal models to adequately mimic human disease and treatment. This important study is one of only a few that have used systematic review methodology to understand how and whether animal studies translate to humans.

Netherlands

Meanwhile, in 2005, veterinarian Merel Ritskes-Hoitinga took up the post of Managing Director of the central animal facility at Radboud University Medical Center, Nijmegen, a post combined with a Professorship in Laboratory Animal Science. In 2006, she founded the 3Rs (Replacement, Reduction, Refinement) Research Centre, which aimed to help researchers implement the 3Rs with a view to improving animal welfare and laboratory animal science. Over time, however, she became disillusioned about the ability of the 3Rs to improve either (Ritskes-Hoitinga 2016). In 2008 she happened to be in Portugal for a conference where she attended a presentation by Malcolm Macleod. By the time Macleod uttered his closing words, Ritskes-Hoitinga had decided that preclinical systematic reviews would provide a more promising way of improving the quality of preclinical science than the 3Rs. She then met with Iain Chalmers, one of the founders of the Cochrane Collaboration, as well as Susan Green from SABRE, who was concerned about the ways in which poorly conducted animal research eventually impacted patients (Green 2015).  Ritskes-Hoitinga went on to found the Systematic Review Centre for Laboratory Animal Experimentation, better known as SYRCLE, in 2012 at Radboud University in the Netherlands.

In 2011, she invited members of the Dutch Parliament to visit her at the central animal facility in Nijmegen to discuss the need for preclinical systematic reviews. The visit made a strong impression on the members and, perhaps also influenced by the Partij voor de Dieren (Party for the Animals) which at the time had almost 2% of the votes and two of the 150 House of Representatives’s seats. A motion was passed in the Dutch Parliament the following year asking the government to make systematic reviews the norm in preclinical research as they had become in the clinical field. A second motion was passed in the Dutch Parliament in 2014, seeking to make systematic reviews a compulsory part of laboratory animal scientists’ training.

SYRCLE

SYRCLE launched with an international symposium in 2012 at Radboud University, Nijmegen. The symposium explored why scientific standards appeared to be lower for preclinical animal research despite its intended application to human health. Speakers focused on the challenges involved in adapting the methodology of systematic review to the preclinical field, so that animal research could be held to the same high standards as clinical research. The symposium, the first of its kind, had Macleod as the keynote speaker, with other speakers including Ian Roberts, Michael Bracken, Marlies Leenaars, Carlijn Hooijmans and Ritskes-Hoitinga. Esther Ouwehand, politician for the Dutch Party for Animals, sent a video message. Chalmers concluded the conference, noting that with more than a hundred participants, the gathering was bigger than the first Cochrane meetings. He also reminded participants of the need to honour Janneke Horn, as one of the first people to conduct a preclinical systematic review. (Five years later she was awarded the FEDERA award for her pioneering preclinical systematic review.)

The Evidence-Based Toxicology Collaboration, which advocated systematic reviews in toxicology, was founded in the United States at around the same time, and a close relationship developed with SYRCLE. The National Toxicology Program in the US actively promotes methods development and the harmonisation of systematic review approaches, and in collaboration with SYRCLE, developed a database of systematic reviews of animal studies (Langendam et al. 2021). SYRCLE also joined Evidence Synthesis International which was formed later to bring together organisations that were generating or using evidence syntheses. CAMARADES also collaborated with these groups, so relationships were formed that spanned the globe.

Realising that laboratory animal scientists would need training and support to conduct and facilitate systematic reviews, the SYRCLE group developed tools such as search filters for researchers to use when searching for animal studies (Hooijmans et al. 2010a; de Vries et al. 2011; de Vries et al. 2013), as well as guidance on performing literature searches (Leenaars et al. 2012). They also provided guidance on evaluating the risk of bias in studies included in systematic reviews (Hooijmans et al. 2014a), performing meta-analysis on animal studies (Hooijmans et al. 2014b), grading the evidence (Hooijmans et al. 2018), and preparing, registering and publishing systematic review protocols (de Vries et al. 2015). As a result of the latter, the publication of preclinical systematic review protocols gained momentum, meaning that in addition to human studies, Prospero began registering and maintaining a database of protocols of preclinical systematic reviews relevant to human health. Additionally, the SYRCLE group published a checklist of items to include when reporting animal studies (both to improve reporting and to make subsequent systematic reviews easier) (Hooijmans et al. 2010b), while a 2013 paper in PlosMedicine highlighted the advantages of preclinical systematic reviews and the progress that had been made in the field (Hooijmans and Ritskes-Hoitinga 2013). This paper emphasised the importance of the routine funding and conduct of preclinical systematic reviews to maximise transparency and avoid waste and unnecessary duplication. It also noted that cooperation between multiple stakeholders, including patients, was essential for further progress. A later paper made the case that preclinical systematic reviews can lead directly to achieving the 3Rs, as well as better science (Ritskes-Hoitinga and Van Luijk, 2019).

Following some negative feedback from academia in response to the motions passed on systematic reviews, the Dutch Parliament commissioned some research to find out why. The research revealed that many scientists were unfamiliar with the methodology or value of systematic reviews (Swankhuisen and Smit, 2014). As a result, the Dutch Health Funder, ZonMw, created a ‘Synthesis of Evidence’ funding stream within the ‘More Knowledge with Fewer Animals’ programme, defining preclinical systematic reviews as animal-free innovations. It funded SYRCLE to offer one day workshops for researchers wanting to learn how to conduct systematic reviews and paid two months’ salary to those who went on to conduct a review. From 2012, when the funding started, until 2020, when the programme was evaluated, around 400 researchers had participated in 21 one-day workshops, learning the theory and practice of systematic reviews. A total of eighty-eight participants began a systematic review with training from SYRCLE, and by the end of 2020, thirty-eight had published their reviews. The programme evaluation concluded that the training raised awareness about the value of systematic reviews for research and researchers, as well as the need to bring greater scientific rigour to the conduct of animal studies, leading to an improvement in animal research quality (Menon et al. 2021).

In 2016, following a request from the Dutch Ministry, SYRCLE began offering e-learning courses in how to conduct preclinical systematic reviews. By 2020 this e-learning was being used by more than 4000 participants from 65 countries globally. An international ambassador network was also established to stimulate the adoption of preclinical systematic review methodology. Currently 40 research ‘ambassadors’ in 16 countries have committed to encouraging its use locally.

Although there was initially some resistance to systematic reviews among preclinical researchers, Ritskes-Hoitinga calculated that there was a 35% reduction in the number of laboratory animals used at Radboud University since it began employing the methodology, while animal use in the Netherlands as a whole reduced over the same period by 15%. (In the UK, however, laboratory animal numbers rose to a peak of 4.4 million in 2015 before declining again, with 2.88 million animals being used in 2020.) In 2017, in recognition of the contribution of systematic reviews to higher scientific standards in the preclinical field, SYRCLE won the Cochrane REWARD 2nd Prize for reducing waste and increasing value in research.

Ritskes-Hoitinga had long wanted the preclinical field to join the Cochrane Collaboration and in 2010 she organised a workshop at a meeting of the Federation of European Laboratory Animal Science Associations in Helsinki to discuss the possibility of joining. A further meeting was held at the 8th World Congress on Alternatives and Animal Use in Life Sciences in 2011; this resulted in the Montreal Declaration on the Synthesis of Evidence to Advance the 3Rs Principles in Science, as well as a paper (Leenaars et al. 2011) on the many advantages of preclinical systematic reviews, but no progress was made towards joining the Cochrane Collaboration. In 2013, SYRCLE was advised to apply to become a Cochrane Animal Study Methods Group in collaboration with CAMARADES. Consequently, an application was prepared, and the two groups met in London, paving the way for membership. Ritskes-Hoitinga and colleagues also made the case for an animal study methods group in an editorial published in the Cochrane Library in 2014 (Ritskes-Hoitinga et al. 2014). By this time, however, the Cochrane Collaboration had started to reorganise, making the application process uncertain and, as yet, the focus of the Cochrane Collaboration remains solely on clinical evidence.

Evidence from systematic reviews

In 2008, Michael Bracken, at Yale University, summarised some of the developments in the field, highlighting the worryingly poor quality of animal studies and making the case for more preclinical systematic reviews (Bracken 2008). Between 2005 and 2010, Korevaar et al. (2011) estimated that 163 preclinical systematic reviews had been published, while 246 were identified between 2009 and 2013 by Mueller et al. (2014). As more and more animal studies were scrutinised as part of the systematic review process, it gradually became apparent that much animal research was conducted to a low standard and was therefore unable to generate robust, reliable data. This made uncomfortable reading for animal researchers, who were found to report low rates of random allocation, allocation concealment, and blinded outcome assessment (Henderson et al. 2015; Perel et al. 2007; Hirst et al. 2014). Studies that take these accepted precautions to reduce biases are less likely to suggest differential effects than studies that do not observe these precautions. It soon became evident that large bodies of animal research had overstated the benefits of their experimental interventions. Tsilidis et al. (2013) demonstrated this clearly in the field of preclinical neurological research, as did Crossley et al. (2008) in the field of preclinical stroke research. The accumulating preclinical systematic reviews also revealed that animal samples are typically small, leading to underpowered and therefore unreliable studies, as Emily Sena, convenor of CAMARADES, showed in her 2014 overview (Sena et al. 2014). In short, systematic reviews provided overwhelming evidence that animal studies suffer from poor experimental design and a lack of scientific rigour, raising doubts about the robustness of their findings and consequently, their clinical relevance.

Selective analysis and biased outcome reporting – the practice of reporting only the most positive outcomes and analyses from among the many performed and studied – was also revealed to be a problem in animal research (Tsilidis et al. 2013). Again, this leads to an overestimate of beneficial treatment effects, ultimately creating a body of evidence with an inflated proportion of studies with positive results. Incomplete reporting was revealed to be another limiting factor. Even basic information, such as the number of animals used in experiments, was found to be missing, as was reporting on attrition (Holman et al. 2016). This – the loss of animals through death or exclusion –  can dramatically alter the results of a study and, again, have the effect of making animal studies appear more positive than they actually are. Publication bias (the phenomenon whereby studies are more likely to be published if they present ‘positive’ findings) was found to be a significant problem (Korevaar et al. 2011; Mueller et al. 2014), leading once more to the benefits of animal studies being overstated (Sena et al. 2010a). And citation bias, first reported in the clinical field (Gotzsche 1987), was found to be an issue in animal research. A German study of 109 investigator brochures, the documents presented to ethics review boards by those applying to conduct Phase I and II trials in humans, revealed that only 6% of the preclinical animal studies referenced in the brochures reported an outcome demonstrating no effect; the vast majority – 82% – were described as reporting positive findings (Wieschowski et al. 2018).

Unsurprisingly then, when scientists from Astra Zeneca reviewed 255 protocols for forthcoming animal experiments, they found that over half needed amending to ensure proper experimental design, appropriate sample sizes, and measures to control bias (Peers et al. 2014). And when pharmaceutical companies conducted in-house validation of data coming from academia, they found that much of it was irreproducible (Prinz et al. 2011), in other words, the experiments did not produce the same results when independently repeated (Begley and Ellis 2012). This problem – which has come to be known as the reproducibility crisis – is due to poor experimental design and poor scientific conduct and is compounded by incomplete reporting. A number of key papers were written on this topic, including by Ioannidis (2012), Leist and Hartung (2013) and Begley and Ioannidis (2015). Although it is beyond the scope of this article to describe them, many initiatives were developed to improve the quality of animal study design and reporting, and to address problems such as publication bias, one of the most famous of which is the ARRIVE guideline (Percie du Sert et al. 2020). Another initiative is EQIPD (European Quality in Preclinical Data), an EU consortium that has assembled preclinical researchers from both academia and industry to identify how the quality of preclinical science could be improved. One of its outputs, for example, is a systematic review of existing guidelines for preclinical animal studies, resulting in 58 recommendations (Vollert et al. 2020).

Poorly conducted, unreliable research has consequences, for humans, animals, and society. It can be dangerous. Corticosteroids, for example, were found to benefit animals with brain injury, and tirilazad was beneficial for animals with acute stroke, but both drugs increased the risk of people dying when they proceeded to clinical trials (Perel et al. 2007). Systematic reviews have also revealed a great deal of redundancy and waste in animal research. In 2010, Sena and colleagues demonstrated in a cumulative meta-analysis that the beneficial effects of tissue plasminogen activator for stroke had been well documented in animal models by 2001, but research using several thousand animals continued for several years afterwards (Sena et al. 2010b). And of course, if the results of preclinical studies are unreliable, then that research is also a waste of time, resources and animals’ lives. In 2014, the Lancet held a conference on research waste in both clinical and preclinical science, highlighting that this could be avoided at every stage of the research process, i.e. funding, conduct, and regulation (Lancet 2014). It was a clear call to action and led, among other things, to the founding of the Ensuring Value in Research (EVIR) Funder Forum, an international group of funders committed to avoiding waste and increasing the value of funded research. Ritskes-Hoitinga made contact with the forum, with the upshot that a preclinical working group was established, with Ritskes-Hoitinga as co-chair.

Also in 2014, Pound returned to the field after an absence of ten years and teamed up with Michael Bracken to review developments (Pound and Bracken 2014). Their paper, again published in the BMJ, provided an overview of the evidence accruing from preclinical systematic reviews. They noted that shortcomings in almost every aspect of the scientific design, conduct, and reporting of animal studies were contributing to an inability to translate into benefits for humans. This time their paper was warmly received, indicating that the scientific climate had changed considerably. No longer was it considered heretical to discuss the limitations or challenge the validity of animal research.

In 2018 a BMJ investigation concluded that an Oxford University research group had been selective in the reporting of their animal study results to gain funding and approval for human trials of a TB booster vaccination (Cohen 2018). The group had gained funding for the human trials, but the trials had ultimately failed. An earlier systematic review of the animal data concluded that insufficient evidence had existed to support claims about the efficacy of the vaccine booster and that the claims had been overstated (Kashangura et al. 2015). Highlighting the problem of selective outcome reporting, the BMJ commented on the group’s ‘pick and mix’ approach, claiming that some of the animal studies showing adverse effects had been omitted from the preclinical evidence. In an accompanying editorial, Ritskes-Hoitinga and colleague Kim Wever outlined steps that needed to be taken to improve the conduct and quality of preclinical research (Ritskes-Hoitinga and Wever 2018).

More recent developments

In January 2017, SYRCLE moved out of the animal facility and into the Department of Health Evidence at Radboud University. The animal facility’s users and research directors had begun to withdraw support for Ritskes-Hoitinga following an interview she gave to the Dutch newspaper Trouw in 2013, in which – having considered the evidence on the poor quality and reporting of animal studies – she stated that animal testing could be reduced by 80%. This had shocked the Dutch animal science community and two colleagues, Professors Frauke Ohl and Coenraad Hendriksen from Utrecht University had disagreed openly in a letter to the newspaper. Ritskes-Hoitinga was advised that, as the manager of an animal facility, her role was to provide a service to users, not comment on the science. In her new department she had greater freedom to investigate the evidence and embarked on a series of studies, including a collaborative project between Utrecht and Radboud Universities which attempted to identify factors contributing to translational success. A scoping review performed as part of this project found that rates of translation from animal to human studies ranged from 0-100 and appeared to be random, with no indication of factors that might increase its likelihood (Leenaars et al. 2019). At a symposium in 2019 to mark the end of the project, epidemiologist John Ioannidis stated his view that animal testing could be reduced by 90% (Ritskes-Hoitinga et al. 2020).

Around this time Ritskes-Hoitinga and Pound began to collaborate. They published on the problem of external validity of animal studies, arguing that even if all the problems of internal validity in animal research were resolved, species differences would continue to make translation to humans unreliable (Pound and Ritskes-Hoitinga 2018). In doing so, they were drawing attention to the problem that Ibn Sina had highlighted a thousand years previously. With colleague Christine Nicol, Pound also conducted a retrospective harm-benefit analysis by reanalysing the animal data from Perel et al.’s 2007 study. Using Bateson’s Cube (Bateson 1986) to weigh the harms to animals used in the research against the benefits to humans that resulted, and taking into account the importance and quality of the research studies, they concluded that fewer than 7% of the 212 animal studies scrutinised were permissible (Pound and Nicol 2018). In a later paper, Pound and Ritskes-Hoitinga highlighted that, while prospective preclinical systematic reviews (i.e. those conducted prior to human trials) allow valuable scrutiny of the preclinical animal data, they are not necessarily able to reliably predict the safety and efficacy of an intervention, or safeguard clinical trial participants (Pound and Ritskes-Hoitinga 2020). A systematic review is only as good as the studies it includes and if the primary animal studies cannot reliably predict safety and efficacy in humans, the systematic review findings will reflect this.

Despite a promising start, the number of preclinical systematic reviews remains disappointingly low, raising questions about the extent to which the evidence-based approach has been accepted within preclinical research. And although two more international symposia on systematic reviews in laboratory animal science were held, one in Edinburgh in 2013 and one in Washington in 2014, no further meetings have taken place. Then in 2021, Radboud University Medical Center suddenly decided to withdraw all funding from SYRCLE; preclinical systematic reviews were apparently no longer a priority for them.

Nevertheless, systematic reviews have been pivotal in preclinical research. In highlighting the shortcomings of animal studies, they enabled, for the first time, an open and constructive debate about the value of animal research – a debate that focused on the science, rather than the ethics of this research. For decades, scientists had side-lined any challenges to the practice of animal research as ethical rather than scientific, referring back to the 3Rs and regulations, but the challenge presented by systematic reviews came from within the scientific community and could not be ignored. Preclinical systematic reviews have not only exposed shortcomings in the internal validity of animal research (i.e. its design, conduct and reporting); in highlighting the poor track record of animal research in translation to humans, they have also exposed its lack of external validity. In this respect, two issues are now clear. First, the inability of many, if not most animal models to replicate complex human diseases; and second, the problem of species differences.

While some of the limitations of animal research can, at least in theory, be addressed over time (i.e. internal validity and some aspects of the animal models themselves), the evolved differences between species present an altogether different problem. Evolutionary theory indicates that species differences will always make the extrapolation of animal findings to humans unreliable (Perlman 2016; Pound and Ritskes-Hoitinga 2018). Bearing in mind that this is a fundamental flaw in the animal research paradigm, and moreover one that is insurmountable, are projects that aim to improve animal models and animal research a good use of scarce funding resources?

Many outside the field of animal research argue that the way forward is not to try to improve this research but to replace it with methodologies and technologies that are directly relevant to humans (Herrmann et al. 2019). New approaches based on human biology include in vitro cell models such as organoids and organs-on-a-chip, as well as those using computer simulations and artificial intelligence. While they face the challenges that any research must deal with, in other words the need to ensure internal validity and reproducibility, they cut out the ‘noise’ that animal studies introduce into clinical translation, producing data that are applicable to humans and that are therefore externally valid. Yet despite these new approaches often performing better than animal studies (see for example Dirven et al. (2021) on the relative performance of in vitro and in vivo methodologies for predicting drug-induced liver injury in humans), scientists appear reluctant to relinquish traditional practices. In a 2020 study, Pound and colleague Rebecca Ram reviewed scientists’ opinions about the limitations of animal models of stroke (Pound and Ram 2020). They found that while many viewed species differences as a significant problem in preclinical stroke research, the vast majority were reluctant to abandon their animal models, with only one of eighty authors advocating a focus on human-relevant research instead.

Consequently, two streams of research are proceeding in parallel and mostly in isolation from each other. On the one hand, research based on human biology is being advanced and developed as a direct response to the perceived limitations of animal research, and – on the other – animal research continues as usual, albeit with an eye on research improvement. In terms of the latter, CAMARADES is still going strong, with several projects underway, including the development of ‘living systematic reviews’. Nevertheless, change seems to be coming. In the UK, a couple of high profile animal laboratories are set to close, the reasons being ‘a move to using alternative technologies’ (Else 2019a), and ‘a changing scientific landscape’ (Else 2019b), while the Medical Research Council’s new ‘Experimental Medicine’ programme funds research that focuses on ‘the human as the ultimate experimental animal for improving human health’, noting that this is now possible due to advances in non-invasive techniques such as medical imaging, sensors, and ex vivo analyses. The Dutch Parliament has a specific transition programme, ‘Transitie naar Proefdiervrije Innovatie’ which aims to ensure that the Netherlands is a frontrunner in the transition to innovate without using laboratory animals. Meanwhile, the United States Environmental Protection Agency has committed to ending the use of mammals in chemical testing, also aiming be a frontrunner in the adoption of human-relevant technologies (Wheeler 2019). And in September 2021, the European Parliament voted by a stunning majority of 667 to 4 to develop a coordinated plan to replace animal experiments with innovative, non-animal methodologies (Haahr 2021). Might this be the rumblings of a scientific revolution?

Acknowledgements

We gratefully acknowledge the help of Shah Ebrahim and Michael Bracken who commented on an earlier draft of this paper.

This James Lind Library article has been republished in the Journal of the Royal Society of Medicine in 2 parts.

Ritskes-Hoitinga, Pound (2022). The role of systematic reviews in identifying the limitations of preclinical animal research, 2000–2022: part 1. JRSM 2022;115:186-192.

Ritskes-Hoitinga, Pound (2022). The role of systematic reviews in identifying the limitations of preclinical animal research, 2000–2022: part 2. JRSM 2022;115:231-235

References

Bateson P (1986). When to experiment on animals. New Scientist 109:30–32. PMID: 11655736.

BBC News website (2004). Scientists doubt animal research. 27^th February.  http://news.bbc.co.uk/1/hi/health/3489952.stm.

Begley CG, Ellis LM (2012). Drug development: Raise standards for preclinical cancer research. Nature 483:531-3. doi: 10.1038/483531a. PMID: 22460880.  https://pubmed.ncbi.nlm.nih.gov/22460880/
https://www.nature.com/articles/483531a

Begley CG, Ioannidis JP (2015). Reproducibility in science: improving the standard for basic and preclinical research. Circ Res 116:116-26. doi: 10.1161/CIRCRESAHA.114.303819. PMID: 25552691. https://pubmed.ncbi.nlm.nih.gov/25552691/

Bracken MB (2008). Why animal studies are often poor predictors of human reactions to exposure. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/why-animal-studies-are-often-poor-predictors-of-human-reactions-to-exposure/) Journal of the Royal Society of Medicine 102:120-122.

Chalmers I, Enkin M, Keirse MJNC (1989). Effective care in pregnancy and childbirth. Oxford: Oxford University Press.

Chalmers I (2006). Archie Cochrane (1909-1988). JLL Bulletin: Commentaries on the history of treatment evaluation. (https://www.jameslindlibrary.org/articles/archie-cochrane-1909-1988/)

Clarke M (2015). History of evidence synthesis to assess treatment effects: personal reflections on something that is very much alive. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/history-of-evidence-synthesis-to-assess-treatment-effects-personal-reflections-on-something-that-is-very-much-alive/).

Cochrane AL (1979). 1931-1971: a critical review, with particular reference to the medical profession. In: Medicines for the year 2000. London: Office of Health Economics, pp 1-11.

Cohen D (2018). Oxford vaccine study highlights pick and mix approach to preclinical research. BMJ. 360:j5845. doi: 10.1136/bmj.j5845. PMID: 29321165. https://pubmed.ncbi.nlm.nih.gov/29321165/.
https://www.bmj.com/content/360/bmj.j5845.long

Crossley NA, Sena E, Goehler J, Horn J, van der Worp B, Bath PMW, Macleod M, Dirnagl U (2008). Empirical evidence of bias in the design of experimental stroke studies: a meta-epidemiologic approach. Stroke 39:929-34. doi: 10.1161/STROKEAHA.107.498725. Epub 2008 Jan 31. PMID: 18239164. https://pubmed.ncbi.nlm.nih.gov/18239164/.

de Vries RB, Hooijmans CR, Tillema A, Leenaars M, Ritskes-Hoitinga M (2011). A search filter for increasing the retrieval of animal studies in Embase. Lab Anim 45:268-70.  doi: 10.1258/la.2011.011056. Epub 2011 Sep 2. PMID: 21890653; PMCID: PMC3175570. https://pubmed.ncbi.nlm.nih.gov/21890653/

de Vries RB, Hooijmans CR, Tillema A, Leenaars M, Ritskes-Hoitinga M (2013). Updated version of the Embase search filter for animal studies. Lab Anim 48:88. doi: 10.1177/0023677213494374. Epub 2013 Jul 8. PMID: 23836850. https://pubmed.ncbi.nlm.nih.gov/23836850/

de Vries RBM, Hooijmans, CR, Langendam MW, van Luijk J, Leenaars M, Ritskes-Hoitinga M, Wever KE (2015). A protocol format for the preparation, registration and publication of systematic reviews of animal intervention studies. Evidence-based Preclinical Medicine 2:1-9 e00007. https://onlinelibrary.wiley.com/doi/full/10.1002/ebm2.7

Dirven H, Vist GE, Bandhakavi S, Mehta J, Fitch SE, Pound P, Ram R, Kincaid B, Leenaars CHC, Chen M, Wright RA, Tsaioun K (2021). Performance of preclinical models in predicting drug-induced liver injury in humans: a systematic review. Sci Rep 11:6403. doi: 10.1038/s41598-021-85708-2. PMID: 33737635; PMCID: PMC7973584. https://pubmed.ncbi.nlm.nih.gov/33737635/

Else H (2019a). Genomics institute to close world-leading animal facility. Nature 569:612. Doi: 10.10138/d41586-019-01685-7. PMID: 31142871. https://pubmed.ncbi.nlm.nih.gov/31142871/
https://www.nature.com/articles/d41586-019-01685-7

Else H (2019b). Proposal to close UK mouse-research centre is ‘major threat’. Nature, 26 June. doi: https://doi.org/10.1038/d41586-019-02002-y https://www.nature.com/articles/d41586-019-02002-y

Gøtzsche PC (1987). Reference bias in reports of drug trials. BMJ 295:654-656. PMID: 3117277; PMCID: PMC1257776
https://pubmed.ncbi.nlm.nih.gov/3117277/

Green SB (2015). Can animal data translate to innovations necessary for a new era of patient-centred and individualised healthcare? Bias in preclinical animal research. BMC Med Ethics 16:53.  https://doi.org/10.1186/s12910-015-0043-7 PMID: 26215508; MPCID: PMC4517563.
https://pubmed.ncbi.nlm.nih.gov/26215508/

Haahr, T. (2021) MEPs demand EU action plan to end the use of animals in research and testing (Press Release) European Parliament. 16 September. https://www.europarl.europa.eu/news/en/press-room/20210910IPR11926/meps-demand-eu-action-plan-to-end-the-use-of-animals-in-research-and-testing.

Henderson, Mark. (2004) Junk medicine: anti-vivisection campaigners. The Times, March 20. https://www.thetimes.co.uk/article/junk-medicine-anti-vivisection-campaigners-vjr6s7zq5n2

Henderson VC, Demko N, Hakala A, MacKinnon N, Federica CA, Fergusson D, Kimmelman J. (2015). A meta-analysis of threats to valid clinical inference in preclinical research of sunitinib. Elife 4:e08351. doi:10.7554/eLife.08351. PMID: 26460544; PMCID: PMC4600817. https://pubmed.ncbi.nlm.nih.gov/26460544/

Herrmann K, Pistollato F, Stephens ML (2019). Beyond the 3Rs: Expanding the use of human-relevant replacement methods in biomedical research. ALTEX 36:343-352. doi: 10.14573/altex.1907031. PMID: 31329258.  https://pubmed.ncbi.nlm.nih.gov/31329258/. https://www.altex.org/index.php/altex/article/view/1301.

Highfield, Roger. (2004) Experiments on animals should end, say doctors. Daily Telegraph, February 27.

Hirst JA, Howick J, Aronson JK, Roberts N, Perera R, Koshiaris C, Heneghan C (2014). The need for randomization in animal trials: an overview of systematic reviews. PLoS One 6;9:e98856. doi: 10.1371/journal.pone.0098856. PMID: 24906117; PMCID: PMC4048216. https://pubmed.ncbi.nlm.nih.gov/24906117/

Hooijmans CR, Tillema A, Leenaars M, Ritskes-Hoitinga M (2010a). Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed. Lab Anim 44:170-5. doi: 10.1258/la.2010.009117. Epub Jun 15. PMID: 20551243; PMCID:
PMC3104815. https://pubmed.ncbi.nlm.nih.gov/20551243/

Hooijmans CR, Leenaars M, Ritskes-Hoitinga M (2010b). A gold standard publication checklist to improve the quality of animal studies, to fully integrate the Three Rs, and to make systematic reviews more feasible. Altern Lab Anim 38:167-82. doi: 10.1177/026119291003800208. PMID: 20507187. https://pubmed.ncbi.nlm.nih.gov/20507187/.

Hooijmans CR, Ritskes-Hoitinga M. (2013) Progress in using systematic reviews of animal studies to improve translational research. PLoS Med 10:e1001482. https://doi.org/10.1371/journal.pmed.1001482. PMID: 23874162; PMCID: PMC3712909
https://pubmed.ncbi.nlm.nih.gov/23874162/

Hooijmans CR, Rovers MM, de Vries RB, Leenaars M, Ritskes-Hoitinga M, Langendam MW (2014a). SYRCLE’s risk of bias tool for animal studies. BMC Med Res Methodol 14:43. doi: 10.1186/1471-2288-14-43. PMID: 24667063; PMCID: PMC4230647.
https://pubmed.ncbi.nlm.nih.gov/24667063/

Hooijmans CR, IntHout J, Ritskes-Hoitinga M, Rovers MM (2014b). Meta-analyses of animal studies: an Introduction of a valuable instrument to further improve healthcare. ILAR Journal 55: 418–426. PMID: 25541544; PMCID: PMC4276598.  https://doi.org/10.1093/ilar/ilu042.
https://pubmed.ncbi.nlm.nih.gov/25541544/

Hooijmans CR, de Vries RBM, Ritskes-Hoitinga M, Rovers MM, Leeflang MM, IntHout J, Wever KE, Hooft L, de Beer H, Kuijpers T, Macleod MR, Sena ES, Ter Riet G, Morgan RL, Thayer KA, Rooney AA, Guyatt GH, Schünemann HJ, Langendam MW (2018). GRADE Working Group. Facilitating healthcare decisions by assessing the certainty in the evidence from preclinical animal studies. PLoS One 13:e0187271. doi: 10.1371/journal.pone.0187271. PMID: 29324741; PMCID:PMC5764235. https://pubmed.ncbi.nlm.nih.gov/29324741/

Holman C, Piper SK, Grittner U, Diamantaras AA, Kimmelman J, Siegerink B, Dirnagl U (2016). Where have all the rodents gone? The effects of attrition in experimental research on cancer and stroke. PLoS Biol 14:e1002331. doi: 10.1371/journal.pbio.1002331. PMID: 26726833; PMCID: PMC4699644. https://pubmed.ncbi.nlm.nih.gov/26726833/.

Horn J, Limburg M (2001). Calcium antagonists for ischemic stroke: a systematic review. Stroke 32:570-6. doi: 10.1161/01.str.32.2.570. PMID: 11157199. https://pubmed.ncbi.nlm.nih.gov/11157199/

Horn J (2001).  PhD thesis. Calcium antagonists in stroke. ISBN 90-9014236-3. https://www.jameslindlibrary.org/horn-j-2001/.

Ibn Sina (1012 CE). The canon of medicine.
https://www.jameslindlibrary.org/ibn-sina-c-1012-ce-c-402-ah/.

Ioannidis JP (2012). Extrapolating from animals to humans.  Sci Transl Med 4:151ps15. doi: 10.1126/scitranslmed.3004631. PMID: 22972841.
https://pubmed.ncbi.nlm.nih.gov/22972841/.

James Lind Library 1.1 Why treatment uncertainties should be addressed. (https://www.jameslindlibrary.org/essays/1-1-why-treatment-uncertainties-should-be-addressed/).

Kashangura R, Sena ES, Young T, Garner P (2015). Effects of MVA85A vaccine on tuberculosis challenge in animals: systematic review. Int J Epidemiol. 44:1970–1981. PMID: 26351306; PMCID: PMC4689998.  https://doi.org/10.1093/ije/dyv142.
https://pubmed.ncbi.nlm.nih.gov/26351306/

Korevaar DA, Hooft L, ter Riet G (2011). Systematic reviews and meta-analyses of preclinical studies: publication bias in laboratory animal experiments. Lab Anim 45:225-30. doi: 10.1258/la.2011.010121. PMID: 21737463. https://pubmed.ncbi.nlm.nih.gov/21737463/

Langendam MW, Magnuson K, Williams AR, Walker VR, Howdeshell KL, Rooney AA, Hooijmans CR (2021). Developing a database of systematic reviews of animal studies. Regul Toxicol Pharmacol 123:104940. doi: 10.1016/j.yrtph.2021.104940. Epub May 6. PMID: 33964349.
https://pubmed.ncbi.nlm.nih.gov/33964349/

Lancet Series (2014). Increasing value, reducing waste. https://www.thelancet.com/series/research.

Leenaars M, Ritskes-Hoitinga M, Griffin G, Ormandy E (2011). Background to the Montréal Declaration on the Synthesis of Evidence to Advance the 3Rs Principles in Science, as Adopted by the 8th World Congress on Alternatives and Animal Use in the Life Sciences, Montréal, Canada, on 25 August, 2011. Altex Proceedings 1/12, Proceedings of WC8, 35-8. https://proceedings.altex.org/data/2012-01/035038_GriffinL41.pdf .

Leenaars M, Hooijmans CR, van Veggel N, ter Riet G, Leeflang M, Hooft L, van der Wilt GJ, Tillema A, Ritskes-Hoitinga M (2012). A step-by-step guide to systematically identify all relevant animal studies. Lab Anim 46:24-31. doi: 10.1258/la.2011.011087. Epub 2011 Oct 28. PMID: 22037056; PMCID: PMC3265183. https://pubmed.ncbi.nlm.nih.gov/22037056/

Leenaars CHC, Kouwenaar C, Stafleu FR, Bleich A, Ritskes-Hoitinga M, De Vries RBM, Meijboom FLB (2019). Animal to human translation: a systematic scoping review of reported concordance rates. J Transl Med 17:223. doi: 10.1186/s12967-019-1976-2. PMID: 31307492; PMCID: PMC6631915. https://pubmed.ncbi.nlm.nih.gov/31307492/.

Leist M, Hartung T (2013). Inflammatory findings on species extrapolations: humans are definitely not 70-kg mice. Arch Toxicol 87:563-7. doi: 10.1007/s00204-013-1038-0. Epub 2013 Mar 19. PMID: 23503654; PMCID: PMC3604596. https://pubmed.ncbi.nlm.nih.gov/23503654/.

Matthews RA (2008). Medical progress depends on animal models – doesn’t it?  J R Soc Med 101:95-8. doi: 10.1258/jrsm.2007.070164. PMID: 18299631; PMCID: PMC2254450. https://pubmed.ncbi.nlm.nih.gov/18299631/.

Menon JML, Ritskes-Hoitinga M, Pound P, van Oort E (2021). The impact of conducting preclinical systematic reviews on researchers and their research: a mixed method case study. PLoS One. 16:e0260619. doi: 10.1371/journal.pone.0260619. PMID: 34898637; PMCID: PMC8668092. https://pubmed.ncbi.nlm.nih.gov/34898637/.

Mueller KF, Briel M, Strech D, Meerpohl JJ, Lang B, Motschall E, Gloy V, Lamontagne F, Bassler D (2014). Dissemination bias in systematic reviews of animal research: a systematic review. PLoS One 9:e116016. doi: 10.1371/journal.pone.0116016. PMID: 25541734; PMCID: PMC4277453.
https://pubmed.ncbi.nlm.nih.gov/25541734/

Nasser M, Tibi A, Savage-Smith E (2007). Ibn Sina’s Canon of Medicine: 11th century rules for assessing the effects of drugs. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/ibn-sinas-canon-of-medicine-11th-century-rules-for-assessing-the-effects-of-drugs/).

Peers IS, South MC, Ceuppens PR, Bright JD, Pilling E (2014). Can you trust your animal study data? Nat Rev Drug Discov 13: 560.  doi: 10.1038/nrd4090-c1. Epub 2014 Jun 6. PMID:24903777. https://pubmed.ncbi.nlm.nih.gov/24903777/

Percie du Sert N, Hurst V, Ahluwalia A, Alam S, Avey MT, Baker M, et al. (2020). The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research. PLoS Biol 18: e3000410. PMID: 32663219; PMCID: PMC7360023 https://doi.org/10.1371/journal.pbio.3000410.
https://pubmed.ncbi.nlm.nih.gov/32663219/

Perel P, Roberts I, Sena E, Wheble P, Briscoe C, Sandercock P, Macleod M, Mignini LE, Jayaram P, Khan KS (2007). Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ 334:197. doi: 10.1136/bmj.39048.407928.BE. Epub 2006 Dec 15. PMID: 17175568; PMCID: PMC1781970. https://pubmed.ncbi.nlm.nih.gov/17175568/ .

Perlman RL (2016). Mouse Models of Human Disease: An Evolutionary Perspective. Evol Med Public Health 1:170-6. doi:10.1093/emph/eow014. PMID: 27121451; PMCID: PMC4875775. https://pubmed.ncbi.nlm.nih.gov/27121451/

Pound P, Ebrahim S, Sandercock P, Bracken MB, Roberts I (2004). Where is the evidence that animal research benefits humans? BMJ 328:514-7. doi: 10.1136/bmj.328.7438.514. PMID: 14988196; PMCID: PMC351856. https://pubmed.ncbi.nlm.nih.gov/14988196/

Pound P, Bracken MB (2014). Is animal research sufficiently evidence based to be a cornerstone of biomedical research? BMJ 348:g3387. doi: 10.1136/bmj.g3387. PMID: 24879816. https://pubmed.ncbi.nlm.nih.gov/24879816/.

Pound P, Nicol CJ (2018). Retrospective harm benefit analysis of pre-clinical animal research for six treatment interventions. PLoS One.13(3):e0193758. doi: 10.1371/journal.pone.0193758. PMID: 29590200; PMCID: PMC5874012. https://pubmed.ncbi.nlm.nih.gov/29590200/.

Pound P, Ritskes-Hoitinga M. (2018) Is it possible to overcome issues of external validity in preclinical animal research? Why most animal models are bound to fail. J Transl Med 16:304. doi: 10.1186/s12967-018-1678-1. PMID: 30404629; PMCID: PMC6223056. https://pubmed.ncbi.nlm.nih.gov/30404629/.

Pound P, Ram R (2020). Are researchers moving away from animal models as a result of poor clinical translation in the field of stroke? An analysis of opinion papers. BMJ Open Science 4:e100041. doi:10.1136/bmjos-2019-100041. PMID: 35047687; PMCID: PMC8749304.
https://pubmed.ncbi.nlm.nih.gov/35047687/

Pound P, Ritskes-Hoitinga M (2020). Can prospective systematic reviews of animal studies improve clinical translation? J Transl Med 18:15. doi: 10.1186/s12967-019-02205-x. PMID: 31918734; PMCID: PMC6953128. https://pubmed.ncbi.nlm.nih.gov/31918734/.

Prinz F, Schlange T, Asadullah K (2011). Believe it or not: how much can we rely on published data on potential drug targets?  Nat Rev Drug Discov 10:712. doi: 10.1038/nrd3439-c1. PMID: 21892149. https://pubmed.ncbi.nlm.nih.gov/21892149/.

Rapid Responses to ‘Pound P, Ebrahim S, Sandercock P, Bracken MB, Roberts I (2004). Where is the evidence that animal research benefits humans? BMJ 328:514-7. https://www.bmj.com/content/328/7438/514/rapid-responses.

Ritskes-Hoitinga M, Leenaars M, Avey M, Rovers M, Scholten R (2014). Systematic reviews of preclinical animal studies can make significant contributions to health care and more transparent translational medicine. Cochrane Database Syst Rev 3: ED000078. doi: 10.1002/14651858.ED000078. PMID: 24719910. https://pubmed.ncbi.nlm.nih.gov/24719910/

Ritskes-Hoitinga M. Public accountability lecture (2016). 23rd April. https://www.ritskes-hoitinga.eu/bestanden/Public%20speech%20Ritskes%20final.pdf.

Ritskes-Hoitinga M, Wever K (2018). Improving the conduct, reporting, and appraisal of animal research. BMJ 360:j4935. doi: 10.1136/bmj.j4935. PMID: 29321149. https://pubmed.ncbi.nlm.nih.gov/29321149/.

Ritskes-Hoitinga M, van Luijk J (2019). How Can systematic reviews teach us more about the implementation of the 3Rs and animal welfare? Anim 9:1163. doi: 10.3390/ani9121163. PMID: 31861205; PMCID: PMC6941037. https://pubmed.ncbi.nlm.nih.gov/31861205/.
https://www.jameslindlibrary.org/ritskes-hoitinga-m-van-luijk-j-2019/.

Ritskes-Hoitinga M, Leenaars C, Beumer W, Coenen-de Roo T, Stafleu F, Meijboom FLB (2020). Improving translation by identifying evidence for more human-relevant preclinical strategies. Animals 10:1170. doi: 10.3390/ani10071170. PMID: 32664195; PMCID: PMC7401546. https://pubmed.ncbi.nlm.nih.gov/32664195/.

Royal Society (2004). The use of non-human animals in research: a guide for scientists. February
https://royalsociety.org/~/media/royal_society_
content/policy/publications/2004/9726.pdf.

Sandercock P, Roberts I (2002). Systematic reviews of animal experiments. Lancet 360:586. doi: 10.1016/S0140-6736(02)09812-4. PMID: 12241927. https://pubmed.ncbi.nlm.nih.gov/12241927/.

Select Committee on Animals in Scientific Procedures (2002). https://publications.parliament.uk/pa/ld200102/ldselect/ldanimal/150/15001.htm.

Sena ES, van der Worp HB, Bath PM, Howells DW, Macleod MR (2010a). Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol 8:e1000344. doi: 10.1371/journal.pbio.1000344. PMID: 20361022; PMCID: PMC2846857.
https://pubmed.ncbi.nlm.nih.gov/20361022/.

Sena ES, Briscoe CL, Howells DW, Donnan GA, Sandercock PA, Macleod MR (2010b). Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: systematic review and meta-analysis. J Cereb Blood Flow Metab 30:1905-1913. doi:10.1038/jcbfm.2010.116. PMID: 20648038; PMCID: PMC3002882. https://pubmed.ncbi.nlm.nih.gov/20648038/

Sena ES, Currie GL, McCann SK, Macleod MR, Howells DW (2014). Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J Cereb Blood Flow Metab 34:737-42. doi: 10.1038/jcbfm.2014.28. Epub 2014 Feb 19. PMID: 24549183; PMCID: PMC4013765. https://pubmed.ncbi.nlm.nih.gov/24549183/.

Swankhuisen C, Smit I (2014). Systematic reviews in the laboratory animal domain.
http://tabularasa.nl/wp-content/uploads/2014/04/Tabula-Rasa-2014-Systematic-reviews-in-the-laboratory-animal-domain.pdf.

Tsilidis KK, Panagiotou OA, Sena ES, Aretouli E, Evangelou E, Howells DW, Al-Shahi Salman R, Macleod MR, Ioannidis JP (2013). Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol 11:e1001609. doi: 10.1371/journal.pbio.1001609. Epub 2013 Jul 16. PMID: 23874156; PMCID: PMC3712913. https://pubmed.ncbi.nlm.nih.gov/23874156/.

Vollert J, Schenker E, Macleod M, Bespalov A, Wuerbel H, Michel M, Dirnagl U, Potschka H, Waldron AM, Wever K, Steckler T, van de Casteele T, Altevogt B, Sil A, Rice ASC, The EQIPD WP3 study group members (2020). Systematic review of guidelines for internal validity in the design, conduct and analysis of preclinical biomedical experiments involving laboratory animals.
BMJ Open Science 4:e100046. doi: 10.1136/bmjos-2019-100046. PMID: 35047688; PMCID: PMC8647591. https://pubmed.ncbi.nlm.nih.gov/35047688/
https://www.jameslindlibrary.org/wp-data/uploads/2020/05/Vollert-et-al-BMJ-2020-SR-of-guidelines-experiments-animals.pdf.

Wheeler AR (2019). Memorandum: Directive to prioritize efforts to reduce animal testing. United States Environmental Protection Agency. 10 September. https://www.epa.gov/sites/default/files/2019-09/documents/image2019-09-09-231249.pdf

Wieschowski S, Chin WWL, Federico C, Sievers S, Kimmelman J, Strech D (2018). Preclinical efficacy studies in investigator brochures: Do they enable risk-benefit assessment?. PLoS Biology 16: e2004879. https://doi.org/10.1371/journal.pbio.2004879. PMID: 29621228; PMCID: PMC5886385. https://pubmed.ncbi.nlm.nih.gov/29621228/.