Introduction
Random assignment of individual participants in clinical trials entails two separate steps: (i) generating an unbiased allocation schedule; and (ii) using the schedule, without foreknowledge of upcoming allocations, to assign participants to treatment comparison groups. Both of these steps were implemented in two controlled trials done under the aegis of the British Medical Research Council (MRC) in the 1940s (Chalmers 2010): the trial of patulin for the common cold (MRC 1944; Clarke 2004), and the better known trial of streptomycin for pulmonary tuberculosis (MRC 1948; Chalmers 2010). However, the two separate steps needed to achieve unbiased treatment assignment were not identified clearly until 1955, when Austin Bradford Hill distinguished them in the 6th edition of his book Principles of Medical Statistics (Chalmers 2005). Shortly thereafter, David Cox (1958) and Peter Armitage (1960) emphasized the importance of implementing an unbiased allocation by using sealed envelopes to conceal an upcoming allocation until after a participant had been irrevocably entered into a trial. Despite early recognition of the importance of the two separate steps in unbiased allocation, however, no widely accepted term denoting the process of concealing upcoming allocations had been adopted more than half a century later.
Methodological terminology is often imprecise. Researchers may abandon the intended meaning, even with common terminology. For example, the terms ‘nomogram’ and ‘case-control’ are widely misused in research (Grimes 2008; Grimes 2009). Jargon (for example, ‘gold standard’) leads to confusing and meaningless terms (Conti 1994). Clearly, accurate communication requires unambiguous terminology.
The plethora of terminology surrounding randomized trials presents particular challenges. Trials suffer from a great deal of arcane terminology. Some examples include ‘random permuted blocks’, ‘participant retention’, ‘double- and triple-blinding’, ’masking’, ‘interim analyses’, ‘alpha spending’, ‘urn randomization’, ‘biased-coin randomization’, and ‘group sequential trials’. While most of these represent important trial processes and methods, even the most common terms may confuse or obfuscate. For example, although blinding terminology seems well ensconced, even apparently simple terminology like ‘double-blinding’ elicits inconsistent connotations. When investigators examined physician interpretations and textbook definitions of ‘double-blinding’, they found 17 unique interpretations and nine different definitions (Devereaux et al. 2001). Reporting only ‘double-blinding’ without proper elaboration leads to ambiguity (Schulz et al. 2002; Schulz and Grimes 2002).
Terminology serves another important function. It can indicate a critical methodological process that might otherwise be neglected in the conduct of a study unless highlighted by terminology. Because lack of attention to detail can impact on the conduct of trials, lack of appropriate terminology may cause harm if an important trial function has been inadequately addressed.
Despite the sheer volume of terminology in randomized trials, we discovered a terminological gap when we were assessing the quality of treatment allocation in reports of controlled trials published in journals of obstetrics and gynecology in the 1990s (Schulz et al. 1994). When writing up that research, we introduced the term ‘allocation concealment’ (Schulz et al. 1994) to denote the second of the two essential steps in achieving unbiased allocation to treatment comparison groups. In this article we have reviewed the precursors of the term ‘allocation concealment’, and its subsequent evolution and adoption from the mid-1990’s onwards.
Recognizing the need for randomization terminology
Over the years, books on randomized trials provided excellent and detailed descriptions of methods to generate randomized allocation sequences. These sequences formed randomization lists. Chapters and sections of chapters had titles such as ‘methods of randomization’, ‘constrained randomization’, ‘adaptive randomization’, ‘unequal treatment allocation’, ‘permuted block designs’, ‘biased-coin design’, ‘urn design’, and ‘stratified randomization’.
In books published before we suggested the term ‘allocation concealment’ in 1994, chapters describing the mechanisms for implementing allocation sequences were rare. Moreover, scant information was devoted to those mechanisms elsewhere in the books. For example, one of the most detailed discussions we found was a ‘Mechanics of Randomization’ section from a book published in 1985, but it was less than two pages in a 17-page chapter (Friedman et al. 1985).
As we have noted, some trialists recognized the importance of mechanisms for implementing allocation sequences long ago, including the use of sealed envelopes (Cox 1958; Armitage 1960; Hill 1962; Chalmers 2005). In meetings with IC and KS in 1992, Richard Peto noted that he had emphasized (Peto 1987) that randomization should incorporate a mechanism to prevent investigators, health care providers, and participants from foreknowledge of upcoming assignments, but he had not come up with a term for the mechanism.
‘Randomization blinding’
Thomas Chalmers and colleagues provided key inspiration. In the early 1980s, they had not only recognised the importance of the assignment mechanism, but had termed it ‘randomization blinding’ (Chalmers et al. 1981). They suspected that bias introduced into “…studies in which assignment of controls is less blinded may be explained by bias in the selection or rejection of patients when the treatment to be given is known or suspected at the time of assignment” (Chalmers et al. 1983). Thus, without adequate ‘randomization blinding’, selection bias could occur. To investigate their suspicions they used trials of treatments for acute myocardial infarction, using case-fatality rates as the outcome. They defined ‘blinded randomization’ as ‘opaque envelopes’, ‘a telephone call to a statistical center’, or ‘a prearranged order of blinded medications labeled consecutively by the pharmacy’ (Chalmers et al. 1983). They defined ‘unblinded randomization’ as “assignment from an open table of random numbers, according to date of birth or chart number, or by some other variably random system in which patients could present for study in a chance order but be selected or rejected after the physician knew the treatment assignment” (Chalmers et al. 1983). They found that trials using unblinded randomization yielded larger estimates of treatment effects, which were more often statistically significant, than those in trials using blinded randomization. Their findings provided the strongest evidence at that time that ‘unblinded randomization’ was associated with empirical evidence suggestive of bias in ‘randomized’ trials.
The ‘blinding’ terms ‘blinded randomization’ and ‘unblinded randomization’ are confusing because they address a mechanism within treatment allocation, not whether a trial is subsequently blinded to treatment identity. For example, a trial may have ‘blinded randomization’ as defined by Thomas Chalmers and his colleagues (Chalmers et al. 1983), but, after allocation, investigators and participants may not have been blinded to treatment assignments. Perhaps because of confusion between ‘randomization blinding’ and ‘treatment blinding’ or ‘outcome measurement blinding’, adoption of the ‘blinded randomization’ and ‘unblinded randomization’ terminology remained rare. Moreover, their definitions of ‘blinded randomization’ and ‘unblinded randomization’ were insufficiently specific. For example, although referring to “opaque envelopes”, they did not mention additional precautions such as sequential numbering, and their definitions and analysis neglected the common occurrence of published reports that had not described the implementation mechanism of the random allocation schedule.
As noted by commentators (Gillman and Runyan 1984), the analysis used by Thomas Chalmers and his colleagues (1983) could not take account of the different types of treatments in their blinded and unblinded randomization categories, thus leaving their results potentially confounded by treatment. For example, most beta-blocker trials were in the blinded randomization group while most antithrombotic agent trials were in the unblinded randomization group (Gillman and Runyan 1984). Second, they did not account for other trial characteristics, such as generation of the allocation sequence and blinding of treatments. Third, their study left unanswered the status of all those trials whose published reports did not provide any information about the implementation mechanism for random allocation. That was a critical issue, because poorly reported trials then predominated in the literature (Mosteller et al. 1980; DerSimonian et al. 1982; Altman and Doré 1990; Tyson et al. 1983), as they do now (Hopewell et al. 2010).
‘Bias-reducing allocation’
In 1990, Altman and Doré improved terminology and definitional clarity (Altman and Doré 1990). Instead of using the terms ‘blinded randomization’ and ‘unblinded randomization’, they termed the process ‘bias-reducing allocation’, using this definition: “The mechanism of treatment allocation should be designed to avoid bias: suitable methods are central randomization, coded drugs prepared by the pharmacy, and use of a series of numbered opaque sealed envelopes”. Their categorization of reports was based on these definitions as ‘yes’, ‘no’, or ‘not specified’.
The Altman and Doré terminology and analysis were improvements over that suggested by Thomas Chalmers and colleagues (1983) because they avoided derivatives of the term ‘blind’, specified that envelopes should be sequentially numbered and sealed, and included a ‘not specified’ option. The latter applied to almost half of the responses in their study and to three-quarters of the reports of randomized trials in PubMed (Hopewell et al. 2010). For anyone appraising the quality of the literature, this ‘not specified’ category provides an important perspective for evaluating quality.
‘Allocation concealment’
We needed a replacement term for ‘randomization blinding’. Although Altman and Doré had improved the terminology by introducing ‘bias-reducing allocation’, some confusion remained. For example, ‘bias-reducing allocation’ could be confused with the process of generating a random allocation sequence, or indeed, the entire randomization process. Accordingly, we sought further terminological improvement in our review of reports of trials in obstetrics and gynecology (Schulz et al. 1994).
Without any recognized method for introducing terminology, we relied upon a subjective, iterative, discussion process. We avoided any terminology associated with ‘blinding’ and explored the dictionary for words to capture the assignment process. After long deliberations, KS, IC, DG, and DA proposed ‘allocation concealment’. This implied a mechanism to prevent foreknowledge of upcoming assignments which avoid any reference to ‘blinding’. We first used the term ‘allocation concealment’ in 1994 (Schulz et al. 1994). Because the term was new, we explained our rationale for preferring ‘allocation concealment’ in detail:
“The reduction of bias in trials depends crucially upon preventing foreknowledge of treatment assignment. Concealing assignments until the point of allocation prevents foreknowledge, but that process has sometimes been confusingly referred to as ‘randomization blinding’. This term, if used at all, has seldom been distinguished clearly from other forms of blinding (masking) and is unsatisfactory for at least three reasons. First, the rationale for generating comparison groups at random, including the steps taken to conceal the assignment schedule, is to eliminate selection bias. By contrast, other forms of blinding, used after the assignment of treatments, serve primarily to reduce ascertainment bias. Second, from a practical standpoint, concealing treatment assignment up to the point of allocation is always possible, regardless of the study topic, whereas blinding after allocation is not attainable in many instances, such as in trials conducted to compare surgical and medical treatments. Third, control of selection bias pertains to the trial as a whole, and thus to all outcomes being compared, whereas control of ascertainment bias may be accomplished successfully for some outcomes, but not for others. Thus, concealment up to the point of allocation of treatment and blinding after that point address different sources of bias and differ in their practicability. In light of those considerations, we refer to the former as ‘allocation concealment’ and reserve the term ‘blinding’ for measures taken to conceal group identity after allocation” (Schulz et al. 1994).
We deemed the following approaches as adequate allocation concealment: “central randomization (e.g., by telephone to a trials office); a pharmacy [drugs or study products prepared by the pharmacy]; numbered or coded containers; and sequentially numbered, opaque, sealed envelopes” (Schulz et al. 1994). We considered non-random, transparent (often called ‘systematic’) approaches, such as “alternate assignment and assignment by odd/even birthdate or hospital number” as inadequate allocation concealment (Schulz et al. 1994). We deemed as ‘unclear allocation concealment’ those approaches in which authors had not reported any allocation approach or reported an approach that was not captured by the ‘adequate’ or ‘inadequate’ concealment categories. Of 206 trial articles, only 23% used adequate allocation concealment, while 48% had not described any mechanism to allocate treatments (Schulz et al. 1994; Schulz et al. 1995a).
Showing that ‘allocation concealment’ matters
Having defined allocation concealment, our next line of inquiry was to investigate whether this construct, as implemented in trials, was associated with indicators of bias. Prior work by Thomas Chalmers and his associates had shown that ‘unblinded randomization’ was associated with larger estimates of effect compared to ‘blinded randomization’ (Chalmers et al. 1983). However, as noted above, their study had several weaknesses.
We addressed those weaknesses by using multiple logistic regression statistical models to analyze 250 trials in 33 meta-analyses published in the Cochrane Pregnancy and Childbirth Database (Schulz et al. 1995b). To address confounding by treatment we examined the association between ‘allocation concealment status’ and estimates of treatment effects within the same treatments across the 33 meta-analyses. To address confounding from other trial characteristics we controlled for the method used to generate the allocation sequence, exclusions after allocation, and blinding of outcome assessments. Lastly, we analyzed trials with unclear allocation concealment (largely those trials for which the published reports had not provided any description of allocation concealment).
Using the substantial amount of relevant data generated by the Cochrane Pregnancy and Childbirth Group (Chalmers 1988), using software that had become available to facilitate our approach, we were able to address weaknesses in the analyses reported by Tom Chalmers and his colleagues (1983) more than a decade earlier. Our more detailed analyses showed that allocation concealment was indeed associated with trial results: on average, estimates of treatment effects were 41% larger in trials with inadequate allocation concealment and 33% larger in those with unclear allocation concealment compared to trials with adequate allocation concealment (Schulz et al. 1995b). Moreover, as a further indication of likely bias, we found that the results of trials with reports of inadequate allocation concealment were more heterogeneous than those with adequate allocation concealment (Schulz et al. 1995b). Our study of randomized trials in pregnancy and childbirth provided empirical evidence that poor allocation concealment was likely associated with bias. We were pleased that the importance of our study was recognized explicitly by others (Figure 1).
Moher replicated our study in additional medical specialties and found similar results (Moher et al. 1998). Repeatedly, evidence was found that allocation concealment mattered. This empirical evidence gave substantial impetus to calls for increased attention to improved reporting, especially on critical items such as allocation concealment. Others had called for improved reporting, but this research provided tangible evidence that indications of poor methods were indeed likely associated with bias. As Drummond Rennie wrote in a 1995 editorial in JAMA commenting on the empirical evidence of bias in our paper, “These admonitions are not new; what is new is the demonstration of the consequences of their neglect” (Rennie 1995).
Establishment and adoption of allocation concealment terminology
Having created the term and validated the concept of ‘allocation concealment’, launching the terminology extended beyond journal publications. We introduced the terminology and concepts into various organizations and groups. Indeed, scientific overlap among all of us aided the process, initiated the Cochrane Centre, which spawned the Cochrane Collaboration. Iain Chalmers enlisted Schulz as the first visiting research fellow at the Cochrane Centre, where they collaborated with Grimes, Altman, and Hayes. Altman and Schulz led the initial development of the Cochrane Statistical Methods Group.
Independently, Moher was organizing a meeting to discuss evaluation of the quality of randomized trials, and he invited Chalmers to attend. However, because our analysis had not been published at the time, Chalmers suggested that Schulz should attend and promote adoption of the term ‘allocation concealment’ and present the evidence that bias is likely to be associated with inadequate allocation concealment. Moher’s meeting resulted in a reporting guideline entitled Standards Of Reporting Trials (SORT) (The Standards of Reporting Trials Group 1994). Following a further meeting to which Moher and Schulz contributed, SORT morphed into another trial reporting guideline – the Consolidated Standards of Reporting Trials (CONSORT) Statement (CONSORT Group 1996). This in turn blosomed into a worldwide initiative to improve the reporting of medicial research – the EQUATOR Network (Altman and Simera 2015).
Schulz promoted appreciation of the importance of allocation concealment with personal accounts of problems he had encountered with assignment schemes and suggested methods for improving allocation concealment (Schulz 1995). Altman joined Moher and Schulz as members of the CONSORT Executive and coauthors of succeeding CONSORT Statements (Moher et al. 2001; Schulz et al. 2010a; Schulz et al. 2010b), and Grimes became part of the CONSORT Group, published a paper setting out the rationale for allocation concealment and providing examples of good reporting (Altman et al. 2001). And in his book on cluster randomized trials, Hayes extended the need for allocation concealment in those trials (Hayes and Moulton 2009). The SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) guidelines for the content of protocols for randomized trials (Chan et al. 2013a; Chan et al. 2013b) drew heavily on the CONSORT Statement.
A PubMed search for the term ‘allocation concealment’ in ‘any field’ during the 22 years before our 1994 paper in JAMA yielded no citations, compared with 1471 citations over the 22 years between 1995 and 2016. The annual number of citations climbed steadily before plateauing around 2012 (Figure 2).
Using Google Scholar to search for ‘allocation concealment’ anywhere in articles published during the 22 years before our 1994 JAMA paper yielded 25 matches, most of which were mistakes. Thirty thousand matches were retrieved from articles published after 1994, and the annual number of matches has increased through 2016 (Figure 3).
Concluding reflections
The term ‘allocation concealment” has been widely adopted by authors and editors. We have less evidence that rigorous definitions are being used for allocation concealment. Although the Cochrane Collaboration uses highly precise definitions (Higgins et al. 2011), we suspect that many authors and editors define allocation concealment imprecisely, similar to their imprecise use of other trial terminology (Devereaux et al. 2001). Furthermore, ‘allocation concealment’ could refer to blinding of outcome assessors. For example, Cox noted in 1958 that “The final stage in which concealment may be advisable is in the making of the observation itself” (Cox 1958). Although the term ‘allocation concealment’ might still be improved to avoid occasional misconceptions about its meaning (for example, extending it to ‘allocation schedule concealment’), we assume that it has been widely adopted by authors and editors because they find the 2-word term useful.
Our modeling and methodological approach (Schulz et al. 1995b) to examine the associations between dimensions of methodological quality and estimates of treatment effects has gained recognition and has led to confirmatory replications. As stated by two prominent methodological researchers (Juni and Egger 2005):
In more recent years, the debate has shifted from anecdotal evidence of bias in single trials to more sophisticated ‘meta-epidemiological’ research, based on many trials and meta-analyses (Sterne et al. 2002). Schulz and colleagues (Schulz et al. 1995) pioneered this approach when they assessed the methodological quality of 250 trials from 33 meta-analyses from the Cochrane Pregnancy and Childbirth Database and examined the association between dimensions of trial quality and estimated treatment effects (Juni and Egger 2005).
This general approach has even been used by Savovic and her colleagues (2012) to incorporate multiple meta-epidemiological studies in an analysis. Although these investigations of associations of allocation concealment extended to other subject areas have yielded the same directions of association as our 1995 analysis (Schulz et al. 1995a), the strength of the associations has varied (Savovic et al. 2012).
Our research on empirical evidence of bias related to allocation concealment provided impetus for the initial development of the SORT and CONSORT reporting guidelines for randomized trials (The Standards of Reporting Trials Group 1994; CONSORT Group 1996; Moher et al. 2001; Schulz et al. 2010a; Schulz et al. 2010b), and the adoption of CONSORT has been associated with improved reporting of randomized trials (Turner et al. 2012a; Turner et al. 2012b). We believe that heightened attention to the process of allocation concealment has also improved the conduct of randomized trials. Indeed, compared with randomized trials published before 1990 those published between 2006 and 2012 were more likely to have reported adequate allocation concealment (Peters et al. 2017; Reveiz et al. 2015). However, much room for improvement remains (Chan and Altman 2005; Hopewell et al. 2010).
Our identification and validation of allocation concealment represents one step towards better conduct and reporting of randomized trials. Further steps must include expanded and enhanced medical research training in design, conduct, and reporting of trials.
Figure 1: Plaque recording the 1996 CDC and ATSDR Statistical Science Award presented to Schulz, Chalmers, Hayes and Altman for the Best Applied Paper.
Figure 2: Number of “allocation concealment” citations retrieved using PubMed, by year, 1994 to 2016.
Figure 3: Number of “allocation concealment” citations (anywhere in article) retrieved using Google Scholar, by year, 1994 to 2016.
This James Lind Library article has been republished in the Journal of the Royal Society of Medicine 2018;111:216-224. Print PDF
References
Altman DG, Doré CJ (1990). Randomisation and baseline comparisons in clinical trials. Lancet 335:149-153.
Altman DG, Simera I (2015). A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/a-history-of-the-evolution-of-guidelines-for-reporting-medical-research-the-long-road-to-the-equator-network/)
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T (2001). The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration. Ann Intern Med 134:663-694.
Armitage P (1960). The construction of comparable groups. In: HILL AB (ed.) Controlled clinical trials. Oxford: Oxford University Press.
Chalmers I, ed (1988). The Oxford Database of Perinatal Trials. Oxford: Oxford University Press.
Chalmers I (2005). Statistical theory was not the reason that randomisation was used in the British Medical Research Council’s clinical trial of streptomycin for pulmonary tuberculosis In: Jorland G, Opinel A, Weisz G eds. Body counts: medical quantification in historical and sociological perspectives. Montreal: McGill-Queens University Press.
Chalmers I (2010). Why the 1948 MRC trial of streptomycin used treatment allocation based on random numbers. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/why-the-1948-mrc-trial-of-streptomycin-used-treatment-allocation-based-on-random-numbers/)
Chalmers TC, Celano P, Sacks HS, Smith HJ (1983). Bias in treatment assignment in controlled clinical trials. N Engl J Med 309:1358-1361.
Chalmers TC, Smith HJ, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A (1981). A method for assessing the quality of a randomized control trial. Control Clin Trials 2:31-49.
Chan AW, Altman DG (2005). Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 365:1159-1162.
Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gotzsche PC, Krleza-Jeric K, Hrobjartsson A, Mann H, Dickersin K, Berlin JA, Dore CJ, Parulekar WR, Summerskill WS, Groves T, Schulz KF, Sox HC, Rockhold FW, Rennie D, Moher D (2013a). SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med:158:200-207.
Chan AW, Tetzlaff JM, Gotzsche PC, Altman DG, Mann H, Berlin JA, Dickersin K, Hrobjartsson A, Schulz KF, Parulekar WR, Krleza-Jeric K, Laupacis A, Moher D (2013b). SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 346: e7586.
Clarke M (2004). The 1944 patulin trial of the British Medical Research Council: an example of how concerted common purpose can get reliable answers to important questions very quickly. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/the-1944-patulin-trial-of-the-british-medical-research-council-an-example-of-how-concerted-common-purpose-can-get-reliable-answers-to-important-questions-very-quickly/)
The CONSORT Group (1996). Improving the quality of reporting of randomized controlled trials. The CONSORT Statement. JAMA 276:637-639.
Conti CR (1994). Confusing and sometimes meaningless terms used in cardiovascular medicine. Clin Cardiol 17:53-54.
Cox DR (1958). Planning of experiments, New York, Wiley.
Dersimonian R, Charette LJ, Mcpeek B, Mosteller F (1982). Reporting on methods in clinical trials. N Engl J Med 306:1332-1337.
Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, Bhandari M, Guyatt, GH (2001). Physician Interpretations and Textbook Definitions of Blinding Terminology in Randomized Controlled Trials. JAMA 285:2000-2003.
Friedman L, Furberg C, Demets D (1985). Fundamentals of Clinical Trials, Littleton, Mass., PSG Publishing.
Gillman MW, Runyan DK (1984). Bias in treatment assignment in controlled clinical trials. N Engl J Med 310:1610-1612.
Grimes DA (2008). The nomogram epidemic: resurgence of a medical relic. Ann Intern Med 149:273-275.
Grimes DA (2009). “Case-control” confusion: mislabeled reports in obstetrics and gynecology journals. Obstet Gynecol 114:1284-1286.
Hayes RJ, Moulton LH (2009). Cluster Randomised Trials, Taylor & Francis.
Higgins, JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343:d5928.
Hill AB (1962). Statistical Methods in Clinical and Preventive Medicine, Edinburgh, E&S Livingstone.
Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG (2010). The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 340:c723.
Juni P, Egger M (2005). Commentary: Empirical evidence of attrition bias in clinical trials. Int J Epidemiol 34:87-88.
Medical Research Council (1944). Clinical trial of patulin in the common cold. Lancet 2:373-5.
Medical Research Council (1948). Streptomycin treatment of pulmonary tuberculosis: a Medical Research Council investigation. BMJ 2:769-782.
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609-613.
Moher D, Schulz KF, Altman DG (2001). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357:1191-1194.
Mosteller F, Gilbert JP, Mcpeek B (1980). Reporting standards and research strategies for controlled trials: agenda for the editor. Controlled Clin Trials 1:37-58.
Peters JPM, Stegeman I, Grolman W, Hooft L (2017). The risk of bias in randomized controlled trials in otorhinolaryngology: hardly any improvement since 1950. BMC Ear Nose Throat Disord 17:3.
Peto R (1987). Why do we need systematic overviews of randomized trials? Stat Med 6: 233-240.
Rennie D (1995). Reporting randomized controlled trials. An experiment and a call for responses from readers. JAMA 273:1054-1055.
Reveiz L, Chapman E, Asial S, Munoz S, Bonfill X, Alonso-Coello P (2015). Risk of bias of randomized trials over time. J Clin Epidemiol 68:1036-1045.
Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, Ioannidis JP, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JA (2012). Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 157:429-438.
Schulz KF (1995). Subverting randomization in controlled trials. JAMA 274:1456-1458.
Schulz KF, Altman DG, Moher D (2010a). CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340:c332.
Schulz KF, Altman DG, Moher D, Fergusson D (2010b). CONSORT 2010 changes and testing blindness in RCTs. Lancet 375:1144-1146.
Schulz KF, Chalmers I, Altman DG (2002). The landscape and lexicon of blinding in randomized trials. Ann Intern Med 136:254-259.
Schulz KF, Chalmers I, Altman DG, Grimes DA, Dore CJ (1995a.) The methodologic quality of randomization as assessed from reports of trials in specialist and general medical journals. Online J Curr Clin Trials, Doc No 197, [81 paragraphs].
Schulz KF, Chalmers I, Grimes DA, Altman DG (1994). Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA 272:125-128.
Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995b). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273:408-412.
Schulz KF, Grimes DA (2002). Blinding in randomised trials: hiding who got what. Lancet 359:696-700.
Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C Egger M (2002). Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 21:1513-1524.
The Standards Of Reporting Trials Group 1994. A proposal for structured reporting of randomized controlled trials. JAMA 272:1926-1931.
Turner L, Shamseer L, Altman DG, Schulz KF Moher D (2012a). Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev, 1, 60.
Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, Dias S, Schulz KF, Plint AC, Moher D (2012b). Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev, 11, Mr000030.
Tyson JE, Furzan JA, Reisch JS, Mize SG (1983). An evaluation of the quality of therapeutic studies in perinatal medicine. J Pediatr 102:10-13.
Schulz KF, Chalmers I, Altman DG†, Grimes DA, Moher D, Hayes RJ (2018). ‘Allocation concealment’: the evolution and adoption of a methodological term.
© Kenneth F. Schulz, FHI 360, Blackwell Street, Suite 200, Durham, NC 27701 Email: KSchulz@fhi360.org.
Cite as: Schulz KF, Chalmers I, Altman DG†, Grimes DA, Moher D, Hayes RJ (2018). ‘Allocation concealment’: the evolution and adoption of a methodological term. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/allocation-concealment-evolution-adoption-methodological-term/)
Introduction
Random assignment of individual participants in clinical trials entails two separate steps: (i) generating an unbiased allocation schedule; and (ii) using the schedule, without foreknowledge of upcoming allocations, to assign participants to treatment comparison groups. Both of these steps were implemented in two controlled trials done under the aegis of the British Medical Research Council (MRC) in the 1940s (Chalmers 2010): the trial of patulin for the common cold (MRC 1944; Clarke 2004), and the better known trial of streptomycin for pulmonary tuberculosis (MRC 1948; Chalmers 2010). However, the two separate steps needed to achieve unbiased treatment assignment were not identified clearly until 1955, when Austin Bradford Hill distinguished them in the 6th edition of his book Principles of Medical Statistics (Chalmers 2005). Shortly thereafter, David Cox (1958) and Peter Armitage (1960) emphasized the importance of implementing an unbiased allocation by using sealed envelopes to conceal an upcoming allocation until after a participant had been irrevocably entered into a trial. Despite early recognition of the importance of the two separate steps in unbiased allocation, however, no widely accepted term denoting the process of concealing upcoming allocations had been adopted more than half a century later.
Methodological terminology is often imprecise. Researchers may abandon the intended meaning, even with common terminology. For example, the terms ‘nomogram’ and ‘case-control’ are widely misused in research (Grimes 2008; Grimes 2009). Jargon (for example, ‘gold standard’) leads to confusing and meaningless terms (Conti 1994). Clearly, accurate communication requires unambiguous terminology.
The plethora of terminology surrounding randomized trials presents particular challenges. Trials suffer from a great deal of arcane terminology. Some examples include ‘random permuted blocks’, ‘participant retention’, ‘double- and triple-blinding’, ’masking’, ‘interim analyses’, ‘alpha spending’, ‘urn randomization’, ‘biased-coin randomization’, and ‘group sequential trials’. While most of these represent important trial processes and methods, even the most common terms may confuse or obfuscate. For example, although blinding terminology seems well ensconced, even apparently simple terminology like ‘double-blinding’ elicits inconsistent connotations. When investigators examined physician interpretations and textbook definitions of ‘double-blinding’, they found 17 unique interpretations and nine different definitions (Devereaux et al. 2001). Reporting only ‘double-blinding’ without proper elaboration leads to ambiguity (Schulz et al. 2002; Schulz and Grimes 2002).
Terminology serves another important function. It can indicate a critical methodological process that might otherwise be neglected in the conduct of a study unless highlighted by terminology. Because lack of attention to detail can impact on the conduct of trials, lack of appropriate terminology may cause harm if an important trial function has been inadequately addressed.
Despite the sheer volume of terminology in randomized trials, we discovered a terminological gap when we were assessing the quality of treatment allocation in reports of controlled trials published in journals of obstetrics and gynecology in the 1990s (Schulz et al. 1994). When writing up that research, we introduced the term ‘allocation concealment’ (Schulz et al. 1994) to denote the second of the two essential steps in achieving unbiased allocation to treatment comparison groups. In this article we have reviewed the precursors of the term ‘allocation concealment’, and its subsequent evolution and adoption from the mid-1990’s onwards.
Recognizing the need for randomization terminology
Over the years, books on randomized trials provided excellent and detailed descriptions of methods to generate randomized allocation sequences. These sequences formed randomization lists. Chapters and sections of chapters had titles such as ‘methods of randomization’, ‘constrained randomization’, ‘adaptive randomization’, ‘unequal treatment allocation’, ‘permuted block designs’, ‘biased-coin design’, ‘urn design’, and ‘stratified randomization’.
In books published before we suggested the term ‘allocation concealment’ in 1994, chapters describing the mechanisms for implementing allocation sequences were rare. Moreover, scant information was devoted to those mechanisms elsewhere in the books. For example, one of the most detailed discussions we found was a ‘Mechanics of Randomization’ section from a book published in 1985, but it was less than two pages in a 17-page chapter (Friedman et al. 1985).
As we have noted, some trialists recognized the importance of mechanisms for implementing allocation sequences long ago, including the use of sealed envelopes (Cox 1958; Armitage 1960; Hill 1962; Chalmers 2005). In meetings with IC and KS in 1992, Richard Peto noted that he had emphasized (Peto 1987) that randomization should incorporate a mechanism to prevent investigators, health care providers, and participants from foreknowledge of upcoming assignments, but he had not come up with a term for the mechanism.
‘Randomization blinding’
Thomas Chalmers and colleagues provided key inspiration. In the early 1980s, they had not only recognised the importance of the assignment mechanism, but had termed it ‘randomization blinding’ (Chalmers et al. 1981). They suspected that bias introduced into “…studies in which assignment of controls is less blinded may be explained by bias in the selection or rejection of patients when the treatment to be given is known or suspected at the time of assignment” (Chalmers et al. 1983). Thus, without adequate ‘randomization blinding’, selection bias could occur. To investigate their suspicions they used trials of treatments for acute myocardial infarction, using case-fatality rates as the outcome. They defined ‘blinded randomization’ as ‘opaque envelopes’, ‘a telephone call to a statistical center’, or ‘a prearranged order of blinded medications labeled consecutively by the pharmacy’ (Chalmers et al. 1983). They defined ‘unblinded randomization’ as “assignment from an open table of random numbers, according to date of birth or chart number, or by some other variably random system in which patients could present for study in a chance order but be selected or rejected after the physician knew the treatment assignment” (Chalmers et al. 1983). They found that trials using unblinded randomization yielded larger estimates of treatment effects, which were more often statistically significant, than those in trials using blinded randomization. Their findings provided the strongest evidence at that time that ‘unblinded randomization’ was associated with empirical evidence suggestive of bias in ‘randomized’ trials.
The ‘blinding’ terms ‘blinded randomization’ and ‘unblinded randomization’ are confusing because they address a mechanism within treatment allocation, not whether a trial is subsequently blinded to treatment identity. For example, a trial may have ‘blinded randomization’ as defined by Thomas Chalmers and his colleagues (Chalmers et al. 1983), but, after allocation, investigators and participants may not have been blinded to treatment assignments. Perhaps because of confusion between ‘randomization blinding’ and ‘treatment blinding’ or ‘outcome measurement blinding’, adoption of the ‘blinded randomization’ and ‘unblinded randomization’ terminology remained rare. Moreover, their definitions of ‘blinded randomization’ and ‘unblinded randomization’ were insufficiently specific. For example, although referring to “opaque envelopes”, they did not mention additional precautions such as sequential numbering, and their definitions and analysis neglected the common occurrence of published reports that had not described the implementation mechanism of the random allocation schedule.
As noted by commentators (Gillman and Runyan 1984), the analysis used by Thomas Chalmers and his colleagues (1983) could not take account of the different types of treatments in their blinded and unblinded randomization categories, thus leaving their results potentially confounded by treatment. For example, most beta-blocker trials were in the blinded randomization group while most antithrombotic agent trials were in the unblinded randomization group (Gillman and Runyan 1984). Second, they did not account for other trial characteristics, such as generation of the allocation sequence and blinding of treatments. Third, their study left unanswered the status of all those trials whose published reports did not provide any information about the implementation mechanism for random allocation. That was a critical issue, because poorly reported trials then predominated in the literature (Mosteller et al. 1980; DerSimonian et al. 1982; Altman and Doré 1990; Tyson et al. 1983), as they do now (Hopewell et al. 2010).
‘Bias-reducing allocation’
In 1990, Altman and Doré improved terminology and definitional clarity (Altman and Doré 1990). Instead of using the terms ‘blinded randomization’ and ‘unblinded randomization’, they termed the process ‘bias-reducing allocation’, using this definition: “The mechanism of treatment allocation should be designed to avoid bias: suitable methods are central randomization, coded drugs prepared by the pharmacy, and use of a series of numbered opaque sealed envelopes”. Their categorization of reports was based on these definitions as ‘yes’, ‘no’, or ‘not specified’.
The Altman and Doré terminology and analysis were improvements over that suggested by Thomas Chalmers and colleagues (1983) because they avoided derivatives of the term ‘blind’, specified that envelopes should be sequentially numbered and sealed, and included a ‘not specified’ option. The latter applied to almost half of the responses in their study and to three-quarters of the reports of randomized trials in PubMed (Hopewell et al. 2010). For anyone appraising the quality of the literature, this ‘not specified’ category provides an important perspective for evaluating quality.
‘Allocation concealment’
We needed a replacement term for ‘randomization blinding’. Although Altman and Doré had improved the terminology by introducing ‘bias-reducing allocation’, some confusion remained. For example, ‘bias-reducing allocation’ could be confused with the process of generating a random allocation sequence, or indeed, the entire randomization process. Accordingly, we sought further terminological improvement in our review of reports of trials in obstetrics and gynecology (Schulz et al. 1994).
Without any recognized method for introducing terminology, we relied upon a subjective, iterative, discussion process. We avoided any terminology associated with ‘blinding’ and explored the dictionary for words to capture the assignment process. After long deliberations, KS, IC, DG, and DA proposed ‘allocation concealment’. This implied a mechanism to prevent foreknowledge of upcoming assignments which avoid any reference to ‘blinding’. We first used the term ‘allocation concealment’ in 1994 (Schulz et al. 1994). Because the term was new, we explained our rationale for preferring ‘allocation concealment’ in detail:
We deemed the following approaches as adequate allocation concealment: “central randomization (e.g., by telephone to a trials office); a pharmacy [drugs or study products prepared by the pharmacy]; numbered or coded containers; and sequentially numbered, opaque, sealed envelopes” (Schulz et al. 1994). We considered non-random, transparent (often called ‘systematic’) approaches, such as “alternate assignment and assignment by odd/even birthdate or hospital number” as inadequate allocation concealment (Schulz et al. 1994). We deemed as ‘unclear allocation concealment’ those approaches in which authors had not reported any allocation approach or reported an approach that was not captured by the ‘adequate’ or ‘inadequate’ concealment categories. Of 206 trial articles, only 23% used adequate allocation concealment, while 48% had not described any mechanism to allocate treatments (Schulz et al. 1994; Schulz et al. 1995a).
Showing that ‘allocation concealment’ matters
Having defined allocation concealment, our next line of inquiry was to investigate whether this construct, as implemented in trials, was associated with indicators of bias. Prior work by Thomas Chalmers and his associates had shown that ‘unblinded randomization’ was associated with larger estimates of effect compared to ‘blinded randomization’ (Chalmers et al. 1983). However, as noted above, their study had several weaknesses.
We addressed those weaknesses by using multiple logistic regression statistical models to analyze 250 trials in 33 meta-analyses published in the Cochrane Pregnancy and Childbirth Database (Schulz et al. 1995b). To address confounding by treatment we examined the association between ‘allocation concealment status’ and estimates of treatment effects within the same treatments across the 33 meta-analyses. To address confounding from other trial characteristics we controlled for the method used to generate the allocation sequence, exclusions after allocation, and blinding of outcome assessments. Lastly, we analyzed trials with unclear allocation concealment (largely those trials for which the published reports had not provided any description of allocation concealment).
Using the substantial amount of relevant data generated by the Cochrane Pregnancy and Childbirth Group (Chalmers 1988), using software that had become available to facilitate our approach, we were able to address weaknesses in the analyses reported by Tom Chalmers and his colleagues (1983) more than a decade earlier. Our more detailed analyses showed that allocation concealment was indeed associated with trial results: on average, estimates of treatment effects were 41% larger in trials with inadequate allocation concealment and 33% larger in those with unclear allocation concealment compared to trials with adequate allocation concealment (Schulz et al. 1995b). Moreover, as a further indication of likely bias, we found that the results of trials with reports of inadequate allocation concealment were more heterogeneous than those with adequate allocation concealment (Schulz et al. 1995b). Our study of randomized trials in pregnancy and childbirth provided empirical evidence that poor allocation concealment was likely associated with bias. We were pleased that the importance of our study was recognized explicitly by others (Figure 1).
Moher replicated our study in additional medical specialties and found similar results (Moher et al. 1998). Repeatedly, evidence was found that allocation concealment mattered. This empirical evidence gave substantial impetus to calls for increased attention to improved reporting, especially on critical items such as allocation concealment. Others had called for improved reporting, but this research provided tangible evidence that indications of poor methods were indeed likely associated with bias. As Drummond Rennie wrote in a 1995 editorial in JAMA commenting on the empirical evidence of bias in our paper, “These admonitions are not new; what is new is the demonstration of the consequences of their neglect” (Rennie 1995).
Establishment and adoption of allocation concealment terminology
Having created the term and validated the concept of ‘allocation concealment’, launching the terminology extended beyond journal publications. We introduced the terminology and concepts into various organizations and groups. Indeed, scientific overlap among all of us aided the process, initiated the Cochrane Centre, which spawned the Cochrane Collaboration. Iain Chalmers enlisted Schulz as the first visiting research fellow at the Cochrane Centre, where they collaborated with Grimes, Altman, and Hayes. Altman and Schulz led the initial development of the Cochrane Statistical Methods Group.
Independently, Moher was organizing a meeting to discuss evaluation of the quality of randomized trials, and he invited Chalmers to attend. However, because our analysis had not been published at the time, Chalmers suggested that Schulz should attend and promote adoption of the term ‘allocation concealment’ and present the evidence that bias is likely to be associated with inadequate allocation concealment. Moher’s meeting resulted in a reporting guideline entitled Standards Of Reporting Trials (SORT) (The Standards of Reporting Trials Group 1994). Following a further meeting to which Moher and Schulz contributed, SORT morphed into another trial reporting guideline – the Consolidated Standards of Reporting Trials (CONSORT) Statement (CONSORT Group 1996). This in turn blosomed into a worldwide initiative to improve the reporting of medicial research – the EQUATOR Network (Altman and Simera 2015).
Schulz promoted appreciation of the importance of allocation concealment with personal accounts of problems he had encountered with assignment schemes and suggested methods for improving allocation concealment (Schulz 1995). Altman joined Moher and Schulz as members of the CONSORT Executive and coauthors of succeeding CONSORT Statements (Moher et al. 2001; Schulz et al. 2010a; Schulz et al. 2010b), and Grimes became part of the CONSORT Group, published a paper setting out the rationale for allocation concealment and providing examples of good reporting (Altman et al. 2001). And in his book on cluster randomized trials, Hayes extended the need for allocation concealment in those trials (Hayes and Moulton 2009). The SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) guidelines for the content of protocols for randomized trials (Chan et al. 2013a; Chan et al. 2013b) drew heavily on the CONSORT Statement.
A PubMed search for the term ‘allocation concealment’ in ‘any field’ during the 22 years before our 1994 paper in JAMA yielded no citations, compared with 1471 citations over the 22 years between 1995 and 2016. The annual number of citations climbed steadily before plateauing around 2012 (Figure 2).
Using Google Scholar to search for ‘allocation concealment’ anywhere in articles published during the 22 years before our 1994 JAMA paper yielded 25 matches, most of which were mistakes. Thirty thousand matches were retrieved from articles published after 1994, and the annual number of matches has increased through 2016 (Figure 3).
Concluding reflections
The term ‘allocation concealment” has been widely adopted by authors and editors. We have less evidence that rigorous definitions are being used for allocation concealment. Although the Cochrane Collaboration uses highly precise definitions (Higgins et al. 2011), we suspect that many authors and editors define allocation concealment imprecisely, similar to their imprecise use of other trial terminology (Devereaux et al. 2001). Furthermore, ‘allocation concealment’ could refer to blinding of outcome assessors. For example, Cox noted in 1958 that “The final stage in which concealment may be advisable is in the making of the observation itself” (Cox 1958). Although the term ‘allocation concealment’ might still be improved to avoid occasional misconceptions about its meaning (for example, extending it to ‘allocation schedule concealment’), we assume that it has been widely adopted by authors and editors because they find the 2-word term useful.
Our modeling and methodological approach (Schulz et al. 1995b) to examine the associations between dimensions of methodological quality and estimates of treatment effects has gained recognition and has led to confirmatory replications. As stated by two prominent methodological researchers (Juni and Egger 2005):
This general approach has even been used by Savovic and her colleagues (2012) to incorporate multiple meta-epidemiological studies in an analysis. Although these investigations of associations of allocation concealment extended to other subject areas have yielded the same directions of association as our 1995 analysis (Schulz et al. 1995a), the strength of the associations has varied (Savovic et al. 2012).
Our research on empirical evidence of bias related to allocation concealment provided impetus for the initial development of the SORT and CONSORT reporting guidelines for randomized trials (The Standards of Reporting Trials Group 1994; CONSORT Group 1996; Moher et al. 2001; Schulz et al. 2010a; Schulz et al. 2010b), and the adoption of CONSORT has been associated with improved reporting of randomized trials (Turner et al. 2012a; Turner et al. 2012b). We believe that heightened attention to the process of allocation concealment has also improved the conduct of randomized trials. Indeed, compared with randomized trials published before 1990 those published between 2006 and 2012 were more likely to have reported adequate allocation concealment (Peters et al. 2017; Reveiz et al. 2015). However, much room for improvement remains (Chan and Altman 2005; Hopewell et al. 2010).
Our identification and validation of allocation concealment represents one step towards better conduct and reporting of randomized trials. Further steps must include expanded and enhanced medical research training in design, conduct, and reporting of trials.
Figure 1: Plaque recording the 1996 CDC and ATSDR Statistical Science Award presented to Schulz, Chalmers, Hayes and Altman for the Best Applied Paper.
Figure 2: Number of “allocation concealment” citations retrieved using PubMed, by year, 1994 to 2016.
Figure 3: Number of “allocation concealment” citations (anywhere in article) retrieved using Google Scholar, by year, 1994 to 2016.
This James Lind Library article has been republished in the Journal of the Royal Society of Medicine 2018;111:216-224. Print PDF
References
Altman DG, Doré CJ (1990). Randomisation and baseline comparisons in clinical trials. Lancet 335:149-153.
Altman DG, Simera I (2015). A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/a-history-of-the-evolution-of-guidelines-for-reporting-medical-research-the-long-road-to-the-equator-network/)
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T (2001). The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration. Ann Intern Med 134:663-694.
Armitage P (1960). The construction of comparable groups. In: HILL AB (ed.) Controlled clinical trials. Oxford: Oxford University Press.
Chalmers I, ed (1988). The Oxford Database of Perinatal Trials. Oxford: Oxford University Press.
Chalmers I (2005). Statistical theory was not the reason that randomisation was used in the British Medical Research Council’s clinical trial of streptomycin for pulmonary tuberculosis In: Jorland G, Opinel A, Weisz G eds. Body counts: medical quantification in historical and sociological perspectives. Montreal: McGill-Queens University Press.
Chalmers I (2010). Why the 1948 MRC trial of streptomycin used treatment allocation based on random numbers. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/why-the-1948-mrc-trial-of-streptomycin-used-treatment-allocation-based-on-random-numbers/)
Chalmers TC, Celano P, Sacks HS, Smith HJ (1983). Bias in treatment assignment in controlled clinical trials. N Engl J Med 309:1358-1361.
Chalmers TC, Smith HJ, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A (1981). A method for assessing the quality of a randomized control trial. Control Clin Trials 2:31-49.
Chan AW, Altman DG (2005). Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 365:1159-1162.
Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gotzsche PC, Krleza-Jeric K, Hrobjartsson A, Mann H, Dickersin K, Berlin JA, Dore CJ, Parulekar WR, Summerskill WS, Groves T, Schulz KF, Sox HC, Rockhold FW, Rennie D, Moher D (2013a). SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med:158:200-207.
Chan AW, Tetzlaff JM, Gotzsche PC, Altman DG, Mann H, Berlin JA, Dickersin K, Hrobjartsson A, Schulz KF, Parulekar WR, Krleza-Jeric K, Laupacis A, Moher D (2013b). SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 346: e7586.
Clarke M (2004). The 1944 patulin trial of the British Medical Research Council: an example of how concerted common purpose can get reliable answers to important questions very quickly. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/the-1944-patulin-trial-of-the-british-medical-research-council-an-example-of-how-concerted-common-purpose-can-get-reliable-answers-to-important-questions-very-quickly/)
The CONSORT Group (1996). Improving the quality of reporting of randomized controlled trials. The CONSORT Statement. JAMA 276:637-639.
Conti CR (1994). Confusing and sometimes meaningless terms used in cardiovascular medicine. Clin Cardiol 17:53-54.
Cox DR (1958). Planning of experiments, New York, Wiley.
Dersimonian R, Charette LJ, Mcpeek B, Mosteller F (1982). Reporting on methods in clinical trials. N Engl J Med 306:1332-1337.
Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, Bhandari M, Guyatt, GH (2001). Physician Interpretations and Textbook Definitions of Blinding Terminology in Randomized Controlled Trials. JAMA 285:2000-2003.
Friedman L, Furberg C, Demets D (1985). Fundamentals of Clinical Trials, Littleton, Mass., PSG Publishing.
Gillman MW, Runyan DK (1984). Bias in treatment assignment in controlled clinical trials. N Engl J Med 310:1610-1612.
Grimes DA (2008). The nomogram epidemic: resurgence of a medical relic. Ann Intern Med 149:273-275.
Grimes DA (2009). “Case-control” confusion: mislabeled reports in obstetrics and gynecology journals. Obstet Gynecol 114:1284-1286.
Hayes RJ, Moulton LH (2009). Cluster Randomised Trials, Taylor & Francis.
Higgins, JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343:d5928.
Hill AB (1962). Statistical Methods in Clinical and Preventive Medicine, Edinburgh, E&S Livingstone.
Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG (2010). The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 340:c723.
Juni P, Egger M (2005). Commentary: Empirical evidence of attrition bias in clinical trials. Int J Epidemiol 34:87-88.
Medical Research Council (1944). Clinical trial of patulin in the common cold. Lancet 2:373-5.
Medical Research Council (1948). Streptomycin treatment of pulmonary tuberculosis: a Medical Research Council investigation. BMJ 2:769-782.
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609-613.
Moher D, Schulz KF, Altman DG (2001). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357:1191-1194.
Mosteller F, Gilbert JP, Mcpeek B (1980). Reporting standards and research strategies for controlled trials: agenda for the editor. Controlled Clin Trials 1:37-58.
Peters JPM, Stegeman I, Grolman W, Hooft L (2017). The risk of bias in randomized controlled trials in otorhinolaryngology: hardly any improvement since 1950. BMC Ear Nose Throat Disord 17:3.
Peto R (1987). Why do we need systematic overviews of randomized trials? Stat Med 6: 233-240.
Rennie D (1995). Reporting randomized controlled trials. An experiment and a call for responses from readers. JAMA 273:1054-1055.
Reveiz L, Chapman E, Asial S, Munoz S, Bonfill X, Alonso-Coello P (2015). Risk of bias of randomized trials over time. J Clin Epidemiol 68:1036-1045.
Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, Ioannidis JP, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JA (2012). Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 157:429-438.
Schulz KF (1995). Subverting randomization in controlled trials. JAMA 274:1456-1458.
Schulz KF, Altman DG, Moher D (2010a). CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340:c332.
Schulz KF, Altman DG, Moher D, Fergusson D (2010b). CONSORT 2010 changes and testing blindness in RCTs. Lancet 375:1144-1146.
Schulz KF, Chalmers I, Altman DG (2002). The landscape and lexicon of blinding in randomized trials. Ann Intern Med 136:254-259.
Schulz KF, Chalmers I, Altman DG, Grimes DA, Dore CJ (1995a.) The methodologic quality of randomization as assessed from reports of trials in specialist and general medical journals. Online J Curr Clin Trials, Doc No 197, [81 paragraphs].
Schulz KF, Chalmers I, Grimes DA, Altman DG (1994). Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA 272:125-128.
Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995b). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273:408-412.
Schulz KF, Grimes DA (2002). Blinding in randomised trials: hiding who got what. Lancet 359:696-700.
Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C Egger M (2002). Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 21:1513-1524.
The Standards Of Reporting Trials Group 1994. A proposal for structured reporting of randomized controlled trials. JAMA 272:1926-1931.
Turner L, Shamseer L, Altman DG, Schulz KF Moher D (2012a). Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev, 1, 60.
Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, Dias S, Schulz KF, Plint AC, Moher D (2012b). Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev, 11, Mr000030.
Tyson JE, Furzan JA, Reisch JS, Mize SG (1983). An evaluation of the quality of therapeutic studies in perinatal medicine. J Pediatr 102:10-13.