Vessey MP (2006). Learning how to control biases in studies to identify adverse effects of drugs.

© Martin Vessey, Clifden Cottage, Burford Road, Fulbrook, Oxon OX18 4BL. Email: martin.vessey@dphpc.ox.ac.uk


Cite as: Vessey MP (2006). Learning how to control biases in studies to identify adverse effects of drugs. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/learning-how-to-control-biases-in-studies-to-identify-adverse-effects-of-drugs/)


I joined the Medical Research Council’s Statistical Research Unit at University College Hospital Medical School, London, in mid-1966, having decided to pursue a career in epidemiology rather than clinical medicine. The director of the Unit was Richard Doll and he asked me to undertake an investigation into the possible relationship between oral contraceptive use and thromboembolic diseaseThis was because the Medical Research Council was concerned about the large number of case reports suggestive of such a relationship which had been sent to the Committee on Safety of Medicines or published in the medical literature.

We decided to use the case-control (case–referent) design to investigate the association. The cases were women admitted to hospital with a diagnosis of thromboembolic disease; the controls were women admitted to hospital with other conditions. It soon became apparent to us that the study would have to be designed and conducted with great care if valid results were to be obtained. We recognised, for example, that women who developed a thromboembolic episode who had an evident predisposing cause (such as having recently given birth or had surgery, or who were suffering from a pre-existing chronic disease) would be less likely to be using oral contraceptives than other women. To design a comparison of cases and controls which endeavoured to take account of these selection biases prompted us to limit our cases to previously healthy women developing a thromboembolic episode ‘out of the blue’ (idiopathic disease).

Likewise, in choosing appropriate hospital controls we recognised that we needed to identify women who had either previously been healthy and had experienced some other acute condition ‘out of the blue’ (for example acute appendicitis or acute pneumonia), or healthy women with some minor disorder requiring only a brief elective admission to hospital (for example excision of a lipoma or extraction of wisdom teeth). The conditions from which the controls suffered had also, of course, to be unrelated (so far as was known) to oral contraceptive use.

We were also concerned that the study results might be distorted by diagnostic bias: as a consequence of their awareness of the published case reports, doctors might have a higher index of suspicion about the possibility of thromboembolic disease in women taking oral contraceptives than in women not doing so.

Our final study design took these and other problems on board and produced what we now know to be the right answers. Indeed, the paper in which we reported our findings (Vessey and Doll 1968) has been widely used for teaching purposes and in due course became a ‘Citation Classic’ (Vessey 1986).

My work on oral contraceptives led to many new work contacts, including with Hershel Jick, Co-Director of the Boston Collaborative Drug Surveillance Program in the United States. This program collected large amounts of data about consecutive admissions to a number of hospitals with a view to identifying and quantifying adverse reactions to drugs. For some analyses, especially those dealing with past drug exposure, the case-control approach was the appropriate one to use. Hershel found himself wrestling with methodological problems similar to those with which Richard Doll and I had been confronted in our oral contraceptive studies. In view of our common interests, Hershel invited me to work with him in Boston, which I did for a period of 5 months in 1972, and for shorter periods on many other occasions. In our various discussions (which sometimes included Olli Miettinen, who was a source of much wisdom), and in the course of conducting numerous analyses, we gradually learned how to control biases in studies to identify adverse effects of drugs.

This experience led to the paper featured in the James Lind Library (Jick and Vessey 1978). The principles set out in the paper appear to be as relevant now as they were at the time they were written, and I have tried to follow them since in the many case-control studies investigating possible adverse effects of drugs in which I have been involved.  The Jick and Vessey paper does not discuss case-control studies nested within cohort studies, which are often of value when they are feasible. Such studies have been used extensively in more recent years by Hershel Jick in research using data in the General Practice Research Database (for example Jick et al. 1995). In addition, since community-based information sources such as the General Practice Research Database have become more available than they were in the past, the Jick and Vessey paper may be more orientated towards hospital based studies than might seem appropriate nowadays. Our joint paper also stressed the need to concentrate on ‘idiopathic’ disease to avoid serious bias, but, in doing so, it may have failed to stress that this focus inevitably yields information only about idiopathic disease – and not about similar side effects in people with predisposing factors. It is often important to know about adverse effects of a drug in people who already have a chronic disease or some other predisposing factor, but the case-control approach cannot usually help in these circumstances.

Although case-control studies cannot directly provide estimates of absolute risk, it is usually possible to come up with some sort of incidence estimate if the study has been conducted in a defined population. As already mentioned, case-control studies of drug effects usually provide information only about idiopathic disease, and this has led to some important underestimates of absolute risks. As an example, consider the effects of hormone replacement therapy (HRT) on venous thromboembolism. The case-control study which originally demonstrated this effect (Daly et al. 1996), which was broadly population based, and was restricted to idiopathic disease and comparison of current HRT users with non-users, yielded a relative risk estimate of 3.5 (95% confidence interval 1.8-7.0). The rate of venous thromboembolism attributable to HRT estimated from the same study was about 20 per 100,000 users per year.

At 2.1 (95% confidence interval 1.6-2.8), the relative risk of venous thromboembolism in users of HRT in comparison with non-user controls in a large randomized controlled trial of HRT (Writing Group for the Women’s Health Initiative Investigators 2002) was fairly close to that obtained in our case-control study (3.5) (Daly et al. 1996). However, the analysis of the randomized trial was not restricted to idiopathic disease, and the rate of venous thromboembolism attributable to HRT estimated from that study was about 180 per 100,000 users per year – nine times higher than the estimate derived from the case-control study. Although the randomized trial involved somewhat older women than the case-control study, this large difference between the two studies cannot be explained merely by the effect of age, and it underlines the need for caution in extrapolating the results of case-control studies to public health statements.

These additional points about the need for caution in interpreting and extrapolating from case-control studies are important. However, I am glad that the basic principles of study design for investigating possible adverse effects of drugs which Hershel Jick and I set out in 1978 appear to have stood the test of time.

This James Lind Library commentary has been republished in the Journal of the Royal Society of Medicine 2007;100:526-7. Print PDF

References

Daly E, Vessey MP, Hawkins MM, Carson JL, Gough P, Marsh S (1996). Risk of venous thromboembolism in users of hormone replacement therapy. Lancet 348:977-80.

Jick H, Vessey MP (1978). Case-control studies in the evaluation of drug-induced illness. American Journal of Epidemiology 107:1-7.

Jick H, Jick SS, Gurewich V, Myers MW, Vasilakis C (1995). Risk of idiopathic cardiovascular death and nonfatal venous thromboembolism in women using oral contraceptives with differing progestagen components. Lancet 346:1589-93.

Vessey MP (1986). This week’s citation classic. Current Contents 13:16.

Vessey MP, Doll R (1968). Investigation of relation between use of oral contraceptives and thromboembolic disease. BMJ 2:199-205.

Writing Group for the Women’s Health Initiative Investigators (2002). Risks and benefits of estrogen plus progestin in healthy postmenopausal women. Principal results from the Women’s Health Initiative randomized controlled trial. Journal of the American Medical Association 288:321-33.