Tribute

Professor Doug Altman (1948-2018) co-edited the first and second editions of this book. Here, Sir Iain Chalmers, founder of The Cochrane Collaboration and the James Lind Alliance, reflects on Altman's seminal contributions to the concept of systematic reviews and the role of meta-analysis.

Doug Altman's Prescience in Recognizing the Need to Reduce Biases before Tackling Imprecision in Systematic Reviews

Iain Chalmers

RECOGNITION OF SHARED INTERESTS AND THE ORIGINS OF A FRIENDSHIP

I came to know Doug Altman during the 1980s when we were both members of the editorial team at the British Journal of Obstetrics and Gynaecology. I was working at the National Perinatal Epidemiology Unit at that time; Doug was at the Division of Medical Statistics at the Medical Research Council's Clinical Research Centre. Our meeting at the BJOG was the beginning of what became a very close friendship.

Doug and I shared an interest in trying to improve the quality of the manuscripts submitted to the BJOG. We commissioned three papers providing reporting guidelines for those submitting reports of controlled trials, assessments of screening and diagnostic tests, and observational studies - early examples of an interest that would become manifested in Doug's creation of the EQUATOR Network (Enhancing the QUAlity and Transparency Of health Research).

We also discovered that we had both become interested in the scientific quality of reviews of research evidence, and the potential for statistical synthesis of estimates derived from several similar studies. I had used this approach in a review of four randomized comparisons of different ways of monitoring fetuses during labour [1], the results of which prompted a very large further controlled trial that confirmed the results of the meta-analysis [2].

Doug's interest in the scientific quality of reviews of research evidence had been stimulated by two papers published in the late 1970s by Richard Peto [3, 4]. These led Doug to prepare a seven-page typescript entitled "Evaluating a series of clinical trials of the same treatment" for presentation at the 1981 meeting of the International Epidemiological Association in Edinburgh [5]. Over the next two years Doug extended the material in the 7-page typescript to a 40-page typescript with the same title [6].

DOUG'S PIONEERING CONCEPTUALIZATION OF SYSTEMATIC REVIEWS AND THE ROLE OF META-ANALYSIS

Doug's 1983 paper is important in the history of systematic reviews because of his prescience of what is important in the science of research synthesis. Unfortunately, it has been hidden from view because it was never formally published. I think Doug first showed me "the almost final version of [his] 1983 paper (complete with handwritten corrections)" at the end of 1986. He said he intended to finalize and submit it for publication, but that did not happen. As he admitted more than two decades later, "I wish I had published my ideas back in 1983" [7]. Since 2011, the typescripts of both papers [5, 6] have been available in the James Lind Library, and the shorter paper, with an accompanying commentary by Doug, is also available in the Cochrane Methods supplement to the Cochrane Database of Systematic Reviews [8].

In both these papers Doug touched on issues that would become more widely recognized as important by the 1990s. In particular, he made clear that techniques of statistical synthesis - "meta-analysis" - were but one element in a science of research synthesis, and usually not the most important. He made clear that, although statistical synthesis could address those elements of between-study variability due to random variation, it could not deal with other sources of variability - differences in entry criteria, study populations, the methods used to generate comparison groups, baseline differences between treatment groups, degrees of blindness achieved, and variations in and deviations from treatment protocols. Doug comments at the beginning of a nine-page section on "Combining the data" in the longer paper that "Since the main purpose of the paper is to discuss the whole issue of whether or not to combine trials rather than to carry out a comparison of the available methods, not all of the possible statistical methods will be described" [6]. Both his papers stressed the likely importance of publication bias and he regretted the lack (then) of hard evidence of the bias and the challenges this posed. He makes the important and too often neglected point:

Although the problem of possible publication bias may appear to be a major restriction on the validity of combining the results from several trials, it is important to realise that any such bias applies to the interpretation of individual studies, although this is always ignored and each study's results taken at face value.

([6], p. 25)

Toward the end of his 1983 paper, Doug presciently identified two desirable developments that would become widely appreciated by the end of the decade. First, the use of individual patient data:

In view of the non-statistical problems in the combination of results from different trials, the choice of statistical method is unlikely to matter greatly, but methods which make use of the raw data are definitely preferable to the combination of probabilities. The pooled estimate of relative risk should be presented with its confidence interval.

([6], p. 33)

Secondly, there is a paragraph in a section of the paper entitled "Ethical considerations" that anticipates developments in thinking and practice during the 1980s and 1990s, which Doug selected for attention after re-reading his paper over 30 years after drafting it [8]. Here's the paragraph that had struck him:

[it] is important to consider whether the results of a series of studies of the same treatment should be accumulated on a regular basis in order to monitor the current state of knowledge about those treatments. Further trials might then be dependent on the combined significance of already completed trials but using a stricter level of statistical significance (say P < 0.001) than is usually applied in single trials. Even without such information trials should perhaps not be given ethical committee approval unless the researchers had analysed the results of published trials in the way suggested in order to demonstrate that there was still uncertainty about the efficacy of the treatment, and the range of uncertainty encompassed clinically relevant benefit. Further, power calculations for a new trial could be conditional on the results of published trials.

([6], p. 27)

THE ORIGINS OF SYSTEMATIC REVIEWS IN HEALTH RESEARCH: META-ANALYSIS IN CONTEXT

Following wider recognition of the need to improve the scientific quality of reviews [9-11], the opening of the Cochrane Centre in Oxford in October 1992 helped to generate interest in the science of research synthesis [12]. I was delighted that Richard Smith, editor of the British Medical Journal, recognized this and proposed an all-day meeting run jointly by the BMJ and The Cochrane Centre. I was very glad that he accepted that the title of the meeting would refer to systematic reviews, and not to meta-analysis, as had been proposed originally. The meeting was held at the Royal Institution on 7 July 1993. Eight presentations covered the development of systematic reviews; doubts about them and the challenge of finding relevant studies; rationale and practicalities; and assessing, updating, and disseminating systematic reviews.

Based on the presentations made at the meeting, a series of articles about systematic reviews began in the 3 September 1994 issue of the BMJ. In his "Editor's Choice" column, Richard Smith noted that systematic review was "one of the most valuable tools in assessing new treatments and technologies" [13]. He was even more supportive in his Editor's Choice column a few weeks later:

Systematic reviews provide the highest quality evidence on treatment. The author of a systematic review poses a clear question, gathers all relevant trials (whether published or not), weeds out the scientifically flawed, and then amalgamates the remaining trials to reach a conclusion. Every stage in the process is crucial, and an article in the journal by Kay Dickersin and her colleagues shows how a careful Medline search for randomised controlled trials will not detect all such trials even in the journals indexed in Medline.

[14]

Richard Smith went on to point out that systematic reviews are also important because - by amalgamating data from similar trials - they can increase the statistical power of treatment comparisons [14]. These succinct explanations of the rationale for systematic reviews made by the Editor-in-Chief of one of the world's most prominent medical journals were heartening to those of us calling for improvements in the scientific quality of reviews of research.

The BMJ's series of articles on systematic reviews was well received and Richard Smith proposed...