A practical guide to network meta-analysis with examples and code
In the evaluation of healthcare, rigorous methods of quantitative assessment are necessary to establish which interventions are effective and cost-effective. Often a single study will not provide the answers and it is desirable to synthesise evidence from multiple sources, usually randomised controlled trials. This book takes an approach to evidence synthesis that is specifically intended for decision making when there are two or more treatment alternatives being evaluated, and assumes that the purpose of every synthesis is to answer the question 'for this pre-identified population of patients, which treatment is 'best'?'
A comprehensive, coherent framework for network meta-analysis (mixed treatment comparisons) is adopted and estimated using Bayesian Markov Chain Monte Carlo methods implemented in the freely available software WinBUGS. Each chapter contains worked examples, exercises, solutions and code that may be adapted by readers to apply to their own analyses.
This book can be used as an introduction to evidence synthesis and network meta-analysis, its key properties and policy implications. Examples and advanced methods are also presented for the more experienced reader.
- Methods used throughout this book can be applied consistently: model critique and checking for evidence consistency are emphasised.
- Methods are based on technical support documents produced for NICE Decision Support Unit, which support the NICE Methods of Technology Appraisal.
- Code presented is also the basis for the code used by the ISPOR Task Force on Indirect Comparisons.
- Includes extensive carefully worked examples, with thorough explanations of how to set out data for use in WinBUGS and how to interpret the output.
Network Meta-Analysis for Decision Making will be of interest to decision makers, medical statisticians, health economists, and anyone involved in Health Technology Assessment including the pharmaceutical industry.
SOFIA DIAS, University of Bristol, UK
A.E. ADES, University of Bristol, UK
NICKY J. WELTON, University of Bristol, UK
JEROEN P. JANSEN, Precision Health Economics, USA
ALEXANDER J. SUTTON, University of Leicester, UK
1. Who Is This Book for?
This book is intended for anyone who has an interest in the synthesis, or 'pooling', of evidence from randomised controlled trials (RCTs) and particularly in the statistical methods for network meta-analysis. A standard meta-analysis is used to pool information from trials that compare two interventions, while network meta-analysis extends this to the comparison of any number of interventions.
Network meta-analysis is one of the core methodologies of what has been called comparative effectiveness research (Iglehart, 2009), and, in view of the prime role accorded to trial evidence over other forms of evidence on comparative efficacy, it might be considered to be the most important.
The core material in this book is largely based on a 3-day course that we have been running for several years. Based on the spectrum of participants we see on our course, we believe the book will engage a broad range of professionals and academics. Firstly, it should appeal to all statisticians who have an interest in evidence synthesis, whether from a methodological viewpoint or because they are involved in applied work arising from systematic reviews, including the work of the Cochrane Collaboration.
Secondly, the methods are an essential component of health technology assessment (HTA) and are routinely used in submissions not only to re-imbursement agencies such as the National Institute for Health and Care Excellence (NICE) in England but also, increasingly, to similar organisations worldwide, including the Canadian Agency for Drugs and Technologies in Health, the US Agency for Healthcare Research and Quality and the Institute for Quality and Efficiency in Health Care in Germany. Health economists involved in HTA in academia and those working in pharmaceutical companies, or for the consultancy firms who assist them in making submissions to these bodies, comprise the largest single professional group for whom this book is intended.
Clinical guidelines are also making increasing use of network meta-analysis, and statisticians and health economists working with medical colleges on guideline development represent a third group who will find this book highly relevant.
Finally, the book will also be of interest, we believe, to those whose role is to manage systematic reviews, clinical guideline development or HTA exercises and those responsible at a strategic level for determining the methodological approaches that should underpin these activities. For these readers, who may not be interested in the technical details, the book sets out the assumptions of network meta-analysis, its properties, when it is appropriate and when it is not.
The book can be used in a variety of ways to suit different backgrounds and interests, and we suggest some different routes through the book at the end of the preface.
2. The Decision-Making Context
The contents of this book have their origins in the methodology guidance that was produced for submissions to NICE. This is the body in England and Wales responsible for deciding which new pharmaceuticals are to be used in the National Health Service. This context has shaped the methods from the beginning.
First and foremost, the book is about an approach to evidence synthesis that is specifically intended for decision-making. It assumes that the purpose of every synthesis is to answer the question 'for this pre-identified population of patients, which treatment is "best"?' Such decisions can be made on any one of a range of grounds: efficacy alone, some balance of efficacy and side effects, perhaps through multi-criteria decision analysis (MCDA) or cost-effectiveness. At NICE, decisions are based on efficacy and cost-effectiveness, but whatever criteria are used, the decision-making context impacts evidence synthesis methodology in several ways.
Firstly, the decision maker must have in mind a quite specific target population, not simply patients with a particular medical condition but also patients who have reached a certain point in their natural history or in their referral pathway. These factors influence a clinician's choice of treatment for an individual patient, and we should therefore expect them to impact how the evidence base and the decision options are identified. Similarly, the candidate interventions must also be characterised specifically, bearing in mind the dose, mode of delivery and concomitant treatments. Each variant has a different effect and also a different cost, both of which might be taken into account in any formal decision-making process. It has long been recognised that trial inclusion criteria for the decision-making context will tend to be more narrowly drawn than those for the broader kinds of synthesis that aim for what may be best described as a 'summary' of the literature (Eccles et al., 2001). In a similar vein Rubin (1990) has distinguished between evidence synthesis as 'science' and evidence synthesis as 'summary'. The common use of random effects models to average over important heterogeneity has attracted particular criticism (Greenland, 1994a, 1994b).
Recognising the centrality of this issue, the Cochrane Handbook (Higgins and Green, 2008) states: 'meta-analysis should only be considered when a group of studies is sufficiently homogeneous in terms of participants, interventions and outcomes to provide a meaningful summary'. However, perhaps because of the overriding focus on scouring the literature to secure 'complete' ascertainment of trial evidence, this advice is not always followed in practice. For example, an overview of treatments for enuresis (Russell and Kiddoo, 2006) put together studies on treatments for younger, treatment-naïve children, with studies on older children who had failed on standard interventions. Not surprisingly, extreme levels of statistical heterogeneity were observed, reflecting the clinical heterogeneity of the populations included (Caldwell et al., 2010). This throws doubt on any attempt to achieve a clinically meaningful answer to the question 'which treatment is best?' based on such a heterogeneous body of evidence. Similarly, one cannot meaningfully assess the efficacy of biologics in rheumatoid arthritis by combining trials on first-time use of biologics with trials on patients who have failed on biologic therapy (Singh et al., 2009). These two groups of patients require different decisions based on analyses of different sets of trials. A virtually endless list of examples could be cited. The key point is that the immediate effect of the decision-making perspective, in contrast to the systematic review perspective, is to greatly reduce the clinical heterogeneity of the trial populations under consideration.
The decision-making context has also made the adoption of Bayesian Markov chain Monte Carlo (MCMC) methods almost inevitable. The preferred form of cost-effectiveness analysis at NICE is based on probabilistic decision analysis (Doubilet et al., 1985; Critchfield and Willard, 1986; Claxton et al., 2005b). Uncertainty in parameters arising from statistical sampling error and other sources of uncertainty can be propagated through the decision model to be reflected in uncertainty in the decision. The decision itself is made on a 'balance of evidence' basis: it is an 'optimal' decision, given the available evidence, but not necessarily the 'correct' decision, because it is made under uncertainty.
Simulation from Bayesian posterior distributions therefore gives a 'one-step' solution, allowing proper statistical estimation and inference to be embedded within a probabilistic decision analysis, an approach sometimes called 'comprehensive decision analysis' (Parmigiani and Kamlet, 1993; Samsa et al., 1999; Parmigiani, 2002; Cooper et al., 2003; Spiegelhalter, 2003). This fits perfectly not only with cost-effectiveness analyses where the decision maker seeks to maximise the expected net benefit, seen as monetarised health gain minus costs (Claxton and Posnett, 1996; Stinnett and Mullahy, 1998), but also with decision analyses based on maximising any objective function. Throughout the book we have used the flexible and freely available WinBUGS software (Lunn et al., 2009) to carry out the MCMC computations required.
3. Transparency and Consistency of Method
Decisions on which intervention is 'best' are increasingly decisions that are made in public. They are scrutinised by manufacturers, bodies representing the health professions, ministries of health and patient organisations, often under the full view of the media. As a result, these decisions, and by extension the technical methods on which they are based, must be transparent, open to debate and capable of being applied in a consistent and fair way across a very wide range of circumstances. In the specific context of NICE, there is not only a scrupulous attention to process (National Institute for Health and Clinical Excellence, 2009b, 2009c) and method (National Institute for Health and Care Excellence, 2013a) but also an explicit rationale for the many societal judgements that are implicit in any health guidance (National Institute for Health and Clinical Excellence, 2008c).
This places quite profound constraints on the properties that methods for evidence synthesis need to have. It encourages us to adopt the same underlying models, the same way of evaluating model fit and the same model diagnostics, regardless of the form...