Network Meta-Analysis for Decision-Making
Description
In the evaluation of healthcare, rigorous methods of quantitative assessment are necessary to establish which interventions are effective and cost-effective. Often a single study will not provide the answers and it is desirable to synthesise evidence from multiple sources, usually randomised controlled trials. This book takes an approach to evidence synthesis that is specifically intended for decision making when there are two or more treatment alternatives being evaluated, and assumes that the purpose of every synthesis is to answer the question "for this pre-identified population of patients, which treatment is 'best'?"
A comprehensive, coherent framework for network meta-analysis (mixed treatment comparisons) is adopted and estimated using Bayesian Markov Chain Monte Carlo methods implemented in the freely available software WinBUGS. Each chapter contains worked examples, exercises, solutions and code that may be adapted by readers to apply to their own analyses.
This book can be used as an introduction to evidence synthesis and network meta-analysis, its key properties and policy implications. Examples and advanced methods are also presented for the more experienced reader.
* Methods used throughout this book can be applied consistently: model critique and checking for evidence consistency are emphasised.
* Methods are based on technical support documents produced for NICE Decision Support Unit, which support the NICE Methods of Technology Appraisal.
* Code presented is also the basis for the code used by the ISPOR Task Force on Indirect Comparisons.
* Includes extensive carefully worked examples, with thorough explanations of how to set out data for use in WinBUGS and how to interpret the output.
Network Meta-Analysis for Decision Making will be of interest to decision makers, medical statisticians, health economists, and anyone involved in Health Technology Assessment including the pharmaceutical industry.
More details
Other editions
Additional editions
Persons
SOFIA DIAS, University of Bristol, UK
A.E. ADES, University of Bristol, UK
NICKY J. WELTON, University of Bristol, UK
JEROEN P. JANSEN, Precision Health Economics, USA
ALEXANDER J. SUTTON, University of Leicester, UK
Content
Preface xiii
List of Abbreviations xxi
About the Companion Website xxv
1 Introduction to Evidence Synthesis 1
1.1 Introduction 1
1.2 Why Indirect Comparisons and Network Meta?-Analysis? 2
1.3 Some Simple Methods 4
1.4 An Example of a Network Meta?-Analysis 6
1.5 Assumptions Made by Indirect Comparisons and Network Meta?-Analysis 9
1.6 Which Trials to Include in a Network 12
1.6.1 The Need for a Unique Set of Trials 12
1.7 The Definition of Treatments and Outcomes: Network Connectivity 14
1.7.1 Lumping and Splitting 14
1.7.2 Relationships Between Multiple Outcomes 15
1.7.3 How Large Should a Network Be? 15
1.8 Summary 16
1.9 Exercises 16
2 The Core Model 19
2.1 Bayesian Meta?-Analysis 19
2.2 Development of the Core Models 20
2.2.1 Worked Example: Meta?-Analysis of Binomial Data 21
2.2.1.1 Model Specification: Two Treatments 21
2.2.1.2 WinBUGS Implementation: Two Treatments 25
2.2.2 Extension to Indirect Comparisons and Network Meta?-Analysis 32
2.2.2.1 Incorporating Multi?-Arm Trials 35
2.2.3 Worked Example: Network Meta?-Analysis 36
2.2.3.1 WinBUGS Implementation 37
2.3 Technical Issues in Network Meta?-Analysis 50
2.3.1 Choice of Reference Treatment 50
2.3.2 Choice of Prior Distributions 51
2.3.3 Choice of Scale 53
2.3.4 Connected Networks 54
2.4 Advantages of a Bayesian Approach 55
2.5 Summary of Key Points and Further Reading 56
2.6 Exercises 57
3 Model Fit, Model Comparison and Outlier Detection 59
3.1 Introduction 59
3.2 Assessing Model Fit 60
3.2.1 Deviance 60
3.2.2 Residual Deviance 61
3.2.3 Zero Counts* 62
3.2.4 Worked Example: Full Thrombolytic Treatments Network 62
3.2.4.1 Posterior Mean Deviance, D¯model 62
3.2.4.2 Posterior Mean Residual Deviance, D¯res 64
3.3 Model Comparison 66
3.3.1 Effective Number of Parameters, pD 68
3.3.2 Deviance Information Criterion (DIC) 69
3.3.2.1 *Leverage Plots 70
3.3.3 Worked Example: Full Thrombolytic Treatments Network 70
3.4 Outlier Detection in Network Meta?-Analysis 75
3.4.1 Outlier Detection in Pairwise Meta?-Analysis 75
3.4.2 Predictive Cross?-Validation for Network Meta?-Analysis 79
3.4.3 Note on Multi?-Arm Trials 85
3.4.4 WinBUGS Code: Predictive Cross?-Validation for Network Meta?-Analysis 86
3.5 Summary and Further Reading 89
3.6 Exercises 90
4 Generalised Linear Models 93
4.1 A Unified Framework for Evidence Synthesis 93
4.2 The Generic Network Meta?-Analysis Models 94
4.3 Univariate Arm?-Based Likelihoods 99
4.3.1 Rate Data: Poisson Likelihood and Log Link 99
4.3.1.1 WinBUGS Implementation 100
4.3.1.2 Example: Dietary Fat 101
4.3.1.3 Results: Dietary Fat 104
4.3.2 Rate Data: Binomial Likelihood and Cloglog Link 105
4.3.2.1 WinBUGS Implementation 107
4.3.2.2 Example: Diabetes 109
4.3.2.3 Results: Diabetes 112
4.3.3 Continuous Data: Normal Likelihood and Identity Link 114
4.3.3.1 Before/After Studies: Change from Baseline Measures 115
4.3.3.2 Standardised Mean Differences 115
4.3.3.3 WinBUGS Implementation 116
4.3.3.4 Example: Parkinson's 117
4.3.3.5 Results: Parkinson's 119
4.4 Contrast?-Based Likelihoods 120
4.4.1 Continuous Data: Treatment Differences 121
4.4.1.1 Multi?-Arm Trials with Treatment Differences (Trial?-Based Summaries) 122
4.4.1.2 *WinBUGS Implementation 123
4.4.1.3 Example: Parkinson's (Treatment Differences as Data) 125
4.4.1.4 Results: Parkinson's (Treatment Differences as Data) 127
4.5 *Multinomial Likelihoods 127
4.5.1 Ordered Categorical Data: Multinomial Likelihood and Probit Link 128
4.5.1.1 WinBUGS Implementation 132
4.5.1.2 Example: Psoriasis 133
4.5.1.3 Results: Psoriasis 137
4.5.2 Competing Risks: Multinomial Likelihood and Log Link 138
4.5.2.1 WinBUGS Implementation 140
4.5.2.2 Example: Schizophrenia 141
4.5.2.3 Results: Schizophrenia 143
4.6 *Shared Parameter Models 146
4.6.1 Example: Parkinson's (Mixed Treatment Difference and Arm?-Level Data) 147
4.6.2 Results: Parkinson's (Mixed Treatment Difference and Arm?-Level Data) 148
4.7 Choice of Prior Distributions 149
4.8 Zero Cells 149
4.9 Summary of Key Points and Further Reading 150
4.10 Exercises 151
5 Network Meta?-Analysis Within Cost?-Effectiveness Analysis 155
5.1 Introduction 155
5.2 Sources of Evidence for Relative Treatment Effects and the Baseline Model 156
5.3 The Baseline Model 158
5.3.1 Estimating the Baseline Model in WinBUGS 158
5.3.2 Alternative Computation Methods for the Baseline Model 162
5.3.3 *Arm?-Based Meta?-Analytic Models 162
5.3.4 Baseline Models with Covariates 164
5.3.4.1 Using Aggregate Data 164
5.3.4.2 Risk Equations for the Baseline Model Basedon Individual Patient Data 165
5.4 The Natural History Model 165
5.5 Model Validation and Calibration Through Multi?-Parameter Synthesis 167
5.6 Generating the Outputs Required for Cost?-Effectiveness Analysis 169
5.6.1 Generating a CEA 169
5.6.2 Heterogeneity in the Context of Decision?-Making 170
5.7 Strategies to Implement Cost?-Effectiveness Analyses 173
5.7.1 Bayesian Posterior Simulation: One?-Stage Approach 174
5.7.2 Bayesian Posterior Simulation: Two?-Stage Approach 174
5.7.3 Multiple Software Platforms and Automation of Network Meta?-Analysis 175
5.8 Summary and Further Reading 177
5.9 Exercises 178
6 Adverse Events and Other Sparse Outcome Data 179
6.1 Introduction 179
6.2 Challenges Regarding the Analysis of Sparse Data in Pairwise and Network Meta?-Analysis 180
6.2.1 Network Structure and Connectivity 182
6.2.2 Assessing Convergence and Model Fit 182
6.3 Strategies to Improve the Robustness of Estimation of Effects from Sparse Data in Network Meta?-Analysis 183
6.3.1 Specifying Informative Prior Distributions for Response in Trial Reference Groups 183
6.3.2 Specifying an Informative Prior Distribution for the Between Study Variance Parameters 184
6.3.3 Specifying Reference Group Responses as Exchangeable with Random Effects 184
6.3.4 Situational Modelling Extensions 185
6.3.5 Specification of Informative Prior Distributions Versus Use of Continuity Corrections 186
6.4 Summary and Further Reading 186
6.5 Exercises 187
7 Checking for Inconsistency 189
7.1 Introduction 189
7.2 Network Structure 190
7.2.1 Inconsistency Degrees of Freedom 191
7.2.2 Defining Inconsistency in the Presence of Multi?-Arm Trials 192
7.3 Loop Specific Tests for Inconsistency 195
7.3.1 Networks with Independent Tests for Inconsistency 195
7.3.1.1 Bucher Method for Single Loops of Evidence 195
7.3.1.2 Example: HIV 196
7.3.1.3 Extension of Bucher Method to Networks with Multiple Loops: Enuresis Example 197
7.3.1.4 Obtaining the 'Direct' Estimates of Inconsistency 199
7.3.2 Methods for General Networks 200
7.3.2.1 Repeat Application of the Bucher Method 201
7.3.2.2 A Back?-Calculation Method 202
7.3.2.3 *Variance Measures of Inconsistency 202
7.3.2.4 *Node?-Splitting 203
7.4 A Global Test for Loop Inconsistency 205
7.4.1 Inconsistency Model with Unrelated Mean Relative Effects 206
7.4.2 Example: Full Thrombolytic Treatments Network 210
7.4.2.1 Adjusted Standard Errors for Multi?-Arm Trials 214
7.4.3 Example: Parkinson's 215
7.4.4 Example: Diabetes 218
7.5 Response to Inconsistency 219
7.6 The Relationship between Heterogeneity and Inconsistency 221
7.7 Summary and Further Reading 223
7.8 Exercises 225
8 Meta?-Regression for Relative Treatment Effects 227
8.1 Introduction 227
8.2 Basic Concepts 229
8.2.1 Types of Covariate 229
8.3 Heterogeneity, Meta?-Regression and Predictive Distributions 232
8.3.1 Worked Example: BCG Vaccine 233
8.3.2 Implications of Heterogeneity in Decision Making 236
8.4 Meta?-Regression Models for Network Meta?-Analysis 238
8.4.1 Baseline Risk 241
8.4.2 WinBUGS Implementation 242
8.4.3 Meta?-Regression with a Continuous Covariate 245
8.4.3.1 BCG Vaccine Example: Pairwise Meta?-Regression with a Continuous Covariate 245
8.4.3.2 Certolizumab Example: Network Meta?-Regression with Continuous Covariate 247
8.4.3.3 Certolizumab Example: Network Meta?-Regression on Baseline Risk 252
8.4.4 Subgroup Effects 255
8.4.4.1 Statins Example: Pairwise Meta?-Analysis with Subgroups 256
8.5 Individual Patient Data in Meta?-Regression 257
8.6 Models with Treatment?-Level Covariates 261
8.6.1 Accounting for Dose 261
8.6.2 Class Effects Models 263
8.6.3 Treatment Combination Models 264
8.7 Implications of Meta?-Regression for Decision Making 266
8.8 Summary and Further Reading 268
8.9 Exercises 269
9 Bias Adjustment Methods 273
9.1 Introduction 273
9.2 Adjustment for Bias Based on Meta?-Epidemiological Data 275
9.3 Estimation and Adjustment for Bias in Networks of Trials 278
9.3.1 Worked Example: Fluoride Therapies for the Prevention of Caries in Children 279
9.3.2 Extensions 285
9.3.3 Novel Agent Effects 286
9.3.4 Small?-Study Effects 287
9.3.5 Industry Sponsor Effects 287
9.3.6 Accounting for Missing Data 288
9.4 Elicitation of Internal and External Bias Distributions from Experts 289
9.5 Summary and Further Reading 290
9.6 Exercises 291
10 *Network Meta?-Analysis of Survival Outcomes 293
10.1 Introduction 293
10.2 Time?-to?-Event Data 294
10.2.1 Individual Patient Data 294
10.2.2 Reported Summary Data 295
10.2.3 Kaplan-Meier Estimate of the Survival Function 295
10.3 Parametric Survival Functions 296
10.4 The Relative Treatment Effect 298
10.5 Network Meta?-Analysis of a Single Effect Measure per Study 300
10.5.1 Proportion Alive, Median Survival and Hazard Ratio as Reported Treatment Effects 300
10.5.2 Network Meta?-Analysis of Parametric Survival Curves: Single Treatment Effect 300
10.5.3 Shared Parameter Models 301
10.5.4 Limitations 302
10.6 Network Meta?-Analysis with Multivariate Treatment Effects 302
10.6.1 Multidimensional Network Meta?-Analysis Model 302
10.6.1.1 Weibull 302
10.6.1.2 Gompertz 303
10.6.1.3 Log?-Logistic and Log?-Normal 303
10.6.1.4 Fractional Polynomial 304
10.6.1.5 Splines 304
10.6.2 Evaluation of Consistency 304
10.6.3 Meta?-Regression 305
10.7 Data and Likelihood 305
10.7.1 Likelihood with Individual Patient Data 305
10.7.2 Discrete or Piecewise Constant Hazards as Approximate Likelihood 306
10.7.3 Conditional Survival Probabilities as Approximate Likelihood 307
10.7.4 Reconstructing Kaplan-Meier Data 307
10.7.5 Constructing Interval Data 308
10.8 Model Choice 308
10.9 Presentation of Results 309
10.10 Illustrative Example 310
10.11 Network Meta?-Analysis of Survival Outcomes for Cost?-Effectiveness Evaluations 319
10.12 Summary and Further Reading 320
10.13 Exercises 322
11 *Multiple Outcomes 323
11.1 Introduction 323
11.2 Multivariate Random Effects Meta?-Analysis 324
11.3 Multinomial Likelihoods and Extensions of Univariate Methods 327
11.4 Chains of Evidence 328
11.4.1 A Decision Tree Structure: Coronary Patency 328
11.4.2 Chain of Evidence with Relative Risks: Neonatal Early Onset Group B Strep 330
11.5 Follow?-Up to Multiple Time Points: Gastro?-Esophageal Reflux Disease 332
11.6 Multiple Outcomes Reported in Different Ways: Influenza 335
11.7 Simultaneous Mapping and Synthesis 337
11.8 Related Outcomes Reported in Different Ways: Advanced Breast Cancer 342
11.9 Repeat Observations for Continuous Outcomes: Fractional Polynomials 344
11.10 Synthesis for Markov Models 345
11.11 Summary and Further Reading 347
11.12 Exercises 349
12 Validity of Network Meta?-Analysis 351
12.1 Introduction 351
12.2 What Are the Assumptions of Network Meta?-Analysis? 352
12.2.1 Exchangeability 352
12.2.2 Other Terminologies and Their Relation to Exchangeability 353
12.3 Direct and Indirect Comparisons: Some Thought Experiments 355
12.3.1 Direct Comparisons 356
12.3.2 Indirect Comparisons 359
12.3.3 Under What Conditions Is Evidence Synthesis Likely to Be Valid? 362
12.4 Empirical Studies of the Consistency Assumption 363
12.5 Quality of Evidence Versus Reliability of Recommendation 365
12.5.1 Theoretical Treatment of Validity of Network Meta?-Analysis 365
12.5.2 GRADE Assessment of Quality of Evidence from a Network Meta?-Analyses 366
12.5.3 Reliability of Recommendations Versus Quality of Evidence: The Role of Sensitivity Analysis 368
12.6 Summary and Further Reading 369
12.7 Exercises 373
Solutions to Exercises 375
Appendices 401
References 409
Index 447
Preface
1. Who Is This Book for?
This book is intended for anyone who has an interest in the synthesis, or 'pooling', of evidence from randomised controlled trials (RCTs) and particularly in the statistical methods for network meta-analysis. A standard meta-analysis is used to pool information from trials that compare two interventions, while network meta-analysis extends this to the comparison of any number of interventions.
Network meta-analysis is one of the core methodologies of what has been called comparative effectiveness research (Iglehart, 2009), and, in view of the prime role accorded to trial evidence over other forms of evidence on comparative efficacy, it might be considered to be the most important.
The core material in this book is largely based on a 3-day course that we have been running for several years. Based on the spectrum of participants we see on our course, we believe the book will engage a broad range of professionals and academics. Firstly, it should appeal to all statisticians who have an interest in evidence synthesis, whether from a methodological viewpoint or because they are involved in applied work arising from systematic reviews, including the work of the Cochrane Collaboration.
Secondly, the methods are an essential component of health technology assessment (HTA) and are routinely used in submissions not only to re-imbursement agencies such as the National Institute for Health and Care Excellence (NICE) in England but also, increasingly, to similar organisations worldwide, including the Canadian Agency for Drugs and Technologies in Health, the US Agency for Healthcare Research and Quality and the Institute for Quality and Efficiency in Health Care in Germany. Health economists involved in HTA in academia and those working in pharmaceutical companies, or for the consultancy firms who assist them in making submissions to these bodies, comprise the largest single professional group for whom this book is intended.
Clinical guidelines are also making increasing use of network meta-analysis, and statisticians and health economists working with medical colleges on guideline development represent a third group who will find this book highly relevant.
Finally, the book will also be of interest, we believe, to those whose role is to manage systematic reviews, clinical guideline development or HTA exercises and those responsible at a strategic level for determining the methodological approaches that should underpin these activities. For these readers, who may not be interested in the technical details, the book sets out the assumptions of network meta-analysis, its properties, when it is appropriate and when it is not.
The book can be used in a variety of ways to suit different backgrounds and interests, and we suggest some different routes through the book at the end of the preface.
2. The Decision-Making Context
The contents of this book have their origins in the methodology guidance that was produced for submissions to NICE. This is the body in England and Wales responsible for deciding which new pharmaceuticals are to be used in the National Health Service. This context has shaped the methods from the beginning.
First and foremost, the book is about an approach to evidence synthesis that is specifically intended for decision-making. It assumes that the purpose of every synthesis is to answer the question 'for this pre-identified population of patients, which treatment is "best"?' Such decisions can be made on any one of a range of grounds: efficacy alone, some balance of efficacy and side effects, perhaps through multi-criteria decision analysis (MCDA) or cost-effectiveness. At NICE, decisions are based on efficacy and cost-effectiveness, but whatever criteria are used, the decision-making context impacts evidence synthesis methodology in several ways.
Firstly, the decision maker must have in mind a quite specific target population, not simply patients with a particular medical condition but also patients who have reached a certain point in their natural history or in their referral pathway. These factors influence a clinician's choice of treatment for an individual patient, and we should therefore expect them to impact how the evidence base and the decision options are identified. Similarly, the candidate interventions must also be characterised specifically, bearing in mind the dose, mode of delivery and concomitant treatments. Each variant has a different effect and also a different cost, both of which might be taken into account in any formal decision-making process. It has long been recognised that trial inclusion criteria for the decision-making context will tend to be more narrowly drawn than those for the broader kinds of synthesis that aim for what may be best described as a 'summary' of the literature (Eccles et al., 2001). In a similar vein Rubin (1990) has distinguished between evidence synthesis as 'science' and evidence synthesis as 'summary'. The common use of random effects models to average over important heterogeneity has attracted particular criticism (Greenland, 1994a, 1994b).
Recognising the centrality of this issue, the Cochrane Handbook (Higgins and Green, 2008) states: 'meta-analysis should only be considered when a group of studies is sufficiently homogeneous in terms of participants, interventions and outcomes to provide a meaningful summary'. However, perhaps because of the overriding focus on scouring the literature to secure 'complete' ascertainment of trial evidence, this advice is not always followed in practice. For example, an overview of treatments for enuresis (Russell and Kiddoo, 2006) put together studies on treatments for younger, treatment-naïve children, with studies on older children who had failed on standard interventions. Not surprisingly, extreme levels of statistical heterogeneity were observed, reflecting the clinical heterogeneity of the populations included (Caldwell et al., 2010). This throws doubt on any attempt to achieve a clinically meaningful answer to the question 'which treatment is best?' based on such a heterogeneous body of evidence. Similarly, one cannot meaningfully assess the efficacy of biologics in rheumatoid arthritis by combining trials on first-time use of biologics with trials on patients who have failed on biologic therapy (Singh et al., 2009). These two groups of patients require different decisions based on analyses of different sets of trials. A virtually endless list of examples could be cited. The key point is that the immediate effect of the decision-making perspective, in contrast to the systematic review perspective, is to greatly reduce the clinical heterogeneity of the trial populations under consideration.
The decision-making context has also made the adoption of Bayesian Markov chain Monte Carlo (MCMC) methods almost inevitable. The preferred form of cost-effectiveness analysis at NICE is based on probabilistic decision analysis (Doubilet et al., 1985; Critchfield and Willard, 1986; Claxton et al., 2005b). Uncertainty in parameters arising from statistical sampling error and other sources of uncertainty can be propagated through the decision model to be reflected in uncertainty in the decision. The decision itself is made on a 'balance of evidence' basis: it is an 'optimal' decision, given the available evidence, but not necessarily the 'correct' decision, because it is made under uncertainty.
Simulation from Bayesian posterior distributions therefore gives a 'one-step' solution, allowing proper statistical estimation and inference to be embedded within a probabilistic decision analysis, an approach sometimes called 'comprehensive decision analysis' (Parmigiani and Kamlet, 1993; Samsa et al., 1999; Parmigiani, 2002; Cooper et al., 2003; Spiegelhalter, 2003). This fits perfectly not only with cost-effectiveness analyses where the decision maker seeks to maximise the expected net benefit, seen as monetarised health gain minus costs (Claxton and Posnett, 1996; Stinnett and Mullahy, 1998), but also with decision analyses based on maximising any objective function. Throughout the book we have used the flexible and freely available WinBUGS software (Lunn et al., 2009) to carry out the MCMC computations required.
3. Transparency and Consistency of Method
Decisions on which intervention is 'best' are increasingly decisions that are made in public. They are scrutinised by manufacturers, bodies representing the health professions, ministries of health and patient organisations, often under the full view of the media. As a result, these decisions, and by extension the technical methods on which they are based, must be transparent, open to debate and capable of being applied in a consistent and fair way across a very wide range of circumstances. In the specific context of NICE, there is not only a scrupulous attention to process (National Institute for Health and Clinical Excellence, 2009b, 2009c) and method (National Institute for Health and Care Excellence, 2013a) but also an explicit rationale for the many societal judgements that are implicit in any health guidance (National Institute for Health and Clinical Excellence, 2008c).
This places quite profound constraints on the properties that methods for evidence synthesis need to have. It encourages us to adopt the same underlying models, the same way of evaluating model fit and the same model diagnostics, regardless of the form...
