Forecast Verification

List of Contributors xi Preface xiii Preface to the First Edition xv 1 Introduction 1 Ian T. Jolliffe and David B. Stephenson 1.1 A brief history and current practice 1 1.1.1 History 1 1.1.2 Current practice 2 1.2 Reasons for forecast verification and its benefits 3 1.3 Types of forecast and verification data 4 1.4 Scores, skill and value 5 1.4.1 Skill scores 6 1.4.2 Artificial skill 6 1.4.3 Statistical significance 7 1.4.4 Value added 8 1.5 Data quality and other practical considerations 8 1.6 Summary 9 2 Basic concepts 11 Jacqueline M. Potts 2.1 Introduction 11 2.2 Types of predictand 11 2.3 Exploratory methods 12 2.4 Numerical descriptive measures 15 2.5 Probability, random variables and expectations 20 2.6 Joint, marginal and conditional distributions 20 2.7 Accuracy, association and skill 22 2.8 Properties of verification measures 22 2.9 Verification as a regression problem 23 2.10 The Murphy-Winkler framework 25 2.11 Dimensionality of the verification problem 28 3 Deterministic forecasts of binary events 31 Robin J. Hogan and Ian B. Mason 3.1 Introduction 31 3.2 Theoretical considerations 33 3.2.1 Some basic descriptive statistics 33 3.2.2 A general framework for verification: the distributions-oriented approach 34 3.2.3 Performance measures in terms of factorizations of the joint distribution 37 3.2.4 Diagrams for visualizing performance measures 38 3.2.5 Case study: verification of cloud-fraction forecasts 41 3.3 Signal detection theory and the ROC 42 3.3.1 The signal detection model 43 3.3.2 The relative operating characteristic (ROC) 44 3.4 Metaverification: criteria for assessing performance measures 45 3.4.1 Desirable properties 45 3.4.2 Other properties 49 3.5 Performance measures 50 3.5.1 Overview of performance measures 51 3.5.2 Sampling uncertainty and confidence intervals for performance measures 55 3.5.3 Optimal threshold probabilities 57 Acknowledgements 59 4 Deterministic forecasts of multi-category events 61 Robert E. Livezey 4.1 Introduction 61 4.2 The contingency table: notation, definitions, and measures of accuracy 62 4.2.1 Notation and definitions 62 4.2.2 Measures of accuracy 64 4.3 Skill scores 64 4.3.1 Desirable attributes 65 4.3.2 Gandin and Murphy equitable scores 66 4.3.3 Gerrity equitable scores 69 4.3.4 LEPSCAT 71 4.3.5 SEEPS 72 4.3.6 Summary remarks on scores 73 4.4 Sampling variability of the contingency table and skill scores 73 5 Deterministic forecasts of continuous variables 77 Michel Deque 5.1 Introduction 77 5.2 Forecast examples 77 5.3 First-order moments 79 5.3.1 Bias 79 5.3.2 Mean Absolute Error 80 5.3.3 Bias correction and artificial skill 81 5.3.4 Mean absolute error and skill 81 5.4 Second- and higher-order moments 82 5.4.1 Mean Squared Error 82 5.4.2 MSE skill score 82 5.4.3 MSE of scaled forecasts 83 5.4.4 Correlation 84 5.4.5 An example: testing the 'limit of predictability' 86 5.4.6 Rank correlations 87 5.4.7 Comparison of moments of the marginal distributions 88 5.4.8 Graphical summaries 90 5.5 Scores based on cumulative frequency 91 5.5.1 Linear Error in Probability Space (LEPS) 91 5.5.2 Quantile-quantile plots 92 5.5.3 Conditional quantile plots 92 5.6 Summary and concluding remarks 94 6 Forecasts of spatial fields 95 Barbara G. Brown, Eric Gilleland and Elizabeth E. Ebert 6.1 Introduction 95 6.2 Matching methods 96 6.3 Traditional verification methods 97 6.3.1 Standard continuous and categorical approaches 97 6.3.2 S1 and anomaly correlation 98 6.3.3 Distributional methods 99 6.4 Motivation for alternative approaches 100 6.5 Neighbourhood methods 103 6.5.1 Comparing neighbourhoods of forecasts and observations 104 6.5.2 Comparing spatial forecasts with point observations 104 6.6 Scale separation methods 105 6.7 Feature-based methods 108 6.7.1 Feature-matching techniques 108 6.7.2 Structure-Amplitude-Location (SAL) technique 110 6.8 Field deformation methods 111 6.8.1 Location metrics 111 6.8.2 Field deformation 112 6.9 Comparison of approaches 113 6.10 New approaches and applications: the future 114 6.11 Summary 116 7 Probability forecasts 119 Jochen Broecker 7.1 Introduction 119 7.2 Probability theory 120 7.2.1 Basic concepts from probability theory 120 7.2.2 Probability forecasts, reliability and sufficiency 121 7.3 Probabilistic scoring rules 122 7.3.1 Definition and properties of scoring rules 122 7.3.2 Commonly used scoring rules 124 7.3.3 Decomposition of scoring rules 125 7.4 The relative operating characteristic (ROC) 126 7.5 Evaluation of probabilistic forecasting systems from data 128 7.5.1 Three examples 128 7.5.2 The empirical ROC 130 7.5.3 The empirical score as a measure of performance 130 7.5.4 Decomposition of the empirical score 131 7.5.5 Binning forecasts and the leave-one-out error 132 7.6 Testing reliability 134 7.6.1 Reliability analysis for forecast A: the reliability diagram 134 7.6.2 Reliability analysis for forecast B: the chi-squared test 136 7.6.3 Reliability analysis for forecast C: the PIT 138 Acknowledgements 139 8 Ensemble forecasts 141 Andreas P. Weigel 8.1 Introduction 141 8.2 Example data 142 8.3 Ensembles interpreted as discrete samples 143 8.3.1 Reliability of ensemble forecasts 144 8.3.2 Multidimensional reliability 152 8.3.3 Discrimination 157 8.4 Ensembles interpreted as probabilistic forecasts 159 8.4.1 Probabilistic interpretation of ensembles 159 8.4.2 Probabilistic skill metrics applied to ensembles 160 8.4.3 Effect of ensemble size on skill 163 8.5 Summary 166 9 Economic value and skill 167 David S. Richardson 9.1 Introduction 167 9.2 The cost/loss ratio decision model 168 9.2.1 Value of a deterministic binary forecast system 169 9.2.2 Probability forecasts 172 9.2.3 Comparison of deterministic and probabilistic binary forecasts 174 9.3 The relationship between value and the ROC 175 9.4 Overall value and the Brier Skill Score 178 9.5 Skill, value and ensemble size 180 9.6 Applications: value and forecast users 182 9.7 Summary 183 10 Deterministic forecasts of extreme events and warnings 185 Christopher A.T. Ferro and David B. Stephenson 10.1 Introduction 185 10.2 Forecasts of extreme events 186 10.2.1 Challenges 186 10.2.2 Previous studies 187 10.2.3 Verification measures for extreme events 189 10.2.4 Modelling performance for extreme events 191 10.2.5 Extreme events: summary 194 10.3 Warnings 195 10.3.1 Background 195 10.3.2 Format of warnings and observations for verification 196 10.3.3 Verification of warnings 197 10.3.4 Warnings: summary 200 Acknowledgements 201 11 Seasonal and longer-range forecasts 203 Simon J. Mason 11.1 Introduction 203 11.2 Forecast formats 204 11.2.1 Deterministic and probabilistic formats 204 11.2.2 Defining the predictand 206 11.2.3 Inclusion of climatological forecasts 206 11.3 Measuring attributes of forecast quality 207 11.3.1 Skill 207 11.3.2 Other attributes 215 11.3.3 Statistical significance and uncertainty estimates 216 11.4 Measuring the quality of individual forecasts 217 11.5 Decadal and longer-range forecast verification 218 11.6 Summary 220 12 Epilogue: new directions in forecast verification 221 Ian T. Jolliffe and David B. Stephenson 12.1 Introduction 221 12.2 Review of key concepts 221 12.3 Forecast evaluation in other disciplines 223 12.3.1 Statistics 223 12.3.2 Finance and economics 225 12.3.3 Medical and clinical studies 226 12.4 Current research and future directions 228 Acknowledgements 230 Appendix: Verification Software 231 Matthew Pocernich A.1 What is good software? 231 A.1.1 Correctness 232 A.1.2 Documentation 232 A.1.3 Open source/closed source/commercial 232 A.1.4 Large user base 232 A.2 Types of verification users 232 A.2.1 Students 233 A.2.2 Researchers 233 A.2.3 Operational forecasters 233 A.2.4 Institutional use 233 A.3 Types of software and programming languages 233 A.3.1 Spreadsheets 235 A.3.2 Statistical programming languages 235 A.4 Institutional supported software 238 A.4.1 Model Evaluation Tool (MET) 238 A.4.2 Ensemble Verification System (EVS) 239 A.4.3 EUMETCAL Forecast Verification Training Module 239 A.5 Displays of verification information 239 A.5.1 National Weather Service Performance Management 240 A.5.2 Forecast Evaluation Tool 240 Glossary 241 References 251 Index 267