Statistical Pattern Recognition

Preface xix Notation xxiii 1 Introduction to Statistical Pattern Recognition 1 1.1 Statistical Pattern Recognition 1 1.2 Stages in a Pattern Recognition Problem 4 1.3 Issues 6 1.4 Approaches to Statistical Pattern Recognition 7 1.5 Elementary Decision Theory 8 1.6 Discriminant Functions 20 1.7 Multiple Regression 27 1.8 Outline of Book 29 1.9 Notes and References 29 Exercises 31 2 Density Estimation - Parametric 33 2.1 Introduction 33 2.2 Estimating the Parameters of the Distributions 34 2.3 The Gaussian Classifier 35 2.4 Dealing with Singularities in the Gaussian Classifier 40 2.5 Finite Mixture Models 46 2.6 Application Studies 63 2.7 Summary and Discussion 66 2.8 Recommendations 66 2.9 Notes and References 67 Exercises 67 3 Density Estimation - Bayesian 70 3.1 Introduction 70 3.2 Analytic Solutions 73 3.3 Bayesian Sampling Schemes 87 3.4 Markov Chain Monte Carlo Methods 95 3.5 Bayesian Approaches to Discrimination 116 3.6 Sequential Monte Carlo Samplers 119 3.7 Variational Bayes 126 3.8 Approximate Bayesian Computation 137 3.9 Example Application Study 144 3.10 Application Studies 145 3.11 Summary and Discussion 146 3.12 Recommendations 147 3.13 Notes and References 147 Exercises 148 4 Density Estimation - Nonparametric 150 4.1 Introduction 150 4.2 k-Nearest-Neighbour Method 152 4.3 Histogram Method 180 4.4 Kernel Methods 194 4.5 Expansion by Basis Functions 204 4.6 Copulas 207 4.7 Application Studies 213 4.8 Summary and Discussion 216 4.9 Recommendations 217 4.10 Notes and References 217 Exercises 218 5 Linear Discriminant Analysis 221 5.1 Introduction 221 5.2 Two-Class Algorithms 222 5.3 Multiclass Algorithms 236 5.4 Support Vector Machines 249 5.5 Logistic Discrimination 263 5.6 Application Studies 268 5.7 Summary and Discussion 268 5.8 Recommendations 269 5.9 Notes and References 270 Exercises 270 6 Nonlinear Discriminant Analysis - Kernel and Projection Methods 274 6.1 Introduction 274 6.2 Radial Basis Functions 276 6.3 Nonlinear Support Vector Machines 291 6.4 The Multilayer Perceptron 298 6.5 Application Studies 314 6.6 Summary and Discussion 316 6.7 Recommendations 317 6.8 Notes and References 318 Exercises 318 7 Rule and Decision Tree Induction 322 7.1 Introduction 322 7.2 Decision Trees 323 7.3 Rule Induction 342 7.4 Multivariate Adaptive Regression Splines 351 7.5 Application Studies 356 7.6 Summary and Discussion 358 7.7 Recommendations 358 7.8 Notes and References 359 Exercises 359 8 Ensemble Methods 361 8.1 Introduction 361 8.2 Characterising a Classifier Combination Scheme 362 8.3 Data Fusion 370 8.4 Classifier Combination Methods 376 8.5 Application Studies 399 8.6 Summary and Discussion 400 8.7 Recommendations 401 8.8 Notes and References 401 Exercises 402 9 Performance Assessment 404 9.1 Introduction 404 9.2 Performance Assessment 405 9.3 Comparing Classifier Performance 424 9.4 Application Studies 429 9.5 Summary and Discussion 430 9.6 Recommendations 430 9.7 Notes and References 430 Exercises 431 10 Feature Selection and Extraction 433 10.1 Introduction 433 10.2 Feature Selection 435 10.3 Linear Feature Extraction 463 10.4 Multidimensional Scaling 484 10.5 Application Studies 493 10.6 Summary and Discussion 495 10.7 Recommendations 495 10.8 Notes and References 496 Exercises 497 11 Clustering 501 11.1 Introduction 501 11.2 Hierarchical Methods 502 11.3 Quick Partitions 510 11.4 Mixture Models 511 11.5 Sum-of-Squares Methods 513 11.6 Spectral Clustering 531 11.7 Cluster Validity 538 11.8 Application Studies 546 11.9 Summary and Discussion 549 11.10 Recommendations 551 11.11 Notes and References 552 Exercises 553 12 Complex Networks 555 12.1 Introduction 555 12.2 Mathematics of Networks 561 12.3 Community Detection 565 12.4 Link Prediction 575 12.5 Application Studies 579 12.6 Summary and Discussion 579 12.7 Recommendations 580 12.8 Notes and References 580 Exercises 580 13 Additional Topics 581 13.1 Model Selection 581 13.2 Missing Data 585 13.3 Outlier Detection and Robust Procedures 586 13.4 Mixed Continuous and Discrete Variables 587 13.5 Structural Risk Minimisation and the Vapnik-Chervonenkis Dimension 588 References 591 Index 637