Statistical Methods for Data Analysis
Description
This third edition expands on the original material. Large portions of the text have been reviewed and clarified. More emphasis is devoted to machine learning including more modern concepts and examples. This book provides the reader with the main concepts and tools needed to perform statistical analyses of experimental data, in particular in the field of high-energy physics (HEP).
It starts with an introduction to probability theory and basic statistics, mainly intended as a refresher from readers' advanced undergraduate studies, but also to help them clearly distinguish between the Frequentist and Bayesian approaches and interpretations in subsequent applications. Following, the author discusses Monte Carlo methods with emphasis on techniques like Markov Chain Monte Carlo, and the combination of measurements, introducing the best linear unbiased estimator. More advanced concepts and applications are gradually presented, including unfolding and regularization procedures, culminating in the chapter devoted to discoveries and upper limits.
The reader learns through many applications in HEP where the hypothesis testing plays a major role and calculations of look-elsewhere effect are also presented. Many worked-out examples help newcomers to the field and graduate students alike understand the pitfalls involved in applying theoretical concepts to actual data.
Reviews / Votes
"The book is important because, as AI and data science continue to shape the future, much interdisciplinary work is being done in many different domains. It is a very good example of interdisciplinary physics research using AI and data science. ... Graduate students are often expected to apply theoretical knowledge. This book will be an invaluable resource for them, to jumpstart their research by getting equipped with the right statistical and data analysis toolsets." (Gulustan Dogan, Computing Reviews, August 8, 2023)More details
Other editions
Additional editions
Previous edition
Person
Luca Lista is full professor at University of Naples Federico II and Director of INFN Naples Unit. He is an experimental particle physicist and member of the CMS collaboration at CERN. He participated in the BABAR experiment at SLAC and L3 experiment at CERN. His main scientific interests are data analysis, statistical methods applied to physics and software development for scientific applications.
Content
Preface to the third edition
Preface to previous edition/s
1 Probability Theory
1.1 Why Probability Matters to a Physicist1.2 The Concept of Probability
1.3 Repeatable and Non-Repeatable Cases1.4 Different Approaches to Probability
1.5 Classical Probability1.6 Generalization to the Continuum
1.7 Axiomatic Probability Definition1.8 Probability Distributions
1.9 Conditional Probability1.10 Independent Events
1.11 Law of Total Probability1.12 Statistical Indicators: Average, Variance and Covariance
1.13 Statistical Indicators for a Finite Sample1.14 Transformations of Variables
1.15 The Law of Large Numbers1.16 Frequentist Definition of Probability
References2 Discrete Probability Distributions
2.1 The Bernoulli Distribution
2.2 The Binomial Distribution2.3 The Multinomial Distribution
2.4 The Poisson DistributionReferences
3 Probability Distribution Functions
3.1 Introduction3.2 Definition of Probability Distribution Function
3.3 Average and Variance in the Continuous Case3.4 Mode, Median, Quantiles
3.5 Cumulative Distribution3.6 Continuous Transformations of Variables
3.7 Uniform Distribution3.8 Gaussian Distribution
3.9 X^2 Distribution3.10 Log Normal Distribution
3.11 Exponential Distribution3.12 Other Distributions Useful in Physics
3.13 Central Limit Theorem
3.14 Probability Distribution Functions in More than One Dimension
3.15 Gaussian Distributions in Two or More Dimensions
References
4 Bayesian Approach to Probability
4.1 Introduction4.2 Bayes' Theorem
4.3 Bayesian Probability Definition4.4 Bayesian Probability and Likelihood Functions
4.5 Bayesian Inference4.6 Bayes Factors
4.7 Subjectiveness and Prior Choice4.8 Jeffreys' Prior
4.9 Reference priors4.10 Improper Priors
4.11 Transformations of Variables and Error PropagationReferences
5 Random Numbers and Monte Carlo Methods
5.1 Pseudorandom Numbers5.2 Pseudorandom Generators Properties
5.3 Uniform Random Number Generators5.4 Discrete Random Number Generators
5.5 Nonuniform Random Number Generators5.6 Monte Carlo Sampling
5.7 Numerical Integration with Monte Carlo Methods5.8 Markov Chain Monte Carlo
References6 Parameter Estimate
6.1 Introduction
6.2 Inference6.3 Parameters of Interest
6.4 Nuisance Parameters6.5 Measurements and Their Uncertainties
6.6 Frequentist vs Bayesian Inference6.7 Estimators
6.8 Properties of Estimators6.9 Binomial Distribution for Efficiency Estimate
6.10 Maximum Likelihood Method6.11 Errors with the Maximum Likelihood Method
6.12 Minimum X^2 and Least-Squares Methods6.13 Binned Data Samples
6.14 Error Propagation6.15 Treatment of Asymmetric Errors
References
7.1 Introduction
7.2 Simultaneous Fits and Control Regions
7.3 Weighted Average
7.4 X^2 in n Dimensions
7.5 The Best Linear Unbiased Estimator
References
8 Confidence Intervals
8.1 Introduction
8.2 Neyman Confidence Intervals8.3 Binomial Intervals
8.4 The Flip-Flopping Problem8.5 The Unified Feldman-Cousins Approach
References9 Convolution and Unfolding
9.1 Introduction
9.2 Convolution9.3 Unfolding by Inversion of the Response Matrix
9.4 Bin-by-Bin Correction Factors9.5 Regularized Unfolding
9.6 Iterative Unfolding9.7 Other Unfolding Methods
9.8 Software Implementations9.9 Unfolding in More Dimensions
References10 Hypothesis Tests
10.1 Introduction10.2 Test Statistic
10.3 Type I and Type II Errors10.4 Fisher's Linear Discriminant
10.5 The Neyman-Pearson Lemma10.6 Projective Likelihood Ratio Discriminant
10.7 Kolmogorov-Smirnov Test10.8 Wilks' Theorem
10.9 Likelihood Ratio in the Search for a New SignalReferences
11 Machine Learning
11.1 Supervised and Unsupervised Learning11.2 Terminology
11.3 Machine Learning Classification from a Statistical Point of View11.4 Bias-Variance tradeo
11.5 Overtraining11.6 Artificial Neural Networks
11.7 Deep Learning
11.8 Convolutional Neural Networks
11.9 Boosted Decision Trees
11.10 Multivariate Analysis ImplementationsReferences
12 Discoveries and Upper Limits
12.1 Searches for New Phenomena: Discovery and Upper Limits12.2 Claiming a Discovery
12.3 Excluding a Signal Hypothesis12.4 Combined Measurements and Likelihood Ratio
12.5 Definitions of Upper Limit12.6 Bayesian Approach
12.7 Frequentist Upper Limits12.8 Modified Frequentist Approach: the CLs Method
12.9 Presenting Upper Limits: the Brazil Plot12.10 Nuisance Parameters and Systematic Uncertainties
12.11 Upper Limits Using the Profile Likelihood12.12 Variations of the Profile-Likelihood Test Statistic
12.13 The Look Elsewhere EffectReferences
Index