Statistics
Description
The book places focus early on continuous measurements, as well as discrete random variables. By invoking simple and intuitive models and geometric probability, discrete and continuous experiments and probabilities are discussed throughout the book in a natural way. Classical probability, random variables, and inference are discussed, as well as material on understanding data and topics of special interest.
Topics discussed include:
* Classical equally likely outcomes
* Variety of models of discrete and continuous probability laws
* Likelihood function and ratio
* Inference
* Bayesian statistics
With the growth in the volume of data generated in many disciplines that is enabling the growth in data science, companies now demand statistically literate scientists and this textbook is the answer, suited for undergraduates studying science or engineering, be it computer science, economics, life sciences, environmental, business, amongst many others. Basic knowledge of bivariate calculus, R language, Matematica and JMP is useful, however there is an accompanying website including sample R and Mathematica code to help instructors and students.
More details
Other editions
Additional editions
Person
DAVID W. SCOTT is the Noah Harding Professor of Statistics at Rice University in Houston, Texas. He is a Fellow of the ASA, IMS, AAAS, an elected member of the ISI and received the 2004 Army Wilks Award and the 2008 ASA Founder's Award. He was formerly the Editor of the Journal of Computational and Graphical Statistics and currently serves as Co-Editor of Wiley Interdisciplinary Reviews: Computational Statistics. He is also the author of Multivariate Density Estimation: Theory, Practice, and Visualization.
Content
Preface xiii
1 Data Analysis and Understanding 1
1.1 Exploring the Distribution of Data 1
1.1.1 Pearson's Father-Son Height Data 2
1.1.2 Lord Rayleigh's Data 3
1.1.3 Discussion 4
1.2 Exploring Prediction Using Data 4
1.2.1 Body and Brain Weights of Land Mammals 5
1.2.2 Space Shuttle Flight 25 5
1.2.3 Pearson's Father-Son Height Data Revisited 7
1.2.4 Discussion 8
Problems 8
2 Classical Probability 11
2.1 Experiments with Equally Likely Outcomes 11
2.1.1 Simple Outcomes 12
2.1.2 Compound Events and Set Operations 12
2.2 Probability Laws 13
2.2.1 Union and Intersection of Events A and B 13
2.2.1.1 Case (i) 14
2.2.1.2 Cases (ii) and (iii) 14
2.2.1.3 Case (iv) 14
2.2.2 Conditional Probability 14
2.2.2.1 Definition of Conditional Probability 15
2.2.2.2 Conditional Probability With More Than Two Events 15
2.2.3 Independent Events 16
2.2.4 Bayes Theorem 17
2.2.5 Partitions and Total Probability 18
2.3 Counting Methods 19
2.3.1 With Replacement 20
2.3.2 Without Replacement (Permutations) 20
2.3.3 Without Replacement or Order (Combinations) 21
2.3.4 Examples 21
2.3.5 Extended Combinations (Multinomial) 22
2.4 Countable Sets: Implications as n ¿ 8 22
2.4.1 Selecting Even or Odd Integers 22
2.4.2 Selecting Rational Versus Irrational Numbers 23
2.5 Kolmogorov's Axioms 23
2.6 Reliability: Series Versus Parallel Networks 24
2.6.1 Series Network 24
2.6.2 Parallel Network 25
Problems 26
3 Random Variables and Models Derived From Classical Probability and Postulates 27
3.1 Random Variables and Probability Distributions: Discrete Uniform Example 27
3.1.1 Toss of a Single Die 28
3.1.2 Toss of a Pair of Dice 28
3.2 The Univariate Probability Density Function: Continuous Uniform Example 30
3.2.1 Using the PDF to Compute Probabilities 32
3.2.2 Using the PDF to Compute Relative Odds 33
3.3 Summary Statistics: Central and Non-Central Moments 33
3.3.1 Expectation, Average, and Mean 34
3.3.2 Expectation as a Linear Operator 35
3.3.3 The Variance of a Random Variable 35
3.3.4 Standardized Random Variables 36
3.3.5 Higher Order Moments 37
3.3.6 Moment Generating Function 37
3.3.7 Measurement Scales and Units of Measurement 38
3.3.7.1 The Four Measurement Scales 38
3.3.7.2 Units of Measurement 39
3.4 Binomial Experiments 39
3.5 Waiting Time for a Success: Geometric PMF 42
3.6 Waiting Time for r Successes: Negative Binomial 43
3.7 Poisson Process and Distribution 43
3.7.1 Moments of the Poisson PMF 44
3.7.2 Examples 45
3.8 Waiting Time for Poisson Events: Negative Exponential PDF 45
3.9 The Normal Distribution (Also Known as the Gaussian Distribution) 46
3.9.1 Standard Normal Distribution 48
3.9.2 Sums of Independent Normal Random Variables 49
3.9.3 Normal Approximation to the Poisson Distribution 49
Problems 50
4 Bivariate Random Variables, Transformations, and Simulations 51
4.1 Bivariate Continuous Random Variables 51
4.1.1 Joint CDF and PDF Functions 51
4.1.2 Marginal PDF 52
4.1.3 Conditional Probability Density Function 52
4.1.4 Independence of Two Random Variables 54
4.1.5 Expectation, Correlation, and Regression 54
4.1.5.1 Covariance and Correlation 55
4.1.5.2 Regression Function 55
4.1.6 Independence of n Random Variables 56
4.1.7 Bivariate Normal PDF 56
4.1.8 Correlation, Independence, and Confounding Variables 57
4.2 Change of Variables 57
4.2.1 Examples: Two Uniform Transformations 57
4.2.2 One-Dimensional Transformations 58
4.2.2.1 Example 1: Negative exponential PDF 59
4.2.2.2 Example 2: Cauchy PDF 60
4.2.2.3 Example 3: Chi-squared PDF with one degree of freedom 60
4.2.3 Two-Dimensional Transformations 60
4.3 Simulations 62
4.3.1 Generating Uniform Pseudo-Random Numbers 62
4.3.1.1 Reproducibility 62
4.3.1.2 RANDU 62
4.3.2 Probability Integral Transformation 63
4.3.3 Event-driven Simulation 63
Problems 64
5 Approximations and Asymptotics 67
5.1 Why Do We Like Random Samples? 67
5.1.1 When u(X) Takes a Product Form 68
5.1.2 When u(X) Takes a Summation Form 68
5.2 Useful Inequalities 69
5.2.1 Markov's Inequality 69
5.2.2 Chebyshev's Inequality 70
5.2.3 Jensen's Inequality 70
5.2.4 Cauchy-Schwarz Inequality 71
5.3 Sequences of Random Variables 72
5.3.1 Weak Law of Large Numbers 73
5.3.2 Consistency of the Sample Variance 73
5.3.3 Relationships Among the Modes of Convergence 74
5.3.3.1 Proof of Result (5.21) 74
5.3.3.2 Proof of Result (5.22) 74
5.4 Central Limit Theorem 75
5.4.1 Moment Generating Function for Sums 75
5.4.2 Standardizing the Sum Sn 75
5.4.3 Proof of Central Limit Theorem 76
5.5 Delta Method and Variance-stabilizing Transformations 77
Problems 78
6 Parameter Estimation 79
6.1 Desirable Properties of an Estimator 80
6.2 Moments of the Sample Mean and Variance 80
6.2.1 Theoretical Mean and Variance of the Sample Mean 81
6.2.2 Theoretical Mean of the Sample Variance 81
6.2.3 Theoretical Variance of the Sample Variance 82
6.3 Method of Moments (MoM) 82
6.4 Sufficient Statistics and Data Compression 83
6.5 Bayesian Parameter Estimation 85
6.6 Maximum Likelihood Parameter Estimation 86
6.6.1 Relationship to Bayesian Parameter Estimation 87
6.6.2 Poisson MLE Example 87
6.6.3 Normal MLE Example 87
6.6.4 Uniform MLE Example 88
6.7 Information Inequalities and the Cramér-Rao Lower Bound 89
6.7.1 Score Function 89
6.7.2 Asymptotics of the MLE 90
6.7.3 Minimum Variance of Unbiased Estimators 91
6.7.4 Examples 91
Problems 92
7 Hypothesis Testing 93
7.1 Setting up a Hypothesis Test 94
7.1.1 Example of a Critical Region 94
7.1.2 Accuracy and Errors in Hypothesis Testing 95
7.2 Best Critical Region for Simple Hypotheses 96
7.2.1 Simple Example Continued 96
7.2.2 Normal Shift Model with Common Variance 97
7.3 Best Critical Region for a Composite Alternative Hypothesis 98
7.3.1 Negative Exponential Composite Hypothesis Test 99
7.3.1.1 Example 99
7.3.1.2 Alternative Critical Regions 99
7.3.1.3 Mount St. Helens Example 100
7.3.2 Normal Shift Model with Common But Unknown Variance: The T-test 102
7.3.3 The Random Variable Tn-1 102
7.3.3.1 Where We Show X and S2 are Independent 102
7.3.3.2 Where We Show That S2 Scaled is ;;2(n - 1) 103
7.3.3.3 Where We Finally Derive the T PDF 103
7.3.4 The One-Sample T-test 104
7.3.5 Example 105
7.3.6 Other T-tests 106
7.3.6.1 Paired T-test 106
7.3.6.2 Two-Sample T-test 107
7.3.6.3 Example Two-Sample T-test: Lord Rayleigh's Data 107
7.4 Reporting Results: p-values and Power 108
7.4.1 Example When the Null Hypothesis Is Rejected 109
7.4.2 When the Null Hypothesis is Not Rejected 109
7.4.3 The Power Function 110
7.5 Multiple Testing and the Bonferroni Correction 111
Problems 111
8 Confidence Intervals and Other Hypothesis Tests 113
8.1 Confidence Intervals 113
8.1.1 Confidence Interval for ;;: Normal Data, ;;2 Known 113
8.1.2 Confidence Interval for ;;: ;;2 Unknown 114
8.1.3 Confidence Intervals and p-values 115
8.2 Hypotheses About the Variance and the F-Distribution 115
8.2.1 The F-Distribution 116
8.2.2 Hypotheses About the Value of the Variance 116
8.2.3 Confidence Interval for the Variance 117
8.2.4 Two-Sided Alternative for Testing ;;2 = ;;2 0 117
8.3 Pearson's Chi-Squared Tests 118
8.3.1 The Multinomial PMF 118
8.3.2 Goodness-of-Fit (GoF) Tests 118
8.3.3 Two-Category Binomial Case 119
8.3.4 m-Category Multinomial Case 120
8.3.5 Goodness-of-Fit Test for a Parametric Model 120
8.3.6 Tests for Independence in Contingency Tables 122
8.4 Correlation Coefficient Tests and CIs 123
8.4.1 How to Test if the Correlation ;; = 0 123
8.4.2 Confidence Intervals and Tests for a General Correlation Coefficient 125
8.5 Linear Regression 125
8.5.1 Least Squares Regression 125
8.5.2 Distribution of the Least-Squares Parameters 126
8.5.3 A Confidence Interval for the Slope 127
8.5.4 A Two-side Hypothesis Test for the Slope 128
8.5.5 Predictions at a New Value 128
8.5.6 Population Interval at a New Value 128
8.6 Analysis of Variance 129
Problems 131
9 Topics in Statistics 133
9.1 MSE and Histogram Bin Width Selection 133
9.1.1 MSE Criterion for Biased Estimators 133
9.1.2 Case Study: Optimal Histogram Bin Widths 134
9.1.3 Examples with Normal Data 137
9.1.4 Normal Reference Rules for the Histogram Bin Width 137
9.1.4.1 Scott's Rule 137
9.1.4.2 Freedman-Diaconis Rule 137
9.1.4.3 Sturges' Rule 138
9.1.4.4 Comparison of the Three Rules 138
9.2 An Optimal Stopping Time Problem 139
9.3 Compound Random Variables 141
9.3.1 Computing Expectations with Conditioning 141
9.3.2 Sum of a Random Number of Random Variables 142
9.4 Simulation and the Bootstrap 143
9.5 Multiple Linear Regression 144
9.6 Experimental Design 145
9.7 Logistic Regression, Poisson Regression, and the Generalized Linear Model 147
9.8 Robustness 148
9.9 Conclusions 150
Appendices 151
A Notation Used in This Book 151
B Common Distributions 153
C Using R and Mathematica For This Text 154
C.1 R Language - The Very Basics 154
C.2 Mathematica - The Basics 155
Bibliography 157
Index 159
1
Data Analysis and Understanding
The field of statistics has a rich history that has become tightly integrated into the emerging field of data sciences. Collaboration with computer scientists, numerical analysts, and decision makers characterizes the field. The role of statistics and statisticians is to find actionable information in a noisy collection of data. Every field of academic endeavor encounters this problem: from the electrical engineer trying to find a signal in a noisy channel to an English professor trying to determine the authorship of a contested newly discovered manuscript.
There are two basic tasks for the statistician. First is to characterize the distribution of possible outcomes using a batch of representative data. An actuary may be asked to find a dollar loss for car accidents that is not exceeded 99.999% of the time. An economist may be asked to provide useful summaries of a collection of income data. The histogram is our primary tool here, an idea that did not appear until the 17th century; see Graunt (1662), who analyzed death records during height of the plague outbreak in Europe.
The second task is that of prediction. A bank may wish to understand how credit risk is related to other information that may be available. A mechanical engineer may wish to understand the risk inherent in a new design under extreme conditions. Methods for performing this task underlie many algorithms today, for example, translating foreign languages or image recognition.
The mathematical backbone of all of our statistical methods is probability theory. Thus we study the basics of probability theory and random variables in the first part of this course. Statistical methods and the basics of statistical decision theory form the core of the middle third of this course. Specific tests and data analysis approaches finish our study.
1.1 Exploring the Distribution of Data
Tukey (1977) introduced a number of data summaries in his book Exploratory Data Analysis. Many are based on quantiles or percentiles of the data vector. Percentiles are particular choices of the sorted data. The middlemost is the median, or the 50th percentile. As a measure of spread, Tukey focused on the distance from the 25th to the 75th percentiles, the so-called interquartile range (IQR). A three-point summary would list these percentiles. Instead Tukey popularized the box-and-whiskers plot, which is a five-point summary. The additional two points are intended to capture 99% of the data. These are drawn at a distance of from the two quartiles. Any points outside these whiskers are plotted as potential outliers.
1.1.1 Pearson's Father-Son Height Data
We illustrate these ideas on a set of data collected by Karl Pearson over a century ago. He recorded the heights of fathers and an adult son. In the left frame in Figure 1.1, we display a box-and-whiskers plot of these data. We see that the sons are taller than their fathers by about an inch. There are also more potential outliers among the sons for some reason.
In the middle frame of Figure 1.1, we show Tukey's stem-and-leaf plot of the 1078 differences of the heights of each son and his father. The range of the data is and the first seven sorted values rounded to one decimal place are . Each data point is decomposed into a stem and a leaf digit. Thus has a stem of and a leaf of 0. The top line is actually , although it is too small to see. With so much data, each stem is broken into two lines to provide more detail. Thus the next two lines show a stem of but no leaves twice. The fourth line shows and the fifth line reads and so on. This figure was generated using the command ; R Core Team (2018). (The default has half as many stems.) Thus the stem-and-leaf plot shows the frequency count of points for each stem as character strings.
In the right frame of Figure 1.1, we show the frequency counts in a histogram. The histogram uses a parameter called the bin width to construct an equally spaced mesh . Then we count the number of points in each interval. These counts are displayed as a bar chart. (The histogram can use any anchor point, although 0 is a common choice.) For the histogram shown, the anchor point selected was 0, and was chosen using Scott's rule ; see Scott (1979). This rule is discussed in Section 9.1.4.1. The default choice in function hist is Sturges' rule, discussed in Section 9.1.4.3, which chooses 11 bins with (not shown).
The choice of is often considered a matter of convenience. The stem-and-leaf plot using one-digit integer stems limits its choices. By way of contrast, any positive real number can be used in a histogram. In Figure 1.2, we show the histograms using by Scott's rule, as well as and . Loosely speaking, the histograms using are missing useful information, while the histograms using display spurious detail. We discuss strategies for finding the best choice of in Section 9.1. In any case, the histogram is a powerful tool for understanding the full distribution of data.
Figure 1.1 Displays of the father-son height data collected by Karl Pearson: (left) box-and-whiskers plot; (middle) stem-and leaf plot; (right) histogram.
Figure 1.2 Histograms of the sons' heights (top row) and fathers' heights (bottom row) using three bin widths: , , from left to right; see text.
1.1.2 Lord Rayleigh's Data
In Exploratory Data Analysis, Tukey (1977) demonstrates the box-and-whiskers plot using the Lord Rayleigh data, which measure the weight of nitrogen gas obtained by various means; see Table 1.1. Discrepancies in the results led to his discovery of the element argon. Rayleigh made measurements from 1892 to 1894, with a mean of 2.30584 and a standard deviation of 0.00537. It is common to assume such measurements of a fundamental quantity are normally distributed. Multiple experiments are run and the results averaged in the presumption that a more accurate estimate will result.
Table 1.1 Lord Rayleigh's 24 measurements (sorted) of the weight of a sample of nitrogen. The first 10 came from chemical samples, while the last 14 came from pure air.
2.29816 2.29849 2.29869 2.29889 2.29890 2.29940 2.30054 2.30074 2.30143 2.30182 2.30956 2.30986 2.31001 2.31010 2.31010 2.31012 2.31017 2.31024 2.31024 2.31026 2.31027 2.31028 2.31035 2.31163Figure 1.3 Displays of Lord Rayleigh's 24 measurements of the atomic weight of nitrogen gas. (Left) Histogram with four bins; (middle) a second histogram; (right) stem-and-leaf display using the command .
In the left frame of Figure 1.3, we display a histogram with four (carefully selected) bins. The histogram is shown on a density scale, rather than a frequency scale, so that the area of the shaded region is 1. We shall see in Problem 1 that this is accomplished by dividing the bin counts by .
The first histogram in Figure 1.3 hides the interesting structure contained in the small dataset. The second histogram and stem-and-leaf plot show the two clusters quite clearly. Charting of data before the 1900s was not common, and looking at a table of the data would typically not reveal this feature. It turned out that Lord Rayleigh had combined various sources of the gas with several purifying agents and extraction methods. The samples originating from "pure air" were "contaminated" with argon. For the discovery of argon, Lord Rayleigh was awarded the Nobel Prize in Physics in 1904.
1.1.3 Discussion
Finding structure in data is a primary goal of data science. Graphical methods are powerful approaches to discovering unexpected or hidden structure. Some of these methods are better suited to small datasets. In a multivariate statistics course, we will learn how to analyze data with more than one variable. Modern genetic datasets often result in more than variables!
1.2 Exploring Prediction Using Data
The second fundamental task of statistics is prediction. Data for this task are typically ordered pairs, . The goal is to predict the value of the variable using the corresponding value of the variable. For example, we might try to predict a son's height () knowing the father's height (). Or a bank contemplating a mortgage loan may use a person's credit score to predict the probability the person will default on the loan.
The initial step is to plot a scatter diagram of the data points in order to determine if there is a strong relationship between and . The relationship, if it exists, is linear or nonlinear. If knowledge of does not convey any information about the value of , then the scatter diagram will have no slope or trends, with values just scattered around their average.
1.2.1 Body and Brain Weights of Land Mammals
In the left...
