An Introduction to Probability and Statistics
Description
Reviews / Votes
"The book is an ideal reference and resource for scientists and engineers in the elds of statistics, mathematics, physics, industrial management, and engineering. The book is also an excellent text for upper-undergraduate and graduate-level students majoring in probability and statistics." (Zentralblatt MATH 2016)The book is an ideal reference and resource for scientists and engineers in the elds of statistics,mathematics, physics, industrial management, and engineering. The book is also an excellenttext for upper-undergraduate and graduate-level students majoring in probability and statis-tics.More details
Other editions
Additional editions
Persons
Content
PREFACE TO THE THIRD EDITION xiii
PREFACE TO THE SECOND EDITION xv
PREFACE TO THE FIRST EDITION xvii
ACKNOWLEDGMENTS xix
ENUMERATION OF THEOREMS AND REFERENCES xxi
1 Probability 1
1.1 Introduction 1
1.2 Sample Space 2
1.3 Probability Axioms 7
1.4 Combinatorics: Probability on Finite Sample Spaces 20
1.5 Conditional Probability and Bayes Theorem 26
1.6 Independence of Events 31
2 Random Variables and Their Probability Distributions 39
2.1 Introduction 39
2.2 Random Variables 39
2.3 Probability Distribution of a Random Variable 42
2.4 Discrete and Continuous Random Variables 47
2.5 Functions of a Random Variable 55
3 Moments and Generating Functions 67
3.1 Introduction 67
3.2 Moments of a Distribution Function 67
3.3 Generating Functions 83
3.4 Some Moment Inequalities 93
4 Multiple Random Variables 99
4.1 Introduction 99
4.2 Multiple Random Variables 99
4.3 Independent Random Variables 114
4.4 Functions of Several Random Variables 123
4.5 Covariance, Correlation and Moments 143
4.6 Conditional Expectation 157
4.7 Order Statistics and Their Distributions 164
5 Some Special Distributions 173
5.1 Introduction 173
5.2 Some Discrete Distributions 173
5.2.1 Degenerate Distribution 173
5.2.2 Two-Point Distribution 174
5.2.3 Uniform Distribution on n Points 175
5.2.4 Binomial Distribution 176
5.2.5 Negative Binomial Distribution (Pascal or Waiting Time Distribution) 178
5.2.6 Hypergeometric Distribution 183
5.2.7 Negative Hypergeometric Distribution 185
5.2.8 Poisson Distribution 186
5.2.9 Multinomial Distribution 189
5.2.10 Multivariate Hypergeometric Distribution 192
5.2.11 Multivariate Negative Binomial Distribution 192
5.3 Some Continuous Distributions 196
5.3.1 Uniform Distribution (Rectangular Distribution) 199
5.3.2 Gamma Distribution 202
5.3.3 Beta Distribution 210
5.3.4 Cauchy Distribution 213
5.3.5 Normal Distribution (the Gaussian Law) 216
5.3.6 Some Other Continuous Distributions 222
5.4 Bivariate and Multivariate Normal Distributions 228
5.5 Exponential Family of Distributions 240
6 Sample Statistics and Their Distributions 245
6.1 Introduction 245
6.2 Random Sampling 246
6.3 Sample Characteristics and Their Distributions 249
6.4 Chi-Square, t-, and F-Distributions: Exact Sampling Distributions 262
6.5 Distribution of (X,S2) in Sampling from a Normal Population 271
6.6 Sampling from a Bivariate Normal Distribution 276
7 Basic Asymptotics: Large Sample Theory 285
7.1 Introduction 285
7.2 Modes of Convergence 285
7.3 Weak Law of Large Numbers 302
7.4 Strong Law of Large Numbers 308
7.5 Limiting Moment Generating Functions 316
7.6 Central Limit Theorem 321
7.7 Large Sample Theory 331
8 Parametric Point Estimation 337
8.1 Introduction 337
8.2 Problem of Point Estimation 338
8.3 Sufficiency, Completeness and Ancillarity 342
8.4 Unbiased Estimation 359
8.5 Unbiased Estimation (Continued): A Lower Bound for the Variance of An Estimator 372
8.6 Substitution Principle (Method of Moments) 386
8.7 Maximum Likelihood Estimators 388
8.8 Bayes and Minimax Estimation 401
8.9 Principle of Equivariance 418
9 Neyman-Pearson Theory of Testing of Hypotheses 429
9.1 Introduction 429
9.2 Some Fundamental Notions of Hypotheses Testing 429
9.3 Neyman-Pearson Lemma 438
9.4 Families with Monotone Likelihood Ratio 446
9.5 Unbiased and Invariant Tests 453
9.6 Locally Most Powerful Tests 459
10 Some Further Results on Hypotheses Testing 463
10.1 Introduction 463
10.2 Generalized Likelihood Ratio Tests 463
10.3 Chi-Square Tests 472
10.4 t-Tests 484
10.5 F-Tests 489
10.6 Bayes and Minimax Procedures 491
11 Confidence Estimation 499
11.1 Introduction 499
11.2 Some Fundamental Notions of Confidence Estimation 499
11.3 Methods of Finding Confidence Intervals 504
11.4 Shortest-Length Confidence Intervals 517
11.5 Unbiased and Equivariant Confidence Intervals 523
11.6 Resampling: Bootstrap Method 530
12 General Linear Hypothesis 535
12.1 Introduction 535
12.2 General Linear Hypothesis 535
12.3 Regression Analysis 543
12.3.1 Multiple Linear Regression 543
12.3.2 Logistic and Poisson Regression 551
12.4 One-Way Analysis of Variance 554
12.5 Two-Way Analysis of Variance with One Observation Per Cell 560
12.6 Two-Way Analysis of Variance with Interaction 566
13 Nonparametric Statistical Inference 575
13.1 Introduction 575
13.2 U-Statistics 576
13.3 Some Single-Sample Problems 584
13.3.1 Goodness-of-Fit Problem 584
13.3.2 Problem of Location 590
13.4 Some Two-Sample Problems 599
13.4.1 Median Test 601
13.4.2 Kolmogorov-Smirnov Test 602
13.4.3 The Mann-Whitney-Wilcoxon Test 604
13.5 Tests of Independence 608
13.5.1 Chi-square Test of Independence-Contingency Tables 608
13.5.2 Kendall's Tau 611
13.5.3 Spearman's Rank Correlation Coefficient 614
13.6 Some Applications of Order Statistics 619
13.7 Robustness 625
13.7.1 Effect of Deviations from Model Assumptions on Some Parametric Procedures 625
13.7.2 Some Robust Procedures 631
FREQUENTLY USED SYMBOLS AND ABBREVIATIONS 637
REFERENCES 641
STATISTICAL TABLES 647
ANSWERS TO SELECTED PROBLEMS 667
AUTHOR INDEX 677
SUBJECT INDEX 679
1
PROBABILITY
1.1 INTRODUCTION
The theory of probability had its origin in gambling and games of chance. It owes much to the curiosity of gamblers who pestered their friends in the mathematical world with all sorts of questions. Unfortunately this association with gambling contributed to a very slow and sporadic growth of probability theory as a mathematical discipline. The mathematicians of the day took little or no interest in the development of any theory but looked only at the combinatorial reasoning involved in each problem.
The first attempt at some mathematical rigor is credited to Laplace. In his monumental work, Theorie analytique des probabilités (1812), Laplace gave the classical definition of the probability of an event that can occur only in a finite number of ways as the proportion of the number of favorable outcomes to the total number of all possible outcomes, provided that all the outcomes are equally likely. According to this definition, the computation of the probability of events was reduced to combinatorial counting problems. Even in those days, this definition was found inadequate. In addition to being circular and restrictive, it did not answer the question of what probability is,it only gave a practical method of computing the probabilities of some simple events.
An extension of the classical definition of Laplace was used to evaluate the probabilities of sets of events with infinite outcomes. The notion of equal likelihood of certain events played a key role in this development. According to this extension, if O is some region with a well-defined measure (length, area, volume, etc.), the probability that a point chosen atrandom lies in a subregion A of O is the ratio measure(A)/measure(O). Many problems of geometric probability were solved using this extension. The trouble is that one can define "at random" in any way one pleases, and different definitions therefore lead to different answers. Joseph Bertrand, for example, in his book Calcul des probabilités (Paris, 1889) cited a number of problems in geometric probability where the result depended on the method of solution. In Example 9 we will discuss the famous Bertrand paradox and show that in reality there is nothing paradoxical about Bertrand's paradoxes; once we define "probability spaces" carefully, the paradox is resolved. Nevertheless difficulties encountered in the field of geometric probability have been largely responsible for the slow growth of probability theory and its tardy acceptance by mathematicians as a mathematical discipline.
The mathematical theory of probability, as we know it today, is of comparatively recent origin. It was A. N. Kolmogorov who axiomatized probability in his fundamental work, Foundations of the Theory of Probability (Berlin), in 1933. According to this development, random events are represented by sets and probability is just a normed measure defined on these sets. This measure-theoretic development not only provided a logically consistent foundation for probability theory but also, at the same time, joined it to the mainstream of modern mathematics.
In this book we follow Kolmogorov's axiomatic development. In Section 1.2 we introduce the notion of a sample space. In Section 1.3 we state Kolmogorov's axioms of probability and study some simple consequences of these axioms. Section 1.4 is devoted to the computation of probability on finite sample spaces. Section 1.5 deals with conditional probability and Bayes's rule while Section 1.6 examines the independence of events.
1.2 SAMPLE SPACE
In most branches of knowledge, experiments are a way of life. In probability and statistics, too, we concern ourselves with special types of experiments. Consider the following examples.
Example 1.
A coin is tossed. Assuming that the coin does not land on the side, there are two possible outcomes of the experiment: heads and tails. On any performance of this experiment one does not know what the outcome will be. The coin can be tossed as many times as desired.
Example 2.
A roulette wheel is a circular disk divided into 38 equal sectors numbered from 0 to 36 and 00. A ball is rolled on the edge of the wheel, and the wheel is rolled in the opposite direction. One bets on any of the 38 numbers or some combinations of them. One can also bet on a color, red or black. If the ball lands in the sector numbered 32, say, anybody who bet on 32 or combinations including 32 wins, and so on. In this experiment, all possible outcomes are known in advance, namely 00, 0, 1, 2,.,36, but on any performance of the experiment there is uncertainty as to what the outcome will be, provided, of course, that the wheel is not rigged in any manner. Clearly, the wheel can be rolled any number of times.
Example 3.
A manufacturer produces footrules. The experiment consists in measuring the length of a footrule produced by the manufacturer as accurately as possible. Because of errors in the production process one does not know what the true length of the footrule selected will be. It is clear, however, that the length will be, say, between 11 and 13 in., or, if one wants to be safe, between 6 and 18 in.
Example 4.
The length of life of a light bulb produced by a certain manufacturer is recorded. In this case one does not know what the length of life will be for the light bulb selected, but clearly one is aware in advance that it will be some number between 0 and 8hours.
The experiments described above have certain common features. For each experiment, we know in advance all possible outcomes, that is, there are no surprises in store after the performance of any experiment. On any performance of the experiment, however, we do not know what the specific outcome will be, that is, there is uncertainty about the outcome on any performance of the experiment. Moreover, the experiment can be repeated under identical conditions. These features describe a random (or a statistical) experiment.
Definition 1.
A random (or a statistical) experiment is an experiment in which
- all outcomes of the experiment are known in advance,
- any performance of the experiment results in an outcome that is not known in advance, and
- theexperiment can be repeated under identical conditions.
In probability theory we study this uncertainty of a random experiment. It is convenient to associate with each such experiment a set O, the set of all possible outcomes of the experiment. To engage in any meaningful discussion about the experiment, we associate with O a s -field , of subsets of O. We recall that a s -field is a nonempty class of subsets of O that is closed under the formation of countable unions and complements and contains the null set F.
Definition 2.
The sample space of a statistical experiment is a pair (O, ), where
- O is the set of all possible outcomes of the experiment and
- is a s -field of subsets of O.
The elements of O are called sample points. Any set A ?is known as an event. Clearly A is a collection of sample points. We say that an event A happens if the outcome of the experiment corresponds to a point in A. Each one-point set is known as a simple or an elementary event . If the set C contains only a finite number of points, we say that (O, ) is a finite sample space . If Ocontains at most a countable number of points, we call (O, ) a discrete sample space. If, however, O contains uncountably many points, we say that (O, )is an uncountable sample space. In particular, if O = k or some rectangle in k , we call it a continuous sample space.
Remark 1. The choice of is an important one, and some remarks are in order. If O contains at most a countable number of points, we can always take to be the class of all subsets of O This is certainly a s -field. Each one point set is a member of and is the fundamental object of interest. Every subset of O is an event. If O has uncountably many points, the class of all subsets of O is still a s -field, but it is much too large a class of sets to be of interest. It may not be possible to choose the class of all subsets of O as . One of the most important examples of an uncountable sample space is the case in which O=or O is an interval in . In this case we would like all one-point subsets of O and all intervals (closed, open, or semiclosed) to be events. We use our knowledge of analysis to specify . We will not go into details here except to recall that the class of all semiclosed intervals (a,b ] generates a class 1 which is a s -field on . This class contains all one-point sets and all intervals (finite or infinite). We take 1. Since we will be dealing mostly with the one-dimensional case, we will write instead of 1. There are many subsets of R that are not in 1, but we will not demonstrate this fact here. We refer the reader to Halmos [42] , Royden [96] , or Kolmogorov and Fomin [54] for further details.
Example 5.
Let us toss a...
