Probability and Random Processes
Description
More details
Other editions
Additional editions
Persons
Content
Preface for the Second Edition xii
Preface for the First Edition xiv
1 Sets, Fields, and Events 1
1.1 Set Definitions 1
1.2 Set Operations 2
1.3 Set Algebras, Fields, and Events 5
2 Probability Space and Axioms 7
2.1 Probability Space 7
2.2 Conditional Probability 9
2.3 Independence 11
2.4 Total Probability and Bayes' Theorem 12
3 Basic Combinatorics 16
3.1 Basic Counting Principles 16
3.2 Permutations 16
3.3 Combinations 18
4 Discrete Distributions 23
4.1 Bernoulli Trials 23
4.2 Binomial Distribution 23
4.3 Multinomial Distribution 26
4.4 Geometric Distribution 26
4.5 Negative Binomial Distribution 27
4.6 Hypergeometric Distribution 28
4.7 Poisson Distribution 30
4.8 Newton-Pepys Problem and its Extensions 33
4.9 Logarithmic Distribution 40
4.9.1 Finite Law (Benford's Law) 40
4.9.2 Infinite Law 43
4.10 Summary of Discrete Distributions 44
5 Random Variables 45
5.1 Definition of Random Variables 45
5.2 Determination of Distribution and Density Functions 46
5.3 Properties of Distribution and Density Functions 50
5.4 Distribution Functions from Density Functions 51
6 Continuous Random Variables and Basic Distributions 54
6.1 Introduction 54
6.2 Uniform Distribution 54
6.3 Exponential Distribution 55
6.4 Normal or Gaussian Distribution 57
7 Other Continuous Distributions 63
7.1 Introduction 63
7.2 Triangular Distribution 63
7.3 Laplace Distribution 63
7.4 Erlang Distribution 64
7.5 Gamma Distribution 65
7.6 Weibull Distribution 66
7.7 Chi-Square Distribution 67
7.8 Chi and Other Allied Distributions 68
7.9 Student-t Density 71
7.10 Snedecor F Distribution 72
7.11 Lognormal Distribution 72
7.12 Beta Distribution 73
7.13 Cauchy Distribution 74
7.14 Pareto Distribution 75
7.15 Gibbs Distribution 75
7.16 Mixed Distributions 75
7.17 Summary of Distributions of Continuous Random Variables 76
8 Conditional Densities and Distributions 78
8.1 Conditional Distribution and Density for P{A} 0 78
8.2 Conditional Distribution and Density for P{A} = 0 80
8.3 Total Probability and Bayes' Theorem for Densities 83
9 Joint Densities and Distributions 85
9.1 Joint Discrete Distribution Functions 85
9.2 Joint Continuous Distribution Functions 86
9.3 Bivariate Gaussian Distributions 90
10 Moments and Conditional Moments 91
10.1 Expectations 91
10.2 Variance 92
10.3 Means and Variances of Some Distributions 93
10.4 Higher-Order Moments 94
10.5 Correlation and Partial Correlation Coefficients 95
10.5.1 Correlation Coefficients 95
10.5.2 Partial Correlation Coefficients 106
11 Characteristic Functions and Generating Functions 108
11.1 Characteristic Functions 108
11.2 Examples of Characteristic Functions 109
11.3 Generating Functions 111
11.4 Examples of Generating Functions 112
11.5 Moment Generating Functions 113
11.6 Cumulant Generating Functions 115
11.7 Table of Means and Variances 116
12 Functions of a Single Random Variable 118
12.1 Random Variable g(X) 118
12.2 Distribution of Y = g(X) 119
12.3 Direct Determination of Density fY(y) from fX(x) 129
12.4 Inverse Problem: Finding g(X) given fX(x) and fY(y) 132
12.5 Moments of a Function of a Random Variable 133
13 Functions of Multiple Random Variables 135
13.1 Function of Two Random Variables, Z = g(X,Y) 135
13.2 Two Functions of Two Random Variables, Z = g(X,Y), W= h(X,Y) 143
13.3 Direct Determination of Joint Density fZW(z,w) from fXY(x,y) 146
13.4 Solving Z = g(X,Y) Using an Auxiliary Random Variable 150
13.5 Multiple Functions of Random Variables 153
14 Inequalities, Convergences, and Limit Theorems 155
14.1 Degenerate Random Variables 155
14.2 Chebyshev and Allied Inequalities 155
14.3 Markov Inequality 158
14.4 Chernoff Bound 159
14.5 Cauchy-Schwartz Inequality 160
14.6 Jensen's Inequality 162
14.7 Convergence Concepts 163
14.8 Limit Theorems 165
15 Computer Methods for Generating Random Variates 169
15.1 Uniform-Distribution Random Variates 169
15.2 Histograms 170
15.3 Inverse Transformation Techniques 172
15.4 Convolution Techniques 178
15.5 Acceptance-Rejection Techniques 178
16 Elements of Matrix Algebra 181
16.1 Basic Theory of Matrices 181
16.2 Eigenvalues and Eigenvectors of Matrices 186
16.3 Vector and Matrix Differentiation 190
16.4 Block Matrices 194
17 Random Vectors and Mean-Square Estimation 196
17.1 Distributions and Densities 196
17.2 Moments of Random Vectors 200
17.3 Vector Gaussian Random Variables 204
17.4 Diagonalization of Covariance Matrices 207
17.5 Simultaneous Diagonalization of Covariance Matrices 209
17.6 Linear Estimation of Vector Variables 210
18 Estimation Theory 212
18.1 Criteria of Estimators 212
18.2 Estimation of Random Variables 213
18.3 Estimation of Parameters (Point Estimation) 218
18.4 Interval Estimation (Confidence Intervals) 225
18.5 Hypothesis Testing (Binary) 231
18.6 Bayesian Estimation 238
19 Random Processes 250
19.1 Basic Definitions 250
19.2 Stationary Random Processes 258
19.3 Ergodic Processes 269
19.4 Estimation of Parameters of Random Processes 273
19.4.1 Continuous-Time Processes 273
19.4.2 Discrete-Time Processes 280
19.5 Power Spectral Density 287
19.5.1 Continuous Time 287
19.5.2 Discrete Time 294
19.6 Adaptive Estimation 298
20 Classification of Random Processes 320
20.1 Specifications of Random Processes 320
20.1.1 Discrete-State Discrete-Time (DSDT) Process 320
20.1.2 Discrete-State Continuous-Time (DSCT) Process 320
20.1.3 Continuous-State Discrete-Time (CSDT) Process 320
20.1.4 Continuous-State Continuous-Time (CSCT) Process 320
20.2 Poisson Process 321
20.3 Binomial Process 329
20.4 Independent Increment Process 330
20.5 Random-Walk Process 333
20.6 Gaussian Process 338
20.7 Wiener Process (Brownian Motion) 340
20.8 Markov Process 342
20.9 Markov Chains 347
20.10 Birth and Death Processes 357
20.11 Renewal Processes and Generalizations 366
20.12 Martingale Process 370
20.13 Periodic Random Process 374
20.14 Aperiodic Random Process (Karhunen-Loeve Expansion) 377
21 Random Processes and Linear Systems 383
21.1 Review of Linear Systems 383
21.2 Random Processes through Linear Systems 385
21.3 Linear Filters 393
21.4 Bandpass Stationary Random Processes 401
22 Wiener and Kalman Filters 413
22.1 Review of Orthogonality Principle 413
22.2 Wiener Filtering 414
22.3 Discrete Kalman Filter 425
22.4 Continuous Kalman Filter 433
23 Probability Modeling in Traffic Engineering 437
23.1 Introduction 437
23.2 Teletraffic Models 437
23.3 Blocking Systems 438
23.4 State Probabilities for Systems with Delays 440
23.5 Waiting-Time Distribution for M/M/c/8 Systems 441
23.6 State Probabilities for M/D/c Systems 443
23.7 Waiting-Time Distribution for M/D/c/8 System 446
23.8 Comparison of M/M/c and M/D/c 448
References 451
24 Probabilistic Methods in Transmission Tomography 452
24.1 Introduction 452
24.2 Stochastic Model 453
24.3 Stochastic Estimation Algorithm 455
24.4 Prior Distribution P{M} 457
24.5 Computer Simulation 458
24.6 Results and Conclusions 460
24.7 Discussion of Results 462
References 462
APPENDICES
A A Fourier Transform Tables 463
B Cumulative Gaussian Tables 467
C Inverse Cumulative Gaussian Tables 472
D Inverse Chi-Square Tables 474
E Inverse Student-t Tables 481
F Cumulative Poisson Distribution 484
G Cumulative Binomial Distribution 488
H Computation of Roots of D(z) = 0 494
References 495
Index 498
1
SETS, FIELDS, AND EVENTS
1.1 SET DEFINITIONS
The concept of sets play an important role in probability. We will define a set in the following paragraph.
Definition of Set
A set is a collection of objects called elements. The elements of a set can also be sets. Sets are usually represented by uppercase letters A, and elements are usually represented by lowercase letters a. Thus
(1.1.1)will mean that the set A contains the elements a1, a2, .?, an. Conversely, we can write that ak is an element of A as
(1.1.2)and ak is not an element of A as
(1.1.3)A finite set contains a finite number of elements, for example, S = {2,4,6}. Infinite sets will have either countably infinite elements such as A = {x:x is all positive integers} or uncountably infinite elements such as B = {x:x is real number =?20}.
Example 1.1.1
The set A of all positive integers less than 7 is written as
Example 1.1.2
The set N of all positive integers is written as
Example 1.1.3
The set R of all real numbers is written as
Example 1.1.4
The set R2 of real numbers x, y is written as
Example 1.1.5
The set C of all real numbers x,y such that x+y =?10 is written as
Venn Diagram
Sets can be represented graphically by means of a Venn diagram. In this case we assume tacitly that S is a universal set under consideration. In Example 1.1.5, the universal set S = {x:x is all positive integers}. We shall represent the set A in Example 1.1.1 by means of a Venn diagram of Fig. 1.1.1.
Empty Set
An empty is a set that contains no element. It plays an important role in set theory and is denoted by Ø. The set A = {0} is not an empty set since it contains the element 0.
Cardinality
The number of elements in the set A is called the cardinality of set A, and is denoted by |A|. If it is an infinite set, then the cardinality is 8.
Example 1.1.6
The cardinality of the set A = {2,4,6} is 3, or |A| = 3. The cardinality of set R = {x:x is real} is 8.
Example 1.1.7
The cardinality of the set A = {x:x is positive integer <7} is |A| = 6.
Example 1.1.8
The cardinality of the set B = {x:x is a real number <10} is infinity since there are infinite real numbers <10.
Subset
A set B is a subset of A if every element in B is an element of A and is written as B?A. B is a proper subset of A if every element of A is not in B and is written as B?A.
Equality of Sets
Two sets A and B are equal if B?A and A?B, that is, if every element of A is contained in B and every element of B is contained in A. In other words, sets A and B contain exactly the same elements. Note that this is different from having the same cardinality, that is, containing the same number of elements.
Example 1.1.9
The set B = {1,3,5} is a proper subset of A = {1,2,3,4,5,6}, whereas the set C = {x:x is a positive even integer =?6} and the set D = {2,4,6} are the same since they contain the same elements. The cardinalities of B, C, and D are 3 and C = D.
We shall now represent the sets A and B and the sets C and D in Example 1.1.9 by means of the Venn diagram of Fig. 1.1.2 on a suitably defined universal set S.
Power Set
The power set of any set A is the set of all possible subsets of A and is denoted by PS(A). Every power set of any set A must contain the set A itself and the empty set Ø. If n is the cardinality of the set A, then the cardinality of the power set |PS(A)| = 2n.
Example 1.1.10
If the set A = {1,2,3} then PS(A) = {Ø, (1,2,3), (1,2), (2,3), (3,1), (1), (2), (3)}. The cardinality |PS(A)| = 8 = 23.
1.2 SET OPERATIONS
Union
Let A and B be sets belonging to the universal set S. The union of sets A and B is another set C whose elements are those that are in either A or B, and is denoted by A?B. Where there is no confusion, it will also be represented as A+B:
(1.2.1)Example 1.2.1
The union of sets A = {1,2,3} and B = {2,3,4,5} is the set C = A?B = {1,2,3,4,5}.
Intersection
The intersection of the sets A and B is another set C whose elements are the same as those in both A and B and is denoted by AnB. Where there is no confusion, it will also be represented by AB.
(1.2.2)Example 1.2.2
The intersection of the sets A and B in Example 1.2.1 is the set C = {2,3} Examples 1.2.1 and 1.2.2 are shown in the Venn diagram of Fig. 1.2.1.
Mutually Exclusive Sets
Two sets A and B are called mutually exclusive if their intersection is empty. Mutually exclusive sets are also called disjoint.
(1.2.3)One way to determine whether two sets A and B are mutually exclusive is to check whether set B can occur when set A has already occurred and vice versa. If it cannot, then A and B are mutually exclusive. For example, if a single coin is tossed, the two sets, {heads} and {tails}, are mutually exclusive since {tails} cannot occur when {heads} has already occurred and vice versa.
Independence
We will consider two types of independence. The first is known as functional independence [58].1 Two sets A and B can be called functionally independent if the occurrence of B does not in any way influence the occurrence of A and vice versa. The second one is statistical independence, which is a different concept that will be defined later. As an example, the tossing of a coin is functionally independent of the tossing of a die because they do not depend on each other. However, the tossing of a coin and a die are not mutually exclusive since any one can be tossed irrespective of the other. By the same token, pressure and temperature are not functionally independent because the physics of the problem, namely, Boyle's law, connects these quantities. They are certainly not mutually exclusive.
Cardinality of Unions and Intersections
We can now ascertain the cardinality of the union of two sets A and B that are not mutually exclusive. The cardinality of the union C = A?B can be determined as follows. If we add the cardinality |A| to the cardinality |B|, we have added the cardinality of the intersection |AnB| twice. Hence we have to subtract once the cardinality |AnB| as shown in Fig. 1.2.2. Or, in other words
(1.2.4a)In Fig. 1.2.2 the cardinality |A| = 9 and the cardinality |B| = 11 and the cardinality |A?B| is 11+9-4 = 16.
As a corollary, if sets A and B are mutually exclusive, then the cardinality of the union is the sum of the cardinalities; or
(1.2.4b)The generalization of this result to an arbitrary union of n sets is called the inclusion-exclusion principle, given by
(1.2.5a)If the sets {Ai} are mutually exclusive, that is, AinAj = Ø for i?j, then we have
(1.2.5b)This equation is illustrated in the Venn diagram for n = 3 in Fig. 1.2.3, where if equals, then we have added twice the cardinalites of and . However, if we subtract once , , and and write
then we have subtracted thrice instead of twice. Hence, adding we get...
