Probability and Random Processes

Preface for the Second Edition xii Preface for the First Edition xiv 1 Sets, Fields, and Events 1 1.1 Set Definitions 1 1.2 Set Operations 2 1.3 Set Algebras, Fields, and Events 5 2 Probability Space and Axioms 7 2.1 Probability Space 7 2.2 Conditional Probability 9 2.3 Independence 11 2.4 Total Probability and Bayes' Theorem 12 3 Basic Combinatorics 16 3.1 Basic Counting Principles 16 3.2 Permutations 16 3.3 Combinations 18 4 Discrete Distributions 23 4.1 Bernoulli Trials 23 4.2 Binomial Distribution 23 4.3 Multinomial Distribution 26 4.4 Geometric Distribution 26 4.5 Negative Binomial Distribution 27 4.6 Hypergeometric Distribution 28 4.7 Poisson Distribution 30 4.8 Newton-Pepys Problem and its Extensions 33 4.9 Logarithmic Distribution 40 4.9.1 Finite Law (Benford's Law) 40 4.9.2 Infinite Law 43 4.10 Summary of Discrete Distributions 44 5 Random Variables 45 5.1 Definition of Random Variables 45 5.2 Determination of Distribution and Density Functions 46 5.3 Properties of Distribution and Density Functions 50 5.4 Distribution Functions from Density Functions 51 6 Continuous Random Variables and Basic Distributions 54 6.1 Introduction 54 6.2 Uniform Distribution 54 6.3 Exponential Distribution 55 6.4 Normal or Gaussian Distribution 57 7 Other Continuous Distributions 63 7.1 Introduction 63 7.2 Triangular Distribution 63 7.3 Laplace Distribution 63 7.4 Erlang Distribution 64 7.5 Gamma Distribution 65 7.6 Weibull Distribution 66 7.7 Chi-Square Distribution 67 7.8 Chi and Other Allied Distributions 68 7.9 Student-t Density 71 7.10 Snedecor F Distribution 72 7.11 Lognormal Distribution 72 7.12 Beta Distribution 73 7.13 Cauchy Distribution 74 7.14 Pareto Distribution 75 7.15 Gibbs Distribution 75 7.16 Mixed Distributions 75 7.17 Summary of Distributions of Continuous Random Variables 76 8 Conditional Densities and Distributions 78 8.1 Conditional Distribution and Density for P{A} 0 78 8.2 Conditional Distribution and Density for P{A} = 0 80 8.3 Total Probability and Bayes' Theorem for Densities 83 9 Joint Densities and Distributions 85 9.1 Joint Discrete Distribution Functions 85 9.2 Joint Continuous Distribution Functions 86 9.3 Bivariate Gaussian Distributions 90 10 Moments and Conditional Moments 91 10.1 Expectations 91 10.2 Variance 92 10.3 Means and Variances of Some Distributions 93 10.4 Higher-Order Moments 94 10.5 Correlation and Partial Correlation Coefficients 95 10.5.1 Correlation Coefficients 95 10.5.2 Partial Correlation Coefficients 106 11 Characteristic Functions and Generating Functions 108 11.1 Characteristic Functions 108 11.2 Examples of Characteristic Functions 109 11.3 Generating Functions 111 11.4 Examples of Generating Functions 112 11.5 Moment Generating Functions 113 11.6 Cumulant Generating Functions 115 11.7 Table of Means and Variances 116 12 Functions of a Single Random Variable 118 12.1 Random Variable g(X) 118 12.2 Distribution of Y = g(X) 119 12.3 Direct Determination of Density fY(y) from fX(x) 129 12.4 Inverse Problem: Finding g(X) given fX(x) and fY(y) 132 12.5 Moments of a Function of a Random Variable 133 13 Functions of Multiple Random Variables 135 13.1 Function of Two Random Variables, Z = g(X,Y) 135 13.2 Two Functions of Two Random Variables, Z = g(X,Y), W= h(X,Y) 143 13.3 Direct Determination of Joint Density fZW(z,w) from fXY(x,y) 146 13.4 Solving Z = g(X,Y) Using an Auxiliary Random Variable 150 13.5 Multiple Functions of Random Variables 153 14 Inequalities, Convergences, and Limit Theorems 155 14.1 Degenerate Random Variables 155 14.2 Chebyshev and Allied Inequalities 155 14.3 Markov Inequality 158 14.4 Chernoff Bound 159 14.5 Cauchy-Schwartz Inequality 160 14.6 Jensen's Inequality 162 14.7 Convergence Concepts 163 14.8 Limit Theorems 165 15 Computer Methods for Generating Random Variates 169 15.1 Uniform-Distribution Random Variates 169 15.2 Histograms 170 15.3 Inverse Transformation Techniques 172 15.4 Convolution Techniques 178 15.5 Acceptance-Rejection Techniques 178 16 Elements of Matrix Algebra 181 16.1 Basic Theory of Matrices 181 16.2 Eigenvalues and Eigenvectors of Matrices 186 16.3 Vector and Matrix Differentiation 190 16.4 Block Matrices 194 17 Random Vectors and Mean-Square Estimation 196 17.1 Distributions and Densities 196 17.2 Moments of Random Vectors 200 17.3 Vector Gaussian Random Variables 204 17.4 Diagonalization of Covariance Matrices 207 17.5 Simultaneous Diagonalization of Covariance Matrices 209 17.6 Linear Estimation of Vector Variables 210 18 Estimation Theory 212 18.1 Criteria of Estimators 212 18.2 Estimation of Random Variables 213 18.3 Estimation of Parameters (Point Estimation) 218 18.4 Interval Estimation (Confidence Intervals) 225 18.5 Hypothesis Testing (Binary) 231 18.6 Bayesian Estimation 238 19 Random Processes 250 19.1 Basic Definitions 250 19.2 Stationary Random Processes 258 19.3 Ergodic Processes 269 19.4 Estimation of Parameters of Random Processes 273 19.4.1 Continuous-Time Processes 273 19.4.2 Discrete-Time Processes 280 19.5 Power Spectral Density 287 19.5.1 Continuous Time 287 19.5.2 Discrete Time 294 19.6 Adaptive Estimation 298 20 Classification of Random Processes 320 20.1 Specifications of Random Processes 320 20.1.1 Discrete-State Discrete-Time (DSDT) Process 320 20.1.2 Discrete-State Continuous-Time (DSCT) Process 320 20.1.3 Continuous-State Discrete-Time (CSDT) Process 320 20.1.4 Continuous-State Continuous-Time (CSCT) Process 320 20.2 Poisson Process 321 20.3 Binomial Process 329 20.4 Independent Increment Process 330 20.5 Random-Walk Process 333 20.6 Gaussian Process 338 20.7 Wiener Process (Brownian Motion) 340 20.8 Markov Process 342 20.9 Markov Chains 347 20.10 Birth and Death Processes 357 20.11 Renewal Processes and Generalizations 366 20.12 Martingale Process 370 20.13 Periodic Random Process 374 20.14 Aperiodic Random Process (Karhunen-Loeve Expansion) 377 21 Random Processes and Linear Systems 383 21.1 Review of Linear Systems 383 21.2 Random Processes through Linear Systems 385 21.3 Linear Filters 393 21.4 Bandpass Stationary Random Processes 401 22 Wiener and Kalman Filters 413 22.1 Review of Orthogonality Principle 413 22.2 Wiener Filtering 414 22.3 Discrete Kalman Filter 425 22.4 Continuous Kalman Filter 433 23 Probability Modeling in Traffic Engineering 437 23.1 Introduction 437 23.2 Teletraffic Models 437 23.3 Blocking Systems 438 23.4 State Probabilities for Systems with Delays 440 23.5 Waiting-Time Distribution for M/M/c/ infinity Systems 441 23.6 State Probabilities for M/D/c Systems 443 23.7 Waiting-Time Distribution for M/D/c/ infinity System 446 23.8 Comparison of M/M/c and M/D/c 448 References 451 24 Probabilistic Methods in Transmission Tomography 452 24.1 Introduction 452 24.2 Stochastic Model 453 24.3 Stochastic Estimation Algorithm 455 24.4 Prior Distribution P{M} 457 24.5 Computer Simulation 458 24.6 Results and Conclusions 460 24.7 Discussion of Results 462 References 462 APPENDICES A A Fourier Transform Tables 463 B Cumulative Gaussian Tables 467 C Inverse Cumulative Gaussian Tables 472 D Inverse Chi-Square Tables 474 E Inverse Student-t Tables 481 F Cumulative Poisson Distribution 484 G Cumulative Binomial Distribution 488 H Computation of Roots of D(z) = 0 494 References 495 Index 498