Probability, Random Variables, Statistics, and Random Processes: Fundamentals & Applications

Preface xiii Acknowledgments xv About the Companion Website xvii Part I Probability 1 1 Basic Concepts of Probability Theory 3 1.1 Statistical Regularity and Relative Frequency 3 1.2 Set Theory and Its Applications to Probability 5 1.3 The Axioms and Corollaries of Probability 12 1.4 Joint Probability and Conditional Probability 18 1.5 Statistically Independent Events and Mutually Exclusive Events 21 1.6 Law of Total Probability and Bayes' Theorem 28 1.7 Summary 32 Problems 32 2 Applications in Probability 37 2.1 Odds and Risk 37 2.2 Gambler's Ruin Problem 41 2.3 Systems Reliability 43 2.4 Medical Diagnostic Testing 47 2.5 Bayesian Spam Filtering 50 2.6 Monty Hall Problem 51 2.7 Digital Transmission Error 54 2.8 How to Make the Best Choice Problem 56 2.9 The Viterbi Algorithm 59 2.10 All Eggs in One Basket 61 2.11 Summary 63 Problems 63 3 Counting Methods and Applications 67 3.1 Basic Rules of Counting 67 3.2 Permutations and Combinations 72 3.2.1 Permutations without Replacement 73 3.2.2 Combinations without Replacement 73 3.2.3 Permutations with Replacement 74 3.2.4 Combinations with Replacement 74 3.3 Multinomial Counting 77 3.4 Special Arrangements and Selections 79 3.5 Applications 81 3.5.1 Game of Poker 81 3.5.2 Birthday Paradox 83 3.5.3 Quality Control 86 3.5.4 Best-of-Seven Championship Series 86 3.5.5 Lottery 89 3.6 Summary 90 Problems 90 Part II Random Variables 95 4 One Random Variable: Fundamentals 97 4.1 Types of Random Variables 97 4.2 The Cumulative Distribution Function 99 4.3 The Probability Mass Function 102 4.4 The Probability Density Function 104 4.5 Expected Values 107 4.5.1 Mean of a Random Variable 107 4.5.2 Variance of a Random Variable 110 4.5.3 Moments of a Random Variable 113 4.5.4 Mode and Median of a Random Variable 114 4.6 Conditional Distributions 116 4.7 Functions of a Random Variable 120 4.7.1 pdf of a Function of a Continuous Random Variable 121 4.7.2 pmf of a Function of a Discrete Random Variable 123 4.7.3 Computer Generation of Random Variables 124 4.8 Transform Methods 125 4.8.1 Moment Generating Function of a Random Variable 125 4.8.2 Characteristic Function of a Random Variable 126 4.9 Upper Bounds on Probability 127 4.9.1 Markov Bound 127 4.9.2 Chebyshev Bound 128 4.9.3 Chernoff Bound 128 4.10 Summary 131 Problems 131 5 Special Probability Distributions and Applications 137 5.1 Special Discrete Random Variables 137 5.1.1 The Bernoulli Distribution 137 5.1.2 The Binomial Distribution 138 5.1.3 The Geometric Distribution 140 5.1.4 The Pascal Distribution 142 5.1.5 The Hypergeometric Distribution 143 5.1.6 The Poisson Distribution 144 5.1.7 The Discrete Uniform Distribution 146 5.1.8 The Zipf (Zeta) Distribution 147 5.2 Special Continuous Random Variables 148 5.2.1 The Continuous Uniform Distribution 148 5.2.2 The Exponential Distribution 149 5.2.3 The Gamma Distribution 151 5.2.4 The Erlang Distribution 152 5.2.5 The Weibull Distribution 152 5.2.6 The Beta Distribution 153 5.2.7 The Laplace Distribution 154 5.2.8 The Pareto Distribution 155 5.3 Applications 156 5.3.1 Digital Transmission: Regenerative Repeaters 156 5.3.2 System Reliability: Failure Rate 157 5.3.3 Queuing Theory: Servicing Customers 158 5.3.4 Random Access: Slotted ALOHA 159 5.3.5 Analog-to-Digital Conversion: Quantization 160 5.4 Summary 161 Problems 162 6 Multiple Random Variables 165 6.1 Pairs of Random Variables 165 6.2 The Joint Cumulative Distribution Function of Two Random Variables 167 6.2.1 Marginal Cumulative Distribution Function 169 6.3 The Joint Probability Mass Function of Two Random Variables 170 6.3.1 Marginal Probability Mass Function 170 6.4 The Joint Probability Density Function of Two Random Variables 171 6.4.1 Marginal Probability Density Function 172 6.5 Expected Values of Functions of Two Random Variables 173 6.5.1 Joint Moments 174 6.6 Independence of Two Random Variables 175 6.7 Correlation between Two Random Variables 178 6.8 Conditional Distributions 185 6.8.1 Conditional Expectations 186 6.9 Distributions of Functions of Two Random Variables 188 6.9.1 Joint Distribution of Two Functions of Two Random Variables 191 6.10 Random Vectors 192 6.11 Summary 197 Problems 198 7 The Gaussian Distribution 201 7.1 The Gaussian Random Variable 201 7.2 The Standard Gaussian Distribution 204 7.3 Bivariate Gaussian Random Variables 210 7.3.1 Linear Transformations of Bivariate Gaussian Random Variables 213 7.4 Jointly Gaussian Random Vectors 215 7.5 Sums of Random Variables 217 7.5.1 Mean and Variance of Sum of Random Variables 217 7.5.2 Mean and Variance of Sum of Independent, Identically Distributed Random Variables 218 7.5.3 Distribution of Sum of Independent Random Variables 218 7.5.4 Sum of a Random Number of Independent, Identically Distributed Random Variables 219 7.6 The Sample Mean 220 7.6.1 Laws of Large Numbers 222 7.7 Approximating Distributions with the Gaussian Distribution 223 7.7.1 Relation between the Gaussian and Binomial Distributions 223 7.7.2 Relation between the Gaussian and Poisson Distributions 225 7.7.3 The Central Limit Theorem 226 7.8 Probability Distributions Related to the Gaussian Distribution 230 7.8.1 The Rayleigh Distribution 230 7.8.2 The Ricean Distribution 231 7.8.3 The Log-Normal Distribution 231 7.8.4 The Chi-Square Distribution 232 7.8.5 The Maxwell-Boltzmann Distribution 232 7.8.6 The Student's t-Distribution 233 7.8.7 The F Distribution 234 7.8.8 The Cauchy Distribution 234 7.9 Summary 234 Problems 235 Part III Statistics 239 8 Descriptive Statistics 241 8.1 Overview of Statistics 241 8.2 Data Displays 244 8.3 Measures of Location 249 8.4 Measures of Dispersion 250 8.5 Measures of Shape 255 8.6 Summary 257 Problems 257 9 Estimation 259 9.1 Parameter Estimation 259 9.2 Properties of Point Estimators 260 9.3 Maximum Likelihood Estimators 265 9.4 Bayesian Estimators 270 9.5 Confidence Intervals 272 9.6 Estimation of a Random Variable 274 9.7 Maximum a Posteriori Probability Estimation 275 9.8 Minimum Mean Square Error Estimation 277 9.9 Linear Minimum Mean Square Error Estimation 279 9.10 Linear MMSE Estimation Using a Vector of Observations 282 9.11 Summary 285 Problems 285 10 Hypothesis Testing 287 10.1 Significance Testing 287 10.2 Hypothesis Testing for Mean 291 10.2.1 p-Value 294 10.3 Decision Tests 300 10.4 Bayesian Test 303 10.4.1 Minimum Cost Test 304 10.4.2 Maximum a Posteriori Probability (MAP) Test 305 10.4.3 Maximum-Likelihood (ML) Test 305 10.4.4 Minimax Test 307 10.5 Neyman-Pearson Test 307 10.6 Summary 309 Problems 309 Part IV Random Processes 311 11 Introduction to Random Processes 313 11.1 Classification of Random Processes 313 11.1.1 State Space 314 11.1.2 Index (Time) Parameter 314 11.2 Characterization of Random Processes 318 11.2.1 Joint Distributions of Time Samples 318 11.2.2 Independent Identically Distributed Random Process 319 11.2.3 Multiple Random Processes 320 11.2.4 Independent Random Processes 320 11.3 Moments of Random Processes 320 11.3.1 Mean and Variance Functions of a Random Process 321 11.3.2 Autocorrelation and Autocovariance Functions of a Random Process 321 11.3.3 Cross-correlation and Cross-covariance Functions 324 11.4 Stationary Random Processes 326 11.4.1 Strict-Sense Stationary Processes 326 11.4.2 Wide-Sense Stationary Processes 327 11.4.3 Jointly Wide-Sense Stationary Processes 329 11.4.4 Cyclostationary Processes 331 11.4.5 Independent and Stationary Increments 331 11.5 Ergodic Random Processes 333 11.5.1 Strict-Sense Ergodic Processes 333 11.5.2 Wide-Sense Ergodic Processes 333 11.6 Gaussian Processes 336 11.7 Poisson Processes 339 11.8 Summary 341 Problems 341 12 Analysis and Processing of Random Processes 345 12.1 Stochastic Continuity, Differentiation, and Integration 345 12.1.1 Mean-Square Continuity 345 12.1.2 Mean-Square Derivatives 346 12.1.3 Mean-Square Integrals 347 12.2 Power Spectral Density 347 12.3 Noise 353 12.3.1 White Noise 353 12.4 Sampling of Random Signals 355 12.5 Optimum Linear Systems 357 12.5.1 Systems Maximizing Signal-to-Noise Ratio 357 12.5.2 Systems Minimizing Mean-Square Error 359 12.6 Summary 362 Problems 362 Bibliography 365 Books 365 Internet Websites 368 Answers 369 Index 387