Probability and Conditional Expectation

Part I Measure-Theoretical Foundations of Probability Theory 1 Measure 3 1.1 Introductory Examples 3 1.2 sigma-Algebra and Measurable Space 4 1.2.1 sigma-Algebra Generated by a Set System 9 1.2.2 sigma-Algebra of Borel Sets on Rn 12 1.2.3 sigma-Algebra on a Cartesian Product 13 1.2.4 intersection -Stable Set Systems That Generate a sigma-Algebra 15 1.3 Measure and Measure Space 16 1.3.1 sigma-Additivity and Related Properties 17 1.3.2 Other Properties 18 1.4 Specific Measures 20 1.4.1 Dirac Measure and Counting Measure 21 1.4.2 Lebesgue Measure 22 1.4.3 Other Examples of a Measure 23 1.4.4 Finite and sigma-Finite Measures 23 1.4.5 Product Measure 24 1.5 Continuity of a Measure 25 1.6 Specifying a Measure via a Generating System 27 1.7 sigma-Algebra That is Trivial With Respect to a Measure 28 1.8 Proofs 28 1.9 Exercises 31 2 Measurable Mapping 41 2.1 Image and Inverse Image 41 2.2 Introductory Examples 42 2.2.1 Example 1: Rectangles 42 2.2.2 Example 2: Flipping two Coins 44 2.3 Measurable Mapping 46 2.3.1 Measurable Mapping 46 2.3.2 sigma-Algebra Generated by a Mapping 51 2.3.3 Final sigma-Algebra 54 2.3.4 Multivariate Mapping 54 2.3.5 Projection Mapping 56 2.3.6 Measurability With Respect to a Mapping 56 2.4 Theorems on Measurable Mappings 58 2.4.1 Measurability of a Composition 59 2.4.2 Theorems on Measurable Functions 61 2.5 Equivalence of Two Mappings With Respect to a Measure 64 2.6 Image Measure 67 2.7 Proofs 70 2.8 Exercises 75 3 Integral 83 3.1 Definition 83 3.1.1 Integral of a Nonnegative Step Function 83 3.1.2 Integral of a Nonnegative Measurable Function 88 3.1.3 Integral of a Measurable Function 93 3.2 Properties 96 3.2.1 Integral of mu-Equivalent Functions 98 3.2.2 Integral With Respect to a Weighted Sum of Measures 100 3.2.3 Integral With Respect to an Image Measure 102 3.2.4 Convergence Theorems 103 3.3 Lebesgue and Riemann Integral 104 3.4 Density 106 3.5 Absolute Continuity and the Radon-Nikodym Theorem 108 3.6 Integral With Respect to a Product Measure 110 3.7 Proofs 111 3.8 Exercises 120 Part II Probability, Random Variable and its Distribution 4 Probability Measure 127 4.1 Probability Measure and Probability Space 127 4.1.1 Definition 127 4.1.2 Formal and Substantive Meaning of Probabilistic Terms 128 4.1.3 Properties of a Probability Measure 128 4.1.4 Examples 130 4.2 Conditional Probability 132 4.2.1 Definition 132 4.2.2 Filtration and Time Order Between Events and Sets of Events 133 4.2.3 Multiplication Rule 135 4.2.4 Examples 136 4.2.5 Theorem of Total Probability 137 4.2.6 Bayes' Theorem 138 4.2.7 Conditional-Probability Measure 139 4.3 Independence 143 4.3.1 Independence of Events 143 4.3.2 Independence of Set Systems 144 4.4 Conditional Independence Given an Event 145 4.4.1 Conditional Independence of Events Given an Event 145 4.4.2 Conditional Independence of Set Systems Given an Event 146 4.5 Proofs 148 4.6 Exercises 150 5 Random Variable, Distribution, Density, and Distribution Function 155 5.1 Random Variable and its Distribution 155 5.2 Equivalence of Two Random Variables With Respect to a Probability Measure 161 5.2.1 Identical and P-Equivalent Random Variables 161 5.2.2 P-Equivalence, PB-Equivalence, and Absolute Continuity 164 5.3 Multivariate Random Variable 167 5.4 Independence of Random Variables 169 5.5 Probability Function of a Discrete Random Variable 175 5.6 Probability Density With Respect to a Measure 178 5.6.1 General Concepts and Properties 178 5.6.2 Density of a Discrete Random Variable 180 5.6.3 Density of a Bivariate Random Variable 180 5.7 Uni- or Multivariate Real-Valued Random Variable 182 5.7.1 Distribution Function of a Univariate Real-Valued Random Variable 182 5.7.2 Distribution Function of a Multivariate Real-Valued Random Variable 184 5.7.3 Density of a Continuous Univariate Real-Valued Random Variable 185 5.7.4 Density of a Continuous Multivariate Real-Valued Random Variable 187 5.8 Proofs 188 5.9 Exercises 196 6 Expectation, Variance, and Other Moments 199 6.1 Expectation 199 6.1.1 Definition 199 6.1.2 Expectation of a Discrete Random Variable 200 6.1.3 Computing the Expectation Using a Density 202 6.1.4 Transformation Theorem 203 6.1.5 Rules of Computation 206 6.2 Moments, Variance, and Standard Deviation 207 6.3 Proofs 212 6.4 Exercises 213 7 Linear Quasi-Regression, Covariance, and Correlation 217 7.1 Linear Quasi-Regression 217 7.2 Covariance 220 7.3 Correlation 224 7.4 Expectation Vector and Covariance Matrix 227 7.4.1 Random Vector and Random Matrix 227 7.4.2 Expectation of a Random Vector and a Random Matrix 228 7.4.3 Covariance Matrix of two Multivariate Random Variables 229 7.5 Multiple Linear Quasi-Regression 231 7.6 Proofs 233 7.7 Exercises 237 8 Some Distributions 245 8.1 Some Distributions of Discrete Random Variables 245 8.1.1 Discrete Uniform Distribution 245 8.1.2 Bernoulli Distribution 246 8.1.3 Binomial Distribution 247 8.1.4 Poisson Distribution 250 8.1.5 Geometric Distribution 252 8.2 Some Distributions of Continuous Random Variables 254 8.2.1 Continuous Uniform Distribution 254 8.2.2 Normal Distribution 256 8.2.3 Multivariate Normal Distribution 259 8.2.4 Central chi2-Distribution 262 8.2.5 Central t -Distribution 264 8.2.6 Central F-Distribution 266 8.3 Proofs 267 8.4 Exercises 271 Part III Conditional Expectation and Regression 9 Conditional Expectation Value and Discrete Conditional Expectation 277 9.1 Conditional Expectation Value 277 9.2 Transformation Theorem 280 9.3 Other Properties 282 9.4 Discrete Conditional Expectation 283 9.5 Discrete Regression 285 9.6 Examples 287 9.7 Proofs 291 9.8 Exercises 291 10 Conditional Expectation 295 10.1 Assumptions and Definitions 295 10.2 Existence and Uniqueness 297 10.2.1 Uniqueness With Respect to a Probability Measure 298 10.2.2 A Necessary and Sufficient Condition of Uniqueness 299 10.2.3 Examples 300 10.3 Rules of Computation and Other Properties 301 10.3.1 Rules of Computation 301 10.3.2 Monotonicity 302 10.3.3 Convergence Theorems 302 10.4 Factorization, Regression, and Conditional Expectation Value 306 10.4.1 Existence of a Factorization 306 10.4.2 Conditional Expectation and Mean-Squared Error 307 10.4.3 Uniqueness of a Factorization 308 10.4.4 Conditional Expectation Value 309 10.5 Characterizing a Conditional Expectation by the Joint Distribution 312 10.6 Conditional Mean Independence 313 10.7 Proofs 318 10.8 Exercises 321 11 Residual, Conditional Variance, and Conditional Covariance 329 11.1 Residual With Respect to a Conditional Expectation 329 11.2 Coefficient of Determination and Multiple Correlation 333 11.3 Conditional Variance and Covariance Given a sigma-Algebra 338 11.4 Conditional Variance and Covariance Given a Value of a Random Variable 339 11.5 Properties of Conditional Variances and Covariances 342 11.6 Partial Correlation 345 11.7 Proofs 347 11.8 Exercises 348 12 Linear Regression 357 12.1 Basic Ideas 357 12.2 Assumptions and Definitions 359 12.3 Examples 361 12.4 Linear Quasi-Regression 366 12.5 Uniqueness and Identification of Regression Coefficients 367 12.6 Linear Regression 369 12.7 Parametrizations of a Discrete Conditional Expectation 370 12.8 Invariance of Regression Coefficients 374 12.9 Proofs 375 12.10Exercises 377 13 Linear Logistic Regression 381 13.1 Logit Transformation of a Conditional Probability 381 13.2 Linear Logistic Parametrization 383 13.3 A Parametrization of a Discrete Conditional Probability 385 13.4 Identification of Coefficients of a Linear Logistic Parametrization 387 13.5 Linear Logistic Regression and Linear Logit Regression 388 13.6 Proofs 394 13.7 Exercises 396 14 Conditional Expectation With Respect to a Conditional-Probability Measure 399 14.1 Introductory Examples 399 14.2 Assumptions and Definitions 404 14.3 Properties 410 14.4 Partial Conditional Expectation 412 14.5 Factorization 413 14.5.1 Conditional Expectation Value With Respect to PB 414 14.5.2 Uniqueness of Factorizations 415 14.6 Uniqueness 415 14.6.1 A Necessary and Sufficient Condition of Uniqueness 415 14.6.2 Uniqueness w.r.t. P and Other Probability Measures 417 14.6.3 Necessary and Sufficient Conditions of P-Uniqueness 418 14.6.4 Properties Related to P-Uniqueness 420 14.7 Conditional Mean Independence With Respect to PZ=z 424 14.8 Proofs 426 14.9 Exercises 431 15 Conditional Effect Functions of a Discrete Regressor 437 15.1 Assumptions and Definitions 437 15.2 Conditional Intercept Function and Effect Functions 438 15.3 Implications of Independence of X and Z for Regression Coefficients 441 15.4 Adjusted Conditional Effect Functions 443 15.5 Conditional Logit Effect Functions 447 15.6 Implications of Independence of X and Z for the Logit Regression Coefficients 450 15.7 Proofs 452 15.8 Exercises 454 Part IV Conditional Independence and Conditional Distribution 16 Conditional Independence 459 16.1 Assumptions and Definitions 459 16.1.1 Two Events 459 16.1.2 Two Sets of Events 461 16.1.3 Two Random Variables 462 16.2 Properties 463 16.3 Conditional Independence and Conditional Mean Independence 470 16.4 Families of Events 473 16.5 Families of Set Systems 473 16.6 Families of Random Variables 475 16.7 Proofs 478 16.8 Exercises 486 17 Conditional Distribution 491 17.1 Conditional Distribution Given a sigma-Algebra or a Random Variable 491 17.2 Conditional Distribution Given a Value of a Random Variable 494 17.3 Existence and Uniqueness 497 17.3.1 Existence 497 17.3.2 Uniqueness of the Functions PY |C ( ·, A') 498 17.3.3 Common Null Set (CNS) Uniqueness of a Conditional Distribution 499 17.4 Conditional-Probability Measure Given a Value of a Random Variable 502 17.5 Decomposing the Joint Distribution of Random Variables 504 17.6 Conditional Independence and Conditional Distributions 506 17.7 Expectations With Respect to a Conditional Distribution 511 17.8 Conditional Distribution Function and Probability Density 513 17.9 Conditional Distribution and Radon-Nikodym Density 516 17.10Proofs 520 17.11Exercises 536 References 541