Multivariate Statistics and Machine Learning

Preface
Acknowledgements

PART I - Preliminaries and Foundations

Chapter 0 - Introduction, Motivation, Pedagogy and Ideas About Learning

0.1. The Paradigm Shift (What Has Changed)
0.1.1. A Wide Divide
0.2. A Unified Vision - The Bridge
0.3. The Data Science and Machine Learning Invasion (Questions and Answers)
0.4. Who Should Read this Book?
0.4.1. Textbook Limbo
0.4.2. Theoretical vs. Applied vs. Software Books vs. "Cookbooks"
0.4.2.1. Watered Down Statistics
0.4.3. Prerequisites to Reading this Book
0.5. Pedagogical Approach and the Trade-Offs of Top-Down, Bottom-Up Learning
0.5.1. Top-Down, Bottom-Up Learning
0.5.2. Ways of Writing a Book: Making it Pedagogical Instead of Cryptic
0.5.3. Standing on the Shoulders of Giants (A Companion to Advanced Texts)
0.5.4. Making Equations "Speak"
0.5.5. The Power of Problems
0.5.6. Computing Languages
0.5.7. Notation Used in the Book
0.6. Nobody Learns a Million Things (The Importance of Foundations and Learning How to Learn)
0.6.1. Essential Philosophy of Science and History
0.6.2. Beyond the Jargon, Beyond the Hype
0.7. The Power and Dangers of Analogy and Metaphor (Ways of Understanding)
0.7.1. The Infinite Regress of Knowledge - A Venture into What it Means to "Understand" Something and Why Epistemology is Important
0.7.1.2. Epistemological Maturity
0.8. Format and Organization of Chapters

Chapter 1 - First Principles and Philosophical Foundations
1.1. Science, Statistics, Machine Learning, Artificial Intelligence
1.1.1. Mathematics, Statistics, Computation
1.1.2. Mathematical Systems as a Narrative to Understanding
1.2. The Scope of Data Analysis and Data Science (Expertise in Everything!)
1.2.1. Theoretical vs. Applied Statistics & Specialization
1.3. The Role of Computers
1.3.1. The Nature of Algorithms
1.3.1.2. Algorithmic Stability
1.4. The Importance of Design, Experimental or Otherwise
1.5. Inductive, Deductive, and Other Logics
1.5.1. Consistency and Goedel's Incompleteness Theorems
1.5.1.2. What is the Relevance of Goedel?
1.6. Supervised vs. Unsupervised Learning
1.6.1. Fuzzy Distinctions
1.7. Theoretical vs. Empirical Justification
1.7.1. Airplanes and Oceanic Submersibles
1.7.2. Will the Bridge Stay Up if the Mathematics Fail?
1.8. Level of Analysis Problem
1.9. Base Rates, Common Denominators and Degrees
1.9.1. Base Rates and Splenic Masses
1.9.2. Probability Neglect
1.9.3. The "Zero Group"
1.10. Statistical Regularities and Perceptions of Risk
1.10.1. Beck Depression Inventory: How Depressed Are You?
1.11. Decision, Risk Analysis and Optimization
1.11.1. The Risk of Making a Wrong Decision
1.11.2. Statistical Lives and Optimization
1.11.3. Medical Decision-Making and Dominating Criteria
1.12. All Knowledge, Scientific and Other, is Tentative
1.13. Occam's Razor
1.13.1. Parsimony vs. Complexity Trade-Off
1.14. Overfitting vs. Underfitting
1.14.1. Solutions to Overfitting
1.14.2. The Idea of Regularization
1.15. The Measurement Problem
1.15.1. What is Data?
1.15.2. The Philosophy and Scales of Measurement
1.15.3. Reliability
1.15.3.1. Coefficient Alpha
1.15.3.2. Test-Retest Reliability
1.15.4. Validity
1.15.5. Scales of Measurement
1.15.6. Likert Scales
1.15.6.1. Statistical Models for Likert Data
1.15.6.2. Models for Ordinal and Monotonically Increasing/Decreasing Data

Overview of Statistical and Machine Learning Concepts
1.16. Probably Approximately Correct
1.17. No Free Lunch Theorem
1.18. V-C Dimension and Complexity
1.19. Parametric vs. Nonparametric Learning Methods
1.19.1. Flexibility and Number of Parameters
1.19.1.1. Concept of Degrees of Freedom
1.19.2. Instance or Memory-Based Learning
1.19.3. Revisiting Classical Nonparametric Tests
1.20. Dimension Reduction, Distance, and Error Functions: Commonalities in Modeling
1.20.1. Dimension Reduction: What's the Big Idea?
1.20.2. The Curse of Dimensionality
1.21. Distance
1.22. Error Minimization
1.23. Training vs. Test Error
1.24. Cross-Validation and Model Selection
1.25. Monte Carlo Methods
1.26. Missing Data
1.27. Quantitative Approaches to Data Analysis
1.28. Chapter Review Exercises


Chapter 2 - Mathematical and Statistical Foundations

2.1. Mathematical "Previews" vs. the "Appendix" Approach (Why Previews are Better)
2.1.2. About Proofs

2.2. Elementary Probability and Fundamental Statistics
2.3. Interpretations of Probability
2.4. Mathematical Probability
2.4.1. Unions and Intersections of Events
2.5. Conditional Probability
2.5.1. Unconditional vs. Conditional Statistical Models
2.6. Probabilistic Independence
2.6.1. Everything is About Independence vs. Dependence!
2.7. Marginal vs. Conditional Distributions
2.8. Independence Implies Covariance of Zero, But Covariance of Zero Does Not (Necessarily) Imply Independence
2.9. Sensitivity and Specificity: More Conditional Probabilities
2.10. Bayes' Theorem and Conditional Probabilities
2.10.1. Bayes' Factor
2.10.2. Bayesian Model Selection
2.10.3. Bayes' Theorem as Rational Belief or Theorizing
2.11. Law of Large Numbers
2.11.1. Law of Large Numbers and the Idea of Committee Machines
2.12. Random Variables and Probability Density Functions
2.13. Convergence of Random Variables
2.14. Probability Density Functions
2.15. Normal (Gaussian) Distributions
2.15.1. Univariate Gaussian
2.15.2. Mixtures of Gaussians
2.15.3. Evaluating Univariate Normality
2.15.4. Multivariate Gaussian
2.15.5. Evaluating Multivariate Normality
2.16. Binomial Distributions
2.16.1. Approximation to the Normal Distribution
2.17. Multinomial Distributions
2.18. Poisson Distribution
2.19. Chi-Square Distributions
2.20. Expectation and Expected Value
2.21. Measures of Central Tendency
2.21.1. The Arithmetic Mean (Average)
2.21.1.1. Averaging Over Cases
(Why Thinking in Terms of Averages Can Be Dangerous)
2.21.2. The Median
2.22. Measures of Variability
2.22.1. Variance and Standard Deviation
2.22.2. Mean Absolute Deviation
2.23. Skewness and Kurtosis
2.24. Coefficient of Variation
2.25. Statistical Estimation
2.26. Bias-Variance Trade-Off
2.26.1. Is Irreducible Error Really Irreducible?
2.27. Maximum Likelihood Estimation
2.27.1. Why ML is so Popular and Alternatives
2.27.2. Estimation and Confidence Intervals
2.28. The Bootstrap (A Way of Estimating Nonparametrically)
2.28.1. Simple Examples of the Bootstrap
2.28.2. Why not Boostrap Everything?
2.28.3. Variations and Extensions of the Bootstrap
2.29. Elements of Classic Null Hypothesis Significance Testing
2.29.1. One-Tailed vs. Two-Tailed Tests
2.29.2. Effect Size
2.29.3. Cohen's d (Measure of Effect Size)
2.29.4. Are p-values that Evil?
2.29.5. Absolute vs. Relative Size of Effect (Context Matters)
2.29.6. Comparability of Effect Sizes Across Studies
2.29.7. Operationalizing Predictors
2.30. Central Limit Theorem
2.31. Covariance and Correlation
2.31.1. Why Does rxy Have Limits -1 to +1?
2.31.2. Covariance and Correlation in R and Python
2.31.3. Correlating Linear Combinations
2.31.4. Covariance and Correlation Matrices
2.32. Z-Scores and Z-Tests
2.32.1. Z-tests and T-tests for the Mean
2.33. Unequal Variances: Welch-Satterthwaite Approximation
2.34. Paired Data
2.35. Review Exercises

2.36 Linear Algebra and Matrices

2.36.1. Vectors
2.36.1.2. Vector Spaces and Fields
2.36.1.3. Zero, Unit Vectors, and One-Hot Vectors
2.36.1.4. Transpose of a Vector
2.36.1.5. Vector Addition and Length
2.36.1.6. Eigen Analysis and Decomposition
2.36.1.7. Points vs. Vectors
2.37. Matrices
2.37.1. Identity Matrix
2.37.2. Transpose of a Matrix
2.37.3. Symmetric Matrices
2.37.4. Matrix Addition and Multiplication
2.37.5. Meaning of Matrices (Matrices as Data and Transformations)
2.37.6. Kernel (Null Space)
2.37.7. Trace of a Matrix
2.38. Linear Combinations
2.39. Determinants
2.40. Means and Variances of Matrices
2.41. Determinant as a Generalized Variance
2.42. Matrix Inverse
2.42.1. Nonexistence of an Inverse and Singularity
2.43. Quadratic Forms
2.44. Positive Definite Matrices
2.45. Inner Products
2.46. Linear Independence
2.47. Rank of a Matrix
2.48. Orthogonal Matrices
2.49. Kernels, the Kernel Trick, and Dual Representations
2.49.1. When are Kernel Methods Useful?
2.50. Systems of Equations
2.51. Distance
2.52. Projections and Basis
2.53. The Meaning of Linearity
2.54. Basis and Dimension
2.54.1. Orthogonal Basis
2.55. Review Exercises

2.56. Calculus and Optimization
2.57. Functions, Approximation and Continuity
2.57.1. Definition of Continuity
2.58. The Derivative
2.58.1. Local Behavior and Approximation
2.58.2. Composite Functions and Basis Expansions
2.59. The Partial Derivative
2.60. Optimization and Gradients
2.60.1. What Does "Optimal" Mean?
2.60.2. Minima and Maxima via Calculus
2.60.3. Convex vs. Non-Convex Functions and Sets
2.61. Gradient Descent
2.61.1. How Does Gradient Descent Find Minima?
2.62. Integral Calculus
2.62.1. Double and Triple Integrals
2.63. Review Exercises

Chapter 3 - R and Python Software

3.1. The Dominance of R and Python
3.2. The R-Project
3.2.1. Installing R
3.2.2. Working with Data
3.2.2.1. Building a Data Frame
3.2.3. Installing Packages in R
3.2.4. Writing Functions in R
3.2.5. Mathematics and Statistics Using R
3.2.5.1. Addition, Subtraction, Multiplication and Division
3.2.5.2. Logarithms and Exponentials
3.2.5.3. Vectors and Matrices
3.2.5.4. Means
3.2.5.5. Covariance and Correlation
3.2.5.6. Sampling with Replacement in R
3.2.5.7. Visualization and Plots
3.2.5.7.1. Boxplots
3.2.6. Further Readings and Resources in R

3.3. Python
3.3.1. Installing Python
3.3.2. Elements of Python
3.3.3. Working With Data
3.3.4. Python Functions for Data Analysis
3.3.4.1. Mathematics Using Python
3.3.4.2. Splitting Data into Train and Test Sets
3.3.4.3. Preprocessing Data
3.3.5. Further Readings and Resources in Python
3.4. Chapter Review Exercises

PART II - Models and Methods

Chapter 4 - Univariate and Multivariate Analysis of Variance Models

4.1. The Classic ANOVA Model
4.1.1. Mean Squares
4.1.2. Expected Mean Squares of ANOVA
4.1.3. Effect Sizes for ANOVA
4.1.4. Contrasts and Post-Hoc Tests for ANOVA
4.1.5. ANOVA in Python
4.1.6. ANOVA in R
4.2. Factorial ANOVA and Higher-Order Models
4.2.1. Factorial ANOVA in Python
4.3. Random Effects and Mixed Models
4.3.1. The Meaning of a Fixed vs. Random Effect
4.3.2. Is the Fixed-Effects Model Actually Fixed? A Look at the Error Term
4.3.3. Mixed Models in Python
4.3.4. Mixed Models in R
4.4. MultiLevel Modeling
4.4.1. A Garbled Mess of Jargon
4.4.2. Why Do Multilevel Models Often Include Random Effects?
4.4.3. A Priori vs. Post-Hoc "Nesting"
4.4.4. Blocking as an Example of Hierarchical/Multilevel Structure
4.4.5. Non-Parametric Random-Effects Model
4.5. Repeated Measures and Longitudinal Models
4.5.1. Classic Repeated Measures Models
4.6. Multivariate Analysis of Variance (MANOVA)
4.6.1. Suitability of MANOVA
4.6.2. Extending the Univariate Model (Hotelling's T2)
4.6.3. Multivariate Test Statistics
4.6.4. Evaluating Equality of Covariance Matrices (The Box-M Test)
4.6.5. MANOVA in Python
4.6.6. MANOVA in R
4.7. Linear Discriminant Analysis (as the "Reverse" of MANOVA)
4.8. Chapter Review Exercises

Chapter 5 - Simple Linear and Multiple Regression Models (and Extensions)
5.1. Simple Linear Regression - Fixed Predictor Case
5.1.1. Parameter Estimates
5.1.2. Simple Linear Regression in R
5.1.3. Simple Linear Regression in Python
5.2. Multiple Linear Regression
5.2.1. Minimizing Squared vs. Absolute Deviations
5.2.2. Hypothesis-Testing in Multiple Regression
5.2.3. Multiple Linear Regression in Python
5.2.4. Multiple Linear Regression in R
5.3. Geometry of Least-Squares
5.4. Gauss-Markov Theorem (What We Like About Least-Squares Estimates)
5.4.1. Are Unbiased Estimators Always Best?
5.5. Time Series (An Example of Correlated Errors)
5.6. Model Selection in Regression (Is There an Optimal Model?)
5.7. Effect Size and Adjusting the Training Error Rate
5.7.1. R2, Adjusted R2, Cp, AIC, BIC
5.7.2. Comparing , , AIC, BIC to Cross-Validation
5.8. Assumptions for Regression
5.8.1. Collinearity
5.8.1.1. Variance Inflation Factor
5.8.2. Collinearity Necessarily Implies Redundancy Only in Terms of Variance
5.9. Variable Selection Methods (Building the Regression Model)
5.9.1. Forward, Backward and Stepwise in R
5.10. Mediated and Moderated Regression Models
5.10.1. Statistical Mediation
5.10.2. Statistical Moderation
5.10.3. Moderated Mediation
5.10.4. Mediation in Python
5.11. Further Directions and a Final Word of Warning on Mediation and Moderation
5.12. Principal Components Regression
5.12.1. What is Principal Components Analysis?
5.12.2. PCA Regression and Singularity
5.12.3. Principal Components Regression in R
5.13. Partial Least-Squares Regression
5.13.1. Partial Least Squares in R
5.13.2. Partial Least Squares in Python
5.14. Multivariate Reduced-Rank Regression
5.15. Canonical Correlation
5.15.1. Canonical Correlation in R
5.16. Chapter Review Exercises

Chapter 6 - Regularization Methods in Regression: Ridge, Lasso, Elastic Net
6.1. The Concept of Regularization
6.1.1. Regularization in Regression and Beyond
6.2. Ridge Regression
6.2.1. Mathematics of Ridge Regression
6.2.2. Consequence of Ridge Estimator
6.2.3. Revisiting the Bias-Variance Tradeoff (Why Ridge is Useful)
6.2.4. A Visual Look at Ridge Regression
6.2.5. Ridge Regression in Python
6.2.6. Ridge Regression in R
6.3. Lasso Regression
6.3.1. Lasso Regression in Python
6.3.2. Lasso Regression in R
6.4. Elastic Net
6.4.1. Elastic Net in Python
6.5. Which Regularization Penalty is Better?
6.6. Least-Angle Regression
6.6.1. Least-Angle Regression in R
6.7. Additional Variable Selection Algorithms
6.8. Chapter Review Exercises

Chapter 7 - Nonlinear and Nonparametric Regression

7.1. Polynomial Regression
7.1.1. Polynomial Regression in Python
7.1.2. Polynomial Regression in R
7.1.3. Polynomial Regression as a Global Strategy
7.1.4. A More Local Alternative
7.1.5. Least-Squares Regression Line as a "Floating Mean" (Toward a Localized Approach)
7.1.5.1. Zooming in on Locality
7.2. Basis Functions and Expansions
7.2.1. Basis Functions and Locality
7.2.2. Neural Networks as a Basis Expansion (Generalizing the Concept)
7.2.3. Regression Splines and the Concept of a "Knot"
7.2.4. Conceptualizing Regression Splines
7.2.5. Problem with Splines and Imposing Constraints
7.2.6. Polynomial Regression vs. Regression Splines
7.3. Nonparametric Regression: Local and Kernel Regression
7.3.1. Motivating Kernel Regression via Local-Averaging
7.3.2. Kernel Regression - "Locally Weighted Averaging"
7.3.3. A Variety of Kernels
7.3.4. Kernel Regression is not Nonlinear; It is Nonparametric
7.3.5. Kernel Regression in R
7.4. Chapter Review Exercises

Chapter 8 - Generalized Linear and Additive Models: Logistic, Poisson, and Related Models

8.1. How to Operationalize the Response
8.1.1. Pros and Cons of Binning
8.1.2. Detecting New Classes or Categories
8.2. The Generalized Linear Model
8.2.1. Intrinsically Linear Models
8.2.2. General vs. Generalized Linear Models
8.3. The Logistic Regression Model
8.3.1. Odds and Odds Ratios
8.3.2. Logistic Regression in R
8.3.3. Logistic Regression in Python
8.4. Generalized Linear Models and Neural Networks
8.5. Multiple Logistic Regression
8.5.1. Multiple Logistic Regression in R
8.6. Poisson Regression
8.6.1. Poisson Regression in R
8.6.2. Poisson Regression in Python
8.7. Generalized Additive Models (A Flexible Nonparametric Alternative)
8.7.1. Why Use a Smoother Instead of Linear Weights?
8.7.2. Deriving the Generalized Additive Model
8.7.3. GAM as a Smooth Extension to GLLM
8.7.4. Generalized Additive Models and Neural Networks
8.7.5. Linking the Logit to the Additive Logistic Model
8.8. Overview and Recap of Nonlinear Approaches for Nonlinear Regression
8.9. Discriminant Analysis
8.9.1. Bayes is Best for Classification
8.9.2. Why Not Always Bayes?
8.9.3. The Linear Discriminant Analysis Model
8.9.4. How LDA Approximates Bayes
8.9.5. Estimating the Prior Probability
8.10. Multiclass Discriminant Analysis
8.11. Discriminant Analysis in a Simple Scatterplot
8.12. Quadratic Discriminant Analysis
8.13. Regularized Discriminant Analysis
8.14. Discriminant Analysis in R
8.15. Discriminant Analysis in Python
8.16. Naive Bayes (Approximating the Bayes Classifier by Assuming (Conditional) Independence)
8.16.1. What Makes Naive Bayes "Naive"?
8.16.2. Naive Bayes in Python
8.17. LDA, QDA, Naive Bayes: Which is Best?
8.18. Nonparametric K-Nearest Neighbors
8.18.1. K-Nearest Neighbor (KNN): Only Looking at Nearby Points
8.18.2. Example of KNN
8.18.3. Disadvantages of KNN
8.19. Chapter Review Exercises

Chapter 9 - Support Vector Machines

9.1. Maximum Margin Classifier
9.1.1. When Sum Does Not Equal Zero
9.1.1.1. So, What's the Problem?
9.1.2. Building the Maximal Margin Classifier
9.2. The Case of No Separating Hyperplane
9.3. Support Vector Classifier for the Non-Separable Case
9.4. Support Vector Machines (Introducing the Kernel for Nonlinearity)
9.4.1. Enlarging the Feature Space with Kernels
9.4.2. Support Vector Machines in Python
9.4.3. Support Vector Machines in R
9.5. Chapter Review Exercises

Chapter 10 - Decision Trees, Bagging, Random Forests and Committee Machines

10.1. Why Combining Weak Learners Works (Concept of Variance Reduction Using Averages)
10.2. Decision Trees
10.2.1. How Should Trees Be Grown?
10.2.2. Optimization Criteria for Tree-Building
10.2.3. Why not Multiway Splits?
10.2.4. Overfitting, Saturation, and Tree Pruning
10.2.5. Cost-Complexity or Weakest-Link Pruning
10.3. Classification Trees
10.3.1. Gini Index
10.3.2. Decision Trees in R
10.4. Committee Machines
10.5. Overview of Bagging and Boosting
10.5.1. Bagging
10.5.2. A Familiar Example (Bagging Samples and the Variance of the Mean)
10.5.3. A Deeper Look at Bagging
10.5.4. Out-of-Bag Error
10.5.5. Interpreting Results from Bagging
10.5.6. Bagging in R
10.6. Random Forests
10.6.1. The Problem with Bagging Decision Trees
10.6.2. Equivalency of Random Forests and Bagging
10.6.3. Random Forests in R
10.7. Boosting
10.7.1. Boosting Using R
10.8. Stacked Generalization
10.9. Chapter Review Exercises

Chapter 11 - Principal Components Analysis, Blind Source Separation, and Manifold Learning

11.1. Dimension Reduction and Jargon
11.2. Deriving Classic Principal Components Analysis
11.2.1. The 2nd Principal Component
11.2.2. PCA as a Least-Squares Technique and Minimizing Reconstruction Error
11.2.3. Choosing the Number of Derived Components
11.2.4. Why Reconstruction Error is Insufficient for Choosing Number of Components
11.2.5. Constraints on Components
11.2.6. Centering Observed Variables
11.2.7. Orthogonality of Components
11.2.8. Proportion of Variance Explained by Each Component (Covariance vs. Correlation Matrices)
11.2.9. Principal Components as a Rotation of Axes
11.2.10 Principal Components, Discriminant Functions, Canonical Variates (Linking Foundations)
11.2.11. Principal Components in Python
11.2.12. Principal Components in R
11.2.13. Cautionary Concerns and Caveats Regarding Principal Components
11.3. Independent Components Analysis
11.3.1. Principal Components vs. Independent Components Analysis
11.4. Probabilistic PCA
11.4.1. Motivation for Probabilistic PCA
11.4.2. Probabilistic PCA in R
11.5. PCA for Discrete, Binary, and Categorical Data
11.6. Nonlinear Dimension Reduction
11.6.1. Kernel PCA
11.6.2. How KPCA Works
11.6.3. Kernalizing and Computational Complexity
11.6.4. Reconstruction Error in Kernel PCA
11.6.5. The Matrices of Kernel PCA
11.6.6. Classical PCA as a Special Case of Kernel PCA
11.6.7. "Kernel Trick" is Not Simply About Cost
11.6.8. Kernel PCA in Python
11.6.9. Kernel PCA in R
11.7. Principal Curves
11.7.1. Principal Components as a Special Case of Principal Curves and Surfaces
11.7.2. Principal Curves in R
11.8. Principal Components Analysis as an Encoder
11.9. Neural Networks and PCA as Autoencoders
11.10. Multidimensional Scaling
11.10.1. Merits of MDS
11.10.2. Metric vs. Non-Metric (Ordinal) Scaling
11.10.3. Weakness of MDS: "Closeness" Can be Arbitrary
11.10.4. Standardization of Distances
11.10.5. MDS in Python
11.10.6. MDS in R
11.11. Self-Organizing Maps
11.12. Manifold Learning
11.12.1. Manifold Hypothesis
11.12.2. Example of a Simple Manifold
11.12.3. Nonparametric Manifolds
11.12.4. Geodesic Distances
11.13. Local Linear Embedding
11.13.1. LLE in Python
11.14. Isomap
11.14.1. Isomap in Python
11.15. Stochastic Neighborhood Embedding (SNE)
11.15.1. SNE in R
11.16. t-SNE
11.16.1. Performance of t-SNE to Other Techniques
11.16.2. t-SNE in Python
11.16.3. t-SNE in R
11.17. Manifold Learning and Beyond
11.18. Chapter Review Exercises

Chapter 12 - Exploratory Factor Analysis

12.1. Why Treat Factor Analysis in its Own Chapter?
12.2. Common Orthogonal Factor Model
12.2.1. Factor Analysis is a Regression Model
12.2.2. Assumptions Underlying the Factor Analysis Model
12.2.3. Implied Covariance Matrix
12.3. The Problem with Factor Analysis
12.3.1. The Problem is the Users, Not the Method
12.3.2. Factor Analysis Generalizes to Machine Learning
12.4. Factor Estimation
12.4.1. Principal Factor (Principal Axis Factoring)
12.4.2. Maximum Likelihood
12.5. Factor Rotation
12.5.1. Varimax
12.5.2. Quartimax
12.6. Bartlett's Test of Sphericity
12.6.1. Factor Analysis in Python
12.6.2. Factor Analysis in R
12.7. Independent Factor Analysis
12.8. Nonlinear Factor Analysis (and Autoencoders)
12.8.1. Unpacking the Autoencoder
12.8.2. Factor Analysis as a Neural Network
12.9. Probabilistic "Sensible" PCA (again)
12.10. Mixtures of Factor Analysis (Modeling Local Linearity)
12.11. Item Factor Analysis
12.12. Sparse Factor Analysis
12.13. Chapter Review Exercises

Chapter 13 - Confirmatory Factor Analysis, Path Analysis and Structural Equation Modeling

13.1. What Makes a Model "Exploratory" vs. "Confirmatory"?
13.2. Why "Causal Modeling" is not Causal at all
13.2.1. Misguided History
13.2.2. Baron and Kenny (1986)
13.3. Is the Variable Measurable? The Observed vs. Unobserved Distinction
13.4. Path Analysis (Extending Regression and Previewing SEM)
13.4.1. Exogenous vs. Endogenous Variables
13.5. Confirmatory Factor Analysis Model
13.6. Structural Equation Models
13.6.1. Covariance Modeling
13.6.2. Evaluating Model Fit
13.6.3. Overall (Absolute) Measures
13.6.4. Incremental Fit Indices
13.7. Structural Equation Modeling with Nonlinear Effects
13.7.1. Example of a Nonlinear SEM
13.7.2. Structural Equation Nonparametric and Semiparametric Mixture Models
13.8. Caveats Regarding SEM Models
13.9. SEM in R
13.10. Chapter Review Exercises

Chapter 14 - Cluster Analysis and Data Segmentation

14.1. Cluster Paradigms and Classifications
14.2. Are Clusters Meaningful?
14.3. Dissimilarity Metrics (The Basis of Clustering Algorithms)
14.4. Association Rules (Market Basket Analysis)
14.5. Why Not Consider All Groups?
14.5.1. What Makes a "Good" Clustering Algorithm?
14.6. Distance and Proximity Metrics
14.7. Is the Data Clusterable?
14.8. Algorithms for Cluster Analysis
14.8.1. K-Means Clustering and K-Representatives
14.8.2. How K-Means Works
14.8.3. Defining Proximity for K-Means
14.8.4. Setting k in K-Means
14.8.5. Weakness of K-Means
14.8.6. K-means vs. ANOVA vs. Discriminant Analysis
14.8.7. Making K-Means Probabilistic via K-Means ++
14.8.8. Using the Data: K-Medoids Clustering
14.8.9. K-Means in Python
14.8.10. K-Means in R
14.9. Sparse and Longitudinal K-Means
14.10. Hierarchical Clustering
14.10.1. Agglomerative Clustering in Python
14.10.2. Agglomerative Clustering in R
14.11. Density-Based Clustering (DBSCAN)
14.11.1. Dense Points and Crowded Regions
14.11.2. DBSCAN in R
14.12. Clustering via Mixture Models
14.12.1. Model Selection for Clustering Solutions
14.13. Cluster Validation
14.14. Cluster Analysis and Beyond
14.15. Chapter Review Exercises

Chapter 15 - Artificial Neural Networks and Deep Learning

15.1. The Rise of Neural Networks: Original Motivation
15.2. Rosenblatt's Perceptron
15.3. Big Picture Overview of Machine Learning and Neural Networks
15.3.1. What is a Neural Network? (Minimizing the Hype)
15.3.2. Neural Networks are Composite Functions
15.4. Single Layer Feedforward Neural Network
15.5. What is an Activation Function?
15.5.1. Activation Functions do not "Activate" Anything
15.5.2. Types of Activation Functions
15.5.3. Saturating vs. Non-Saturating Activation Functions
15.5.3.1. The Problem with ReLU
15.5.3.2. LeakyReLU
15.5.4. Which Activation Function to Use?
15.6. The Multilayer Perceptron - A Deeper Look at Neural Networks
15.7. Training Neural Networks
15.7.1. Backpropagation and Minimizing Error Sums of Squares
15.8. How Many Hidden Nodes and Layers to Include?
15.9. Overfitting in Neural Networks
15.9.1. Early Stopping
15.9.2. Dropout Method
15.9.3. Regularized Network
15.10. Types of Networks
15.11. The Universal Approximation Theorem (The Appeal of Neural Networks)
15.11.1.Visualizing the Universal Approximation Theorem
15.12. Neural Networks and Projection Pursuit
15.12.1. Projection Pursuit Regression and Relation to Neural Networks
15.13. Summary, Warnings and Caveats of Neural Networks
15.14. Neural Networks in Python
15.15. Neural Networks in R
15.16. Chapter Review Exercises

Concluding Remarks

References

Index