Product Analytics

Preface xvii
Acknowledgments xxiii
About the Author xxv

Part I: Qualitative Methodology 1
Chapter 1: Data in Action: A Model of a Dinner Party 3
1.1 The User Data Disruption 4
1.2 A Model of a Dinner Party 7
1.3 What's Unique about User Data? 13
1.4 Why Does Causation Matter? 23
1.5 Actionable Insights 24

Chapter 2: Building a Theory of the Social Universe 25
2.1 Building a Theory 25
2.2 Conceptualization and Measurement 36
2.3 Theories from a Web Product 40
2.4 Actionable Insights 46

Chapter 3: The Coveted Goalpost: How to Change Human Behavior 47
3.1 Understanding Actionable Insight 47
3.2 It's All about Changing "Your" Behavior 50
3.3 A Theory about Human Behavioral Change 55
3.4 Change in a Web Product 59
3.5 What Are Realistic Expectations for Behavioral Change? 61
3.6 Actionable Insights 66

Part II: Basic Statistical Methods 67
Chapter 4: Distributions in User Analytics 69
4.1 Why Are Metrics Important? 69
4.2 Actionable Insights 82

Chapter 5: Retained? Metric Creation and Interpretation 85
5.1 Period, Age, and Cohort 85
5.2 Metric Development 91
5.3 Actionable Insights 106

Chapter 6: Why Are My Users Leaving? The Ins and Outs of A/B Testing 107
6.1 An A/B Test 107
6.2 The Curious Case of Free Weekly Events 109
6.3 But It's Correlated ... 113
6.4 Why Randomness? 117
6.5 The Nuts and Bolts of an A/B Test 119
6.6 Pitfalls in A/B testing 132
6.7 Actionable Insights 135

Part III: Predictive Methods 137
Chapter 7: Modeling the User Space: k-Means and PCA 139
7.1 What Is a Model? 139
7.2 Clustering Techniques 140
7.3 Actionable Insights 150

Chapter 8: Predicting User Behavior: Regression, Decision Trees, and Support Vector Machines 151
8.1 Predictive Inference 151
8.2 Much Ado about Prediction? 152
8.3 Predictive Modeling 154
8.4 Validation of Supervised Learning Models 169
8.5 Actionable Insights 172

Chapter 9: Forecasting Population Changes in Product: Demographic Projections 173
9.1 Why Should We Spend Time on the Product Life Cycle? 174
9.2 Birth, Death, and the Full Life Cycle 174
9.3 Different Models of Retention 177
9.4 The Art of Population Prediction 183
9.5 Actionable Insights 203

Part IV: Causal Inference Methods 205
Chapter 10: In Pursuit of the Experiment: Natural Experiments and Difference-in-Difference Modeling 207
10.1 Why Causal Inference? 208
10.2 Causal Inference versus Prediction 208
10.3 When A/B Testing Doesn't Work 211
10.4 Nuts and Bolts of Causal Inference from Real-World Data 213
10.5 Actionable Insights 222

Chapter 11: In Pursuit of the Experiment, Continued 225
11.1 Regression Discontinuity 226
11.2 Estimating the Causal Effect of Gaining a Badge 229
11.3 Interrupted Time Series 234
11.4 Seasonality Decomposition 238
11.5 Actionable Insights 241

Chapter 12: Developing Heuristics in Practice 243
12.1 Determining Causation from Real-World Data 243
12.2 Statistical Matching 244
12.3 Problems with Propensity Score Matching 251
12.4 Matching as a Heuristic 253
12.5 The Best Guess 254
12.6 Final Thoughts 257
12.7 Actionable Insights 258

Chapter 13: Uplift Modeling 259
13.1 What Is Uplift? 259
13.2 Why Uplift? 260
13.3 Understanding Uplift 261
13.4 Prediction and Uplift 261
13.5 Difficulties with Uplift 262
13.6 Actionable Insights 275

Part V: Basic, Predictive, and Causal Inference Methods in R 277
Chapter 14: Metrics in R 279
14.1 Why R? 279
14.2 R Fundamentals: A Very Basic Introduction to R and Its Setup 280
14.3 Sampling from Distributions in R 285
14.4 Summary Statistics 290
14.5 Q-Q Plot 291
14.6 Calculating Variance and Higher Moments 293
14.7 Histograms and Binning 294
14.8 Bivariate Distribution and Correlation 301
14.9 Parity Progression Ratios 305
14.10 Summary 307

Chapter 15: A/B Testing, Predictive Modeling, and Population Projection in R 309
15.1 A/B Testing in R 309
15.2 Clustering 320
15.3 Predictive Modeling 324
15.4 Population Projection 333
15.5 Actionable Insights 342

Chapter 16: Regression Discontinuity, Matching, and Uplift in R 343
16.1 Difference-in-Difference Modeling 343
16.2 Regression Discontinuity and Time-Series Modeling 346
16.3 Statistical Matching 357
16.4 Uplift Modeling 370
16.5 Actionable Insights 383

Conclusion 387

Bibliography 391
Index 397