Methods Matter

1 The Challenge for Educational Research 1.1 The Long Quest 1.2 The Quest is World-Wide 1.3 What this Book is About 1.4 What to Read Next 2 The Importance of Theory 2.1 What is Theory? 2.2 Theory in Education 2.3 Voucher Theory 2.4 What Kind of Theories? 2.5 What to Read Next 3 Designing Research to Address Causal Questions 3.1 Conditions to Strive for in All Research 3.2 Making Causal Inferences 3.3 Past Approaches To Answering Causal Questions in Education 3.4 The Key Challenge of Causal Research 3.5 What to Read Next 4 Investigator-Designed Randomized Experiments 4.1 Conducting Randomized Experiments 4.1.1 An Example of a "Two-Group" Experiment 4.2 Analyzing Data from Randomized Experiments 4.2.1 Better Your Research Design, the Simpler Your Data-Analysis 4.2.2 Bias and Precision in the Estimation of Experimental Effects 4.3 What to Read Next 5 Challenges in Designing, Implementing, and Learning from Randomized Experiments 5.1 Critical Decisions in the Design of Experiments 5.1.1 Defining the Treatment Being Evaluated 5.1.2 Defining the Population from Which Participants Will Be Sampled 5.1.3 Deciding Which Outcomes to Measure 5.1.4 Deciding How Long To Track Participants 5.2 Threats to Validity of Randomized Experiments 5.2.1 Contamination of the treatment-control contrast 5.2.2 Cross-Overs 5.2.3 Attrition from the sample 5.2.4 Participation in an Experiment Itself Affects Participants' Behavior 5.3 Gaining Support for Conducting Randomized Experiments: Examples from India 5.3.1 Evaluating an Innovative Input Approach 5.3.2 Evaluating an Innovative Incentive Policy 5.4 What to Read Next 6 Statistical Power and Sample Size 6.1 Statistical Power 6.1.1 Reviewing the Process of Statistical Inference 6.1.2 Defining Statistical Power 6.2 Factors Affecting Statistical Power 6.2.1 The Strengths and Limitations of Parametric Tests 6.2.2 The Benefits of Covariates 6.2.3 The Reliability of the Outcome Measure Matters 6.2.4 The Choice between One-Tailed and Two-Tailed Tests 6.3 What to Read Next 7 Experimental Research When Participants Are Clustered within Intact Groups 7.1 Using the Random-Intercepts Multilevel Model to Estimate Effect Size When Intact Groups of Participants Were Randomized To Experimental Conditions 7.2 Statistical Power When Intact Groups of Participants Were Randomized To Experimental Conditions 7.2.1. Statistical Power of the Cluster-Randomized Design and Intraclass Correlation 7.3 Using Fixed-Effects Multilevel Models to Estimate Effect Size When Intact Groups of Participants are Randomized To Experimental Conditions 7.3.1 Specifying a "Fixed-Effects" Multilevel Model 7.3.2. Choosing Between Random- and Fixed-Effects Specifications 7.2 What to Read Next 8 Using Natural Experiments To Provide "Arguably Exogenous" Treatment Variability 8.1 Natural- and Investigator-Designed Experiments: Similarities and Differences 8.2 Two Examples of Natural Experiments 8.2.1 The Vietnam Era Draft Lottery 8.2.2 The Impact of an Offer of Financial Aid for College 8.3 Sources of Natural Experiments 8.4 Choosing the Width of the Analytic Window 8.5 Threats to Validity in Natural Experiments with a Discontinuity Design 8.5.1 Accounting for the Relationship between the Forcing Variable and the Outcome in a Discontinuity Design 8.5.2 Actions by Participants Can Undermine Exogenous Assignment to Experimental Conditions in a Natural Experiment with a Discontinuity Design 8.6 What to Read Next? 9 Estimating Causal Effects Using a Regression-Continuity Approach 9.1 Maimonides' Rule and the Impact of Class Size on Student Achievement 9.1.1 A Simple "First Difference" Analysis 9.1.2 A "Difference-in-Differences" Analysis 9.1.3 A Basic "Regression-Discontinuity" Analysis 9.1.4 Choosing an Appropriate "Window" or "Bandwidth" 9.2 Generalizing the Relationship between Outcome and Forcing Variable 9.2.1 Specification Checks Using Pseudo-Outcomes and Pseudo-Cutoffs 9.2.2 RD Designs and Statistical Power 9.3 Additional Threats to Validity in an RD Design 9.4 What to Read Next 10 Introducing Instrumental Variables Estimation 10.1 Introducing Instrumental Variables Estimation 10.1.1 Investigating the Relationship Between an Outcome and a Potentially- Endogenous Question Predictor Using OLS Regression Analysis 10.1.2 Instrumental Variables Estimation 10.2 Two Critical Assumptions That Underpin Instrumental Variables Estimation 10.3 Alternative Ways of Obtaining the IV Estimate 10.3.1 Obtaining an IV Estimate by the Method of Two-Stage Least-Squares 10.3.2 Obtaining an IVE by Simultaneous Equations Estimation 10.4 Extensions of the Basic IVE Approach 10.4.1 Incorporating Exogenous Covariates into IV Estimation 10.4.2 Incorporating Multiple Instruments into the First-Stage Model 10.4.3 Examining the Impact of Interactions between the Endogenous Question Predictor and Exogenous Covariates in the Second-Stage Model 10.4.4 Choosing Appropriate Functional Forms for the Outcome/Predictor Relationships in the First- and Second-Stage Models 10.5 Finding and Defending Instruments 10.5.1 Proximity of Educational Institutions 10.5.2 Institutional Rules and Personal Characteristics 10.5.3 Deviations from Cohort Trends 10.5.4 The Search Continues 10.6 What To Read Next 11 Using IVE to Recover the Treatment Effect in a Quasi-Experiment 11.1 The Notion of a "Quasi-Experiment" 11.2 Using IVE to Estimate the Causal Impact of a Treatment in a Quasi-Experiment 11.3 Further Insight into the IVE (LATE) Estimate, in the Context of Quasi- Experimental Data 11.4 Using IVE to Resolve "Fuzziness" in a Regression Discontinuity Design 11.5 What To Read Next 12 Dealing with Bias in Treatment Effects Estimated from Non-Experimental Data 12.1 Reducing Observed Bias by the Method of Stratification 12.1.1 Stratifying on a Single Covariate 12.1.2 Stratifying on Covariates 12.2 Reducing Observed Bias by Direct Control for Covariates Using Regression Analysis 12.3 Reducing Observed Bias Using A "Propensity Score" Approach 12.3.1 Estimation of the Treatment Effect by Stratifying on Propensity Scores 12.3.2 Estimation of the Treatment Effect by Matching on Propensity Scores 12.3.3 Estimation of the Treatment Effect by Weighting by the Inverse of the Propensity Scores 12.4 A Return to the Substantive Question 12.5 What to Read Next 13 Substantive Lessons from High-Quality Evaluations of Educational Interventions 13.1 Increasing School Enrollments 13.1.1 Reduce Commuting Time 13.1.2 Reduce Out-of-Pocket Educational Costs 13.1.3 Reduce Opportunity Costs 13.1.4 Does Increasing School Enrollment Necessarily Lead To Improved Long-Term Outcomes? 13.2 Improving School Quality 13.2.1 Provide More or Better Educational Inputs 13.2.1.1 Provide More Books 13.2.1.2 Teach Children in Smaller Classes 13.2.1.3 Recruit Skilled Teachers or Provide Training to Enhance Teachers' Effectiveness 13.2.2 Improve Incentives For Teachers 13.2.3 Improving Incentives for Students 13.2.4 Increase Families' Schooling Choices 13.3 Summing Up 14 Methodological Lessons from the Long Quest 14.1 Be Clear About Your Theory of Action 14.2 Learn about Culture, Rules, and Institutions in the Research Setting 14.3 Understand the Counterfactual 14.4 Worry about Selection Bias 14.5 Measure All Possible Important Outcomes 14.6 Be On the Lookout for Longer-Term Effects 14.7 Develop a Plan for Examining Impacts on Subgroups 14.8 Interpret Your Research Results Correctly 14.9 Pay Attention to Anomalous Results 14.10 Recognize That Good Research Always Raises New Questions 14.11 Final Words