Social Experiments in Practice: The What, Why, When, Where, and How of Experimental Design and Analysis
New Directions for Evaluation, Number 152
Published on 7. March 2017
Book
Paperback/Softback
144 pages
978-1-119-34888-7 (ISBN)
Description
This issue considers social experiments in practice and how recent advances improve their value and potential applications. Although controversial, it is clear they are here to stay and are in fact increasing.
With their greater abundance, experimental evaluations have stretched to address more diverse policy questions, no longer simply providing a treatment-control contrast but adding multiarm, multistage, and multidimensional (factorial) designs and analytic extensions to expose more about what works best for whom. Social experiments are also putting programs under the microscope when they are most ready for testing, enhancing the policy value of their findings.
This volume provides new developments in all these areas from scholars instrumental to recent scientific advances. In some instances, established ideas are given new attention, connecting them to new opportunities to learn and inform policy. By all means, this issue aims to encourage stronger and more informative social experiments in the future.
This is the 152nd issue in the New Directions for Evaluation series from Jossey-Bass. It is an official publication of the American Evaluation Association.
With their greater abundance, experimental evaluations have stretched to address more diverse policy questions, no longer simply providing a treatment-control contrast but adding multiarm, multistage, and multidimensional (factorial) designs and analytic extensions to expose more about what works best for whom. Social experiments are also putting programs under the microscope when they are most ready for testing, enhancing the policy value of their findings.
This volume provides new developments in all these areas from scholars instrumental to recent scientific advances. In some instances, established ideas are given new attention, connecting them to new opportunities to learn and inform policy. By all means, this issue aims to encourage stronger and more informative social experiments in the future.
This is the 152nd issue in the New Directions for Evaluation series from Jossey-Bass. It is an official publication of the American Evaluation Association.
More details
Series
Language
English
Place of publication
New York
United States
Publishing group
John Wiley & Sons Inc
Target group
Professional and scholarly
Dimensions
Height: 226 mm
Width: 147 mm
Thickness: 8 mm
Weight
181 gr
ISBN-13
978-1-119-34888-7 (9781119348887)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Person
Laura R. Peck is a principal scientist at Abt Associates Inc., Social and Economic Policy Division and director of the Research and Evaluation Expertise Center.
Content
EDITOR'S NOTES. SOCIAL EXPERIMENTS IN PRACTICE: INTRODUCTION, FRAMING, AND CONTEXT 9
Laura Peck, Issue Editor, and Naomi Goldstein New approaches to designing and conducting experimental evaluations can improve their utility by enhancing the generalizability of their findings as well as allowing them to determine which specific aspects of a program or policy produce impacts.
1. On the "Why" of Social Experiments: Some Lessons on Overcoming Barriers from 45 Years of Social Experiments 19
Howard Rolston
The history of social policy experiments has demonstrated their utility and value and produced methods for overcoming practical barriers to their application.
2. On the "When" of Social Experiments: The Tension Between Program Refinement and Abandonment 33
Diana Epstein, Jacob Alex Klerman
This chapter considers how to proceed when a process analysis shows that a program does not satisfy the intermediate steps of its own logic model or when an impact analysis does not show clear evidence of impact: when to further refine the program model and when to abandon the program model.
3. On the "Where" of Social Experiments: The Nature and Extent of the Generalizability Problem 47
Stephen H. Bell, Elizabeth A. Stuart
This chapter describes research strategies for investigating how much nonrepresentative site selection biases affect findings produced by rigorous evaluations of social programs.
4. On the "Where" of Social Experiments: Selecting More Representative Samples to Inform Policy 61
Robert B. Olsen, Larry L. Orr
This chapter describes how social experiments can evaluate interventions or programs in samples that are more representative of the population of interest from a policy perspective.
5. On the "How" of Social Experiments: Using Implementation Research to Get Inside the Black Box 73
Lawrence M. Mead
Implementation research describes the actual character of evaluated programs, analyzes why local sites vary in performance, and helps identify additional programs that might be promising.
6. On the "How" of Social Experiments: Analytic Strategies for Getting Inside the Black Box 85
Laura R. Peck
Analysis of postrandom assignment (endogenous) events or experiences in experimental evaluation data is becoming increasingly widespread. This chapter highlights some analytic strategies for revealing mediators of program impacts.
7. On the "How" of Social Experiments: Experimental Designs for Getting Inside the Black Box 97
Stephen H. Bell, Laura R. Peck
This chapter discusses several experimental evaluation designs that randomize individuals-including multiarmed, multistage, and factorial designs-to permit estimating impacts for specific policy design features or program elements within a larger examination of an overall program.
8. Program and Policy Evaluations in Practice: Highlights from the Federal Perspective 109
Rebecca Maynard, Naomi Goldstein, Demetra Smith Nightingale
From the vantage point of federal agencies that commission a large share of domestic program evaluations, this chapter explores the conditions under which particular methodological strategies are and are not helpful in advancing the job of policy analysts, policymakers, and program administrators.
INDEX 137
Laura Peck, Issue Editor, and Naomi Goldstein New approaches to designing and conducting experimental evaluations can improve their utility by enhancing the generalizability of their findings as well as allowing them to determine which specific aspects of a program or policy produce impacts.
1. On the "Why" of Social Experiments: Some Lessons on Overcoming Barriers from 45 Years of Social Experiments 19
Howard Rolston
The history of social policy experiments has demonstrated their utility and value and produced methods for overcoming practical barriers to their application.
2. On the "When" of Social Experiments: The Tension Between Program Refinement and Abandonment 33
Diana Epstein, Jacob Alex Klerman
This chapter considers how to proceed when a process analysis shows that a program does not satisfy the intermediate steps of its own logic model or when an impact analysis does not show clear evidence of impact: when to further refine the program model and when to abandon the program model.
3. On the "Where" of Social Experiments: The Nature and Extent of the Generalizability Problem 47
Stephen H. Bell, Elizabeth A. Stuart
This chapter describes research strategies for investigating how much nonrepresentative site selection biases affect findings produced by rigorous evaluations of social programs.
4. On the "Where" of Social Experiments: Selecting More Representative Samples to Inform Policy 61
Robert B. Olsen, Larry L. Orr
This chapter describes how social experiments can evaluate interventions or programs in samples that are more representative of the population of interest from a policy perspective.
5. On the "How" of Social Experiments: Using Implementation Research to Get Inside the Black Box 73
Lawrence M. Mead
Implementation research describes the actual character of evaluated programs, analyzes why local sites vary in performance, and helps identify additional programs that might be promising.
6. On the "How" of Social Experiments: Analytic Strategies for Getting Inside the Black Box 85
Laura R. Peck
Analysis of postrandom assignment (endogenous) events or experiences in experimental evaluation data is becoming increasingly widespread. This chapter highlights some analytic strategies for revealing mediators of program impacts.
7. On the "How" of Social Experiments: Experimental Designs for Getting Inside the Black Box 97
Stephen H. Bell, Laura R. Peck
This chapter discusses several experimental evaluation designs that randomize individuals-including multiarmed, multistage, and factorial designs-to permit estimating impacts for specific policy design features or program elements within a larger examination of an overall program.
8. Program and Policy Evaluations in Practice: Highlights from the Federal Perspective 109
Rebecca Maynard, Naomi Goldstein, Demetra Smith Nightingale
From the vantage point of federal agencies that commission a large share of domestic program evaluations, this chapter explores the conditions under which particular methodological strategies are and are not helpful in advancing the job of policy analysts, policymakers, and program administrators.
INDEX 137