Designing Small Evaluation Studies
1st Edition
Published on 10. July 2025
Book
Paperback/Softback
352 pages
978-1-0719-5415-7 (ISBN)
Description
This text describes how to design and analyze small efficacy or evaluation studies, typically carried out as part of the development of programs or interventions in areas such as education. The authors provide guidance on designs for small, randomized trials, and also non-randomized causal designs. The final section compares research designs, discusses approaches to choosing a design, and provides guidance on reporting.
Reviews / Votes
The book will be an important addition to instruction in designs for causal inference in the field of education. It is long overdue.-- Thomas J. Lipscomb
More details
Language
English
Place of publication
Thousand Oaks
United States
Publishing group
Sage Publications Inc Ebooks
Target group
College/higher education
Dimensions
Height: 251 mm
Width: 177 mm
Thickness: 20 mm
Weight
666 gr
ISBN-13
978-1-0719-5415-7 (9781071954157)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Persons
Larry V. Hedges is one of eight Board of Trustees Professors at Northwestern, the university's most distinguished academic position. He holds appointments in statistics and data science, psychology, education and social policy, and medical social sciences. Previously, he was the Stella M. Rowley Distinguished Service Professor at the University of Chicago. He received the Yidan Prize in Education Research in 2018 and the Jose Vasconcelos World Award of Education from the World Culture Council in 2023 He is an elected member of the National Academy of Education and is a fellow of the American Academy of Arts and Sciences, the American Statistical Association, the American Psychological Association, the Association for Psychological Science, and the American Educational Research Association. He is also a member of the Society for Research on Educational Effectiveness, which established an annual endowed lecture in his honor. He was selected to give the inaugural Hedges Lecture in 2016. Hedges was nominated by President Barack Obama to the Board of Directors of the National Board for Education Sciences, was confirmed by the U.S. Senate in June 2012, and was elected Chair of the Board in 2016. Widely published, he has authored or co-authored 250 journal articles and 13 books including Coe, Waring, Hedges, & Day Ashley, Research Methods and Methodologies in Education, 3rd edition (2021, Sage London). He has also co-directed research training institutes on research design funded by the US Institute of Education Sciences, the National Institutes of Health, and the National Science Foundation for the last 20 years.
Elizabeth (Beth) Tipton is Professor of Statistics and Data Science at Northwestern University, where she is also holds an appointment in the School of Education and Social Policy and as a Faculty Fellow at the Institute for Policy Research. She is an elected fellow of both the American Statistical Association and the American Educational Research Association. Her research has been recognized with early career awards from the American Educational Research Association, the American Psychological Association, and the Society for Research Synthesis Methods. In 2021, she received the Frederick Mosteller award for Distinguished Contributions to Research Synthesis from the Campbell Collaboration. In 2025, she will serve as both the President of the Board of the Society for Research on Educational Effectiveness and the Chair of the Social Statistics Section of the American Statistical Association. She has previously served as an Associate Editor of the Journal of Educational and Behavioral Statistics and is a member of the editorial board at several other journals, including Psychological Bulletin. Her work has been supported by the National Science Foundation, the Institute for Education Sciences, the Spencer Foundation, and the Raikes Foundation. Her research focuses on the development of methods for experimental design, meta-analysis, and the translation of evidence. She has also co-directed a research training institute funded by the National Science Foundation on research design for STEM education researchers.
Elizabeth (Beth) Tipton is Professor of Statistics and Data Science at Northwestern University, where she is also holds an appointment in the School of Education and Social Policy and as a Faculty Fellow at the Institute for Policy Research. She is an elected fellow of both the American Statistical Association and the American Educational Research Association. Her research has been recognized with early career awards from the American Educational Research Association, the American Psychological Association, and the Society for Research Synthesis Methods. In 2021, she received the Frederick Mosteller award for Distinguished Contributions to Research Synthesis from the Campbell Collaboration. In 2025, she will serve as both the President of the Board of the Society for Research on Educational Effectiveness and the Chair of the Social Statistics Section of the American Statistical Association. She has previously served as an Associate Editor of the Journal of Educational and Behavioral Statistics and is a member of the editorial board at several other journals, including Psychological Bulletin. Her work has been supported by the National Science Foundation, the Institute for Education Sciences, the Spencer Foundation, and the Raikes Foundation. Her research focuses on the development of methods for experimental design, meta-analysis, and the translation of evidence. She has also co-directed a research training institute funded by the National Science Foundation on research design for STEM education researchers.
Content
Preface
About the Author
Chapter 1: Introduction
A. What Is an Intervention?
B. Examples
Section I: Background Concepts
Chapter 2: Introduction to Section I: Is a Small Efficacy Study Right for You?
A. What Is the Logic of An Efficacy Study?
B. How Can This Test Be Operationalized?
C. What About Small Efficacy Studies?
D. What Questions Can't a Small Efficacy Study Answer Well?
E. What Are Other Options?
F. Moving Forward
Chapter 3: Research Design
A. Logic of Inquiry
B. Principles of Research Design
C. Types of Quantitative Research Designs
Questions to Test Your Knowledge
Chapter 4: Validity of Research Designs
A. Causal Inference
B. Internal validity
C. External Validity
D. Statistical Conclusion Validity
E. Construct Validity of Explanation
F. Conclusion
Questions to Test Your Knowledge
Chapter 5: A Brief Review of Statistics
A. Populations and Samples
B. Random Sampling
C. Models and Notations
D. Estimators and Sampling Distributions
E. Statistical Inference
F. Design Sensitivity
G. Design Complexities and Statistical Models
H. Multiple Regression Analysis
I. Multilevel Statistical Models
J. Research Designs
Questions to Test Your Knowledge
Chapter 6: A Brief Review of Measurement
A. What is Measurement?
B. Measurement Theory
C. Outcome Concept Domains
D. Types of Measures
E. Scoring of Measures
F. Choosing an Outcome Measure
Questions to Test Your Knowledge
Chapter 7: Choosing an Appropriate Effect Size
A. What is an Effect Size?
B. Choosing an Effect Size Value
C. Other Measurement and Statistical Considerations
Questions to Test Your Knowledge
Section II: Randomized Designs
Chapter 8: Introduction to Section II: What is Randomization?
A. Randomization
B. Theoretical Objections to Random Assignment
C. Small Efficacy Studies
D. Overview of The Next Four Chapters
Chapter 9: Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. The Statistical Analysis of the Design
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 10: Multisite Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multisite Individually Randomized Designs With Fixed Site Effects
D. Increasing Design Sensitivity
E. Multisite Individually Randomized Designs with Random Site Effects
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 11: Multisite Cluster Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multi-Site Cluster Randomized Design
D. Increasing Design Sensitivity
E. Examples
F. Conclusions
Appendix
Chapter 12: Cluster Randomized Designs
A. General Approach
B. Statistical Model and Analysis
C. Increasing Design Sensitivity
D. Examples
E. Conclusions
Questions to Test Your Learning
Appendix
Section III: Quasi-Experimental Designs
Chapter 13: Intro to Section III: What Is A 'Quasi' Experiment?
A. Threats to Internal Validity in QEDs
B. Why Pre-Post Designs are Not Adequate
C. Statistical Analyses of QEDs
D. QEDs for Small Efficacy Studies
Chapter 14: Nonequivalent Control Group Designs
A. General Approach
B. Statistical Models and Analysis
C. Selection Bias
D. Confounders and Covariate Selection
E. Matching Models
F. Statistical Adjustment Methods
G. Designing a NECD Study
H. Examples
I. Conclusions
Questions to Test Your Learning
Chapter 15: The Difference in Differences Design
A. General Approach
B. Comparisons of Individuals (No Nesting)
C. Multiple Subgroups Within Each Treatment Group
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 16: Interrupted Time Series Designs
A. General Approach
B. Model and Notation
C. Estimation and Analysis of the Design
D. Design Sensitivity
E. Variants of This Design
F. Conclusions
Questions to Test Your Learning
Appendix
Section IV: Tools and Reporting
Chapter 17: Introduction to Section IV: Tools and Considerations for Practice
A. Principles of Small Efficacy Studies
B. Randomized Designs
C. Quasi-Experimental Designs
D. Tools for Use in the Field
Chapter 18: Choosing a Research Design
A. Know Your Setting
B. Is a Comparison Group Possible?
C. Is Randomization Possible?
D. Can an Adequately Matched Comparison Group Be Formed?
E. Which Designs are Possible?
F. What If None of These Designs Can Be Implemented?
Questions to Test Your Learning
Chapter 19: Worksheets for Comparing Designs
A. Overview of Worksheet 1
B. Overview of Worksheet 2
Chapter 20: Best Practices for Reporting Small Efficacy Study Results
A. Reporting Standards
B. Special Considerations for Small Efficacy Studies
List of Tables and Figures
Glossary
References
Index
About the Author
Chapter 1: Introduction
A. What Is an Intervention?
B. Examples
Section I: Background Concepts
Chapter 2: Introduction to Section I: Is a Small Efficacy Study Right for You?
A. What Is the Logic of An Efficacy Study?
B. How Can This Test Be Operationalized?
C. What About Small Efficacy Studies?
D. What Questions Can't a Small Efficacy Study Answer Well?
E. What Are Other Options?
F. Moving Forward
Chapter 3: Research Design
A. Logic of Inquiry
B. Principles of Research Design
C. Types of Quantitative Research Designs
Questions to Test Your Knowledge
Chapter 4: Validity of Research Designs
A. Causal Inference
B. Internal validity
C. External Validity
D. Statistical Conclusion Validity
E. Construct Validity of Explanation
F. Conclusion
Questions to Test Your Knowledge
Chapter 5: A Brief Review of Statistics
A. Populations and Samples
B. Random Sampling
C. Models and Notations
D. Estimators and Sampling Distributions
E. Statistical Inference
F. Design Sensitivity
G. Design Complexities and Statistical Models
H. Multiple Regression Analysis
I. Multilevel Statistical Models
J. Research Designs
Questions to Test Your Knowledge
Chapter 6: A Brief Review of Measurement
A. What is Measurement?
B. Measurement Theory
C. Outcome Concept Domains
D. Types of Measures
E. Scoring of Measures
F. Choosing an Outcome Measure
Questions to Test Your Knowledge
Chapter 7: Choosing an Appropriate Effect Size
A. What is an Effect Size?
B. Choosing an Effect Size Value
C. Other Measurement and Statistical Considerations
Questions to Test Your Knowledge
Section II: Randomized Designs
Chapter 8: Introduction to Section II: What is Randomization?
A. Randomization
B. Theoretical Objections to Random Assignment
C. Small Efficacy Studies
D. Overview of The Next Four Chapters
Chapter 9: Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. The Statistical Analysis of the Design
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 10: Multisite Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multisite Individually Randomized Designs With Fixed Site Effects
D. Increasing Design Sensitivity
E. Multisite Individually Randomized Designs with Random Site Effects
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 11: Multisite Cluster Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multi-Site Cluster Randomized Design
D. Increasing Design Sensitivity
E. Examples
F. Conclusions
Appendix
Chapter 12: Cluster Randomized Designs
A. General Approach
B. Statistical Model and Analysis
C. Increasing Design Sensitivity
D. Examples
E. Conclusions
Questions to Test Your Learning
Appendix
Section III: Quasi-Experimental Designs
Chapter 13: Intro to Section III: What Is A 'Quasi' Experiment?
A. Threats to Internal Validity in QEDs
B. Why Pre-Post Designs are Not Adequate
C. Statistical Analyses of QEDs
D. QEDs for Small Efficacy Studies
Chapter 14: Nonequivalent Control Group Designs
A. General Approach
B. Statistical Models and Analysis
C. Selection Bias
D. Confounders and Covariate Selection
E. Matching Models
F. Statistical Adjustment Methods
G. Designing a NECD Study
H. Examples
I. Conclusions
Questions to Test Your Learning
Chapter 15: The Difference in Differences Design
A. General Approach
B. Comparisons of Individuals (No Nesting)
C. Multiple Subgroups Within Each Treatment Group
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 16: Interrupted Time Series Designs
A. General Approach
B. Model and Notation
C. Estimation and Analysis of the Design
D. Design Sensitivity
E. Variants of This Design
F. Conclusions
Questions to Test Your Learning
Appendix
Section IV: Tools and Reporting
Chapter 17: Introduction to Section IV: Tools and Considerations for Practice
A. Principles of Small Efficacy Studies
B. Randomized Designs
C. Quasi-Experimental Designs
D. Tools for Use in the Field
Chapter 18: Choosing a Research Design
A. Know Your Setting
B. Is a Comparison Group Possible?
C. Is Randomization Possible?
D. Can an Adequately Matched Comparison Group Be Formed?
E. Which Designs are Possible?
F. What If None of These Designs Can Be Implemented?
Questions to Test Your Learning
Chapter 19: Worksheets for Comparing Designs
A. Overview of Worksheet 1
B. Overview of Worksheet 2
Chapter 20: Best Practices for Reporting Small Efficacy Study Results
A. Reporting Standards
B. Special Considerations for Small Efficacy Studies
List of Tables and Figures
Glossary
References
Index