Preface

Our main objective in this handbook has been and continues to be to make the practice of program evaluation as accessible and practical as possible. This fourth edition introduces many changes aimed at considerably improving the contents so as to better meet this objective.

We are pleased to note that we have added a number of chapters on new subjects, further improving this handbook's practicality and usefulness. These new subjects address:

• Culturally Responsive Evaluation (Chapter Twelve),
• Adopting designs and tools to evaluate multi-service community change programs (Chapter Eleven), and
• Using role playing to collect data (Chapter Fifteen).

In addition, we have added new materials about important topics to existing chapters on:

• Using cognitive interviewing to pre-test surveys (Chapter Fourteen),
• Coding qualitative data (Chapter Twenty-Two), and
• Employing data visualization techniques to present quantitative data (Chapter Twenty-Three).

Intended Audience

The intended audience for this handbook includes (1) managers, management analysts, policy analysts, and evaluators in federal, state, and local governments and school districts; (2) managers and analysts in foundations and nonprofit organizations; (3) independent evaluators, auditors, and management consultants; and (4) faculty members and students in professional schools, such as schools of public policy, public administration, business administration, education, public health, and social work.

The information presented here is intended to help those involved in program evaluation, those who fund programs and evaluation studies, those who operate programs, those who are staff members in the legislative and executive branches of government, those in universities, and those in the consulting world-both individuals new to evaluation and experienced evaluators (who may find new ideas to add to their current toolkit). Even though the language is U.S. English and the authors are primarily people who live and work in the United States, the material presented here should be applicable in any country.

Scope

Considerable diversity exists in the training and skills possessed both by those charged with evaluating public and nonprofit programs and by program managers and staff members seeking to collect useful data on their programs.

Evaluators and program managers may have a variety of evaluation objectives in mind. They may have specific questions, or they may be unsure of how to frame useful questions about their programs. Careful analysis of the program to be evaluated and the context in which it operates is a significant precursor to the planning and design of any evaluation endeavor. Identification of the theory underlying the program and the contextual factors that affect its operations and success is critical.

This handbook covers a variety of approaches to analyzing the operations and results of past and current programs. Guidance for designing ongoing program performance monitoring systems is provided, along with advice on designing and implementing studies of program processes and program outcomes. A variety of evaluation approaches are discussed, including both qualitative and quantitative approaches for evaluating program operations and impact. (Note that, except in discussions about ways for evaluators to develop options and recommendations for program improvement, the term program evaluation as used in this handbook does not include assessing the effectiveness of future programs.)

The program evaluation approaches and tools covered here may provide feedback on program expenditures, program operations, or program results. They can be useful in developing new legislative proposals and in reauthorizing existing programs; in developing, debating, and deciding among budget alternatives; in implementing, operating, and improving public programs and programs operated by for-profit or nonprofit organizations; and in managing, auditing, and reporting on the uses of public funds.

Need for Program Evaluation

The demand for program evaluation, and "evidence-based" information on results, has become extraordinary. National and state legislatures, local legislative bodies, public agencies, foundations, and other funding agencies are increasingly demanding information on how program funds have been used and what funded programs have produced. Both program advocates and fiscal conservatives need information on program results. Performance management (or managing for results) initiatives are involving increasing numbers of program managers and staff in developing performance goals, monitoring performance, and then reporting on performance to inform decision making. The public is also becoming more demanding about how their tax dollars and fees are being used.

National governments, such as those in the United States, Australia, Canada, Chile, France, Mexico, Peru, the Scandinavian countries, and the United Kingdom, have developed experience with and expertise in program evaluations, including performance audits, also known as value-for-money audits. In the United States, the Government Performance and Results Act (GPRA) of 1993, along with the GPRA Modernization Act of 2010, require federal agencies to develop and update strategic plans, to establish annual performance targets, and to report annually on program performance. The Chief Financial Officers Act of 1990 requires federal agencies to provide a list of their program evaluations as well as to provide "systematic measurement of performance" and information on the "results of operations." The World Bank and other multilateral and regional banks have been pressing for evaluations of the work for which they provide support.

Handbook Organization

This handbook is divided into four parts. These address evaluation planning and design, data collection, data analysis, and evaluation use. In Part One, the authors of Chapters One through Twelve explore a variety of approaches to planning and designing evaluation studies and performance monitoring systems. Evaluation planning and design should ensure that the benefits of evaluation outweigh its costs. The chapters in Part One provide an overview of program evaluation; suggest strategies for involving intended users and other key stakeholders in evaluation planning and design; and discuss the use of logic models, evaluability assessment and other exploratory evaluation approaches, performance measurement systems, case studies, and various experimental and quasi-experimental designs. The evaluation designs presented provide information on how to analyze program theories-including program activities, outputs, program outcomes, and contextual factors affecting the extent to which program activities have produced desired program results. Chapter authors discuss the purpose of each evaluation design, the types of questions that can be answered, and requirements that must be met to use each design properly. In addition, Part One contains chapters on recruitment and retention of evaluation study participants, multisite evaluations, complex community change programs, and culturally responsive evaluation.

In Part Two, the authors of Chapters Thirteen through Twenty-One describe practical data collection procedures, that is, methods for collecting data on program performance within tight time and resource constraints. They describe both well-established and newer procedures for collecting information on program performance, including the use of agency records, surveys, role playing, ratings by trained observers, the Internet, focus groups, and semi-structured interviewing and the collection of data in the field and of "stories" for evaluation studies. They discuss uses of these data collection procedures in program evaluations and indicate the types of quality control needed to ensure that the resulting data are valid and reliable.

In Part Three, the authors of Chapters Twenty-Two to Twenty-Five provide advice on important methods for data analysis, looking at methods for analyzing qualitative data, use of appropriate statistics and statistical tests, cost-effectiveness and cost-benefit analysis, and meta-analyses and evaluation syntheses. They discuss the requirements that must be met to use these data analysis techniques and present examples illustrating their application.

In Part Four, the authors of Chapters Twenty-Six to Thirty describe methods for getting evaluation results used. They offer advice on avoiding evaluation pitfalls, developing options and recommendations for policy and program change, reporting findings persuasively, contracting for evaluations, and overcoming political and bureaucratic challenges to the use of evaluation findings. The final chapter discusses challenges and issues in evaluation, including quality control of the evaluation process, selection and training of evaluators, and evaluation standards and ethics, and it also examines the relationships between performance measurement systems and evaluation studies. This chapter-and the handbook-closes with a discussion of current and future trends in program evaluation.

Acknowledgments

The editors and authors acknowledge the significant impact Joseph Wholey has had on all of our thinking about evaluation. While Joe has retired, and did not participate as actively in this edition, his influence remains...