Program Evaluation in Practice
Description
More details
Other editions
Additional editions
Person
Content
List of Tables, Figures, Exhibits, and Boxes xiii
Preface xvii
Acknowledgments xxi
The Author xxiii
Part One: Introduction 1
Chapter One: Foundations of Program Evaluation 3
What is Program Evaluation? 5
Internal and External Evaluators 11
How to Use This Book 13
The Evaluation Objective 15
Designing and Developing an Evaluation Matrix 19
Data Collection 20
Triangulation of Data 31
Writing the Evaluation Report 31
Dissemination and Use of Evaluation Findings 35
Summary 37
Key Concepts 38
Discussion Questions 38
Class Activities 39
Suggested Reading 39
Chapter Two: Ethics in Program Evaluation and an Overview of Evaluation Approaches 41
Ethics in Program Evaluation 41
What is an Evaluation Approach? 43
Objectives-Based Approach 44
Decision-Based Approach 47
Participatory Approach 50
Consumer-Oriented Approach 55
Expertise-Oriented Approach 55
Eclectic Approach 56
Summary 56
Key Concepts 57
Discussion Questions 57
Class Activities 58
Suggested Reading 58
Chapter Three: In-depth Look at the Objectives-based Approach to Evaluation 59
Objectives-Based Approach 59
How to Use Evaluation Objectives 64
Summary 66
Key Concepts 67
Discussion Questions 67
Class Activities 67
Suggested Reading 67
Part Two: Case Studies 69
Chapter Four: Improving Student Performance in Mathematics Through Inquiry-based Instruction 71
The Evaluator 71
The Program 76
The Evaluation Plan 78
Summary of Evaluation Activities and Findings 81
Final Thoughts 86
Key Concepts 86
Discussion Questions 87
Class Activities 87
Suggested Reading 88
Chapter Five: Evaluation of a Community-based Mentor Program 89
The Evaluator 89
The Program 91
The Evaluation Plan 92
Summary of Evaluation Activities and Findings 97
Final Thoughts 97
Key Concepts 97
Discussion Questions 98
Class Activities 98
Suggested Reading 99
Chapter Six: Teacher Candidates Integrating Technology into their Student Teaching Experience 101
The Evaluators 101
The Program 102
The Evaluation Plan 102
Summary of Evaluation Activities and Findings 108
Final Thoughts 108
Key Concepts 108
Discussion Questions 108
Class Activities 109
Suggested Reading 110
Chapter Seven: Evaluation of a Professional Development Technology Project in a Low-performing School District 111
The Evaluator 111
The Program 112
The Evaluation Plan 113
Summary of Evaluation Activities and Findings 116
Final Thoughts 117
Key Concepts 117
Discussion Questions 117
Class Activities 118
Suggested Reading 119
Chapter Eight: Expansion of a High School Science Program 121
The Evaluators 121
The Program 122
The Evaluation Plan 122
Summary of Evaluation Activities and Findings 124
Final Thoughts 126
Key Concepts 126
Discussion Questions 126
Class Activities 127
Suggested Reading 127
Chapter Nine: Evaluation of a Proven Practice for Reading Achievement 129
The Evaluators 129
The Program 130
The Evaluation Plan 131
Summary of Evaluation Activities and Findings 131
Final Thoughts 133
Key Concepts 133
Discussion Questions 133
Class Activities 134
Suggested Reading 135
Chapter Ten: Project Plan for Evaluation of a Statewide After-school Initiative 137
The Evaluator 137
The Program 138
The Evaluation Plan 140
Summary of Evaluation Activities and Findings 143
Final Thoughts 144
Key Concepts 144
Discussion Questions 145
Class Activities 145
Suggested Reading 145
Chapter Eleven: Evaluation of a Training Program in Mathematics for Teachers 147
The Evaluators 147
The Program 148
The Evaluation Plan 149
Summary of Evaluation Activities and Findings 151
Final Thoughts 154
Key Concepts 155
Discussion Questions 155
Class Activities 156
Suggested Reading 156
Chapter Twelve: An Evaluator-in-Training's Work on a School Advocacy Program 159
The Evaluator 159
The Program 160
The Evaluation Plan 162
Summary of Evaluation Activities and Findings 162
Final Thoughts 168
Key Concepts 168
Discussion Questions 168
Class Activities 169
Suggested Reading 170
Chapter Thirteen: Evaluation of a School Improvement Grant to Increase Parent Involvement 171
The Evaluators 171
The Program 171
The Evaluation Plan 172
Summary of Evaluation Activities and Findings 176
Final Thoughts 177
Key Concepts 178
Discussion Questions 178
Class Activities 178
Suggested Reading 179
Chapter Fourteen: Evaluating the Impact of a New Teacher Training Program 181
The Evaluators 181
The Program 182
The Evaluation Plan 182
Summary of Evaluation Activities and Findings 185
Final Thoughts 185
Key Concepts 186
Discussion Questions 186
Class Activities 186
Suggested Reading 186
References 187
Index 189
Chapter 2
Ethics in Program Evaluation and an Overview of Evaluation Approaches
Learning Objectives
After reading this chapter you should be able to
- Understand ethical dilemmas faced by evaluators
- Understand the Joint Committee on Standards for Educational Evaluation's standards and how evaluators may use them in the profession
- Understand the key similarities and differences among the various evaluation approaches
- Understand the key benefits and challenges of the different evaluation approaches
Ethics in Program Evaluation
When conducting an evaluation, a program evaluator may face not only methodological challenges (for example, what data collection instrument to use) but ethical challenges as well. Ethics in program evaluation refers to ensuring that the actions of the program evaluator are in no way causing harm or potential harm to program participants, vested stakeholders, or the greater community.
In some cases, evaluators may find themselves in an ethical dilemma because of the report they have created. For example, an evaluator might be tempted to suppress negative findings from a program evaluation for fear of angering the client and losing the evaluation contract. In other cases, evaluators may find themselves in a dilemma not because of their report, per se, but because of how others use it. For example, how should an evaluator move forward if he or she knows that a report supports one stakeholder group over another and will no doubt spark a situation? For example, a school superintendent who finds an after-school program too expensive might use the evaluation report to support canceling the program even though parents and students find the program beneficial. Evaluators faced with a multitude of ethical challenges each day turn to the Joint Committee on Standards for Educational Evaluation for guidance (Newman & Brown, 1996).
Established in 1975, the Joint Committee was created to develop a set of standards to ensure the highest quality of program evaluation in educational settings. The Joint Committee is made up of several contributing organizations, one of which is the American Evaluation Association (AEA). Although the AEA, which sends delegates to Joint Committee meetings, has not officially adopted the standards, the organization does recognize the standards and support the work of the committee. The standards are broken down into five main areas: utility, feasibility, propriety, accuracy, and evaluation accountability.
Utility standards. The purpose of these standards is to increase the likelihood that stakeholders will find both the process and the product associated with the evaluation to be valuable. These standards include, for example, making sure the evaluation focuses on the needs of all stakeholders involved in the program, making sure the evaluation addresses the different values and perspectives of all stakeholders, and making sure that the evaluation is not misused.
Feasibility standards. The purpose of these standards is to ensure that the evaluation is conducted using appropriate project management techniques and uses resources appropriately.
Propriety standards. These standards are designed to support what is fair, legal, and right in program evaluation. These standards include, for example, ensuring that the human rights and safety of program participants are upheld and maintained indefinitely throughout the evaluation process; that reports provide a comprehensive evaluation that includes a summary of goals, data collection methods, findings, and recommendations; and that evaluations are conducted for the good of the stakeholders and the community.
Accuracy standards. The purpose of these standards is to ensure that evaluations are dependable and truthful in their data collection and findings. These standards include making sure that the evaluation report is both reliable and valid, and that data collection tools and methodologies were sound and rigorous in nature.
Evaluation accountability standards. These standards call for both the rigorous documentation of evaluations and the use of internal and external meta-evaluations to improve the ongoing processes and products associated with evaluation.
A complete list of the standards can be found at www.eval.org/evaluationdocuments/progeval.xhtml.
What is an Evaluation Approach?
As noted in Chapter One, program evaluation is the process of systematically collecting data to determine if a set of objectives has been met. This process is done to determine a program's worth or merit (see Figure 2.1).
Figure 2.1 Determining a Program's Worth or Merit
evaluation approach The model that an evaluator uses to undertake an evaluation
The evaluation approach is the process by which the evaluator goes about collecting data. Two evaluators working to evaluate the same program not only may use different methods for collecting data but also may have very different perspectives on the overall purpose or role of the evaluation. Although many beginning evaluators may believe that simply changing the type of data being collected (for example, from quantitative to qualitative) is changing the approach, in reality an evaluation approach is based on more than simply data collection techniques. Changing an approach to program evaluation entails a shift not only in philosophy but also in the "reason for being" or purpose of the evaluation.
In this chapter you will read about some of the main approaches used in program evaluation today (see Figure 2.2 for an overview). When considering these approaches, think about the criteria used to evaluate a program and who will ultimately judge the program using the criteria.
Figure 2.2 Overview of Evaluation Approaches
Objectives-Based Approach
Just as there are many applied research approaches, there are several different approaches to program evaluation. The most common approach program evaluators can use is the objectives-based approach, which involves objectives written by both the creators of the program and the evaluator. An evaluation objective is a written statement that depicts an overarching purpose of the evaluation and clearly states the types of information that will be collected. Often these objectives are further supported through the use of benchmarks. A benchmark is more detailed than an objective in that it specifically states what quantitative goals the participants in the program need to reach for the program to be successful. Box 2.1 presents an evaluation objective followed by a benchmark.
objectives-based approach An evaluation model whereby the evaluator focuses on a series of preestablished objectives, and then collects only the necessary data to measure whether the program met those objectives
benchmarks Specific outcomes that define the success or worth of a program
Box 2.1. Example of an Evaluation Objective and Benchmark
Evaluation objective: To document middle school students' changes in academic achievement, particularly in the area of reading and literacy skills.
Benchmark: Students in fifth through eighth grade will show a 10 percent gain on the English language arts (ELA) state assessment in year one, and there will be a 20 percent increase in students passing the ELA in program years two and three.
Evaluators will often start with the objectives for the evaluation and build evaluation data collection activities from those objectives. Evaluation objectives may guide either formative or summative data collection. Either way, quantitative or qualitative data, or both, is collected, and findings are compared to the project's objectives. Objectives are certainly helpful in shaping the evaluation, but there is a risk that evaluators may become so focused on the objectives that they lose sight of other unanticipated outcomes or benefits to participants as a result of the program.
Although objectives assist in guiding an evaluation, there is another method-the goal-free approach -that doesn't prescribe using evaluation objectives. This approach is guided by the perspective that there are many findings and outcomes that do not fall within the strict confines of the goals and objectives established by both the project developers and the evaluator. Those who practice goal-free evaluation believe that the unforeseen outcomes may be more important than outcomes that the program developers emphasize. One difficulty in conducting a goal-free evaluation is that projects that receive funding are required to show specific outcomes based on objectives. If the outcomes are not included in the evaluation, the appropriate data to present to funding bodies may not end up being collected.
goal-free approach An evaluation model designed to control for bias, whereby the evaluator purposely does not learn the goals and objectives of the program being evaluated but tries to determine these through careful data collection
Early Objectives-Based Approach
Evidence of program evaluation in the United States dates back to the early 1800s, but the Tylerian approach, named after its creator Ralph Tyler, was the first to focus on the use of behavioral objectives as a method of determining or judging the worth of a program. Beginning in 1932, Tyler, now considered the father of education...
