Preface

Evaluation is a task that every program and project manager will face at one time or another: you may manage the funding process for an organization, or direct a self-study that requires evaluation, or write a grant proposal that includes an evaluation segment. Occasions vary, but they are inevitable.

Still, endorsing evaluation is a lot like endorsing regular visits to the dentist. People are quick to endorse both activities, but when it comes to doing either one, most people are very uncomfortable. The ABCs of Evaluation will reduce your discomfort by doing two things: First, the book will demystify the process of designing and conducting an evaluation by helping you understand the components of an evaluation design. Second, and more important, the book's aim is to convince you that you play an integral part in any evaluation process, and therefore you are a beneficiary of the results.

Who is the you that this book is designed to assist in the process of evaluation? You might be a manager, supervisor, team facilitator, analyst, or evaluator in the private sector. Then again, you might be in the public or private sector as an administrator, coordinator, facilitator, consultant, or evaluator. Or you might be a faculty member or student in any of a host of disciplines (education, management, human services, psychology, public affairs, labor relations, or health). For any of these situations, the book will provide you with a basic understanding of the steps to follow to design, conduct, and supervise a program evaluation.

This volume has been designed to serve as an introduction to evaluation for program and project managers who have little experience in this task. It presents time-tested evaluation principles and techniques, but it does not purport to reflect the totality of the professional field of evaluation, which encompasses a far greater range of techniques and technologies than are suited to the needs and interests of this book's readership.

Demonstrating Success

As a professional, you know that your programs need to show evidence of success. For example, regardless of how important a communication skills training seminar might be to the overall team-building effort of a company, if you cannot show how it affected the employees' teamwork skills or how improved teamwork added to company performance, the program might not be funded again. This realization may have originated with you, or the organization may require evaluations and demand accountability. If you have always assumed that evaluations are just too difficult for you to do, that you don't want to tackle difficult data collection or unfathomable data analysis, think again. The ABCs of Evaluation will dispel these misconceptions and show you what to do.

Both the person who is designing and performing an evaluation and the person who is participating in and receiving the findings of an evaluation need to prepare themselves to understand the basic processes involved. The evaluator, by understanding the thoughts, concerns, and questions of the evaluated, can better anticipate and prepare to address them up front. The evaluated, by anticipating the processes, the probing, and the data requests of an evaluator, can position themselves to make better use of the evaluation process and results.

Too often people feel that evaluation is something that is done to them rather than for them. Indeed, evaluation may be seen as something to get through and get around (Gray, 1998).

The ABCs of Evaluation is an easy-to-read explanation of the concepts and methods of the evaluation process. The chapters are designed to teach readers what the process offers, what it consists of, and how it is designed and implemented. In addition, the book will guide managers step-by-step throughout the process of conducting an evaluation, from the early question, "Why evaluate?" to the later one, "What did the evaluation tell me?" Consequently, the book will be useful to those who design and conduct the evaluation, as well as to those who supervise others in these tasks.

You may be experiencing downsizing, reorganization, or the introduction of high-performance work teams where you work. These new circumstances may require you to possess evaluation skills. Although you now find yourself in a position that requires the skills, you may not have them, or you may not have been trained to use them.

Overview

At the beginning of each chapter, a scenario sets the stage. Then the chapter presents new evaluation concepts followed by concluding exercises designed to challenge your understanding.

Chapter One defines evaluation, keeping in mind that it is considered both an art and a science. The chapter proposes that readers learn principles and theories as a first step and then apply them in actual situations.

In Chapter One, you will learn that evaluation, both quantitative and qualitative, means measuring and collecting data against some standard. The evaluation may be of people, any activity, an entire program, or all of the above.

After an introduction to the formal reasons to evaluate (more on this in Chapter Two), the chapter explores the ideas of setting standards, using project cycles, and determining levels in evaluation.

Finally, there is an introduction to the evaluation design format that the reader will learn to use throughout the book.

Chapter Two encourages you to think about the following: where there is a program, there must be evaluation. The "Why?" of evaluation emanates from different sources or audiences-from a need you have, or from a requirement that administration or management has for planning, policymaking, funding, or ongoing research. Whatever its purposes and objectives, evaluation is an integral part of most programs and should be planned from the outset. The evaluation design format introduced in Chapter One comes into use here to address your need to define your audience and to eventually report to that audience.

Chapter Three explores the importance of management buy-in to decision making and explores how staff, subject-matter experts, and other stakeholders are involved in the preplanning, planning, and application of the evaluation. The chapter also addresses this question: How can I evaluate a program I did not help create?

Evaluation decisions flow through the various stages in the evaluation. Consequently, in this chapter you will learn whom to involve in the evaluation, and how and why they should be involved. You will learn how to monitor evaluation data and engage in process (formative) evaluation as well as product (summative) evaluation.

Chapter Four addresses a number of questions regarding the evaluator's program description: Why do you need it? How does it relate to your objectives and activities? The chapter will include discussion of the importance of gleaning information from program staff, for example, goals and objectives, the activities planned to achieve the goals and objectives, and the measures that will be used to evaluate the results. The discussion may also involve more than one level of goals and objectives, such as those of the organization, the staff members, and the clients.

The purpose of Chapter Five is to describe different models of evaluation and to illustrate the circumstances under which each is used. Certain design components are shared by all models: evaluation questions, activities to observe, data sources, whom you collect data from or about, how and when you collect the data, how you analyze the data, and who is responsible for each piece of the evaluation.

In Chapter Six, you will learn to identify the data sources that will help you, as an evaluator, to determine whether the program has met its objectives. Two important questions are answered: How do data sources relate to the evaluator's program description? Are paper-and-pencil instruments the only form of measurement? This chapter also introduces logic models as an additional means of understanding project concept and project flow. If a logic model exists, it is advisable that an evaluator discusses it with the project staff so that there is total understanding of the intent, process, and intended outcomes. If one does not exist, it might be helpful for the evaluator to construct one with the project staff so that there is understanding and agreement among all stakeholders of the project staffs' intent, process, and intended outcomes.

Chapter Six discusses many forms of data collection, among them using existing data, such as previously collected public records, and collecting new data using surveys, checklists, interview schedules, objective tests and scales, projective measures, and observational analysis.

Chapter Six also differentiates between data collection formats that are obtrusive (when people know you are collecting information) and those that are unobtrusive (when people are unaware that you are collecting information).

In Chapter Seven, data analysis will become an understandable concept for you. Specifically, Chapter Seven introduces data analysis by levels of measurement: nominal, ordinal, interval, and ratio. Data analysis terms such as measures of central tendency and analysis of variance, which are ways of analyzing differences (growth and change), are discussed by level.

The purpose of Chapter Eight is to help you distinguish between evaluation and research. The chapter discusses both quantitative and qualitative...