2
Using Tests

Testing is important for measuring learning in program evaluations. Pre- and post-program comparisons using tests are common. An improvement in test scores shows the change in skills, knowledge, or attitude attributed to the program. The principles of test development are similar to those for the design and development of questionnaires and surveys. This chapter presents information on types of formal testing instruments and on informal tools used to assess knowledge acquisition.

Types of Tests

Three types of formal tests are commonly used in program evaluation: norm-referenced tests, criterion-referenced tests, and performance tests. In this section, we will examine each of these types of tests.

Norm-Referenced Tests

Norm-referenced tests compare participants with each other or with other groups rather than against specific instructional objectives. Norm-referenced tests use data to compare the participants with the norm or average. Although in some evaluations, norm-referenced tests can be of only limited use, they may be useful in programs in which large numbers of participants are involved and average scores and relative rankings are important. In some situations, participants who score highest on the exams receive special recognition or awards or are made eligible for other special activities.

Criterion-Referenced Tests

A criterion-referenced test is an objective test in which a predetermined cut-off score indicates acceptable performance. A criterion-referenced test is a measure against carefully written objectives for a specific program. In a criterion-referenced test, the interest lies in whether participants meet the desired minimum standard, not in how participants rank against one another. The primary concern is measuring, reporting, and analyzing participant performance as it relates to the instructional objectives. Table 2.1 shows a printout of the results of a criterion-referenced test.

Criterion-referenced testing is a popular measurement method (Shrock and Coscarelli, 2001). Its use is becoming widespread, with frequent use in e-learning. Criterion-referenced tests have the advantages of being objective, precise, and relatively easy to administer. However, their use requires programs with clearly defined objectives that can be measured by tests.

Performance Tests

Performance testing allows a participant to exhibit a skill or, occasionally, knowledge or attitudes that have been learned in a program. The skill can be manual, verbal, analytical, or a combination of the three. Performance testing is used frequently in job-related training in which the participants need to demonstrate what they have learned. In supervisory and management training, performance testing comes in the form of skill practices or role-playing. Participants are asked to demonstrate the discussion or problem-solving skills that they have acquired.

Table 2.1. Reporting Format for Data from a Criterion-Referenced Test

Source: Phillips, 1997, p. 124. Used with permission.

To design and administer an effective performance test, as well as other types of tests, follow these recommendations:

• Make sure that the test accurately represents the content of the program and allows the participants to demonstrate as many of the skills that were taught in the program as possible.
• Thoroughly plan every phase of the test, including collection of necessary materials and tools, preparation of the participants, use of time, and evaluation of results.
• Prepare thorough and consistent instructions. As in other types of tests, the quality of the instructions can affect the outcome of a performance test. All participants should be provided with the same instructions.
• Develop standards for a performance test so that participants know in advance what must be accomplished in order for their performance to be considered satisfactory.
• Include relevant information that will keep participants on track and maintain objectivity in responses.

By following these general guidelines, you can develop performance tests into effective tools for program evaluation. Although they are more costly than written tests, performance tests are essential in situations in which the test conditions must mirror the work environment.

Simulations

Job simulations are another technique for measuring learning. This method involves creating and implementing a procedure or task that simulates or models the activity that the program teaches. The simulation is designed to represent the actual job situation as closely as possible. Simulations may be used as an integral part of the program's instruction as well as for evaluation. When a simulation is used for evaluation, participants have an opportunity to try performing the simulated activity as well as to have their performance evaluated. Simulations may be used during a program, at the end of a program, or as part of follow-up evaluation. A variety of simulation techniques that are used to evaluate program results will be discussed in this section.

Electromechanical Simulation

Electromechanical simulation uses a combination of electronic and mechanical devices to simulate real-life situations; this technique is often used in conjunction with programs to develop operational and diagnostic skills.

Task Simulation

As its name implies, task simulation requires participants, as part of the evaluation process, to perform a simulated task similar to what they would do on the job.

Business Games

Business games have grown in popularity in recent years. Business games simulate part or all of a business enterprise; participants change the variables of the business and observe the effects of those changes. The game not only reflects the real-world situation but also is a synopsis of the program of which it is a part.

In-Basket Simulation

The in-basket simulation is particularly useful for assessing learning in supervisory and management training programs. Portions of a supervisor's job are simulated through a series of items that would normally appear in the participant's in-basket, such as memos, notes, letters, and reports, which simulate real-life conditions that the supervisor will face. The participant's performance in the in-basket simulation represents an evaluation of the program.

Case Study

Another popular technique is the case study. A case study gives a detailed description of a problem and usually includes a list of several questions. The participant is asked to analyze the case and determine the best course of action. Case studies allow participants to analyze a real-life situation. Although case studies are often ideal for assessing some programs (for evaluating a project or the leadership style of a corporate executive, for example), it is difficult to actually place a score on a person's level of knowledge acquisition. However, case studies do require that participants examine various aspects of program content and transfer that information to the case situation.

Role-Playing

In role-playing (sometimes referred to as skill practice), participants practice a newly learned skill while they are being observed by other individuals. Participants are given specific instructions about their assigned roles, which may include an ultimate course of action. Participants then practice the skill with the other assigned individuals, trying to accomplish the desired objectives and demonstrate the knowledge that they acquired during the program.

Simulations come in many varieties. They offer opportunities for participants to practice what was taught in a program and to have those performances observed in conditions similar to those they will face on the job. Job simulations can provide extremely accurate evaluations if objective, clearly measurable criteria are set for performance in the simulation.

Informal Tests

In some situations, an informal check of learning is necessary to provide some assurance that participants acquired the desired skills, knowledge, or perhaps changes in attitudes as a result of a program. This approach is appropriate when levels of...