PREFACE: ASKING AND ANSWERING

When word of the people's discontent reached the grand vizier, he had trouble sleeping, for, you see, he was a conscientious ruler who had the welfare of his people at heart. He called upon the captain of guards and directed him to question members of the populace.

Dutifully, the captain of the guards rounded up one hundred people, men and women, and asked them a series of questions to discover the cause of their discontent. Now the captain was an imposing fellow, some six feet four inches tall and muscular. He wore a scimitar in a scabbard, and the medals he had won for his bravery in battle adorned his waistcoat. When he addressed the citizens, his voice had a hard and commanding tone, and he looked at them with the steely glare of a man ready for combat, which of course was how he approached all tasks.

"The grand vizier has commanded that you provide information about the quality of life in our kingdom. First, I want to know if you believe that taxes are too high?"

In unison all one hundred citizens responded, "No," although records of this session suggest that some citizens responded less enthusiastically than others.

"Ah yes, good." replied the captain of the guards. "Now tell me, do you think that the laws of the land are administered fairly?"

In unison all one hundred citizens responded, "Yes," although it was noted that one citizen had recently lost property in a dispute with the vizier's second cousin's nephew.

"Splendid," said the captain of the guards, and he exhaled a sigh of relief, for he did not relish bringing bad news to the grand vizier. "And do you citizens agree with the new law prohibiting mules in the marketplace?"

In unison all one hundred citizens responded, "Yes," although several were merchants whose businesses were hurt by the new law.

And so the questioning went, and the captain of the guard was pleased that each time the citizens replied in unison and each time there was complete agreement to his question. Armed with this information (for the captain of the guards never went anywhere without being fully armed), he returned to the grand vizier. "Good news, my ruler," he declared. "Although there is some grumbling among the populace, the people I spoke to, who were a diverse lot, all had good things to say about you and your administration. Sleep soundly now, for truly the kingdom is in good hands."

The grand vizier did indeed sleep soundly, at least for a day or two, until word again reached him of the people's discontent. Now this sorely vexed the vizier, for he had the best interests of his people at heart. But he was also confused, because what the people had told the captain of the guards was clearly at odds with what he continued to hear. For that reason he sent for Halcolm, a teacher and counselor, and asked him to go forth among the people to determine the source of their displeasure. Halcolm agreed to this task with the understanding that he would speak to the people in confidence, so that they could speak freely without fear of reprisal.

Now Halcolm was a man of simple and plain means, very undiscerning in appearance. His voice was calm and reassuring, and people tended to feel comfortable in his presence. While buying some fruit in the marketplace, he asked the vendor, "I've been thinking of opening a stand in the marketplace to sell candlesticks and other brass goods. Do you think I could make a profit at that?" The vendor smiled and replied, "Yes and no. Surely you should have a good market for your wares. But as to making a profit, it will be hard given the current tax rate. When the vizier decided to build an addition to the palace, he upped our taxes by 20 percent! I'll be happy when that project is over, and the taxes are reduced."

Further down the marketplace, Halcolm spoke with a rug vendor and asked the same question. "Yes," the rug vendor replied, "you will certainly have a market for brass goods. However, the laws of the land are often applied unevenly, and they create a lot of headaches for us. For example, you might be fined if your cart is too close to the street. Also, since the vizier passed the rule outlawing mules in the marketplace, it is becoming increasingly difficult to carry our wares into town to sell. If changes aren't soon made, many of us will go out of business. In the end, we, the vizier, and ultimately the entire kingdom will suffer under these rules. Now don't get me wrong. I have great respect for the grand vizier. He has been a good and just ruler in the past. It is just that recently I think he has lost touch with the people."

Halcolm continued his project and met with a number of people in the marketplace, at religious shrines, and at the theater. After several days of asking his questions he returned to the grand vizier and shared his findings. "Perhaps I have been out of touch," sighed the vizier. "I will immediately revise my plans for constructing and financing the palace addition. Also, I will establish a committee to examine our laws and rewrite them so that they are fair and just. Thank you, Halcolm, for finding out what really troubled our citizens. I am pleased that the people were so honest in their responses. If I had depended on what people told the captain of the guards, my kingdom might have gone down in ruin," and as he spoke those words, he glared at the captain of the guards, who was also present at this meeting.

"Thank you for your kind words," responded Halcolm, "but I would like to add that as a teacher and counselor, it is my job to know how to ask the right questions and how to ask questions right. You would not, however, want me to lead your army into battle. For that, you would want your captain of the guards, who is an excellent tactician and great warrior." With that, the grand vizier smiled, the captain of the guards smiled, and Halcolm smiled, for in addition to knowing how to ask a good question, Halcolm knew how to phrase a good answer.

(With special thanks to Michael Quinn Patton, for providing the evaluation profession a role model in the guise of Halcolm.)

Increasingly, individuals and organizations are being asked to collect, manage, and use information for decision making, particularly to improve the quality of services and products. Rather than being based on intuition or hunches, decision making is viewed as being a data-driven process, one that is systematic and produces trustworthy information.

The purpose of this book is to provide the reader with a systematic, nontechnical and commonsense approach to developing instruments for data collection and analysis. We have written the book as a guide for both those who are using or developing instruments for the first time and those with experience who want to hone their skills, people ranging from students to agency personnel to program managers to researchers. The book does not require any technical expertise and is written for all levels of readers.

Throughout the text we use the term instrument generically to describe any format for collecting data, such as attitudinal questionnaires, checklists, and political polls. And as we note in the text, the process for developing an instrument is the same whether that instrument will be used for self-rating (by a respondent) or whether another person (an observer) will fill it out.

There are literally thousands of instruments that have been developed and marketed to fulfill distinct needs for information. However, we have found that informational needs are often unique to a particular person, organization, situation, time, or event. Consequently, these needs can be met only by designing and developing a questionnaire for that specific purpose and situation. Additionally, finding a previously developed instrument may be time consuming, and the cost of purchasing the instrument may prove to be greater than the cost of developing a new instrument that uniquely fits your needs.

The approach presented here is based on the underlying assumption that the process of constructing an instrument is both a creative and a technical venture. It involves not only being very familiar with the content or substance of the topic of interest but also developing good questions or items and presenting them in a format accessible to the people who will have to complete the instrument. Consequently, this book is designed to help you create an instrument that will obtain the information you seek.

Throughout we emphasize the need to ensure that an instrument will produce trustworthy and accurate data. To that end, we provide guidelines for reviewing and revising to enhance data validity and reliability. Additionally, we stress the importance of involving, throughout the process of instrument construction, the different groups of people who will be affected by the data generated. These groups, referred to as stakeholders, include the instrument designer(s), decision makers (such as administrators, policymakers, and funding agencies), agency personnel, clients, and raters or respondents.

The ideas in this book are based on the authors' experience in several hundred evaluation projects over the past twenty-five years. In addition, they have been presented in a variety of settings, from the university classroom to workshops on questionnaire construction. Moreover, in writing this text, we have addressed you, the reader,...