Chapter One
The Social Context of Question Asking

The precise wording of questions plays a vital role in determining the answers given by respondents. This fact is not appreciated as fully as it should be, even in ordinary conversation. For example, a colleague mentioned that he needed to pick out granite for a kitchen countertop. The only day he could make the trip was the Saturday before Labor Day. Although he called on Friday to make certain the store was open, he arrived at the store on Saturday only to find a sign on the door that said "Closed Labor Day Weekend." When asked if he remembered what question he had asked the clerk at the store, he said, "I asked him what hours he was open on Saturday, and he replied 'Nine to five.'"

This story illustrates the basic challenge for those who engage in the business of asking questions. It illustrates not only the importance of the golden rule for asking questions-Ask what you want to know, not something else-but also, more important, the ambiguities of language and the powerful force of context in interpreting the meaning of questions and answers. Our colleague had unwittingly asked a perfectly ambiguous question. Did the question refer to Saturdays in general or the next Saturday specifically? The clerk obviously interpreted the question as referring to Saturdays in general. Our colleague meant the next Saturday and did not think his question could mean anything else until he arrived at the store and found it closed.

In everyday life, these types of miscommunications happen all the time. Most of the time they are corrected by further conversation or by direct questions that clarify their meaning. Sometimes they only get corrected when expected behavior does not occur, as was the case when the store turned out to be closed. But the stylized form of question asking used in surveys does not often provide feedback about ambiguities or miscommunications. We must depend on pretesting to weed out ambiguities and to help reformulate questions as clearly as possible-to ask about what we want to know, not something else.

The thesis of this book is that question wording is a crucial element in surveys. The importance of the precise ordering of words in a question can be illustrated by another example.

Two priests, a Dominican and a Jesuit, are discussing whether it is a sin to smoke and pray at the same time. After failing to reach a conclusion, each goes off to consult his respective superior. The next week they meet again. The Dominican says, "Well, what did your superior say?"

The Jesuit responds, "He said it was all right."

"That's funny," the Dominican replies. "My superior said it was a sin."

The Jesuit says, "What did you ask him?"

The Dominican replies, "I asked him if it was all right to smoke while praying."

"Oh," says the Jesuit. "I asked my superior if it was all right to pray while smoking."

Small Wording Changes that Made Big Differences

The fact that seemingly small changes in wording can cause large differences in responses has been well known to survey practitioners since the early days of surveys. Yet, typically, formulating the questionnaire is thought to be the easiest part of survey research and often receives too little effort. Because no codified rules for question asking exist, it might appear that few, if any, basic principles exist to differentiate good from bad questions. We believe, however, that many such principles do exist. This book provides principles that novices and experienced practitioners can use to ask better questions. In addition, throughout the book we present examples of both good and bad questions to illustrate that question wording and the question's social context make a difference.

Loaded Words Produce Loaded Results

Suppose a person wanted to know whether workers believed they were fairly compensated for their work. Asking "Are you fairly compensated for your work?" is likely to elicit a very different answer than asking "Does your employer or his representative resort to trickery in order to defraud you of part of your earnings?" One would not be surprised to find that an advocate for improving the situation of workers asked the second question. Clearly the uses of words like "trickery" and "defraud" signal that the author of the question does not have a high opinion of employers. Indeed, this was a question asked by Karl Marx on an early survey of workers.

Questionnaires from lobbying groups are often perceived to be biased. A questionnaire received by one of the authors contained the following question: "The so-called 'targeted tax cuts' are a maze of special interest credits for narrow, favored groups. Experts agree the complex, loophole-ridden tax code makes it easy for Big Government liberals to raise taxes without the people even realizing it. Do you feel a simpler tax system-such as a single flat rate or a national sales tax with no income tax-would make it easier for you to tell when politicians try to raise your taxes?"

Even an inexperienced researcher can see that this question is heavily loaded with emotionally charged words, such as "so-called," "loophole-ridden," and "Big Government liberal." The authors of this questionnaire are clearly interested in obtaining responses that support their position. Although the example here is extreme, it does illustrate how a questionnaire writer can consciously or unconsciously word a question to obtain a desired answer. Perhaps not surprisingly, the questionnaire was accompanied by a request for a contribution to help defray the cost of compiling and publicizing the survey. Surveys of this type, sometimes called frugging (fundraising under the guise) surveys, are often primarily intended to raise funds rather than to collect survey information. The American Association for Public Opinion Research has labeled fundraising surveys deceptive and unethical, but they are unfortunately not illegal.

Wording questions to obtain a desired answer is not the only type of problem that besets survey authors. Sometimes questions are simply complex and difficult to understand. Consider this example from a British Royal Commission appointed to study problems of population (cited in Moser and Kalton, 1972): "Has it happened to you that over a long period of time, when you neither practiced abstinence, nor used birth control, you did not conceive?" This question is very difficult to understand, and it is not clear what the investigators were trying to find out.

The Nuances of Politically Charged Issues

Yet even when there are no deliberate efforts to bias the question, it is often difficult to write good questions because the words to describe the phenomenon being studied may be politically charged. The terms used to describe the area of concern may be so politically sensitive that using different terms changes the response percentages considerably. A question asking about welfare and assistance to the poor from the 1998 General Social Survey (Davis, Smith, and Marsden, 2000) produced quite different opinions.

We are faced with many problems in this country, none of which can be solved easily or inexpensively. I am going to name some of these problems, and for each one I'd like you to tell me whether you think we're spending too much money on it, too little money, or about the right amount. Are we spending too much money, too little money or about the right amount on .

Not all wording changes cause changes in response distributions. For example, even though two old examples of questions about government responsibility to the unemployed were worded differently, 69 percent of respondents answered "yes." Perhaps this is because the questions were fairly general. One question, from a June 1939 Roper survey, asked, "Do you think our government should or should not provide for all people who have no other means of subsistence?" (Hastings and Southwick, 1974, p. 118).

A differently worded question, this one from a Gallup poll of January 1938, asked, "Do you think it is the government's responsibility to pay the living expenses of needy people who are out of work?" (Gallup, 1972, p. 26).

Respondents are less likely to agree as questions become more specific, as illustrated by three Gallup questions from May to June 1945:

Do you think the government should give money to workers who are unemployed for a limited length of time until they can find another job? (Yes 63%)

It has been proposed that unemployed workers with dependents be given up to $25 per week by the government for as many as 26 weeks during one year while they are out of work and looking for a job. Do you favor or oppose this plan? (Favor 46%)

Would you be willing to pay higher taxes to give people up to $25 a week for 26 weeks if they fail to find satisfactory jobs? (Yes 34%)

...