Biased data are bad data: How to think about question order

The order in which you ask questions can make a huge difference in your data. If you’re not careful, you can inadvertently anchor your respondents and bias your data. And no one wants that.

A classic example of the impact of question order occurred during the Cold War.[1] A survey was conducted in which American respondents were asked whether or not Soviet journalists should be allowed to visit the United States to write articles for Soviet newspapers. The same respondents were then asked whether or not American journalists should be allowed into the Soviet Union to write articles for American newspapers. The responses indicated lower support for American reporters among those that had opposed allowing the Soviet journalists to cover the U.S.

Interestingly, when these same questions were asked in reverse order—this time with American journalists referenced first—there was greater support among American respondents for allowing both American and Soviet journalists to cover the other’s country.

Why the change? Respondents often seek to provide answers that are consistent with their prior responses. In this case specifically, respondents were likely looking to conform to the norm of evenhandedness.

There are many other reasons that respondents may be influenced by question order. That’s why it’s so important to be aware of when you are designing your surveys.

Even seemingly innocuous questions may influence each other. For example, asking respondents which flavor of ice cream is their favorite and then immediately following up with a question about which dessert is their favorite is likely to result in more respondents indicating a preference for ice cream. This is an example of priming, where the respondent was made to think about something before being asked about a broader class of items that contains the thing they were just ‘primed’ to think about.

How to combat question order bias

1. Pre-test your surveys. Ask a small group of friends or colleagues to take the survey and provide feedback about the order of questions or anything else that might come up as confusing or out of place. This is an opportunity to catch problems with the questionnaire, including question order effects.
2. Randomizing the order of unrelated questions. Randomizing the order of unrelated questions reduces bias. For example, you would want to randomize questions like “what is your favorite sport?” and “what is your favorite dessert?” But you won’t want to randomize everything. Respondents expect related questions to be asked together. For example, you wouldn’t want to mix your demographic questions in with your ad-testing questions.
3. Create groups of related questions. People expect related questions to appear together, but that doesn’t mean they have to be in the same order all the time. Group related questions together and then randomize the order of questions within that group, or block, to reduce bias. Where it makes sense, you can even randomize the order of some of the blocks, which is easy to implement in Qualtrics.

While randomization can help reduce bias introduced by question order effects, it is not a perfect solution. Some questions need to be asked in a particular order to make sense, and other questions, including branched questions, need to be asked in a specific order to be effective.. For these types of questions, it is best not to attempt randomization and simply ask the questions in order.