Surveys – how to do a questionnaire the right way

Two days ago I wrote about an example of how surveys should not be done.

But how to make a good survey?

This is an astonishingly hard topic, so I can only give you some basics, the things you have to look out for if you want to create a questionnaire. That also means you know how to spot a bad one, as a bonus. Two in one, nice!

Open and closed ended questions

An open ended question is one where the person(s) asked offer answers themselves.

What country do you currently live in?
Answer: XYZ

Closed questions have the answers provided. In some cases that is all you need.

Do you want to eat pizza?
A) Yes
B) No

But in many cases this is not sufficient. Consider this question:

What is your religion?
A) Christian
B) Muslim
C) FSM

It is easy to see that this question leaves out a lot of people who cannot give a correct answer. First of all, the majority of people on earth believe in something else then those three religions, and there are also a lot of Atheists.

The reason that almost always closed questions are used (or the mix with several answers and then “something else: XYZ” is that is is a lot less work – fully automated – to analyze the answers if you have a closed question. Manual answers may have different names for the same thing and they may be written with wrogn speeling.

And of course it is way easier to manipulate with closed questions ;)

Order effects

But even if a closed question is all you need, you have to be careful.

The answers may differ depending on how the options are ordered. In phone interviews, the last answers are more likely to be taken, while on a ballot sheet the first party always has a ~3% vote bonus (multi-party systems, I don’t know about the US where you basically only have two).

That is why answers should be randomized, if people should answer questions where the answer is not set in stone.
Your favorite color may not change in your whole life and most people can name it instantly.
But what about a question like “What is the most healthy food” with 10 possible answers?

There were just so many nice pizza photos I could not decide

Of course, giving too many answers is not good either. People might even abort the questionnaire if it gets too stressy with all those decisions!

Loading

Another problem is that one question can influence the answers on a different question.

If you ask “Name the first politician that comes to mind” then the answer may be very different if you have asked before “Did you see Trump’s latest idiocy?” compared to “Did you know Gandhi beat his wife?”

Same is true for sounds or pictures. How hungry are you right now?

I guess more hungry than at the start of this post, right?

Wording

Here we enter the most murky and sometimes dark area of asking a question.

The words you choose for your question can have a high influence on the answers, even if they mean the same or very similar things.

Several experiments have shown that there is much greater public support for expanding “assistance to the poor” than for expanding “welfare.” (*from source)

And even more important may be the (intentional) mentioning or not mentioning of a related fact. The numbers of YES votes for the question

taking military action in Iraq to end Saddam Hussein’s rule

got 68%, but

taking military action in Iraq to end Saddam Hussein’s rule even if it meant that U.S. forces might suffer thousands of casualties

got only 43% approval. Just not mentioning the possible causalities (you are going for a war, duh, did you think there would be none?) increased the number of YES answers by 58%! (*from source)

While this example might be easy to spot, how about this?

Do you want more border security to save lives from terrorist attacks?
A) Yes
B) No

I didn’t even say only Islamistic terror (which most people nowadays think off). All terrorists are meant. Would you say yes? Really?
Let me add a tiny bit of information to the question.

Do you want more border security to save lives from terrorist attacks even if that is very expensive and the same money could be used somewhere else to save 200 times more people?
A) Yes
B) No

Yes, Anti-Terror actually kills people!

Still want to vote for yes, you murderer?

Conclusion

Do not trust a survey where you don’t know the question. Be careful when answering in questionnaires that may be intentionally bad. Be even more careful when making your own! (And I hope you don’t do it to manipulate.)

Always use your brain for critical thinking – and that includes thinking critical about yourself and your critical thinking!

source: Pew Research Center, with more (general) information.)