This is the second in a series of 3 blogs about the principles of questionnaire design. If you missed the previous blog with the first 5 tips, click on this link: https://www.liezelkorf.co.za/dissertation-assistance/5-tips-for-questionnaire-design1/
Let’s continue with the principles:
6. Questions which measure one construct (an underlying theme – something you cannot see) should clearly relate to this theme. Take the following questions which were all designed to measure the state of Leadership in an organization:
- My manager leads by example.
- I receive positive feedback from my subordinates on my leadership style.
- Leadership in this company is transparent.
- I believe myself to be a good leader.
The first question refers to leadership of one’s immediate superior, the second the feedback on the leadership style of the respondent, the third to leadership in the company, and the last one to the respondents’ perception of their own leadership again. Although this may be an extreme example, it is taken from an actual student questionnaire, and I hope it illustrates the point. These questions do not measure a coherent construct called “Leadership”. If all these aspects are relevant, it should rather be broken up into 3 sections – Company leadership, Leadership of immediate manager and Self – leadership – each with 3-4 questions.
7. The last question in the example above leads us to the next issue, which is the social desirability of answers. Would anyone really concede that they are not a good leader? Maybe they will, but what about a question like: “I always treat people with respect” or “I behave in an ethical manner at work”. It is in human nature to present oneself in a positive light, and someone is hardly likely to admit that they behave in an unethical manner at work – even if anonymity of the questionnaire is guaranteed!
How does one get around this? Apart from steering clear of such questions, which would be the obvious choice, another option is to ask the question about the person’s peers or colleagues. That way respondents feel as if this is not directed towards themselves, but if a representative sample in a groups answers this question, one does get a sense of what this particular aspect is like in your population of interest. For example: “Most students I know will not practice plagiarism” or “People in my team generally uphold company values”. You get the idea? Freud would have been impressed with this use of projection 😊
8. Scaled answers vs Yes / No answers. The type of response scale is determined by the type of question. If the essence of the question is the extent to which a respondent agrees with the statement, then a scale using some form of agreement is appropriate. However, if the question would be better addressed by a measure of frequency, it is better to use some scale of frequency (See blog 1 on questionnaire design). Some questions are, by their very nature, answered by Yes / No options. These usually refer to a respondent’s knowledge of a certain phenomenon or a state of affairs. When these are asked using a scale, the results are often not meaningful. A question such as: “Our company has a policy for dealing with staff grievances” cannot be answered on a scale – the company either has such a policy or it does not. To allow for respondents who may not know whether this is the case, a “don’t know” option can be added.
More principles of questionnaire design will be discussed in the next blog.