Question Wording and Practical Tips
Clear survey questions and prompts help ensure the reliability of survey data. By looking at a survey as an inquisitory conversation between a survey writer and the survey respondent, we can apply linguistics principles generate survey questions that are clear and convey the desired meaning.
The Four Maxims
According to the “cooperative principle,” four maxims, or elements, are automatically assumed to be true upon entering into communications (Parker and Riley, 2000):
| Quantity: The necessary information is included but no unnecessary information has been added. |
| Quality: The information is truthful. |
| Relation: The information is relevant to the context of the situation. |
| Manner: The information is delivered clearly and succinctly with no underlying meaning. |
When writing your survey questions, you can use these four maxims as checks to help reduce possible misinterpretation.
Be Specific!
If you want to know more specific details, then you must design your questions to elicit those response types.
Consider this sample Likert-type question:
Table 1: Sample prompt with less clarity.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| Strongly Disagree | Disagree | Slightly Disagree | Niether Agree Nor Disagree | Slightly Agree | Agree | Strongly Agree |
A student may respond to the first question with a “Disagree,” but this tells you nothing about specific elements they felt unprepared for. The student may feel ambivalent about the preparedness or may be considering several factors that fall into differing degrees of preparedness. To avert this issue, you must be specific in your inquiries. What about upper-level courses do you want to know if they are prepared for?
Instead, try:
Table 2: Sample prompt with more clarity.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| Strongly Disagree | Disagree | Slightly Agree | Neither Agree Nor Disagree | Slightly Agree | Agree | Strongly Agree |
The survey respondent may still respond “neither agree nor disagree,” but you can now be sure the response targets a specific detail rather than possibly comingling many factors.
Don't Combine Questions! The Problem with "and" & "or"
You may be tempted to merge similar questions together using “and” or “or” statements to shorten the survey. This, however, complicates data collection and makes analysis less accurate and more difficult.
For example, let’s say you want to know about how well your course prepared your students in two categories: rigor and content understanding.
Table 3: Example prompt using AND.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| Strongly Disagree | Disagree | Slightly Disagree | Neither Agree Nor Disagree | Slightly Agree | Agree | Strongly Agree |
In this example, you will have no way of knowing if the student felt prepared for the rigor only, the content only, both, or neither. By merging inquiries into a single survey item, you lose the ability to clearly interpret the data obtained since you must then take individual respondents’ interpretations into account. For example:
-
Student A feels prepared for the rigor but not the content. -> Responds disagree because they took the “and” seriously.
-
Student B also feels prepared for the rigor but not the content. -> Responds agree because the positive outweighed the negative to them and did not take the “and” statement at face value.
Like “and” statements, “or” statements also preclude clear data interpretation. Using the above example question and same student opinions with the “and” swapped for an “or,” we can again see where this introduces problems in data analysis as the results flip without the driving data ever changing.
Table 4: Example prompt using OR.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| Strongly Disagree | Disagree | Slightly Disagree | Neither Agree Nor Disagree | Slightly Agree | Agree | Strongly Agree |
-
Student A felt prepared for the rigor but not the content. -> Responded agree because they adhered to the limitation of the “or.”
-
Student B also felt prepared for the rigor but not the content. -> Responded disagree because the negative outweighed the positive to them and did not adhere to the “or” phrasing.
The Solution: Separate Inquiries, Separate Survey Questions
Instead, break up these statements into two separate survey items:
Tables 5 & 6: Eliminating the AND/OR confusion.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| Strongly Disagree | Disagree | Slightly Disagree | Neither Agree Nor Disagree | Slightly Agree | Agree | Strongly Agree |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| Strongly Disagree | Disagree | Slightly Disagree | Neither Agree Nor Disagree | Slightly Agree | Agree | Strongly Agree |
Both Student A and Student B will now respond to the first option with “agree” and the second with a “disagree” as there is no ambiguity as to what the response is tied to, and there is less room for individual interpretation. This makes your data analysis more streamlined and the results clearer.
Avoid Negation Words: It Doesn't Not Hurt Your Survey...Right?
When writing your questions, avoid using phrasing with a negation component. Questions containing negative wording require longer processing time and increase the instances of respondent mistakes, so format questions in the affirmative rather than the negative (Lietz, 2010).
Consider the following two questions: “Do you think SI attendance should be a requirement?” and “Do you think SI attendance should not be a requirement?” Both are simple sentence questions that adhere to all the previous outlined requirements, yet the second question introduces a negation which complicates the question. If respondents miss the “not” of the question, their answer now takes on the opposite meaning as what you were asking. In addition, if you are working with a Likert model, your respondents must now mentally calculate all the double negative possibilities in the answer. “I agree” is a much clearer answer than “I do not disagree” and makes the data more reliable by eliminating areas of confusion.
References
Bunce, D. M., & Cole, R. S. (Eds.). (2008). Nuts and Bolts of Chemical Education Research (Vol. 976). American Chemical Society. https://doi.org/10.1021/bk-2008-0976
Lietz, P. (2010). Research into Questionnaire Design: A Summary of the Literature. International Journal of Market Research, 52(2), 249–272. https://doi.org/10.2501/S147078530920120X
Parker, F., & Riley, K. L. (2000). Linguistics for Non-linguists: A Primer with Exercises. Allyn & Bacon.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.