Developing Questions
Developing questions can take some time and there are many wrong ways to ask questions. Ambiguity, bias, or lack of appropriate options will affect the quality of your results. Choosing the wrong question types, incorrectly setting up logic, or asking questions people don’t understand, will affect your responses. If you’re designing your own questions and want to avoid some of the common pitfalls, here are some pointers:
- Use a short introduction. Detail why you are collecting this information. Consider the need to re-enforce that a survey is anonymous.
- Have a vision and a plan. It’s critical to have a clear understanding of what your objective is. With quantitative research it is much harder to go back and adjust once you have collected your data. With each question it is worth writing down what you are hoping to use the results for.
- Provide respondents with enough information. Respondents need to be able to make informed decisions. Information can be provided in the survey or as additional supporting documents. It is important to provide balanced information so that the question is not biased.
- Avoid biased questions. It’s okay to use statements and ask if people agree or disagree but it is not ok to ask if they agree. Agreeable people usually agree!
- Start general, then get specific. When designing the flow, imagine an inverted pyramid. Start out broad, then narrow as needed. This is a logical way for the respondent to think, making it easier for them to answer the questions e.g., 1. How was the event overall? 2. Now tell me about the specifics.
- Avoid order bias. It’s critical not to give away the answer to a question before asking it (e.g., sharing brand names before asking unaided brand awareness). Ensure earlier questions do not influence responses to later questions.
- Use clear language and keep your audience in mind. If your terminology or the language used isn’t clear to you or your project group, it won’t be clear to your audience. If there’s any ambiguity, it’s likely to creep into your results.
- Use the right tone. Ensure that the tone of the wording is appropriate to the target audience.
- Use conditional questions to reduce the length. Electronic forms can be formatted to skip questions that are not relevant to the respondent. In a paper questionnaire you can use ‘If Yes’ or ‘Skip to question…’.
- Keep length in mind. Always time your questionnaire to ensure you are within 2-5 minutes. The number of completed surveys drops off drastically when over 5 minutes. Stick to the questions you need to ask. If the survey has to be longer, consider the use of a ‘save’ button.
- Bear in mind nonresponse bias. This is when people who respond to your survey aren’t representative, and have different opinions or characteristics of the people who don’t respond. That means you are talking to the wrong people which can give you bad data.
- Consider open and closed questions. Surveys often use majority quantitative ( questions with defined answers to select. Quantitative (information that can be shown in numbers and amounts) questions provides statistical information that can be quickly analysed. Numbers of completed responses are important here for statistical significance. Most surveys should also offer the opportunity to add qualitative (the explanation of the quality of experience/s which is not as easy to measure) comment.
- Keep your questionnaire neat and orderly. Surveys that are ‘bitty’, poorly formatted, and illogical are a ‘turn off’ for respondents. Keep related questions together.
- Consider whether taking an email address is appropriate/necessary for feedback or further involvement. It is good practice to provide feedback. Let the respondent know how you are planning to feedback the results. Don’t forget to thank the respondent and tell them how their engagement will be used. They have completed this voluntarily and if they are to engage in the future, they need to know that their time was appreciated.
- Place an end date and return address on all paper questionnaires
- Review all questions to ensure they are not leading
- Pre-test the questionnaire. Probably your best failsafe to ensure that all of the above points can be considered by someone else. Ask testers to provide serious answers and analyse them. This may identify where a question can be ambiguous or highlight where you need to ask an additional question or for more information. Testing also gives you a better idea of how long it will take to complete so you can let respondents know.
Demographic data collection on surveys
Data that identifies people’s features is usually required for cross-analysis of the data, to make sure that respondents match the demographic features of the area and for Equality and Diversity reporting. It is advisable to adopt a consistent approach and agreed demographic questions. You can use options like don’t know or prefer not to say in your demographic questions, but only when they are appropriate. These are especially useful on especially sensitive questions and allow respondents a way out of answering without abandoning your survey. See our Equality and Diversity section for more information.
Promoting surveys/questionnaires
Advertising is a key part of the success of surveys. The Council’s minimum communication standard is covered in the section ‘Communicating ways to get involved’ but it can and should be enhanced for key target audiences. You could take your survey to out into community venues or community groups and complete it face to face with people. Most venues will require you to follow protocol such as getting permission, booking a slot, and completing a risk assessment. Allow sufficient time to meet with their calendars. Think about supermarkets, markets, schools, colleges, universities, primary care centres, GPs, workplaces, community centres, interest groups (for example sports, health, hobbies), voluntary, community and social enterprise organisations. You could also:
• Use paid for advertising on Facebook to place it on the newsfeed of your target audience
• Use posters and flyers
• Incentivise completion (e.g., enter into a prize draw)
• Ask respondents to share with their friends and family.
More information on finding where to go and who to talk to is available in the ‘Talking‘ section of this toolkit, under ‘Other ways to find the right stakeholders for your project’.
Page last updated: 06 Oct 2023