How do we tell which problem our survey had?
You can tell which problem your survey had by working through it in the same order the survey was built. First, try to state the decision the results were supposed to inform; if your team can't agree on one, the problem started before anyone wrote a question. Next, read each question as a participant would and look for wording that suggests an answer or asks two things at once. Then compare the people who responded with the people your program serves, by site, session, or how long they've taken part. Finally, look at the answers themselves: if nearly everyone chose the most positive option, or the open-ended answers are too brief or too varied to group, the response options or the questions may not have given people a way to say what they meant.
A disappointing survey often has more than one of these problems, and naming them is the first step toward a better survey next time.
What makes a survey question produce useless answers?
A survey question produces useless answers when people can't tell what it's asking, when it suggests the answer it expects, or when the answer can't be connected to anything you need to know. Some of the most common problems:
- Questions that push toward an answer. A made-up example: "How much did you enjoy our helpful new workshop?" assumes the workshop was helpful and enjoyable. Survey research organizations such as Pew Research Center and the American Association for Public Opinion Research advise keeping question wording free of language that pushes respondents toward a particular response.
- Two questions in one. "Was the session useful and well organized?" can't be answered accurately by someone who found it useful but disorganized. Ask only one question at a time.
- Vague words. "Regularly," "often," and "satisfied" can mean different things to different people, so the answers are difficult to compare or interpret.
- Answer options that don't fit. If someone's honest answer isn't among the choices, they will pick the closest one or skip the question.
Why does it matter who answered the survey?
It matters who answered the survey because the results only describe the people who responded, and those people may differ in important ways from everyone your program serves. If you sent the survey at the last session, the people who stopped coming earlier, who may have the most to tell you about what isn't working, weren't there to answer. If very few people respond, you may be missing the experiences of some groups entirely.
Who asks the questions matters as well, because participants may answer differently when the person asking is the staff member who provides their services and they don't want to seem ungrateful or cause problems for someone they rely on. Explaining what you're asking, why you're asking it, and how the answers will be used, and then reporting results so no one can be identified, can reduce that pressure. An outside evaluator asking the questions and maintaining privacy removes that power dynamic.
How do we design the next survey so the results are usable?
You can design the next survey so the results are usable by deciding what you need to learn, and what you'll do with each answer, before you write a single question. For each question you draft, ask what you would do differently depending on how people answer; if the answer is nothing, cut it. Keep the survey short, and decide in advance how you'll summarize the answers, so the format of each question fits the analysis you plan to do.
Before sending it, test the survey with a few people similar to the ones who will receive it, and ask them what they understood each question to mean, since survey researchers treat this kind of pretesting as an essential step in designing a questionnaire. Consider, too, whether a survey is the right method for every question. Numbers show patterns and change, and interviews or small group conversations help explain why, so each question should determine the method you use to answer it.
Survey design is also one of the stages where an evaluator's expertise matters most, because a question that's poorly designed can't be fixed after people have answered it. What free or low-cost tools help a small nonprofit measure impact? covers the tools, and Can a small nonprofit afford real evaluation help? describes smaller forms of help, such as a workshop on writing survey questions.
The Grantibly perspective
Lauren's perspective
One of the most common mistakes in evaluation is jumping immediately to data collection. "We need a survey." "We should do focus groups." "We need pre- and post-tests." Maybe. But first: What are we trying to understand?
The evaluation question should drive the method, not the other way around. So instead of asking, "What questions should we put on our survey?" ask, "What do we need to learn, and what evidence would help us answer that?"
