How do we tell whether the data we collect is the right data?
You can tell whether the data you collect is the right data by connecting each piece of information to a question your organization needs answered. Make a short list of what you need to learn, then note beside each question the information that would help answer it and whether you already collect that information. A question with nothing beside it represents a gap. Information you collect that doesn't connect to any question on the list may represent work for staff and participants that nobody is using.
An example of connecting questions to evidence
| What you need to learn | Information that could help answer it |
|---|
| Who participated, and who didn't? | Enrollment and attendance records, compared with the population the program was designed to reach |
| What did participants experience? | Surveys, interviews, or focus groups with participants and staff |
| What changed for participants? | Information gathered at more than one point in time, such as at the beginning and the end of the program |
| What helped, and what got in the way? | Conversations with participants and staff, considered alongside the records |
Your own questions will depend on what the program is designed to accomplish and what your board and funders need to understand. Do we need new surveys, or can we use what we already collect? covers what to do when a question has no information behind it, and What data do I actually need to collect for my grant evaluation? covers the requirements of a funded grant.
Why do we need information about how the program is carried out as well as its results?
You need information about how the program is carried out because results are difficult to interpret without it. If outcomes fall short, that doesn't automatically mean the underlying idea was ineffective: the program may not have been fully implemented, participants may not have received enough of it, external conditions may have changed, or the model may need refinement. Likewise, positive outcomes don't automatically tell you what produced them. If your results do suggest the program isn't working as you hoped, What if the evaluation shows my program isn't working? covers what to do next.
Information about implementation answers questions such as whether the program was carried out as intended, who participated and how much of the program they received, what changed along the way, what barriers emerged, and whether participants at different sites or in different groups had the same experience. Organizations often already hold some of this information in attendance logs, staff notes, and meeting records, without recognizing it as evaluation data.
What can our data show about whether the program is working, and what can't it show?
Your data can show whether the changes you hoped for are occurring, for whom, and alongside which parts of the program, and it can provide evidence that supports a conclusion about your program's progress. What it usually can't do on its own is establish that the program caused those changes, because participants' lives are shaped by many influences at once, and a change that follows a program isn't automatically a result of it.
That's one reason "Did the program work?" is more difficult to answer than it appears, and why the perspective below suggests a more answerable question. How confident any conclusion about cause can be depends on how the evaluation is designed, which is why that design belongs to an evaluator's expertise at the beginning of the work, before the information has been gathered.
The difference between counting what a program delivered and showing what changed is a separate question. What's the difference between outputs and outcomes, and why does it matter to funders? explains it.
The Grantibly perspective
Lauren's perspective
Evaluation shouldn't begin with the assumption that a project is successful and then look for evidence to support that conclusion. The purpose is to learn. Sometimes the data show strong results, sometimes they show mixed results, and sometimes an approach doesn't work as expected. Often the most useful information reveals both strengths and opportunities for improvement.
That's why we encourage organizations to ask different questions. Instead of asking how to show the funder that you're successful, ask what you're genuinely learning about your progress toward the outcomes you promised. And instead of asking whether the program worked, ask what worked, for whom, under what conditions, and what evidence supports that conclusion.
Understanding how a program operates is essential to understanding its results. We want clients to see evaluation as a source of information instead of a judgment.
