Why are some outcomes so hard to measure?
Some outcomes are hard to measure because they take years to appear, happen after participants have left the program, depend on many influences besides the program, or describe personal changes that are difficult to put into numbers. A program working toward stable employment, stronger family relationships, or a student's sense of belonging may not see that outcome during the grant period, and when it does appear, many other things in a participant's life may have contributed to it.
Measuring these outcomes can also place real demands on participants and staff. Following up with people years later takes time and resources. Asking about sensitive parts of their lives requires care, consent, and a clear reason. More data aren't necessarily better data, so collect enough of the right evidence to answer the questions that matter.
What can we report before the long-term outcome happens?
Before the long-term outcome happens, you can report how the program was implemented and the short-term and intermediate changes your program expects to lead toward that outcome. Among its other parts, a logic model lays out those expected steps: the activities, the short-term changes they are meant to produce (such as new knowledge or skills), the intermediate changes that should follow (such as changes in behavior or circumstances), and the long-term outcomes the program is working toward. If you don't yet have one, What's a logic model, and does my proposal really need one? explains what it is.
Implementation belongs in the report too. Was the program carried out as planned? Who participated, and who didn't? How much of the program did participants actually receive? What barriers came up, and what changed along the way? Without that information, any outcome is hard to interpret, because a result that falls short may reflect how the program was carried out rather than whether the idea behind it works.
Keep in mind that a logic model describes how a program is expected to work. Seeing the short-term changes it predicts is encouraging evidence of progress, but it doesn't establish that the long-term outcome will follow or that the program alone produced the change. Say that plainly when you report interim results.
Do participants' own accounts count as evidence?
Participants' own accounts count as evidence when they are gathered and analyzed systematically. Interviews, focus groups, and open-ended survey questions can document changes in confidence, relationships, or how people see their own future that a number would miss, and they can help explain why a pattern in the numbers occurred.
What makes that evidence credible is the process behind it. Ask consistent questions, hear from a range of participants and not only the most enthusiastic, look for patterns across many accounts, and report experiences that don't fit the pattern as well as those that do. Quantitative and qualitative data tell different parts of the story, and used together they give a more complete picture than either one alone.
Individual stories have a place too. What makes an impact story credible instead of just emotional? covers how to use individual stories responsibly.
How do we explain what we can't measure without it sounding like an excuse?
You explain what you can't measure without it sounding like an excuse by saying specifically what the evidence does show, what it doesn't show yet, and what it would take to know more. A sentence such as "participants reported stronger job-search skills at the end of the program, and we will not know for another two years how many found stable work" (an illustrative example) is specific, honest, and useful to a funder or board.
Professional evaluators are expected to report limitations this way. The American Evaluation Association's guiding principles ask evaluators to communicate all aspects of an evaluation, including its limitations, in a transparent, accurate, and accessible manner. A report that is clear about its limits gives readers a sound basis for trusting the findings it does present. It also helps to say what you're doing about the gap, such as how you plan to follow up with participants.
The Grantibly perspective
Lauren's perspective
Projects are often eager to know whether their program worked. That's an important question, but it's difficult to interpret outcomes without understanding implementation: whether the program was carried out as intended, who participated, how much of the program they actually received, and what changed along the way.
If outcomes fall short, that doesn't automatically mean the underlying idea was ineffective. The program may not have been fully implemented, participants may not have received enough of it, external conditions may have changed, or the model may need refinement. Likewise, positive outcomes don't automatically tell us what produced them.
We'd rather help a team ask what worked, for whom, under what conditions, and what evidence supports that conclusion. That's a question you can begin answering long before the final outcome is in.
