What should "working" mean for a work-based learning program?
What "working" should mean for a work-based learning program depends on what the program was set up to change, and each group involved may have its own answer. For participants, it might mean building skills, earning a credential, deciding whether the field is right for them, or getting hired. For employers, it might mean a steadier supply of people prepared for their roles. For your organization, it might mean stronger employer partnerships or more students or participants completing a pathway.
Start with the questions you need answered and the decisions they will inform, not with a survey. A program that wants to know whether to expand to more employers needs different evidence from one that wants to know why participants at one site progress more slowly than at another. Writing down what you expect the program to change, and why you expect it, also makes clear which assumptions the evidence should test. Those expectations describe how the program is supposed to work; they don't establish that it does.
What can we look at while the long-term outcomes are still years away?
While the long-term outcomes are still years away, you can look at interim markers that show whether the program is moving participants in the right direction. Those markers include who enrolls and who doesn't, whether participants reach each step of the program on time, whether they're placed and stay through their placements, how often they meet with mentors, which skills they demonstrate, and whether they complete the program. Later, you can look at whether completers move into related jobs or further education.
Interim markers are useful because they let the program improve while there's still time to act. A drop in progression at one stage, or at one employer, is a signal to find out what's happening there. Markers suggest whether the program is on track, but they don't show on their own that the program caused a change, so read them alongside what you learn about how the program is running.
Why do we need to understand how the program runs before judging the results?
You need to understand how the program runs before judging the results because outcomes are hard to interpret without knowing what participants received. A work-based learning program can look different from one employer or site to the next. One placement may give a participant a prepared mentor and real work, while another gives them little supervision and little to do. If outcomes fall short, that doesn't automatically mean the idea behind the program is weak; the program may not have been delivered as designed, or conditions may have changed.
The reverse is also true. Good completion numbers don't tell you which parts of the program produced them or whether every group of participants benefited equally. Looking at who participated, what they experienced, and how the program changed along the way tells you what worked, for whom, and under what conditions. That's the information you need to decide what to keep, change, or expand.
What do participants' and mentors' perspectives add that a completion rate can't?
Participants' and mentors' perspectives add the reasons behind the numbers, which a completion rate can't supply. A completion rate can show that one group finishes less often than another; the people involved can tell you why, whether it's scheduling, transportation, a placement that didn't fit, or mentoring that didn't happen. Supervisors, employers, and program staff add what they see from their own positions.
Gathering those perspectives well takes methods designed for them, such as surveys, focus groups, or participatory listening sessions, where participants discuss their experiences together. Ripple Effects Mapping is one participatory method for tracing what a program set off, including what nobody planned. Participants share stories of what changed and map how one effect led to another. Who to ask, what to ask, and how to protect people's ability to answer candidly are design decisions an evaluator makes with you. If your program is grant-funded and you need to know what data the funder requires, What data do I actually need to collect for my grant evaluation? covers that. If you're weighing doing the work yourselves, Can we evaluate our own program without hiring anyone? looks at what's realistic.
The Grantibly perspective
Lauren's perspective
Projects are often eager to know whether their program worked. That's an important question, but it's difficult to interpret outcomes without understanding implementation. Was the program implemented as intended? Who participated? How much of the program did participants receive? Did participants at different sites have the same experience?
We don't see quantitative and qualitative methods as competing approaches. Numbers can show patterns, change, and differences between groups. Talking with participants, mentors, and employers can help explain experiences, barriers, and why a pattern may be occurring. Administrative data might show that one group has lower completion rates, and participant feedback may reveal barriers the numbers alone can't identify.
So instead of asking only whether the program worked, we'd ask what worked, for whom, under what conditions, and what evidence supports that conclusion.
