Why shouldn't we claim on our own that the program caused the results?
You shouldn't claim on your own that the program caused the results you see because a change that follows a program doesn't show that the program produced it. Participants may have changed for other reasons, such as other services they received, changes in their circumstances, or changes in the community around them. The people who took part may also differ from those who didn't in ways that explain the difference. Separating the program's contribution from everything else requires a design built for that purpose, and designing one well is specialized work.
What your own evaluation can say honestly is still valuable: what changed, for whom, how the program was carried out, and what participants say helped them. Describing those things accurately, without claiming more than they show, protects your organization's credibility. If a funder or board wants to know whether the program is responsible for the results, that's a question to answer with an evaluator from the start.
When does an outside audience need an independent evaluation?
An outside audience needs an independent evaluation when its decision depends on trusting that the people reporting the results had no stake in how they turned out. That includes funders whose solicitation requires an independent evaluator, decisions about continuing or expanding a program, and results you plan to report publicly as evidence of impact. In those situations a self-evaluation, however careful, may not be accepted.
Independence also affects what you learn. An external evaluator asking questions and maintaining privacy removes the power dynamic between program staff and participants, who may answer differently when the person asking provides their services. Why do funders want an external evaluator instead of self-reporting? explains the funder's side of this, and Should I hire an external evaluator or handle evaluation in-house? covers the decision when a requirement is in play.
Do we need ethics review to evaluate our own program?
Whether you need ethics review to evaluate your own program isn't a decision to make on your own, because an institutional review board (IRB) decides whether an evaluation needs review. Under the federal rule, research is a systematic investigation designed to contribute to generalizable knowledge. An evaluation meant only to understand and improve your own program may not meet that definition, but a plan to publish the findings or to use the data in a broader study can change the answer.
Ask before you collect anything, and take particular care when participants include children or other groups the federal rule gives additional protections. If your organization doesn't have an IRB of its own, ask your funder or any college or university you partner with how review is handled. Does my program evaluation need IRB approval? explains how the decision is made, and Can we survey minors or vulnerable participants in our program? covers the additional protections.
Which evaluation designs need an evaluator's expertise?
The evaluation designs that need an evaluator's expertise are the ones where the structure of the study determines what the results can mean. Examples include comparing participants with people who didn't take part, combining results from several sites or partners, following the same people over several years, and analyses that go beyond counts, percentages, and themes in what participants said. Each requires choices about who to include, when to collect information, and how to account for differences between groups, and mistakes in those choices are difficult to correct after the data comes in.
Many small programs don't need designs like these, and a sophisticated method is never the point. When one of your questions does require one, bring in an evaluator to design the evaluation before data collection begins. Your team can still do much of the routine work, such as tracking attendance and participation, while the evaluator designs the evaluation, analyzes the data, and interprets what it means. Can we evaluate our own program without hiring anyone? describes the parts your team can take on.
The Grantibly perspective
Lauren's perspective
Evaluation should not begin with the assumption that a project is successful and then look for evidence to support that conclusion. The purpose is to learn.
Sound methods, careful analysis, and honest interpretation matter. We don't overstate what a survey, interview, dataset, or evaluation can establish. But rigor doesn't require inaccessible language or a technical report no one uses, and evidence should be explained in plain, accurate language.
