What should I ask an evaluator about how they'll design the evaluation?
The questions to ask an evaluator about design are the ones that show whether the evaluation will be built around your project or around a method they already prefer.
- "How will you decide what to measure?" A good answer starts with what your project is trying to understand and what decisions the findings will support, because the evaluation question should drive the method. Be cautious if the first thing you hear is a list of surveys.
- "How will you keep data collection manageable?" Every additional measure creates work for participants, project staff, and evaluators, so a good answer describes collecting enough of the right data to answer the questions that matter and using information your organization already has before collecting more.
- "How will you look at how the program is being carried out, not only its results?" Outcomes are difficult to interpret without understanding who participated, how much of the program they received, and what changed along the way, so a good answer includes both implementation and results.
- "What happens to the evaluation plan if our project changes?" Programs evolve during a grant as staff, partners, and timelines change, and a good answer treats documenting and understanding those changes as part of the evaluator's work. Our program changed mid-grant. Can we change the evaluation plan too? covers the details.
What should I ask an evaluator about sharing findings?
The questions to ask an evaluator about sharing findings tell you whether you'll hear what the evaluation is learning in time to use it.
- "When will our team hear findings?" A good answer includes sharing findings during the project, in meetings or brief summaries, while there's still time to act, as well as the reports your funder requires.
- "Can we see an example of how you report findings?" A good answer is a sample, with client details removed if needed, that a reader outside evaluation could understand and use.
- "What do you do when the results aren't what we hoped?" A good answer is that they'll report what the evidence shows, explain what it means for the project, and help your team determine what to do about it. Be cautious about an evaluator who promises a positive conclusion before the work begins, because an evaluation shouldn't start by assuming the project is successful. What can we ask our evaluator to change after we've reviewed a report? explains what's reasonable to ask for once a report is drafted.
- "How will you help our team understand our own data?" A good answer describes explaining what they're doing and why, making findings accessible, and helping your team become stronger users of evidence over time.
What should I ask an evaluator about working together?
The questions to ask an evaluator about working together show you what the relationship will be like for your team over the next several years.
- "Who will do the work, and who will be our main contact?" A good answer names the people who'll lead the evaluation, collect and analyze the data, and handle scheduling and day-to-day questions.
- "How often will we meet?" A good answer gives a regular rhythm, such as quarterly meetings, and says it can change with the project's needs. We've received notice that our project was funded. When do I need to get my evaluator involved? covers the first months.
- "What will you need from us to get started?" A good answer lists specific items: your proposal or award narrative, a logic model if you have one, your reporting calendar and timeline, the names of your project team, and any institutional steps such as vendor setup.
- "How do you handle IRB review and data agreements?" A good answer explains how they'll work with your IRB office, whether their staff should be listed on the application, and what data use or confidentiality agreements may be needed. Does my program evaluation need IRB approval? explains when review applies.
What answers should make me cautious?
The answers that should make you cautious are the ones that suggest the evaluation will serve the evaluator's routine more than your project. Be careful if an evaluator recommends methods before asking what your project needs to learn, proposes to measure everything, describes the work only as an annual report, or suggests the results will be positive before any data is collected. A vague answer to "What will you need from us?" is also worth a follow-up, because unclear expectations at the start can create extra work for your staff later.
What an evaluator charges is its own question, answered in What does an external evaluator actually cost?, and credentials are covered in Does our evaluator need a Ph.D. or specific credentials?
The Grantibly perspective
Lauren's perspective
We don't collect data simply because it might be interesting, and we don't believe more measures automatically mean a more rigorous evaluation. We're not interested in producing enormous reports no one reads.
We also don't tell clients that everything is going wonderfully when the evidence suggests otherwise, and we don't want clients dependent on us to make sense of their work. Those are the answers we'd hope to hear from any evaluator you're considering, including us.
