What assumptions sit behind an evaluation quote?
The assumptions behind an evaluation quote are the evaluator's estimate of how much work the evaluation will take, and two firms may estimate it differently. A quote rests on assumptions about:
- how many sites and participants the evaluation will cover
- how many rounds of surveys, interviews, or focus groups there will be, and when
- whether the evaluator will analyze institutional records or other data you already have
- what condition that data is in, and how much cleaning or combining it needs
- how many reports, meetings, and presentations are expected each year
- how much travel is involved
- how many years the evaluation runs
A change in any one of these changes the evaluator's time, and therefore the price. If one firm assumed two focus groups a year and another assumed six, their quotes will differ for that reason alone. When a quote doesn't state its assumptions, ask for them. What drives each of these costs is covered in What does an external evaluator actually cost?
How much does it matter who does the work?
Who does the work can matter a great deal to the price, because an evaluation's cost is mostly people's time and different people's time costs different amounts. In one firm, senior evaluators may design the evaluation, lead the focus groups, analyze the data, and write the reports. In another, a senior evaluator may design the plan while less experienced staff carry out much of the data collection and analysis. Both arrangements can work well, and they're likely to be priced differently.
Ask each firm who will do each part of the work, who your main contact will be, and who will interpret the findings and present them to your team, because the answers tell you what you're comparing. Comparing different kinds of evaluators, such as an independent consultant, a firm, and a university evaluation center, is its own question, covered in Independent consultant, evaluation firm, or a university center: which kind of evaluator do we want?
What might a quote leave out?
A quote might leave out work that another firm included, which can make it look less expensive than it is. Items that can sit inside one quote and outside another include:
- preparing an institutional review board (IRB) submission or a data use agreement
- cleaning or reorganizing existing data before analysis
- travel to sites
- meetings with the project team beyond a set number
- presentations to an advisory board, leadership, or the funder
- additional reports or report formats your funder requires
- revisions to reports after the first draft
None of these is a reason to rule out a firm, but each is a reason to ask, so that the quote you choose covers the work you'll need.
How do I compare evaluation quotes like to like?
To compare evaluation quotes like to like, give every firm the same description of the project and ask each to state what its quote covers. Your description should include the solicitation's evaluation requirements, the project's goals, activities, and outcomes, the sites and participants, the timeline, the data you already have, and the reports your funder expects. Then line the quotes up by the work each one describes.
Expect the designs to differ, because designing the evaluation is part of an evaluator's expertise, and firms may propose different questions and methods for the same project. That's useful information, so compare whether each design answers the questions that matter to your project and your funder, whether its data collection is realistic for your staff and participants, and whether the scope and the budget are in line with each other. If your institution requires a competitive selection, such as an RFP, a clear project description also makes the responses easier to compare. Price is one consideration among several; How do I choose between two evaluation firms that both look qualified? covers the others.
The Grantibly perspective
Lauren's perspective
It's easy for evaluation plans to grow. Add another survey, add another measure, ask another group, collect data every semester. Every additional measure creates work for participants, project staff, and evaluators, and it shows up in the price.
Part of what we do when we scope an evaluation is help clients distinguish between what they need to know to evaluate the project well and what would be interesting to study. Both can be worthwhile. They simply may not belong in the same scope of work, and that difference alone can separate two quotes.
Price also isn't the only thing you're choosing. You'll work with this evaluator for the next three, four, or five years, or however long the funding period is, so you want someone you have a good relationship with, who supports your work, and who you can trust to give you honest feedback on your program.
