What should I look for beyond a firm's qualifications?
Beyond a firm's qualifications, look at how it thinks about your project, because that's what shapes the evaluation you'll get. A strong evaluation partner starts with the question, not the instrument. Before they recommend a survey, focus groups, or pre- and post-tests, they ask what your project is trying to understand and what decisions the findings will support, and then they determine what evidence would answer those questions.
Listen also for proportion. More data isn't necessarily better data, and every added measure creates work for participants, project staff, and the evaluator. A firm that proposes to collect enough of the right data to answer the questions that matter is planning an evaluation your team can sustain. A firm that proposes to measure every possible outcome may be planning one that creates survey fatigue and data no one has time to use. In a grant-funded project at a college or university, also notice whether the firm distinguishes what you need to know to evaluate the project from what would be interesting to study. Both can be worthwhile, but they may not belong in the same scope of work.
How do I compare their scopes of work and budgets?
To compare two firms' scopes of work and budgets, start by checking whether each scope fits its own budget and fits your project's goals, activities, and outcomes. A plan that promises a large scope with too little budget to carry it out can't be delivered as written, and reviewers notice that mismatch in a proposal. A budget much larger than the planned work is the less common problem, and it's just as worth asking about.
Then compare what each scope says about timing and roles. Check when data will be collected and whether that timing makes sense for your activities, then look at how often the evaluator will meet with your team, who at the firm will do the work, and what the firm will need from your staff. If one quote is much higher or lower than the other, Why do evaluation quotes vary so much between firms? explains what usually drives the difference, so you can compare like with like.
How can I tell how a firm will share findings with us?
You can tell how a firm will share findings by asking it directly and by looking at examples of its work. Ask when your team will hear what the evaluation is learning: only in the annual report, or in project meetings and interim summaries while there's still time to act on it. An evaluation is most useful when its findings become part of planning conversations and implementation decisions, and a firm that builds that feedback into its plan is designing for that use.
Ask each firm for a sample report, summary, or presentation, with client details removed if needed. Look at whether a reader outside evaluation could understand what the findings mean and what they suggest the project should do next. A percentage in a table may be technically accurate and still leave your team asking, "So what?"
How much should the working relationship matter?
The working relationship should matter a great deal, because you'll work with this evaluator for three, four, or five years, or however long the funding period is. Choose a firm whose people you'd be comfortable calling when something in the project changes or a result surprises you. They should also be willing to raise concerns or share uncomfortable findings in service of the project. You also shouldn't have to manage your evaluator. If the conversation already feels like work, it's worth noticing.
If the two firms are different kinds of evaluator, such as an independent consultant and a university evaluation center, the comparison changes. Independent consultant, evaluation firm, or a university center: which kind of evaluator do we want? covers that choice, and What questions should I ask an evaluator before hiring them? gives you questions to use in both conversations.
The Grantibly perspective
Lauren's perspective
We are partners, not vendors. We care about the success of the work, not simply completing the evaluation tasks listed in a scope of work.
That doesn't mean telling clients what they want to hear. A good evaluation partner should be willing to raise questions, identify concerns, challenge assumptions, and share findings that may be uncomfortable, and we do that in service of the project. We want clients to call us when they're considering a change, struggling with participation, trying to understand an unexpected result, or preparing for a funder conversation.
