How do I read what the reviewers said about the evaluation?
To read what the reviewers said about the evaluation, look for which of the common problems their comments point to. Here's one example of what a declined applicant receives. When the National Science Foundation declines a proposal, the principal investigator receives copies of the reviews and the panel summary, if there was one, without reviewers' names. If the reasons still aren't clear, NSF allows the investigator to ask the program officer for more information. Other funders share different amounts of feedback, so use whatever yours provides.
| What the comments point to | What it looks like in the plan | How to fix it |
|---|
| The evaluation only looks at the end | The plan evaluates the program only after it's finished, with no feedback while it's running | Build in evaluation methods and regular feedback to the project team throughout the project |
| The plan isn't aligned with the project | It collects too much, too little, or the wrong information for the project's goals and outcomes | Rewrite the evaluation questions from the goals, activities, and outcomes, then match the data to each question |
| The scope and the budget don't match | The plan promises more than the budget can pay for, or the budget is large for little work | Bring the scope and the budget into line |
The first problem is about timing. Reviewers like to see the evaluation used to improve the program as it's carried out, which is called formative evaluation, alongside the summative evaluation at the end. For example, NSF's Advanced Technological Education (ATE) program describes the evaluator's primary role as informing the project team's decisions about changes during the project, with a summative evaluation at the end. What's the difference between formative and summative evaluation, and why does my funder want both? explains the two.
What does it mean when reviewers call an evaluation plan vague?
When reviewers call an evaluation plan vague, it often means the plan doesn't show how each evaluation question connects to a goal, what data will answer it, from whom, and when. Spell those out. Start from the project's goals, activities, and outcomes, write the evaluation questions from them, and make sure the data you propose to collect and analyze addresses each one adequately.
Then check two more things. Does the plan collect both quantitative and qualitative data where a question needs both? And does the timing of the data collection make sense for the timing of the activities? A survey scheduled before participants have finished the activity it asks about, for example, can't answer the question it was meant to answer. What should the evaluation section of my grant proposal include? walks through what each part of the section needs to say.
Should I bring in an evaluator before I resubmit?
You should consider bringing in an evaluator before you resubmit. An evaluator can align the evaluation plan with the project's goals and the budget with the scope, and can write the evaluation section. A common problem when the project team doesn't work with an evaluator during the proposal phase is that the evaluation gets under-priced, which is one of the problems reviewers notice.
If you worked with an evaluator on the declined proposal, consider whether a different one would serve you better. Bring them the reviews and the panel summary along with the proposal, so the revision starts from what the reviewers said. What can an evaluator actually do for me before I submit the proposal? describes that work, and How do I find an external evaluator for my federal grant? covers finding one.
The Grantibly perspective
Lauren's perspective
We believe in sharing useful findings while there's still time to act on them. Annual reports and final reports matter, but project teams shouldn't have to wait until the end of a grant to learn something important. Evaluation shouldn't run on a separate track from the program. It should create a cycle that happens throughout the life of a project: program, data, learning, action, and program improvement.
More data isn't necessarily better data, either. We would rather have a focused evaluation that produces useful evidence than an enormous evaluation plan that creates survey fatigue, burdens project staff, and generates data no one has time to use.
