How do I find out what data my funder requires?
To find out what data your funder requires, read your solicitation, your award terms, and the funder's reporting template, if it has one. Often the funder doesn't require a type of data but rather wants an answer to a question.
NSF's Advanced Technological Education program is one example. Its solicitation asks the evaluation plan to identify the specific data sources, data collection tools, and methods the evaluation will use, which leaves those choices to the project team and its evaluator. If your funder does name particular measures or participant information it wants reported, they'll be in those documents. If something is unclear, ask your program officer before you design around a guess.
Which data do we already have that we can use?
The data you already have that you can use for a grant evaluation may include participation and attendance records, institutional data, applications, and surveys you already run. Start with what exists, check whether it answers any of your evaluation questions, and collect only what's missing.
Using records you already keep can still take some setup. Institutional data held by your college or university may need a data use agreement before an outside evaluator can work with it, and some uses may need review by your institution's IRB. Does my program evaluation need IRB approval? covers that question.
Making sense of years of data you've already collected, outside a grant's requirements, is a different question. Are we collecting the right data to show that our program is working? and Do we need new surveys, or can we use what we already collect? take it up for any organization.
Do we need both surveys and interviews?
You don't always need both surveys and interviews for a grant evaluation, because each evaluation question decides the method. Numbers show patterns and change, and interviews or focus groups help explain why.
For example, a survey might show that participants report lower satisfaction with one part of a program, and interviews or focus groups might help explain the reasons. Records might show that one group completes the program at a lower rate, and feedback from participants might reveal barriers the numbers alone can't show. Used together, the two can give a much more complete picture of a program. But if a question can be answered well from participation records, adding interviews may simply create more work for the people you ask.
The Grantibly perspective
Lauren's perspective
It's easy for evaluation plans to grow. Add another survey. Add another measure. Ask another group. Every additional measure creates work for participants, project staff, and evaluators. We'd rather have a focused evaluation that produces useful evidence than an enormous evaluation plan that creates survey fatigue and generates data no one has time to use.
One of the most common mistakes in evaluation is jumping straight to data collection. "We need a survey." Maybe. But first: what are we trying to understand? Once we know what a project needs to learn, we can determine what evidence would answer that question and the best way to collect it. Sometimes that's a survey. Sometimes it's an interview, a focus group, participation records, observations, institutional data, or a combination of methods.
