How do we take stock of the data we already have?
To take stock of the data you already have, make a simple inventory before anyone begins analyzing it. For each set of data, record what it is, which years it covers, whom it was collected from, what information it requested, and where it's stored. Include the records your organization may not think of as data, such as attendance and enrollment logs, intake or application forms, and staff notes, alongside the surveys.
An example of a one-page data inventory
| What it is | Years | Collected from | What it asked or recorded | Where it lives |
|---|
| End-of-program survey | Several grant cycles | Participants who completed the program | Satisfaction and self-reported change | A shared drive, one file per year |
| Attendance records | Each program year | Every participant | Sessions attended | The program database |
| Intake forms | Each program year | Every new participant | Background and goals | Paper files and spreadsheets |
The rows here are only an example, and your own inventory will reflect your programs. A completed inventory often reveals that some data sets can be connected, such as attendance records with survey responses, and that others were collected in ways that limit what they can demonstrate.
Which questions should we try to answer first?
The questions to answer first are the ones connected to a change your organization could realistically make or a conversation it has to have soon, and that your existing data might be able to address. A question such as "Are participants who attend more sessions more likely to complete the program?" can often be explored with records you already hold. A question such as "Why do participants leave before finishing?" usually can't be answered from attendance records alone, and recognizing that limitation early helps your organization determine what additional information it would need.
Once you have a result, continue questioning it: what it means, why the organization might be seeing it, whether it confirms what the team expected or challenges an assumption, whether there's an important difference between groups of participants, and whether it suggests something the program should change. A number becomes useful when your team can answer those questions about it and decide what, if anything, to do differently.
What can years of old survey results tell us, and what can't they?
Years of old survey results can tell you how the answers of the people who responded changed over time, as long as the same questions were asked in a similar way. They can't tell you much when the wording changed from one year to the next, when different groups were surveyed in different years, or when only some participants responded, because then a change in the results may reflect a change in who answered or what was asked instead of a change in the program.
End-of-program surveys have a further limitation, because they usually reach only the participants who finished, and the experiences of people who left early may be the most important information missing from the folder. Surveys also record what people reported without usually explaining why they reported it, which is where conversations with participants and staff can contribute.
If your organization is an employer trying to make sense of HR and payroll records instead of program surveys, the starting point is similar but the data is different. We have plenty of HR data and still can't answer basic questions. Why? covers that case.
Is it worth bringing in an evaluator to make sense of years of data?
Bringing in an evaluator to make sense of years of data is worth it when the questions matter, the data is more than your team has time to analyze, or interpreting it responsibly requires methods your team doesn't use every day. Making sense of data an organization already has, deciding what additional information it needs, and identifying what is working and what could improve are central parts of an evaluator's work.
Before you sign an agreement, describe the condition of the data to the evaluator honestly. Cleaning data takes time, and if an evaluator plans and budgets to start with a clean data set and then learns the data needs substantial cleaning, the scope and the budget will need to change. Will an evaluator judge us if our data is a mess? answers the question many organizations ask first.
The Grantibly perspective
Lauren's perspective
Data don't make decisions. People do. Our job is to help organizations turn information into understanding and action. A percentage in a table may be technically accurate but still leave a project team asking, "So what?" We want to go further: what the result means, why we might be seeing it, and what it suggests the program should do differently.
We also want clients to understand their own data. That means explaining what we're doing and why, making results accessible, and helping teams develop stronger evaluation thinking over time. The goal isn't to turn every client into an evaluator. It's to help them become stronger users of evidence.
