Do we need clean data before we hire an evaluator?
You don't need clean data before you hire an evaluator. If you've collected information but haven't cleaned it, organized it, or analyzed it yet, an evaluator can do that work with you when it's included in the scope of work. Many organizations are in exactly this position because their staff are busy delivering services every day, and keeping data tidy usually comes second to the work itself.
An evaluator can also help you determine whether the information you're collecting fits the questions you want answered and suggest improvements once they're familiar with it. Reviewing the evidence you already have is usually the right first step before anyone collects anything new, and it can show that you have more to work with than you thought. We have years of data and no idea what it tells us. Where do we start? covers that situation in more detail.
What does an evaluator need from us at the start?
At the start, an evaluator needs to understand what your organization wants to learn and how your program works. That usually means:
- what you want to understand, decide, or communicate, such as whether a program is reaching the people it's meant to serve or what to tell your board or funder
- the documents that describe the program, such as a grant proposal or award narrative, a logic model or program framework if you have one, and a timeline of key activities
- any reporting deadlines you already have with funders
- the names of the people involved and one main contact for routine communication
- a description of the data you have, where it's kept, and who can share it
If your evaluation involves participants' personal information, the evaluator will also talk with you about privacy, any IRB review your organization or funder requires, and any agreements needed to share data. None of this has to be polished, and a good evaluator will help you sort out what's missing.
How much detail should we give an evaluator about the condition of our data?
Give an evaluator enough detail about the condition of your data to plan the right amount of time: what data you have, where it lives, and what state it's in, shared honestly before you sign an agreement. Cleaning and organizing data takes time. If an evaluator plans and budgets for a dataset that's ready to analyze on the first day, and then learns after the agreement is signed that it needs substantial cleaning, the scope and budget will need to change, and that isn't fair to your evaluator.
Being specific helps. Saying that attendance was tracked on paper for two years and in a spreadsheet since then, or that a survey was given every year with slightly different questions, lets the evaluator plan the right amount of time. Those details aren't a reason to wait. They're what the evaluator needs to give you a realistic plan.
Is there anything worth doing before we reach out to an evaluator?
Before you reach out to an evaluator, it's worth gathering what you already have and talking as a team about what you most want to learn. You don't need to fix anything first. A short list of where your program's information lives, how complete each source is, and two or three questions your leadership, board, or funder keeps asking is enough to start a useful conversation.
What isn't usually worth doing first is designing and launching a new survey on your own. It's easy to collect information that doesn't answer the questions that matter, and an evaluator can help you decide what's worth collecting before your staff and participants spend time on it. If you're not sure your organization needs a full evaluation, When is a full evaluation more than we actually need? helps you decide, and Can we evaluate our own program without hiring anyone? covers doing some of the work yourselves.
The Grantibly perspective
Angelika's perspective
Organizing data is one of the biggest challenges that organizations face, because they're too busy doing the good work that they do every day. And rightly so.
There's a reason organizations hire people to do the kinds of things they don't know how to do, especially data analysis and organizational improvement, and that reason is that they aren't evaluators. Leave the evaluation work to us. You're in good hands. Go do what you do best, and let us take care of making sense of the data you already have, collecting the data you need, learning what is working great and where your areas of opportunity for improvement are, and helping you make improvements.
