What information does our institution already collect that might answer our questions?
The information your institution already collects that might answer your program's questions usually sits in several offices. Depending on the institution, the registrar or institutional research office may hold enrollment, course completion, grades, persistence, and graduation or credential records. Advising, tutoring, and support services keep their own records, and your staff keep program applications, attendance, and participation logs. Another office may already conduct a campus-wide or district-wide survey.
Records like these can often answer questions about who participated, how much of the program they received, and what happened to them afterward, such as whether they completed a course, continued into the next term, or earned a credential. They're much less likely to tell you what participants experienced, what helped or got in the way, or why a pattern is occurring.
Whether your institutional research office can also conduct the evaluation is a different question. Can our internal institutional researcher do the evaluation to save money? covers the tradeoffs.
When is a new survey worth it?
A new survey is worth it when your program needs to answer a question that no existing source addresses, and when a survey is the best way to answer it. That usually means questions about experiences, perceptions, or reasons, such as how participants experienced a mentoring component or what made it difficult for them to attend. It can also mean the existing information covers the wrong people or the wrong period, for example when a campus-wide survey doesn't identify which respondents participated in your program.
A survey isn't the only option. Interviews or focus groups can explore reasons and context in more depth, and observations or a review of documents may suit other questions. The evaluation question should determine the method. If a question can be answered well from records the institution already holds, a new survey may simply add work for the people you ask.
What makes institutional data harder to use than it looks?
Institutional data can be harder to use than it looks for several practical reasons. Access usually requires permission, and using student or participant records for an evaluation may require a data use agreement and, in some cases, review by your institution's IRB. Offices may define the same term differently, so "retention" or "completion" in one report may not match another. Records may become available only after a term ends, which can be later than your reporting deadline. And when a program serves a small number of students or participants, results for small groups can shift a great deal from year to year. They may also need careful reporting to protect privacy.
None of these is a reason to avoid institutional data. They're reasons to plan for it early, ideally while the evaluation is being designed. Does my program evaluation need IRB approval? covers the IRB question in more detail.
How do we avoid surveying the same students or participants too often?
To avoid surveying the same students or participants too often, coordinate with the other offices that survey them, ask only the questions your program will actually use, and schedule data collection around the academic calendar instead of during its busiest weeks, such as final exams. Sometimes your program's questions can be added to a survey another office already conducts, which reduces the number of requests students receive. A focused set of questions respects people's time and helps avoid the survey fatigue that comes from asking too much.
The Grantibly perspective
Lauren's perspective
It's easy for evaluation plans to grow: another survey, another measure, another group to ask, data collected every semester. We don't collect data simply because it might be interesting, and we don't believe more measures automatically mean a more rigorous evaluation. We also don't assume every program needs the most sophisticated possible methodology.
Sometimes the evidence a project needs comes from a survey. Sometimes it's an interview, a focus group, administrative data, participation records, observations, institutional data, or a combination of methods. An evaluator can also help you determine whether you're collecting the appropriate data for your questions and offer suggestions for improvement once they've familiarized themselves with your data.
