What do we need to be clear about before we can tell whether a change helped?
Before you can tell whether a change helped, your organization needs to be clear about which problem the change was intended to address and what you expected to be different afterward. That sounds obvious, and it's a step that is often skipped. Organizations often make a change because something felt wrong, and several months later nobody can say exactly what the change was supposed to improve.
Three things to write down
- The problem. Describe what the organization was experiencing and where it was occurring. For example, new employees on second shift were leaving within their first 90 days.
- What you expected to be different. Identify which numbers or employee experiences should look different if the change is working, and keep the list short.
- Where you started. Record what those numbers or employee experiences looked like before the change was made. This gives you a starting point, or baseline, that you can compare with what happens afterward to see what changed.
If you're reading this before making a change, document these three things now. It doesn't take long, and it makes every later comparison easier. If the change is already in place, the next section describes what is still possible.
What if we didn't measure anything before we made the change?
If you didn't measure anything before making a change, you can still learn a lot about whether it helped, and many organizations are in this position. Your organization likely has records from before the change, even if nobody collected them for this purpose. Turnover, absences, overtime, schedule changes, and exit conversations are all places to look for information that can help you understand what things looked like before the change and what happened afterward.
Next, ask the employees who experienced the change what has been different for them since then. They can tell you whether anything is different for them, what has improved, what has stayed the same, and what the change didn't reach. The people closest to the work often know things the data can't tell you on its own.
Be honest about the limits of what you can conclude from the evidence. Looking back is less reliable than planning ahead, and you may only be able to say that conditions moved in the right direction. That's still worth knowing, and it's worth saying plainly when you share what you found.
How do we separate our change from everything else that happened?
You usually can't completely isolate the effects of your change from everything else that was happening in your organization, but you can make a stronger assessment by identifying what else changed during the same period and which employees did and didn't experience the change. Most workplaces won't be able to answer that question exactly, but several practical comparisons can help.
Comparisons that help
- Before and after, within the same group. Compare the same department or shift before and after the change. When possible, compare similar time periods so seasonal differences don't get mistaken for an effect of the change.
- Groups that experienced the change and groups that didn't. If the change was introduced in one facility or department first, compare its results with facilities or departments where the change hasn't yet been introduced..
- Other factors that could have affected the results A new supervisor, a pay adjustment, layoffs elsewhere in your industry, or a different approach to hiring can also affect the outcomes you're measuring
- Whether the change happened as planned. Consider who actually experienced the change and whether it was implemented consistently. If some employees were affected by the change and others weren't, you need to know that when interpreting the results. And a policy that exists on paper but isn't consistently followed can't be expected to produce the intended results.
That last comparison matters more than it may seem. If the results are disappointing, that doesn't automatically mean the underlying idea was ineffective. The change may not have been fully implemented, or it may have reached some employees and not others. Similarly, when results improve, that doesn't automatically tell you what produced the improvement.
What should we do if the change didn't help, or only helped some employees?
If the change didn't help, or helped only some employees, treat that as information and act on it. A change that helped one group of employees and not another can help you identify differences in their needs, circumstances, or experiences with the change. A more useful question than "Did it work?" is "What worked, for whom, and under what conditions?"
Next, follow up with employees. Explain what the organization tried, what you learned, and what will happen next, whether that means continuing the change, adjusting it, or trying something different. A specific person should be responsible for that communication and for making sure the next steps happen .
If the change you made was a training program, How do I know if our training is worth what we spend on it? covers that case. If you now need to explain the results to your leadership team, How do I show our leadership team this work is worth continuing? covers how.
The Grantibly perspective
Lauren's perspective
Organizations are often eager to know whether something "worked." That's an important question, but it's difficult to interpret results without understanding how the change was carried out. Which employees experienced it, whether they experienced it consistently, and what changed along the way? I taught research methods for more than a decade, and those implementation questions are where I'd begin.
We also don't begin with the assumption that a change was successful and then look for evidence to support that. The purpose is to learn. Sometimes the data shows strong results, sometimes mixed results, and sometimes an approach doesn't work as expected. Finding something early enough to improve it can be one of the most valuable outcomes.
We want leaders to see this kind of follow-up as a source of information rather than a judgment. Learning what happened after a change is part of the process, not an afterthought.
Follow up to determine whether the changes made a difference, and then tell employees what you learned.
