How does shared measurement end up flattening local differences?
Shared measurement ends up flattening local differences when the design keeps only what every site has in common and the results are read as one number. A common measure has to fit every site, so it tends to capture the most general part of the work, and the outcomes a particular site was built to achieve may never be reported. When results are then combined into a single portfolio figure, a site doing exceptionally well and a site that's struggling can cancel each other out, and neither one is visible.
Comparing sites on a shared number without context can flatten them in a different way. Sites serve different people, under different conditions, with different resources, and a lower result can reflect any of those, so an outcome that falls short at one site doesn't automatically mean the approach was ineffective there. The program may not have been fully implemented, participants may not have received enough of it, or local conditions may have changed.
How do we keep what makes each site different in the picture?
You keep what makes each site different in the picture by collecting context alongside the shared measures and by looking at each site before you look at the whole. In practice, that means:
- A short shared list, plus measures each site chooses. The shared measures give you a common thread, and the local ones capture what each site is there to do.
- Information about implementation. Who participated, who didn't, how much of the program they received, and what changed along the way at each site.
- What people say as well as what the numbers show. Interviews, focus groups, or open-ended survey questions can explain why a result at one site differs from another.
- Results broken apart by site, and by groups within sites, before anyone combines them.
- Interpretation with the sites. Share each site's results with the people who run it, and ask what they think explains them before the portfolio account is written.
None of this has to mean more data collection. Much of the context already exists in grantees' own records and reports, and using it first keeps the burden proportionate.
Should we compare sites with each other?
You should compare sites with each other when the comparison helps you learn, and be careful about comparisons that turn into rankings. A difference between two sites is a question worth asking: what's different about the people served, the way the work is carried out, or the conditions at each one? The answer can tell you more about what makes the approach work than either site's result on its own.
A comparison used to judge or rank grantees changes how they report. Organizations that expect to be ranked have understandable reasons to report cautiously, and the relationship between funder and grantee can suffer as a result. Before you share any comparison, tell grantees what the measures are for, who will see the results, and how they'll be used. What's the difference between grantee support and compliance monitoring? covers where that line sits.
The Grantibly perspective
Lauren's perspective
We are sociologists first. We naturally think about individuals, groups, organizations, systems, context, relationships, and patterns, and we look at groups and patterns: who is having different experiences, and where.
When we evaluate a program, one of the questions we ask is whether participants at different sites had the same experience. If outcomes fall short, that doesn't automatically mean the underlying idea was ineffective. The program may not have been fully implemented, participants may not have received enough of the intervention, external conditions may have changed, or the model may need refinement. Likewise, positive outcomes don't automatically tell us what produced them.
