In short: Whether an evaluation needs IRB review depends on what's asked, of whom, and what happens to the answers, and the ethical obligations hold either way.
Studying the people in a program comes with obligations. These pages cover when IRB review applies, common ethics mistakes, and evaluating responsibly.
A fundable evaluation section names the questions, the design, a defensible budget, and the evaluator, and it's written while the proposal is still being built.
A grant evaluation runs across the whole award, from the first question asked to the funder's annual report, and the plan can change when the program does.
An evaluation that finds a program isn't working is still useful, and there are honest ways to report it that protect the program and the funder relationship at once.