Article chapter 07 of 08
Make the controlled release decision
From Evaluating an AI workflow before customers depend on it
Write the release criteria before the final run. That way it's harder to reinterpret a disappointing result afterwards. The criteria should cover task risk, critical failures, review effort and the controls you'll have during the release.
I wouldn't lean on one combined percentage. A high overall number can hide a complete failure on a small task family that happens to be consequential. Set minimums per family where the risks differ. Say which critical failures block release regardless of the overall result, and include requirements for permissions, logging, recovery and reviewer capacity.
A controlled release decision should name:
- which users and task families are included
- any inputs, actions or source collections that are excluded
- the approved model, prompt and workflow versions
- the evaluation set and result you're using as evidence
- unresolved failure modes and the controls on them
- where human review and approval are required
- the monitoring signals and who'll respond to them
- the condition for pausing or rolling back
The first release should keep exposure manageable. Limit the task boundary, the eligible users or the volume, wherever those limits reduce the damage an unseen failure could do. And enforce the limitation in the application itself, because a note in the release plan won't stop an unsupported task reaching the model.
Go through the remaining failures with whoever is responsible for operations, security and the affected business process. The person who'll have to deal with a failed action should confirm that the logs give them enough to work with and that the correction path is actually available, since product approval on its own doesn't check either of those.
Write the decision down, including why you accepted the risks you did. That gives you a baseline for later changes, and it stops a good demo from turning into informal permission to widen the workflow past the boundary you evaluated.