Measure review and exception work

Human review can be a large uncertain cost. Prototypes tend to use attentive project staff and a small set of clean cases, while routine use introduces repeated decisions, interruptions and work that needs specialist knowledge.

Split review into activities instead of applying one time estimate to every case. A reviewer may read source material, verify a proposed field, investigate a conflicting result, correct the output, approve an external action and record a reason. Some cases need only a quick confirmation. Others move to a specialist or back to the requester.

During controlled operation, collect:

• when a case entered and left each review state;
• active handling time where it can be measured without intrusive monitoring;
• wait time separately from work time;
• reason codes for escalation or rejection;
• the role required to resolve the case;
• whether the model output was accepted, edited or discarded;
• whether the correction suggests a source, rule or product problem.

Averages can conceal a small group of expensive cases. Keep the distribution and review the slowest cases individually. They may reveal unsupported file types, ambiguous policy, poor source documents or a task that should never have been automated.

Define the review policy as a cost input. Sampling a proportion of low-risk outputs, reviewing every consequential action and escalating uncertain cases produce different workloads. The budget should use the approved policy rather than assume human review will shrink automatically as the model improves.

Account for reviewer calibration. People need examples, guidance and a way to resolve disagreements. When the method changes, they may need to review new fixtures before working on live cases. This is operating work, especially where consistency matters.

A queue also creates service cost. If required reviewers are unavailable, cases wait or another role is interrupted. Set service targets only after measuring the staffing needed to meet them. When approval is the constrained stage, reducing model response time will not shorten most cases.