22 September 2026 / Agent operations / 8 chapters

Changes you can't just undo

From Recovering an automated workflow after a partial failure

If an external record was created with the wrong values, recovering the workflow might mean making a correction in that system. Deleting the local run won't remove the remote record or undo anything that's already used it.

Write down which actions can be reversed and which need a separate compensating action. Releasing a reservation, cancelling a draft and issuing a correction all have different business consequences, and what you're allowed to do depends on the system and your organisation's rules. An email that's already been delivered can't be treated as if nobody got it.

A compensating action should have its own approval where that's needed, its own operation identifier and a result you can check, and it should be linked to the action it's fixing. Compensation can fail too, so it gets the same uncertain-result handling as the original operation.

Before you restore an old value, check whether anything has changed in between. Someone might have fixed the destination record by hand after the failed run, and blindly writing the earlier snapshot back would wipe out their work. Where the destination supports it, use its version or conditional-update mechanism to catch that conflict. Otherwise compare against the current record and ask for a review when it no longer matches what the recovery assumed.

Keep recovery limited to the affected case or a known batch. If an integration is still producing bad writes, pause that write path through the approved operational control and keep the evidence. Record which work you're holding so it can be accounted for when things resume. Restarting everything at once makes it harder to tell the original problem apart from the new attempts.

Before any destructive correction, have the responsible person look over the affected records and the recovery plan, and take a backup or export where the system lets you. I'd put that approval requirement in the runbook at the exact step where the operator is about to need it.

All articles