Article chapter 08 of 08
Exercise the gaps between successful steps
From Recovering an automated workflow after a partial failure
Use a test environment with controlled inputs and destinations to interrupt the workflow at its consequential boundaries. Test recovery by inspecting stored state and destination records. A worker returning successfully is only one part of the result.
For the example workflow, useful cases include:
- the destination accepts a write, but its response is lost;
- the response arrives, but the worker stops before saving completion locally;
- a duplicate event arrives during recovery;
- two workers attempt to claim the same case;
- the approved input changes while a case is waiting;
- a notification fails after the external record is confirmed;
- the destination lookup is unavailable or temporarily incomplete;
- a compensating action fails after it has been authorised.
For each case, state the expected destination records and the remaining work before running the test. Inspect for duplicates as well as missing changes. Check that an uncertain result remains visible, an unapproved change cannot proceed, and another operator can recover the case using the saved evidence.
Exercise the delayed path too. An immediate retry may fall inside a provider's deduplication window while next week's manual recovery does not. Use the provider's documented limits to decide what your own records must retain and when a fresh lookup or approval is required.
Finally, compare the recovered result with the source and the approved proposal. The note on checking an integration after the job says it finished describes that read-only comparison. Keep unresolved differences assigned and visible, even if the execution itself has finished.
Before enabling recovery for live work, have another authorised operator run the lost-response test from the procedure. Ask them to identify the existing destination record, complete only the outstanding steps and show the final comparison. Keep their result with the workflow release record, including any part that still requires manual investigation.