Article chapter 02 of 08
Keep uncertain results open for reconciliation
From Recovering an automated workflow after a partial failure
A timeout tells the caller that it did not receive a response in time. It does not establish whether the destination performed the action. Record that result as uncertain until there is enough evidence to resolve it.
Malcolm Featonby's explanation of idempotent APIs in the Amazon Builders' Library describes this problem using a resource-creation request. The caller has to determine whether the resource already exists before risking another creation. The same issue applies to a document workflow that has lost the response containing its new record identifier.
Use the destination's supported lookup mechanism. It may allow a search by the caller's reference, a request-status lookup or a read using an identifier recorded before the timeout. Confirm the account and environment as part of that lookup. An empty result from a different account is no evidence about the original operation.
Also check the consistency behaviour of the lookup. A newly created record may take time to appear in a search index. If that is how the destination works, one immediate empty search cannot establish absence. Follow the provider's documented behaviour and leave the case unresolved when the available evidence is insufficient.
Reconciliation should produce a usable finding: the intended record exists and matches, it exists with conflicting values, absence is established within the destination's contract, or the result remains unknown. Save the observation time and source with the finding. A later operator should be able to distinguish a confirmed result from someone's suggested explanation.
An agent can help collect those observations and prepare a short case summary. Keep the evidence links beside its conclusion. If it cannot determine whether the action happened, let it report that uncertainty and assign the case for review. A confident paragraph cannot supply a missing destination record.