27 November 2025 / Applied AI / 8 chapters

Make task state survive worker restarts

From Self-hosting an AI agent: architecture, observability and recovery

A worker can stop after almost any step: after the model responds, while a tool is running, after an external request leaves, or just before it stores the result. Durable task state needs to explain what another worker may safely do next.

Create a task_id when the request is accepted and a run_id for each execution attempt. Store the task objective, authenticated actor, scope, creation time, current status and required dependency class. A practical status set might include queued, running, waiting_for_approval, waiting_for_dependency, completed, failed, cancelled and unresolved. Keep the transition history rather than replacing it with the latest label alone.

Workers should claim work with a lease. The lease has an owner, token and expiry. Completion is accepted only from the current lease holder. If a worker disappears, the scheduler waits for expiry and then evaluates the task's last durable step. It should not assume that every expired lease is safe to run again.

External actions need a stable action identity. Create the action record before sending the request and use its ID as an idempotency key when the destination supports one. After a timeout, reconcile through that key or the destination's object reference. A fresh request could duplicate an email, job or record when the first request actually succeeded.

Keep waiting states out of active worker capacity. A task waiting for human approval or a repaired dependency should release its worker lease and retain a durable wake-up condition. The scheduler can resume it when an approval event arrives, a dependency changes state or a review time is reached. Polling loops inside workers waste capacity and disappear during restart.

Cancellation also becomes stateful. Stop new model and tool calls, mark requests that may still be in flight, and reconcile their outcomes. The interface can then distinguish cancellation_requested from a task whose external effects have all been accounted for.

All articles