27 November 2025 / Applied AI / 8 chapters

Treat partial failure as an ordinary operating state

From Self-hosting an AI agent: architecture, observability and recovery

The agent interface loads, accepts a request and displays a working indicator. Behind it, the queue is no longer dispatching jobs. Another installation may still run model calls while memory retrieval times out. A third can prepare an action, then discover that its tool credential expired overnight. In each case the application process is alive and the task is stuck or working with missing context.

This is the starting point for self-hosting. A process check proves that one process answered. It says little about the path a task takes through storage, scheduling, model calls and tools. The system needs to report the state of that path, including degraded states where only some work remains safe.

Write down a handful of task types before drawing the deployment. A read-only research task has a different path from a scheduled task that uses durable memory, and both differ from a task that waits for approval before changing an external record. For each type, identify:

  • where the request first becomes durable;
  • which stores and services it must reach;
  • which credentials it uses;
  • where work can wait;
  • which effects can escape the host;
  • what evidence marks completion.

That exercise finds weak dependencies quickly. If the memory service is optional for one task, the agent can continue while stating that memory was unavailable. If current memory is required, the task should stop with a dependency error. Quietly substituting an empty retrieval result makes an outage look like a genuine absence of information.

Give every dependency an owner and a failure policy. The policy should state whether new work is rejected, queued, run in a reduced mode or sent for operator review. Avoid one global healthy flag. A status page that says the system is degraded because scheduled work is delayed is far more useful than a green badge beside an interface that cannot complete its advertised jobs.

All articles