Operate freshness, failures and review

Once the service is in use, source owners change documents, permissions move, connectors fail and users discover gaps. Give operators visible state for each of those events.

Monitor the ingestion pipeline by collection and stage. Show the last successful discovery, fetch and index times, plus outstanding failures. A green application health check can coexist with a stale knowledge index, so data freshness needs its own status.

Set service expectations around the source. If a published procedure should become searchable within a defined period, measure the actual source version present in the index. Do the same for revocation and deletion. Technical uptime says little about whether the answer set is current.

Keep a feedback route that captures the question, answer, authorised source references, configuration version and the user's issue. Avoid asking users to paste private content into an unrelated ticket. Feedback categories might include wrong source, outdated source, missing source, unsupported statement, unclear answer and access problem. Route source problems to content owners and system problems to the delivery team.

Prepare for partial failure. If the embedding service is unavailable, new documents may wait while existing search continues. If the permission service is unavailable, the safe response may be to deny retrieval rather than use cached access beyond its approved window. If citation links fail, the interface should not pretend the answer remains fully verifiable.

Review source coverage regularly. Collections accumulate abandoned drafts, duplicates and documents with no owner. Remove or quarantine material through an approved process, then reconcile the index. More documents can reduce usefulness when authority and applicability are unclear.

Before release, confirm that:

• each collection has a business and technical owner;
• current-version and deletion rules are defined;
• extraction quality has been sampled by format;
• access is enforced during retrieval;
• chunks retain source and version metadata;
• ingestion can restart and reconcile;
• evaluation includes unsupported, conflicting and restricted questions;
• citations open the exact permitted source location;
• operational views expose stale and failed items;
• feedback can be traced to the configuration and evidence used.

Run that review again when adding a source collection, changing the embedding or generation model, altering chunking, or widening the audience. For the first implementation, select one maintained collection, write its answer and access contract, and build the evaluation set before indexing anything else.