Putting an agent into production

An agent that performs well in a demo has been tested against the questions we thought to ask it. Production traffic asks the other ones. This note sets out the checks our engineering team runs before an LLM agent is allowed to touch real users or real data.

We write the evaluation set before we write the demo

The first artefact on an agent project is an evaluation file. We collect 80 to 150 real cases from the client's own material: support tickets, filed documents, historical queries and the answers given at the time. Each case sits in version control with the input, the behaviour we expect, and the rule that grades it.

Most cases are graded without a model in the loop. A citation check compares document identifiers against a labelled set, structured output is validated against its JSON schema, and a refusal case asserts that the agent declined and routed the request to a person. Free text gets a model grader with a rubric, checked against hand labelled examples. The suite runs in continuous integration, and a frozen holdout set runs before release.

We measure retrieval quality apart from model quality

When an answer is wrong, the first useful question is whether the agent was given the right material. We test the retriever on its own against labelled query and document pairs and report recall at the cut-off used in the prompt. Chunk size, overlap, the embedding model, hybrid search and a reranking pass are each measured against that set.

Every generation case then runs twice, once with the live retriever and once with the correct passages injected by hand. If the answer is right with the gold passages and wrong with the live ones, the work belongs in the retrieval pipeline rather than the prompt.

Tools are typed, scoped and validated on the server

Every tool the agent can call is a typed function with an explicit schema, and read tools are separated from write tools at the interface. The agent runs under a service account with access to the tables, indexes and endpoints its task requires and nothing further. The tenant identifier comes from the authenticated session on our server, never from the arguments the model produced.

• Write operations carry an idempotency key, so a retried call cannot post the same record twice.
• Operations that change client records return a proposed change for approval until the workflow has earned more autonomy.
• Arguments are validated on the server against the schema and against business rules, because a model is an untrusted caller.
• Each session has a call budget, and the agent loop stops after a fixed number of steps.

A named reviewer and an audit trail

The first release usually runs in suggest mode. The agent drafts, a named reviewer in the client team accepts, edits or rejects, and the decision is recorded beside the draft. It also gives us labelled data about where the system is weak.

Each run writes an immutable record: prompt version, model identifier, the identifiers and hashes of the retrieved passages, every tool call with its arguments and result, the final output, the reviewer decision, and timestamps. Retention follows the client's own records policy.

If a decision cannot be reconstructed six months after it was made, we treat the system as unfinished.

Cost and latency budgets are set per request

Before we build, we write down what a request may cost and how long it may take. An interactive answer might carry a 95th percentile target of six seconds and a ceiling of a few cents, while a queued document task might be allowed ninety seconds. Those figures are agreed with the client and then instrumented.

We record input and output tokens per step, latency for each tool call and for retrieval, and cost per completed task rather than per model call. Embeddings and stable prompt prefixes are cached, and we load test at the concurrency the client expects.

What the agent does when something fails

We define the behaviour for each dependency failure and test those paths alongside the successful ones.

• Retrieval returns nothing above the relevance threshold: the agent states that it cannot answer from the available material and routes the request to a person.
• The model returns malformed structured output: one reparse attempt with the schema error appended, then a clean failure.
• A tool times out or errors: a bounded retry with jitter, then a message naming the unavailable system.
• The provider is rate limiting or unavailable: requests move to a secondary provider or a queue, and interactive users are told rather than left waiting.
• Error rates cross a threshold: a circuit breaker opens, and a feature flag disables the agent for a tenant without a deployment.

Observability and replay of a failed run

Every run emits a structured trace with a span for each retrieval, model call and tool call, tied to a correlation identifier that is visible in the interface. A reviewer who reports a poor answer gives us that identifier.

Our replay harness runs a stored trace again against the current prompt and model with the same retrieved context, so a fix is demonstrated on the input that failed. Alerts watch refusal rate, tool error rate, 95th percentile latency, daily cost and reviewer rejection rate.

A failed production run becomes a case in the evaluation set before anyone edits the prompt that caused it.

Hand-over and the runbook

The system belongs to the client team, so hand-over is engineering work with its own acceptance criteria. The runbook lives in the repository beside the code: architecture and data flows, where secrets are held, how to run the evaluation suite locally, how to roll a prompt version forward and back, how to disable the agent, and a decision tree for each alert. It also states the limits plainly, including the query types the agent does not handle and the document formats it parses poorly.

We then run two working sessions with the client's engineers, walking through a real incident from the correlation identifier to the replayed run to the fix. They deploy a change and roll it back themselves while we watch. Hand-over is complete when an engineer on their side has run the evaluation suite, shipped a prompt change, and cleared an alert without calling us.