AWS’s nine-stage method tests AI agents before and after deployment

A production AI agent can stay online and still become less useful after a prompt, tool or data change. AWS has published a nine-stage evaluation workflow designed to catch those silent regressions before and after release. The method comes from an internal field-sales system, so its reported improvements are company results rather than an independent benchmark.

One broken release exposed the monitoring gap

The AWS case study describes a multi-agent system that routes work among specialists for performance, customer information and text-to-SQL data queries. In preproduction, related data, tool and prompt changes arrived out of sequence. The pass rate fell from 78% to 38%, showing that component-level tests had missed a system-level failure.

AWS’s answer is a lifecycle rather than a single score. Teams first define acceptable behavior and compare versions, then run small regression suites at code merge and a complete “golden set” before deployment. A golden set is a versioned collection of representative questions, expected outcomes and known failure cases. After release, canary checks, adversarial tests, drift monitoring, live dashboards and user feedback continue the same evaluation loop.

Measure tools, specialists and the complete agent

The framework scores three layers: individual tool calls, specialist agents and the combined workflow. It covers safety, answer quality, speed, cost and operational control. Deterministic checks are used where a correct value is known; model-based judges assess qualities such as helpfulness. AWS’s AgentCore Evaluations documentation says these evaluators can score unified traces from supported agent frameworks, including custom criteria.

For the live system, AWS says it sampled 10% of traffic in 30-minute windows and alerted only after three consecutive threshold breaches. That rule aims to expose persistent degradation without paging on every noisy result. In one example, a customer-identification score slipped from 98% to about 80% because real queries contained ticker symbols, shorthand and misspellings absent from offline tests.

The useful lesson—and its limit

AgentCore Observability sends OpenTelemetry-compatible traces and metrics to CloudWatch, according to AWS documentation. AWS reports that this combination cut detection time from days or weeks to under four hours. It also attributes improved grounding and fewer hallucination events to one prompt change.

Those numbers describe one AWS environment; the post does not establish the same gains or operating cost elsewhere. The transferable idea is architectural: test known cases before release, sample real behavior afterward, and preserve traces so a falling score can be connected to the exact tool, prompt or routing step that changed.