Thumbnail for What Will the Agent Check Its Result With?

What Will the Agent Check Its Result With?

Published: 2026-11-21

What Will the Agent Check Its Result With?

Before handing a task to a coding agent, I'd first check what the agent can verify its result with. When the answer is "nothing", I'd make building that check the first task and do the actual change after it.

In a project I worked on, the agent had a local snapshot of real data and scripts that simulated the process. It could try an idea, run it and report what actually happened. It worked on a static copy, without access to the live production database. We also wrote tests around business rules, and before changing existing behaviour we wrote characterization tests that recorded what the system did at that moment. Some problems looked fine in the code and showed up only when the solution ran against real data.

Below is an example from AWS that shows the same idea at a much bigger scale, and a list of things I'd give an agent when a project has nothing for it to check against.

Four nights of supervision, then about two hours

Swami Sivasubramanian from AWS described this in a guest post on Werner Vogels' blog (11 November 2026). He was rebuilding the replication engine of Alf, a Paxos-based lock service, with an agent. The original build had taken about four months. He supervised the agent for four nights before he saw that it had no way to check its results. Once it had a spec and a test environment, the rebuild took about two hours.

It's one person's story, told by the company that built the tool, so I treat it as one example. From what I've seen, when the agent can't run anything that tells it whether it's right, the person supervising it ends up checking every step by hand.

AWS Emulator: the test is checked against the real service first

The same approach is behind AWS Emulator, a local copy of more than 200 AWS services written and tested by agents. The agents start from AWS's public API models, written in Smithy, and write end-to-end fidelity tests from them. First they correct each test until it passes against real AWS, so the test records how the real service behaves. Then they build the emulator until it passes the same test. When a test fails, it points to a gap in the emulator.

Every change also goes through a merge gate: automated checks, several security agents doing adversarial reviews, a general code reviewer, and deterministic checks that stop agents from changing code they shouldn't touch.

The post is open about the limits. The internal version came together over a weekend, and making it robust took much more work. The repository has a things-that-do-not-work folder that lists behaviour that doesn't match real AWS yet.

For me the most useful part is the order of steps. The test is checked against the real system before anyone builds the thing that should pass it, so the agent always has a target that reflects reality.

Swami also mentions how the same team tested Alf about twenty years ago. They ran the servers inside a JVM with a simulated network, injected faults like dropped packets, and used TLA+ for formal reasoning. Most concurrency bugs were gone before testing on real servers. The way I read it, agents can now take on building this kind of test environment, which used to be a big piece of work in itself.

It works when there is something to compare against

The AWS approach depends on having a reference: the real cloud and a machine-readable description of every API. Other references could be an existing implementation, a spec the agent can turn into checks, or a snapshot of real data with known results. Without any of these, the agent has nothing to get closer to. It writes code that looks right, and someone has to check it by hand.

In my project I also learned to look at where the expected values in a check come from. If the test and the implementation share the same wrong assumption, they agree with each other and the test passes anyway. So I'd always ask where the expected values come from: the real system, source data or a rule someone confirmed.

What I'd give the agent as the first task

When the agent has nothing to check its result with, I'd make one of these the first task, before the actual change:

  • A test against the real system. For an integration, a test that runs against the real service, on a test account or in a sandbox, and records how it actually responds. Once it passes, the agent can write the code.
  • Characterization tests for existing code. Before changing behaviour, tests that record what the system does today. They may record bugs too, and they show when the agent changes something it shouldn't.
  • A small test environment. A local snapshot of real data plus a script that runs the process end to end, so the agent can run an idea and look at the result.
  • A fake checked against the real thing. If calling the real system is slow, expensive or risky, a fake that passes the same tests as the real system. This is the AWS Emulator approach on a small scale.
  • A script that calls the endpoint. For a change in an API, sometimes a simple script that the agent runs after each change is enough.
  • A spec the agent can read. An API model, a schema or a list of business rules written so that the agent can turn them into checks.

I'd tell the agent the order directly: first build the check and show that it gives the right result against the real system, then implement the change.

These are my suggestions, and how far to go depends on the project. A one-off script probably doesn't need a fake service. For code the agent will change many times, I'd expect the check to be worth the extra work.

Check verification before changing the model

Addy Osmani gives similar advice in a post on claude.dev about choosing a model (9 November 2026). When a model isn't good enough for a task, the first step he lists is to check that the model can verify its own work, with a test, a build or a script that calls the endpoint. After that you raise the effort one level, and then you move to a bigger model.

I like this order. Switching to a bigger model is the easiest step to take, and I think it helps little when the agent can't tell whether its result is right.

The part that stays with the engineer

Swami describes the engineer's job with agents as giving them room to work on their own and building the specs, tests and merge gates that decide when the work is correct. That matches how I'd split the work. The agent can write a lot of the code and a lot of the tests. Deciding what counts as correct, and where the expected values come from, stays with the person who runs the project.

So before the next task I hand to an agent, I'd ask what it will check its result with. If the answer is nothing, building that check is the first task.

Back to Blog