Nine Lines of Verification That Beat a Six-Agent AI Swarm

A single agent beat 1,200 lines of orchestration by 13× on wall clock. That result had a hole in it, and it took another day to find: I had only ever measured orchestration on the strongest model available.

Of course a review panel adds nothing on top of a model that doesn't need review. The honest question isn't "does orchestration help?" — it's "does orchestration rescue a weak model?" If a framework could take a model from 19/22 to 22/22, then orchestration's domain is exactly the models that can't go it alone, and my headline was an artefact of testing only the model that made review redundant.

That's a single cell in a 2×2, and it decides the whole thing.

The 2×2

All runs from the same 17/22 baseline with a 13/13 holdout:

The only run to reach the full contract is the only one where the gate engaged. It tried to stop 52 times, was refused 51, and finished 22/22 in 72 minutes and 197 steps — beating the six-agent panel on score, wall clock and cost, using one agent and nine lines instead of 1,200.

The deciding cell landed at 19/22. The six-agent orchestration did not rescue the weak model. Nine lines of verification did.

What the nine lines do

When the agent declares it's finished, run the criteria. If any fail, refuse the stop and hand back the failures.

That's it. No reviewer, no panel, no synthesis, no adversarial second model. The only reason it can exist is that the contract runs in two seconds — at 28 seconds this would be unaffordable at 52 invocations.

The gate has no opinion about the code. It doesn't review, doesn't suggest, doesn't reason. It compares a claim against a contract, and the only thing it contributes is refusal.

The ungated row is the part worth staring at

Without the gate, the same model announced "successfully implemented all required functionality" — at 12/22, with 7 of 13 holdout probes broken. It had deleted a field the application depends on, and every request that touched it now raised KeyError: 'due_date'.

It was not lying. It had no way to know. It was asked to judge its own completion and it did what anything does in that position: consulted its own account of its work and found it satisfactory.

This is not a prompting problem. You cannot instruct your way out of it, because the information that would change the answer isn't in the context at all. It's in the running application, and only something that runs the application can go and get it.

Corroboration from an uncomfortable direction

While writing this up I went looking at how production coding agents handle the same problem, and found that a widely-used commercial agent architecture has no verification gate at all — all verification is left to the orchestrating model. No reviewer agent, no adversarial panel.

I had built 1,200 lines of exactly the layer the reference implementation deliberately omits, and then found that the nine lines it also omits were the ones that mattered. I don't think that reflects well on either of us, but it does suggest the gap is real rather than idiosyncratic to my setup.

Two caveats that travel with this

I want these attached to the finding permanently, because it's the sort of result that gets quoted without them.

It is n=1 on the engaged condition. The other two gated runs never declared DONE at all, so their 20/22 says nothing either way about the gate. One run reached the phase where the mechanism could fire. One.

The weak-model row ran at the wrong sampling parameters. A per-model profile store existed and nothing read it, so every model ran at a single temperature that matched almost none of them. That's its own write-up, and it means these numbers need re-running before they're load-bearing.

I nearly made a third error on top of both. Mid-run I wrote that the gate "fired once in six runs, so it's variance" — a count taken before the one run that mattered reached the phase where it began declaring DONE. A snapshot generalised into a structural claim, and it inverted the actual finding.

What I'd tell a team

Never let an agent be the judge of its own completion. Not because it will lie, but because completion is a claim about the world and the agent only has access to its own account of the world.

And when you're deciding what to build, ask which layer is doing the work. I built decomposition, ownership, scheduling, review, synthesis and gating — and the thing that produced the result was the smallest and least interesting of them. Fan-out was the part that felt like engineering. Refusal was the part that worked.

What's next

The mechanism question is settled enough for now. The one I still can't answer is the one I bought the hardware for: what does a fix actually cost on machines I own, and how many agents can I run at once before that number stops falling?

I have the measurement. It's better than I expected, and it changes which model I'd choose.