Two DGX Sparks

Last month I killed an experiment because of the bill, not because it stopped being interesting. I've been chewing on that sentence ever since.

The experiment was dull by design. Point a coding agent at a repository, have it propose a change and verify the change, then do that a few hundred times and see what fraction of the proposals survive. All the value is in the "few hundred times." One run tells you nothing — agents are stochastic and a single result is a coin flip you've mistaken for a measurement. A hundred runs tell you where the model is reliable and where it's guessing.

Every one of those runs is metered.

So I did what everybody does. I ran twelve instead of two hundred, drew a conclusion from twelve, and moved on. That isn't research. That's a sampling strategy chosen by a price list.

The workload is the problem, not the price

What I actually want is agents running continuously over my own codebase — tightening the slow paths, hardening the parts that touch untrusted input. Not a demo. A standing process that goes looking for work.

That workload is inherently wasteful, and it should be. You sweep the whole tree. You re-run the same file with different framing. You generate five candidate patches and throw four away. The ratio of useful output to tokens burned is terrible by construction, because that's what search looks like from the outside.

Per-token pricing is precisely the wrong instrument for search. It prices the tokens you burn, and search burns tokens on purpose. The meter doesn't punish waste — waste is the method. It punishes thoroughness, which is the one property I need.

The tell is that I wasn't optimising the experiment. I was optimising the invoice, and calling it judgement.

Capital instead of consumption

The alternative is to stop renting the compute and buy it. Two DGX Sparks — GB10, 121 GB of unified memory each — run about $9,400 for the pair. Once they're on the desk, running an experiment costs electricity and time. The marginal cost of curiosity goes to roughly zero, which is the entire point.

But "roughly zero" is a feeling, not a number, and I've been burned by that feeling before. So here's the arithmetic I'm actually buying against.

I'm writing off 70% of the purchase against this project — the boxes have other uses and some residual value, and pretending otherwise would flatter the result. Amortise that over 36 months, and assume 200 genuinely busy hours a month.

About ninety-five cents per machine-hour, of which power is four percent. That surprised me. I'd assumed running a box hard meant the electricity would show up somewhere. It doesn't — at these wattages the meaningful cost is the purchase, spread thin.

Which means the whole economic case rests on one number, and it isn't a hardware number.

The load-bearing assumption is me

Two hundred busy hours a month is the assumption everything hangs from. At 200 hours the boxes cost $0.91 an hour. At 20 hours a month — a weekend habit that fizzles by October — the identical hardware costs $9.14 an hour, ten times worse, and I'd have been better off renting.

Buying hardware is not a bet on the hardware. It's a bet on your own consistency. The failure mode isn't that the boxes turn out to be slow; it's that they turn out to be idle. Every hour a Spark sits dark, the effective price of every hour it did work goes up.

That reframes the risk in a useful way. I'm not asking "is this hardware fast enough." I'm asking "will I really run this thing two hundred hours a month," and the honest answer is that I don't know yet.

Cost per hour is still the wrong unit

Here's where I nearly stopped, and shouldn't have.

A machine-hour that produces nothing costs exactly the same as a machine-hour that fixes a bug. If I report dollars-per-hour I've measured the engine, not the work. The denominator has to be the artifact I actually wanted.

Cost per solved task. That's the unit.

That unit has a nasty property I want on the record before I have any data: it is undefined when the task never gets solved. Not large — undefined. A model that gets eighty percent of the way through a repair has not produced eighty percent of a fix. It has produced a workspace I throw away. There's no partial credit in a denominator.

I suspect that's going to matter more than any speed difference between models, and I'd rather write that prediction down now, while it can still turn out to be wrong.

What I'd tell a team

Pick your denominator before you buy your numerator.

If you measure dollars per hour, you will optimise utilisation, and you'll congratulate yourself for keeping expensive hardware busy with garbage. If you measure dollars per token, you'll optimise brevity, which is exactly backwards for a workload whose whole value is in running more of it. Choose the unit that names the thing you wanted, then let it embarrass you.

And be suspicious of any hardware decision that only works at a utilisation you haven't demonstrated. Mine currently doesn't work at all — it works at two hundred hours a month, hypothetically, starting whenever the boxes turn up.

Where this falls apart

One failure mode kills the whole argument: if the local models can't finish the task at all, cost per solved task is undefined at any utilisation, and I've bought two very expensive space heaters. No amount of amortisation rescues a denominator of zero.

So that's the first thing to measure, and I can't measure it until the hardware lands.

What I need before then isn't a benchmark. It's a way to tell whether a repair actually happened — a grader I trust more than I trust the agent's own account of itself. That turns out to be much harder than it sounds, and it's what I'm building next.

The hardware isn't here yet. That makes this the only honest moment to write down what would count as success, because a bar set after you've seen the results isn't a bar — it's a description of the results with a bar drawn around it.

I've done the dishonest version before. You run a thing, it produces something that looks pretty good, and you go looking for a metric that agrees with you. The metric is always there. There are enough plausible ways to measure a codebase that at least one of them will flatter any given afternoon's work.

So: three requirements, written now, while I have no data to protect.

One — the oracle drives the app, it does not inspect it

The tempting way to grade an agent's repair is to look at the code. Does the class exist. Does the module define that function. Does the schema have that column. These checks are fast, they're objective, and you can write hundreds of them in an afternoon.

They are also blind in a specific and dangerous way: they verify declarations, not behaviour. A structural check confirms that something was written. It cannot confirm that the running system does the right thing when you poke it.

That gap isn't academic, and it's exactly where the defects I care about live. The things I want an agent hunting — a slow path, an input that isn't validated, a delete that leaves a dangling reference behind it — are all behavioural. None of them are visible in the shape of the code. You find them by running the application and asking it questions.

So requirement one: the grader must never import a model, read a class name, or check a table. It starts the app and asks it questions a person would ask. Can you add a member? Can you delete one who's assigned to a task? What's left afterwards?

If the contract can be satisfied without the application working, the contract is not a contract.

Two — something the agent has never seen, that detects damage

Any score the agent can see, it will optimise. That isn't cheating; it's the job. Point an optimiser at a number and it moves the number, and if there's a cheaper route to the number than actually fixing the software, it will find that route because it's cheaper.

The standard answer is a held-out set, and I want one — but with a twist I think matters more than the holding-out itself.

The holdout has to start at full marks.

If the held-out probes pass completely on the broken starting code, then they can't measure improvement — and that's the point. Their only job is to detect damage. A run that raises the visible score while dropping a single held-out probe has broken something to satisfy the grader, and I want that to be loud and unambiguous rather than buried in a net-positive total.

This inverts the usual instinct. I'm not looking for a second opinion on progress. I'm looking for a tripwire, and a tripwire that's already tripped tells you nothing.

Three — the agent does not get to say when it's finished

This is the one I expect pushback on, so let me be precise about the claim.

An agent that has been working on a problem for fifty steps has, by then, a long context full of its own reasoning about why its approach is correct. Ask it whether it's done and you're asking a witness to corroborate itself. It will say yes, sincerely, and it will not be lying — it genuinely has no way to know. Nothing in its context contains the information that would contradict it.

The fix isn't a better prompt. It's structural: when the agent declares it's finished, run the contract, and if anything fails, refuse the stop and hand the failures back. Completion becomes something the harness decides on evidence, not something the agent asserts.

I want to state the prediction plainly so it can be wrong: I think this single mechanism will turn out to matter more than model choice, more than prompt engineering, and more than any orchestration I might build on top. I don't have data. I have a suspicion and a place to write it down.

What I'm deliberately not requiring

Some things I could ask for and won't, because they'd measure the wrong thing:

• Code style, structure, or file layout. If the application behaves correctly, I don't care how it's arranged. Grading layout is how you end up rewarding an agent for tidiness while it ships a broken delete.
• That the agent explain itself. Explanations are cheap and persuasive and correlate poorly with correctness. Evidence over claims.
• Speed, at this stage. A fast wrong answer isn't a partial credit. Timing matters when the results are trustworthy, not before.

What I'd tell a team

Write the scorecard while you still have nothing to lose by it. Every hour you wait, the bar gets more negotiable, because by then you have a result you're fond of.

And check one property of every metric you adopt: can this number be moved without doing the work? If yes, something the optimiser can't see needs to be watching. That's the entire design principle, and it applies well beyond agents — it's the same reason you don't let the team that builds the thing write the acceptance test alone.

Where this could be wrong

The obvious risk is that I've over-engineered a scorecard for a problem that doesn't need one — that the models turn out to be either clearly capable or clearly not, and all this apparatus is ceremony around an answer visible in ten minutes.

The less obvious risk, and the one I actually believe: the scorecard measures the harness as much as the model. Every one of these mechanisms is code I have to write. If my contract has a hole, a model will find it and I'll record the result as a fact about the model. I don't have a good defence against that yet, beyond suspicion and a habit of checking.

That's the bar. Next I need to turn it into sixteen requirements concrete enough to build against — and every one of them, it turns out, has a scar behind it.

I've built a multi-agent coding system before. Not well — that's the point of this post. Before the new hardware arrives I've been going back through where the old one broke, and turning each break into a requirement I can build against and, more importantly, test.

The rule I set myself: nothing goes on the list because it sounds like good architecture. Every requirement has to trace to a specific failure I watched happen. Sixteen survived that filter. Here are the ones that cost me the most.

R1 — Decompose by cause, not by count

The old orchestrator took the failing checks and dealt them into buckets like a hand of cards. N failures, N agents, evenly split.

Then a run came in with 163 checks failing — all of them from a single environment error. The correct response is one agent fixing one thing. What a cause-grouping orchestrator does instead is notice that 163 symptoms share one root, size the work accordingly, and launch one agent. What mine did was carve the 163 symptoms into buckets and launch a fleet to fix a problem that didn't exist in any of them.

Acceptance: given at least fifty failing criteria where thirty or more share a root cause, produce six work units or fewer, with at least one unit covering ten or more criteria. Given failures with N distinct causes, produce roughly N units. The requirement is worthless without that second half — a system that always answers "one" passes the first test.

R2 — Disjoint file ownership, enforced in code

This is the one that still bothers me.

An earlier from-scratch swarm ran 258 tasks. 238 of them wrote a file that another task also wrote. Last writer won, every time, silently. The system destroyed roughly 92% of its own output while reporting healthy progress the whole way through, because the score went up — the surviving writes were real improvements, and nothing anywhere measured the work that had been overwritten.

That's the property that makes it dangerous: last-writer-wins is invisible to any score-based gate. A gate compares before and after. It cannot see a change that existed for four minutes in between.

The fix can't be a request. Asking a model nicely to stay in its lane fails the moment it decides a neighbouring file needs a one-line import. The harness computes the overlap from the plan and removes it before anything runs, and it records that it did so.

R3 — Dependency-aware scheduling, kept separate from decomposition

A competent decomposition can still be in the wrong order. A separate planning pass on the old system caught exactly this and said so plainly: "This plan will fail in execution. Unit 3 fixes route registrations in modules that import from app.api.tasks, which unit 2 is fixing."

Run those concurrently and unit 3 burns its entire budget chasing a symptom of a bug another agent is in the middle of repairing. Independent units go in one wave; dependent ones get sequenced. Every unit appears in exactly one wave.

R4 — Stated intent before work

A diff on its own can only be judged on style. Against a stated plan it can be judged on fidelity — did you do what you said you'd do.

That distinction caught two failures the diff alone would have waved through: an agent whose plan promised _clean(data: dict) -> dict and whose implementation returned something else entirely, and another that planned to use APIRouter — a framework this project doesn't use anywhere. Both diffs looked fine. Both plans were checkable.

No file modification is accepted from an agent that produced no plan, and the plan is stored alongside the diff.

R11 — Parser tolerance

This one has the least glamour and, I suspect, the highest cost.

A model emits something structurally reasonable that my parser doesn't accept, the parser silently discards it, and the model looks incapable. I've fixed that same shape of bug six separate times in code I wrote myself. It never presents as a parser error. It presents as a bad model.

The requirement is that the harness tolerates the reasonable variations — and that when it can't parse something, it says so loudly instead of returning an empty result that looks like a lazy agent.

R16 — Cost and portability

The system must run entirely against a local endpoint, with no dependency on a hosted model, and any framework adopted must not assume a provider.

That's the whole project stated as one requirement. If a candidate framework only really works against one vendor's API, it fails here regardless of how good the rest of it is.

The remaining ten, briefly

What I'd tell a team

Write the requirement and its acceptance test in the same sitting. Half of these started as a sentence that felt obviously correct and only became useful when I forced myself to say what measurement would prove it. R1 is the clearest case: "decompose sensibly" is unfalsifiable, and "given fifty failures with thirty sharing a cause, produce six units or fewer" is something I can run.

And note what the list is missing. There's no requirement about model quality, prompt design, or context management — the things people argue about. Every scar I have is from the plumbing: ownership, ordering, parsing, logging. That may say more about my engineering than about agents in general, but it's what the evidence says, so it's what I'm building against.

Next: sixteen requirements is also a scoring rubric, so I'm pointing it at the frameworks everyone else already built — before I write another line of my own.

I have sixteen requirements and no hardware yet. That's a good week to find out whether somebody has already built this.

So I scored ten candidates against R1–R16: LangGraph, Pydantic AI, DSPy, the OpenAI Agents SDK, MCP, CrewAI, AG2 (AutoGen), SWE-agent, OpenHands, and — as candidate zero — the roughly 1,500 lines of dependency-free Python I already have. The rubric was written before the survey, and the survey was explicitly allowed to conclude that the incumbent wins. A comparison that can't return "keep what you have" isn't a comparison; it's a procurement document.

I want to be upfront about the method's boundary, because it turns out to matter: this was done by reading. Documentation, source, and issue trackers. Nothing was run.

The one-line finding

Every framework surveyed leaves R1–R10 — the actual orchestration logic — as custom code you write either way.

Not "partially supports." None of them have a concept of root-cause decomposition, harness-enforced disjoint file ownership, dependency-aware wave scheduling, a plan-before-work gate, blind multi-aspect review with synthesis, a verdict structured enough to name files for partial revert, or dual-oracle grading. That isn't a gap in how hard I looked — it's confirmed against current docs and source for every candidate.

What adopting one would actually buy

Strip out the parts every framework leaves to you, and what's left is the generic LLM-calling boilerplate — roughly 10–20% of the system. That's the purchase. In exchange you take on a dependency.

Which would be a fine trade, except for where the dependency is weakest.

Every framework that attempts structured output has open, sometimes maintainer-declined issues in exactly one code path: tool-call and structured-output parsing against a non-frontier, locally served model. LangChain #31335 and #31403, and equivalents in the others. That is not an obscure corner for me — it's the only configuration I will ever run, and per R11 it's the bug class I've already found and fixed six times in my own code.

There's a version of this finding that sounds smug, and I want to avoid it. The frameworks aren't badly built. They're built for a different shape of problem — a hosted frontier model, a conversational or tool-using agent, a team that wants scaffolding rather than control. Judged against that, several are excellent. Judged against sixteen requirements derived from my own failures on local hardware, they mostly don't engage.

The picks

#1 — Keep the hand-rolled system. R1–R10 are the substance, no candidate implements any of them, and every one of them would leave me writing the same ~1,200 lines of orchestration on top of a dependency whose weakest path is my only path. The incumbent also already handles a set of parser quirks that a 15-line function absorbs today with zero dependencies.

#2 — LangGraph, if the decision changes. It's the only candidate whose primitives structurally resemble what R3 and R5 need: Send() gives genuine runtime fan-out and fan-in rather than a fixed-N gather, state reducers give a real per-branch-then-converge mechanism, and it reached 1.0 GA this year with the most mature tooling around it — checkpointing, visualisation, tracing. It satisfies R1, R2, R4 and R7 no better than the others, and its default structured-output path is precisely the buggy one. But it's the only candidate where adopting it and building the missing 60% on top would plausibly produce less new code than building the same logic on a gather and a hand-rolled state dict.

What would have to become true for it to actually win: crash recovery mattering more than debuggability. My system has no answer to "the process died at round 3 of 6." LangGraph's checkpointing does. Today I'd rather be able to read the whole thing.

Pydantic AI did not make the top two. R1–R7, R9, R10 and R12 all unsupported. It's a well-made library aimed at a different job.

The errata, which I'm leaving in

Two API details in this survey were wrong, caught while writing it, and corrected in place rather than quietly fixed. Both were cases where the documentation described a behaviour the source didn't implement quite that way.

I'm keeping them visible because they're evidence about the method, not just about those two APIs. A survey conducted by reading inherits every inaccuracy in the material it reads, and the errors it produces look exactly like the correct parts — confident, sourced, wrong.

What I'd tell a team

Score against requirements you derived from your own failures, not against a feature matrix from the vendor. Half of what these frameworks advertise is real and irrelevant to me; the things I need aren't in anyone's feature list because they're not features, they're decisions about how work is decomposed and owned.

And let the incumbent compete honestly. If your evaluation can't return "keep what you have," you've already decided.

Where this is weakest

I know exactly where the soft spot is, because it's structural: I audited what these systems claim to provide. Docs and source describe intent. A benchmark describes behaviour. Those come apart precisely where I care most — the local-model path that every project's test suite exercises least.

There's an uncomfortable symmetry here I noticed too late to design around. I spent last week arguing that a grader which reads code instead of running it is blind to the defects that matter. Then I evaluated ten frameworks by reading them.

So this ranking is a hypothesis, not a result. When the hardware lands, I'm going to build the top candidates for real, run them against the same contract on identical workspaces, and find out how much of this survives contact. I'd put decent odds on the ranking holding and the reasoning behind it turning out to be wrong.

The boxes turned up on a Tuesday. Two DGX Sparks, 121 GB of unified memory each, small enough that both of them plus a laptop take less desk than a monitor stand used to.

I'd spent three weeks writing about what I'd need before I trusted an agent with real code. Within about a day of the hardware arriving I had my answer to whether the scorecard was any good, and the answer was no.

What I built first

Not a benchmark. A grader — because everything I'd written beforehand said the measurement is the thing that has to exist before any result means anything.

So: a deliberately broken Flask application, a spec, and 229 acceptance checks. The checks are structural. Does class Team declare __tablename__ = 'teams'. Does this module define that column. Does that route get registered. They run in about 28 seconds, they're completely objective, and I could write a hundred more in an afternoon.

Then I pointed three very different systems at it: a six-agent local fleet running on the new hardware, and two frontier hosted models working the same task on their own.

All three scored 229 out of 229

Which felt like a great result for about ten minutes.

All three shipped the identical defect. Delete a member who's assigned to a task, and task.assignee_id still points at the deleted row. The application accepts the delete, reports success, and leaves a task pointing into a hole.

Three independent systems, three perfect scores, one bug. That's not three coincidences. When three different approaches produce the same wrong answer, the thing they have in common is the problem — and the only thing they had in common was my grader.

Why 229 checks couldn't see it

I went through the checks afterwards to find out how a suite that exhaustive misses something that basic. The answer is embarrassing in how simple it is.

Not one of the 229 checks ever deletes a record that something else references.

They confirm structure exists. Classes, columns, relationships, routes, imports, signatures. Every one of them is a question about the code at rest. Not one of them is a question about what the running system does when you take something away.

A defect only appears in a suite if some check walks a path that touches it. Adding more checks of the same kind extends the suite along an axis where it's already strong. It buys nothing on the axis where it has no extent at all.

That's the part I'd genuinely not internalised: exhaustive and blind are independent properties. Exhaustive is about count. Blind is about kind. I'd been treating a big number as evidence of coverage.

The fifteen probes that found it in seconds

The thing that caught it wasn't clever. It was a small set of behavioural probes I'd kept outside every workspace — the tripwire from the scorecard post. Fifteen of them, none ever copied where an agent could see them.

They found the dangling reference immediately, because one of them does exactly what a person would do: add a member, assign them a task, delete the member, then ask what's left.

Fifteen questions beat 229 checks, and it wasn't close. Not because fifteen is a better number, but because those fifteen drive the application and the 229 read its source.

What I'd tell a team

Count how many of your acceptance checks change state and then look at what happened. If the answer is zero — if every check is an assertion about structure — you have a suite that will pass a broken system, and its size will make you confident while it does.

The corollary is harsher and I think it's right: a check that never destroys anything can't verify cleanup. Deletion, rollback, cascade, eviction, expiry, revocation — the entire family of "and then it goes away" behaviours is invisible to any suite that only ever builds up. That family is also where the defects with real consequences tend to live.

And when several independent attempts converge on the same wrong answer, stop investigating the attempts. Look at what they share.

Where I was lucky

I want to name this rather than let the post read as foresight: I only caught this because I'd built the holdout first, for reasons that were mostly abstract at the time. If I'd built the 229 checks and stopped, I'd have shipped "three systems solved it perfectly" as a finding, and I'd have believed it, and every number after it would have been built on top.

The holdout wasn't insight. It was insurance I bought before I knew what for, and it paid out on day one.

What's next

The 229 checks are done. Not extended, not supplemented — replaced. What goes in their place took about an hour to write and did more for this project than the two days of orchestration machinery I'd built around them, which is a sentence I'd have found irritating a week ago.

Three independent systems scored 229 out of 229 on my grader while shipping the same broken delete. The obvious move is to add checks. I deleted them instead.

What replaced them took about an hour to write. It has done more for this project than the two days of orchestration machinery I'd built around the old grader, and I want to be precise about why, because "write better tests" is not the lesson.

Three files

SPEC.md the contract. Every endpoint named. Deletion rules stated outright: refuse with 409 or clean up, never leave orphans.

accept.py 22 criteria that DRIVE the running app. Never imports a model, never reads a class or table name. "Can you add a member? Can you delete one who is assigned to a task? What is left?"

holdout2.py 13 probes one ring further out, never copied into a workspace. Baseline 13/13, so it detects DAMAGE, not improvement.

That's the whole thing. The most important line in it is the one in SPEC.md that states the deletion rule outright — because the defect that started all this existed in a gap between what I assumed and what I'd written down. The old spec never said what should happen to a task whose assignee is deleted. Three systems each guessed, all three guessed the same way, and all three were wrong in a way I couldn't grade because I'd never decided.

The rule that makes it work

accept.py is not allowed to import anything from the application. Not a model, not a schema, not a constant.

It starts the app and talks to it over HTTP like any other client. That single constraint is what makes it impossible to satisfy structurally — you cannot pass a criterion by declaring something, because the criterion never looks at your declarations. It creates a member, assigns a task, deletes the member, and reads back what survived.

Both acceptable endings are in the spec. The agent may refuse the delete or clean up after it — I don't care which, and saying so removes an entire class of argument about implementation choice while keeping the behaviour pinned.

It got faster, which I did not expect

Scoring went from 28 seconds to 2.

That surprised me — 229 checks replaced by 22 that boot a web application ought to be slower. It isn't, because the old suite was doing static analysis and reflection across the whole tree while the new one makes a couple of dozen HTTP calls against an already-running process.

The speed turns out to matter more than the accuracy in one specific way: at 2 seconds, the contract can run after every edit. At 28 it could only run at round boundaries. A grader that's cheap enough to run continuously stops being a report card and becomes a feedback signal — and that, it turns out, is what makes it possible to refuse an agent's claim that it has finished.

What the 13 probes are for

holdout2.py sits one ring further out and is never copied into a workspace. No agent has seen it.

Its baseline is 13/13 on the broken starting code. That's deliberate, and it's the design decision I'd defend hardest: since the probes already pass before any work happens, they cannot measure improvement. They can only ever go down. A run that raises the visible score while dropping one holdout probe has broken something to satisfy the grader, and that shows up as an unmistakable signal rather than a smaller net gain.

I set this up before I had any reason to think I'd need it. It has since caught things I would otherwise have recorded as successes.

The part that isn't about testing

Here's what I actually take from this.

The 22 criteria are the same questions I'd have asked in a code review. Can you add one. Can you delete one that's referenced. What's left afterwards. There's no sophistication in them at all — the entire content of the change is that I stopped asking about the shape of the code and started asking about what it does.

I'd built 229 checks because 229 felt like diligence. It was the opposite: it was a way to avoid deciding what the software was supposed to do, by measuring a hundred things I could verify mechanically instead of the six things that actually mattered. Structural checks are what you write when you haven't written the spec.

What I'd tell a team

Write the spec first, and make it state the awkward cases outright — the deletions, the conflicts, the "what happens to the thing that pointed at this." Every defect that survived my old grader lived in a case the spec never named. The checks weren't the root problem; the undecided behaviour was.

Then set one hard constraint on the grader: it may not import the code it's grading. That single rule forces every criterion to be behavioural, and it costs nothing to enforce.

And measure how long your acceptance suite takes, because that number silently decides how often it can run, which decides what kind of instrument it can be.

What I got wrong

I framed this as "the grader was broken." That's too kind to me. The grader did exactly what it was built to do — the problem is that I built it to check declarations because that was the easy thing to automate, and then treated its output as though it meant something else.

There's also a consequence I hadn't seen coming when I made the change: the five-expert review panel I'd spent two days building existed specifically to catch what a structural oracle couldn't see. With the contract in place, that whole apparatus was now insurance on a risk that no longer existed. I didn't notice for another day, and it took a control run to make it undeniable.

Last week I scored ten agent frameworks by reading their documentation and picked a top two: keep the hand-rolled system, with LangGraph as the framework to adopt if the decision changed. Pydantic AI didn't make the cut — R1–R7, R9, R10 and R12 all unsupported.

Then I built all three and ran them on identical workspaces, same worker, same prompts, same review panel, same gate, with the orchestration wiring as the only variable.

arm                            result  holdout    time      turns
armA-handrolled                 22/22   13/13   237.5 min    881
armB-langgraph                  22/22   13/13   190.6 min    802
armC-pydantic                   21/22   13/13   160.0 min    979

The paper ranking held. The guide's #1 ties for first, and the candidate it excluded from the top two is the only arm that didn't finish the contract.

I'd love to report that as a win for careful desk research. It isn't, and the reason is more interesting than the table.

First, a correction I have to make about myself

I wrote up this result before it was finished.

At the point I published, armA was still running and the numbers I had said the hand-rolled arm was trailing. So I wrote "the guide's top pick came last" — a great line, a clean reversal, exactly the kind of finding that makes a post worth reading. Then armA closed at 22/22 in round 3 and the ranking collapsed into a tie.

That is precisely the error I'd spent the previous week criticising in my own grader: reading a partial state as a result. I did it on the same day, about the same experiment, one level up. The claim was interesting, which is exactly why I should have waited.

Where the survey was actually wrong

Not the ranking. The reasoning underneath it.

The survey scored frameworks on whether they implement R1–R10 as primitives, found that none of them do, and concluded there was nothing to gain from adopting one.

The first half was correct and the measurement confirmed it. Arm B is 84% shared hand-rolled code. Only 32 lines in the entire arm touch a LangGraph primitive.

The second half did not follow.

The gain didn't come from primitives — it came from those 32 lines being harder to get wrong. Five of the thirty harness bugs I hit during this work lived in exactly that surface: a wave loop that died with KeyError: '_unit' on the second wave, a reducer that couldn't be reset between rounds, a field read as problems where the schema emitted concerns. LangGraph's version of those 32 lines has been debugged by more people than mine has.

A framework audit that counts features cannot see that, because the thing being bought isn't a feature. It's the accumulated debugging of a small, fiddly surface that I would otherwise get wrong five times.

What LangGraph actually bought

Not a better outcome. Both arms reached 22/22 with the holdout intact.

What it bought was convergence speed — 22/22 in round 2 versus round 3. That's worth about 9% on turns and 20% on wall clock. Real, worth having, and much less than the framing "which framework wins" implies.

I originally reported this as "10% better outcome from 1.1% of the code." That claim is retracted too — it was computed when armA was still trailing. The correct version is: same outcome, 20% faster, from 1.1% of the code. Less quotable. Actually true.

Pydantic AI, fairly

It came last, and the survey predicted it would. But it also ran the fastest wall clock of the three arms at 160 minutes, and its holdout is a clean 13/13 — it did not break anything, it just didn't finish everything. 21/22 with an intact holdout is a respectable failure mode, and a much better one than 22/22 with damage.

It's a well-built library aimed at a different job. The survey's verdict was that it doesn't do what I need, not that it's bad, and the measurement agrees.

What I'd tell a team

When a paper evaluation turns out to be right, check whether it was right for its stated reasons. Mine reached the correct ranking through an argument that measurement contradicted. If I'd taken the ranking as validation of the method, I'd have kept using a method that scores the wrong property — and the next time, when the reasoning and the answer diverge, I'd have had no warning.

And be suspicious of your own best lines. The strongest sentence I wrote about this experiment was the one that was false, and it was strong because it was false — a clean reversal is more shareable than a tie. That's an incentive worth naming out loud, because it doesn't announce itself while you're typing.

The bigger problem with this whole post

All three arms are competing on a task where the orchestration itself may not be earning its place.

While these were running, I set up a control I should have run first: a single agent, no orchestration at all, same contract. I'm writing that up next, and it reframes everything above — not by making the framework comparison wrong, but by making it a careful measurement of something that turned out not to matter much for this task.

For eight hours yesterday, Nemotron-70B was recorded in my results as "no score — not serviceable on this hardware." Seven failed attempts. A negative result, written up and filed.

It was serviceable the whole time. I was doing it wrong in a way I'd been told about, in writing, by my own test output.

The setup that seemed obvious

The checkpoint on disk is BF16 — about 132 GB. Each Spark has 121 GB of unified memory. The arithmetic is immediate: it doesn't fit on one box, and I have two boxes. Split it with tensor parallelism, TP=2, and get on with the day.

Every attempt wedged at the same moment: the instant three agents sent prompts of roughly 10k tokens. Not at load — loading was fine. Not on a small probe — a 30-token generation came back happily. It died specifically when real agent-sized work arrived, and then it stayed dead: one node pinned at 96% GPU utilisation, the other at exactly 0%, indefinitely. No error, no timeout, no crash. Just two boxes, one of them very busy doing nothing.

Seven things I tried

In order, each one reasonable, each one wrong:

1. Bound Gloo to loopback — plausible, because cross-node collectives are exactly where address binding bites.
2. Raised the RPC timeout past its 300-second default, on the theory it was a slow collective rather than a dead one.
3. Found my own watchdog holding a flock that was blocking relaunches — a real bug, genuinely fixed, entirely unrelated.
4. Swapped Ray for mp as the distributed executor backend.
5. Chunked the prefill, since the failure only appeared at large prompt sizes.
6. Applied the vendor's documented NCCL settings for this hardware.
7. Added --disable-custom-all-reduce.

That last one changed nothing. Which was the answer, had I been listening.

Disabling the custom all-reduce path and observing no change tells you the problem isn't in the custom all-reduce path. I read that as "another thing didn't work" and moved to the next idea, when it was the piece of evidence that should have sent me back to the premise instead of further down the list.

The premise was wrong

Here's the thing I should have noticed on attempt one: every other large model in this sweep runs quantised, and most of them run on a single node. A 120B parameter model fits on one Spark. I had already proven that, repeatedly, on the same hardware, that same week.

There was never a reason a 70B needed two nodes — beyond the accident that BF16 was the format I happened to have on disk.

Quantised to FP8, around 70 GB, TP=1, one node: it runs. Scores 20/22, 2/22, 18/22, calling DONE at steps 21 and 28 on the two good runs. Its best run clears the wall that five other models never got past.

The right response to "the weights don't fit" was to shrink the weights, not to distribute them across a fabric with a documented collective bug.

The cost I hadn't counted

There's a second reason TP=2 was the wrong instinct, and it has nothing to do with the bug.

Splitting a model across both Sparks doesn't just add a fragile collective to every forward pass — it collapses my fleet to a single agent. Both boxes are now serving one model. For a one-off benchmark that's fine. For what I actually want, which is many agents sweeping a codebase continuously, it's the worst possible configuration: maximum coordination cost, minimum parallelism.

A model that fits on one node leaves the other node free. That's not a footnote to the capability comparison — for my purposes it may be the whole comparison.

About that 2/22

The middle run is the ugliest number I have, and it's real — a genuine model error, which I checked carefully before writing down, given how many of my numbers have turned out to be mine.

The model was trying to stop members being orphaned when a team is deleted. It wrote a delete-orphan cascade onto Member.team — the many side of a many-to-one. SQLAlchemy accepts that at import time and raises ArgumentError on the first query. So the application still builds. It still registers all 21 routes. And it fails every single request.

None of my three build guards fired, because all three prove that create_app() returns, not that the app can serve anything. The model had 25 further steps and never recovered.

That's a guard gap worth closing, and the fix is obvious in hindsight: one request probe after each edit would have caught it immediately and handed the model an error it could act on.

What I'd tell a team

When a fix changes nothing, that's data — treat it as such. A change with no effect eliminates a hypothesis, and eliminating a hypothesis should send you back up the tree, not sideways to the next leaf. I had seven leaves and never revisited the root.

And check whether the constraint you're routing around is one you accepted or one you chose. "It doesn't fit" was true of a specific file in a specific precision that happened to be on my disk. I treated a property of my download as a property of the model, and then spent a day engineering around it.

The tell was available from the start: I was building elaborate machinery to work around a constraint that nothing else in the project was subject to. When your solution is the only one of its kind in the system, ask why your problem is the only one of its kind.

Where it stands

The BF16 TP=2 attempt is kept in the results, marked void with the reason recorded, rather than deleted — because "seven approaches that didn't work" is the useful part, and quietly replacing it with the run that worked would make the FP8 number look like a first attempt.

One caveat that travels with the score permanently: that row is FP8, quantised on load. The checkpoint is BF16. It is not a BF16 result and it must never be quoted as one.

I spent two days building the orchestration. File-partitioned workers, a five-expert review panel, wave scheduling, a synthesiser to turn five opinions into one verdict, repair loops, score backstops. About 1,200 lines of coordination machinery, most of it satisfying to write.

Then I ran a single agent against the same task as a control.

run                result  holdout   time        turns   completion tokens
armA-handrolled     22/22    13/13   237.5 min    881         300k
armB-langgraph      22/22    13/13   190.6 min    802         285k
armC-pydantic       21/22    13/13   160.0 min    979         190k
solo-control2       22/22    13/13    12.0 min     28         5.4k
solo-control3       22/22    13/13    12.0 min     21         5.4k
solo-control4       22/22    13/13    16.0 min     17         9.4k

Three independent single agents each solved the full contract with the holdout intact, in 12–16 minutes, using 30–50× fewer turns and roughly 40× fewer completion tokens than the orchestrated arms. Same baseline workspace, same oracle, same engine load, no answer key anywhere.

The orchestration bought nothing measurable. Not correctness — solo matched the best arm and beat one of them. Not safety — the holdout is 13/13 everywhere. It cost about 13× the wall clock to arrive in the same place.

That's the headline, and it's the least interesting thing here. The interesting part is why it lost.

The orchestration was insurance against a problem I'd already fixed

The five-expert panel existed for a specific reason. My original grader was 229 structural checks — does this class have that attribute, does that module define this column — and three separate workspaces scored 229/229 while shipping the identical defect. Delete a member who's assigned to a task and the task still points at the deleted row. Nothing in 229 checks ever deleted an assignee, so nothing ever saw it.

Given an oracle that blind, building something else to look is a coherent design. The panel wasn't stupid. It was compensating.

Then I replaced the grader with 22 criteria that drive the running application, and the panel's entire reason for existing evaporated. The class of defect it was built to catch is now caught by the oracle itself, in two seconds, every round.

The orchestration didn't lose because multi-agent is a bad idea. It lost because it was insurance against a blindness that no longer existed. Nobody re-evaluated the insurance after fixing the leak — I certainly didn't, until the control run made it unavoidable.

That's the transferable lesson, and it has nothing to do with agents. When you fix a root cause, go back and delete the machinery you built to survive it. Otherwise you keep paying the premium forever, and — worse — you keep attributing your outcomes to it.

What this does not prove

I want to be careful here, because the overreach is very tempting and this project has punished me for it before.

This was a small repair task with a complete contract: five defects, 22 criteria, everything fitting inside one context window. That is precisely the case where decomposition can only add overhead, which makes it the weakest possible ground on which to defend orchestration. Concluding "multi-agent doesn't work" from this data would be exactly the kind of claim I'd criticise in someone else's post.

Three places where coordination may still pay, all of them untested here:

• Beyond one context. The finding that started this whole project was a from-scratch swarm where 238 of 258 tasks wrote a file another task also wrote, destroying about 92% of its own output. Coordination isn't optional there — it is the problem. And a real codebase, which is what I actually want to point this at, exceeds one context by definition.
• Incomplete or ambiguous specs. My contract named every endpoint and stated the deletion rules outright. A review panel is most defensible exactly where the agent has to guess.
• Adversarial regressions, where a locally sensible fix breaks something distant.

The framework ranking, briefly

Three of those arms were a framework bake-off — hand-rolled, LangGraph, Pydantic AI — and I've written that up separately, because the result there is subtler than this one and deserves its own space. The short version: the paper survey's ranking held up, and its reasoning didn't.

It's worth noting what the control run does to that comparison, though. All three arms are now competing to lose to a single agent by a factor of thirteen. Ranking them is still meaningful — but it answers a question that turned out not to matter for this task.

What I'd tell a team

Run the trivial control first. Not because the simple thing usually wins, but because the number it produces is the only thing that makes the complicated result legible. I had a 22/22 from an orchestrated arm and I was pleased with it. That number meant nothing until I knew that one agent got the same score in a twentieth of the time.

I had every opportunity to run it earlier. It cost twelve minutes. I didn't run it because I'd already spent two days on the sophisticated version and the control felt like a formality — which is precisely the psychology that makes it worth mandating rather than leaving to judgement.

And notice the shape of the mistake: I never measured whether the problem still existed. I measured the solution, repeatedly and carefully, against a baseline I'd stopped questioning.

What's next

If fan-out isn't what bought the result, something else did. The next run isolates the smallest possible mechanism — nine lines that refuse to let the agent decide it's finished — and puts it up against the six-agent panel directly.

I did not expect that comparison to go the way it went.

A single agent beat 1,200 lines of orchestration by 13× on wall clock. That result had a hole in it, and it took another day to find: I had only ever measured orchestration on the strongest model available.

Of course a review panel adds nothing on top of a model that doesn't need review. The honest question isn't "does orchestration help?" — it's "does orchestration rescue a weak model?" If a framework could take a model from 19/22 to 22/22, then orchestration's domain is exactly the models that can't go it alone, and my headline was an artefact of testing only the model that made review redundant.

That's a single cell in a 2×2, and it decides the whole thing.

The 2×2

All runs from the same 17/22 baseline with a 13/13 holdout:

The only run to reach the full contract is the only one where the gate engaged. It tried to stop 52 times, was refused 51, and finished 22/22 in 72 minutes and 197 steps — beating the six-agent panel on score, wall clock and cost, using one agent and nine lines instead of 1,200.

The deciding cell landed at 19/22. The six-agent orchestration did not rescue the weak model. Nine lines of verification did.

What the nine lines do

When the agent declares it's finished, run the criteria. If any fail, refuse the stop and hand back the failures.

That's it. No reviewer, no panel, no synthesis, no adversarial second model. The only reason it can exist is that the contract runs in two seconds — at 28 seconds this would be unaffordable at 52 invocations.

The gate has no opinion about the code. It doesn't review, doesn't suggest, doesn't reason. It compares a claim against a contract, and the only thing it contributes is refusal.

The ungated row is the part worth staring at

Without the gate, the same model announced "successfully implemented all required functionality" — at 12/22, with 7 of 13 holdout probes broken. It had deleted a field the application depends on, and every request that touched it now raised KeyError: 'due_date'.

It was not lying. It had no way to know. It was asked to judge its own completion and it did what anything does in that position: consulted its own account of its work and found it satisfactory.

This is not a prompting problem. You cannot instruct your way out of it, because the information that would change the answer isn't in the context at all. It's in the running application, and only something that runs the application can go and get it.

Corroboration from an uncomfortable direction

While writing this up I went looking at how production coding agents handle the same problem, and found that a widely-used commercial agent architecture has no verification gate at all — all verification is left to the orchestrating model. No reviewer agent, no adversarial panel.

I had built 1,200 lines of exactly the layer the reference implementation deliberately omits, and then found that the nine lines it also omits were the ones that mattered. I don't think that reflects well on either of us, but it does suggest the gap is real rather than idiosyncratic to my setup.

Two caveats that travel with this

I want these attached to the finding permanently, because it's the sort of result that gets quoted without them.

It is n=1 on the engaged condition. The other two gated runs never declared DONE at all, so their 20/22 says nothing either way about the gate. One run reached the phase where the mechanism could fire. One.

The weak-model row ran at the wrong sampling parameters. A per-model profile store existed and nothing read it, so every model ran at a single temperature that matched almost none of them. That's its own write-up, and it means these numbers need re-running before they're load-bearing.

I nearly made a third error on top of both. Mid-run I wrote that the gate "fired once in six runs, so it's variance" — a count taken before the one run that mattered reached the phase where it began declaring DONE. A snapshot generalised into a structural claim, and it inverted the actual finding.

What I'd tell a team

Never let an agent be the judge of its own completion. Not because it will lie, but because completion is a claim about the world and the agent only has access to its own account of the world.

And when you're deciding what to build, ask which layer is doing the work. I built decomposition, ownership, scheduling, review, synthesis and gating — and the thing that produced the result was the smallest and least interesting of them. Fan-out was the part that felt like engineering. Refusal was the part that worked.

What's next

The mechanism question is settled enough for now. The one I still can't answer is the one I bought the hardware for: what does a fix actually cost on machines I own, and how many agents can I run at once before that number stops falling?

I have the measurement. It's better than I expected, and it changes which model I'd choose.

Five weeks ago I wrote down the arithmetic for buying two Sparks and said the unit that mattered was cost per solved task — and that I couldn't measure it until the hardware arrived.

Here it is.

The measurement

Fire N concurrent bug-fix requests at one node. Real 8 KB prompts, no token cap, graded serially afterwards because the working tree can only hold one patch at a time. Model is Qwen3.6-35B-A3B, running on a single Spark.

Sixteen agents, sixteen bugs, sixteen fixes, two-thirds of a cent each. Nothing degraded as N climbed — the solve rate stayed perfect and the cost per solve fell monotonically. And this used one of the two boxes.

The dollar figures come from the same cost model I set out before buying: 70% of a $9,400 purchase amortised over 36 months at 200 busy hours, plus 240 W at $0.15/kWh — about $0.95 per machine-hour.

The mechanism is that this machine is bandwidth-bound. Weights get read once per forward pass no matter how many sequences share the batch, so N concurrent generations cost roughly the wall clock of one. Sixteen times the work for 3.6× the time.

The ceiling was mine, not the machine's

Here's the part that stung.

I'd been running everything at three concurrent agents and had started treating that as what the hardware could do. It isn't. It's modeltest.sh taking runs as its third argument with a default of 3 — and the comments say exactly why: "THREE RUNS PER MODEL, not one. Single-agent runs vary... one sample would have implied a precision that is not there."

That 3 is a statistical sample size for a benchmark. It was never a fleet. Worse, those three runs aren't even collaborating — they're three isolated workspaces attempting the same task independently. For the thing I actually want, I didn't have a fleet of three. I had one agent, measured three times.

I'd also let the engine config drift. --max-num-seqs appears nowhere in profiles.json — the canonical per-model config with an audit gate in front of it — but the serve scripts set it inconsistently: 8 in three of them, 32 in one, 64 in two. Whichever script last started a box silently decided its concurrency ceiling.

Which model, once fleet size is the question

This reframes the model comparison completely.

DeepSeek-V4-Flash is the most reliable model I've tested — three out of three on the full contract. It also needs both Sparks, split with tensor parallelism, and in practice tops out around three concurrent agents. Qwen3.6-35B-A3B matches the ceiling twice out of three on one node, and runs sixteen agents while doing it.

That's roughly ten times the agents per box. For a one-off repair, DeepSeek's reliability is worth having. For continuously sweeping a real codebase — the thing I bought the hardware for — it's disqualifying: you'd pay more per fix to run a tenth of the agents.

Capability per box, not capability, is the number that decides what you can actually do.

Where I stopped measuring

Two gaps, both mine, both worth stating plainly rather than letting the table imply completeness.

N=32 was never run. The script's default level list goes 1,4,8,16,32; the run that produced this table stopped at 16. The engine was serving at --max-num-seqs 64 at the time, so there is headroom above my best number that nobody has looked at. 16 is where I stopped, not where it broke.

Best-of-N on the design case never finished. The log ends with the header for part B and no table under it. It's still open.

And one for the tally: the first attempt at this measurement reported 0/N unsolved at every level in 0.0 seconds — nothing executed at all. That's a harness bug, not a result, and it cost the better part of a night.

What I'd tell a team

Check whether your concurrency setting is a capacity limit or a leftover default. Mine was a benchmark's sample size that followed me into normal use, and it cost me roughly 5× the throughput I could have had for free. The number was right for measuring and wrong for working, and nothing complained.

And measure cost per solved task at several concurrency levels before you choose a model. Ranked serially, the biggest model wins. Ranked by what a box can actually run, a smaller one wins by a factor that no capability benchmark will ever show you.

What's next

Two-thirds of a cent is one model on one task. The obvious question is what the other eight models cost per solved task — and the answer turns out to be that most of them don't have a price at all.

Every model I could serve on two Sparks, run three times against the same 22-criterion behavioural contract from the same 17/22 baseline, with a 13-probe holdout the agent never sees.

Ranked the way I said I'd rank things before I bought the hardware: by what a solved repair costs.

Costs use the model I set out before buying: ~$0.95 per machine-hour, doubled for anything occupying both boxes, charged for the wall clock the node was actually busy.

Seven of them have no price

Not "expensive." Undefined. They never produce the artifact, so there's nothing to divide by.

This is the entire argument for the unit, and it took having the data in front of me to feel it properly. On a score ranking, gpt-oss-20b looks respectable and thrifty: 17/22, ten minutes, sixteen cents. On a cost-per-task ranking it's the same product as Devstral-2-123B, which burned $4.21 and 266 minutes to arrive in the same place. The 26× price gap between them doesn't matter. Neither one hands you a repaired application.

The wall at 17/22

Five models fail the same five criteria: B3, B4, B5, C3 and C5 — deleting a team without orphaning its members or tasks, foreign-key validation, and the priority enum. Referential integrity, in a phrase.

This is not a size effect. gpt-oss at 20B and Nemotron at 120B fail the identical set. Devstral-123B spends 121 minutes a run to arrive in the same place. Six times the parameters and twelve times the wall clock buy nothing here.

The models that clear the wall aren't the biggest ones — they're the ones that keep checking their own work instead of declaring victory.

Worth noticing which five criteria. Every one is about what survives when something is removed — the same family of behaviour that my original 229 structural checks were constitutionally unable to see. The hardest thing for these models is the thing my first grader couldn't measure at all.

The two that have a real price

Qwen3.6-35B-A3B at $0.087 — matches the ceiling twice at 35B, on one node, in eleven minutes, and stops when it's done rather than burning the budget. Nothing else comes close on capability per unit of cost.

DeepSeek-V4-Flash at $0.169 — the only model to solve the full contract three times out of three. It's also the heaviest thing in the fleet and the only one needing both nodes, so you pay 1.9× per solved contract for that reliability, and you get a fleet of about three agents instead of sixteen.

Don't read the table as "Qwen is good." The 3.6 generation is doing the work — the Coder variants sit mid-pack at 17–20, and Qwen3-Coder-Next lands at a flat 17.

Caveats that travel with this table

• The protocol column isn't uniform. DeepSeek's 22×3 is a text-protocol score; Qwen3.6's is tools. Five models moved when I switched them, so these aren't strictly like-for-like. DeepSeek's own tools-mode run scored 20/19/0 — but sampling changed at the same time, so it attributes nothing and is excluded.
• Best-of-three flatters the variable models. Laguna spans 17–22 across three runs. DeepSeek and Qwen3.6-35B are the only two consistently at the top.
• Nemotron-70B's row is FP8, quantised on load. The checkpoint on disk is BF16. It is not a BF16 score and must never be quoted as one.
• The 2/22 in that row is real and I checked it carefully: the model wrote a delete-orphan cascade onto the many side of a many-to-one. SQLAlchemy accepts it at import and raises on the first query, so the app builds, registers all 21 routes, and fails every request. That story has its own post.
• Everything here is one task. A 22-criterion Flask repair contract is not a benchmark suite, and a model that wins it is not thereby the best model.

What I'd tell a team

Put the undefined column in your table and don't fill it with a large number. The temptation is to write "$∞" or to rank the never-finishers by how cheaply they failed. Both re-import the gradient the unit is designed to remove. Seven rows saying never is uncomfortable to look at, which is the point — it's the honest shape of the result.

And check what your hardest failures have in common before blaming capability. Mine were all one behaviour: cleanup after deletion. That's a much more actionable finding than "the small models are worse," and a score ranking sorted by size would have hidden it completely.

What's next

I've now got a ranking table where six of the numbers turned out to be measuring my own code rather than the models. That accounting is the last post in this arc, and it's the one I'd want to read if someone else had written it.

The ranking table is the output of this project. This is the part I'd actually want to read if someone else had written it.

Six of the numbers in that table were measuring my harness, not the models. Here's the accounting, and then the running tally, which is the real result.

Three flat 17s, three different causes

Three models scored a dead-flat 17/22 three times running. All three were harness faults — and each was a different fault, which is why it took so long to see the pattern.

Every one of them was being served with a tool-call parser and then driven through a plain-text protocol. They emitted tool syntax into content, my parser threw it away, and they looked incapable.

Laguna went from a model I'd written off to one that solves the full contract. Same weights. The only thing that changed was that I stopped discarding its output.

Qwen3.6-35B is the one that should sting most. Its first run produced 82 files of prose and read as a model that couldn't code. That was reasoning leaking into content. Given the right reasoning parser and the tools protocol, the same weights solve the full contract twice out of three and turn in the cheapest cost per solved task in the fleet.

Three zeroes, three holes in the same guard

Three separate 0/22 results were also mine, and each exposed a different hole in the same build guard.

• The guard checked import app but never called create_app(). A model wrote valid Python that raised only when the factory ran. 17/22 → 0/22, holdout 0/13, reported as (patched app/__init__.py) — a success.
• The guard covered EDIT but not RUN. A model rewrote a file with RUN python3 << 'EOF', bypassing the edit path entirely.
• The guard fingerprinted only *.py. A RUN that wrote a non-Python file the app depends on was invisible to it.

All three are closed with regression tests now. The pattern across them is worth more than the individual fixes: each time I plugged the hole at the exact spot the failure appeared, and each time the next failure came through a door two feet to the left. The guard was verifying a proxy — that something imports, that a .py file changed — rather than the property I cared about, which is that the application can serve a request.

The fix that would have caught all three from the start: one request probe after each edit. Not a smarter guard. A guard that asks the actual question.

The four corrections that changed the outcome

Four times, someone told me in one sentence what was wrong, and each time it took me hours to implement and changed the result.

1. "you gave it a cheat code" — my first fast solo benchmark named the stack, showed a worked example lifted near-verbatim from a check, and told the agent to poll the grader. I spent hours arguing the number was contaminated. It was contaminated — and it was also right, which I never considered.
2. "it should be accept, revert, or change" — a third verdict turned the reviewer's deletions into repairs.
3. "i am your reviewer... stop, fix your code, resubmit" — the repair loop. It produced the first ACCEPT after nine straight rejections.
4. "restructure your tests... can you add a member, can you delete a member" — the behavioural contract. The single most effective change in the project.

I notice all four are about the measurement, not the models or the orchestration. Every hour I spent on the parts that felt like engineering was worth less than any one of those sentences.

Two claims I had to retract

"10% better outcome from 1.1% of the code." Reported from two-thirds of the data. The third arm then closed to 22/22 and the framework ranking collapsed into a tie. The honest version — same outcome, 20% faster — is much less quotable.

"The verification gate fired once in six runs, so it's variance." A count taken before the one run that mattered reached the phase where it began declaring DONE. A snapshot generalised into a structural claim, and it inverted the actual finding.

Reading a partial log as a result is the same error as scoring a workspace mid-flight. I did both, on the same day, about the same experiment.

The tally

Forty-four harness bugs. Zero confirmed model limitations.

The dominant shape, over and over: a protocol specified in the prompt and enforced nowhere. The model emits something reasonable, the harness silently discards it, and the model gets blamed.

The sharpest instance is worth the whole project. The solo agent's system prompt said it was graded by accept.py; its kickoff message said check.py. It followed the concrete instruction it received first, ran 60 steps, reported 163/173 (94%) against a grader that isn't the contract, and moved the real criteria from 17/22 to 17/22.

Zero progress, reported as success — because an agent optimises the feedback loop it can actually see. Which is the thesis of this entire project, reproduced by accident, at my expense.

What I'd tell a team

A model that looks broken is a harness bug until proven otherwise. Not as a slogan — as a default. Forty-four to zero is not a close call, and every single time I assumed the other way I was wrong.

Before recording any bad result, check three things: that you can see what the model actually emitted, that your parser accepted it, and that your guard tests the property you care about rather than a proxy for it. All six of my bad numbers would have been caught by the first check alone.

And log what the model received, not just what it sent. My worst bug was two files disagreeing about the grader's name, and it was invisible from either side alone.

What's next

The coding arc closes here. The same method — write the prediction before you look — turns out to be even more necessary when the output is a picture, because a picture is persuasive in a way a score isn't.

Two models make pictures on this box: FLUX.1-schnell and SDXL-Turbo. Twenty-seven images later, the headline is one sentence.

FLUX can spell. SDXL cannot.

Everything else is detail — but the detail is where the interesting parts live, including two conclusions from an earlier round that turned out to be wrong about the models and right about my own code, which by now is the most predictable outcome in this project.

The method, which matters more than the pictures

Every probe carries three fields written in this order: the prompt, what to expect, and what we got. The expectation is written from the prompt alone, before generating anything.

This sounds like bureaucracy. It's the only thing standing between an honest result and a page of pretty pictures with captions retrofitted to match. Diffusion output is enormously persuasive at a glance, and a caption written after looking will always find something to praise. Writing the target first means the page can't quietly move it.

Every prompt is also chosen to break something. Counting, spatial relations and attribute binding are the three most reliable ways to embarrass a diffusion model, so the funny prompts and the hard prompts are the same prompts. A rubber duck at a cash machine is a counting test wearing a joke.

The headline: legible text

Three exact strings, scored character by character.

FLUX: one exact, one off by a character, one partial. SDXL: 0 for 3.

The failure mode is the interesting part. SDXL's problem isn't blur — the letter shapes are individually crisp. Its signature failure is repetition: it renders the phrase two or three times, each version differently wrong. On the chalkboard it eventually wrote 44 — after also writing 445 and 4.54. It isn't failing to draw letters. It's failing to stop.

FLUX has a related tic. Asked for a book cover with a title and nothing else, it produced a clean product shot with a correct spine and shadow — and invented a two-line subtitle of pure gibberish. Unprompted text is where even the good model reverts to texture. That was in the written prediction, and it arrived on schedule.

6.3× the time and 3.2× the memory. Which sounds like a trade-off and isn't. If the picture contains words, SDXL is not the cheaper option — it isn't an option. Spend the 7.6 seconds or change the brief.

Resolution is a per-prompt decision, not a setting

The most useful practical finding here is one I nearly deleted.

SDXL-Turbo is a 512-native model. Pushed to 1024 it duplicates single subjects. Asked for "a lone lighthouse," it produced two — one on the cliff, one on a sea stack behind. The prompt contains the word lone, and the extra resolution overrode it.

The obvious lesson would be "run SDXL at 512." That lesson is wrong. On the same sweep, at the same 1024:

• the overgrown data centre gained vines across the ceiling and green LED text on the racks — a clear win
• the clockwork bird schematic became, arguably, the most detailed image produced in the whole set — from the model that loses to FLUX everywhere text is involved

The difference is whether the prompt contains a countable subject. With one lighthouse to duplicate, 1024 breaks the image. With no single subject to tile, 1024 buys real detail. Same model, same setting, opposite outcomes, decided by the noun in the prompt.

The 512 set is the control that makes this visible at all. I deleted it during a rewrite for being redundant and was caught. Without it, "SDXL duplicates at 1024" is an assertion; with it, it's a comparison.

The hard set: six probes designed to fail

FLUX: 3 pass, 1 near, 2 partial, 0 fail. SDXL: 0 for 6.

Neither model can count. Asked for exactly five ducks, FLUX gave six and SDXL gave about nineteen. Diffusion models have no counting mechanism; they render "a queue of ducks" and the number falls out of the composition. Off by one versus off by a factor of four is a real difference in degree, but neither is a model you can ask for a specific number of things.

Attribute bleed is the cleanest divider. Given three penguins with three different accessories, FLUX bound two of three correctly and let the red bow tie bleed one position right. SDXL applied the bow tie to all four penguins — it miscounted too — and dropped the other two attributes entirely. The textbook failure in miniature versus the textbook failure at full strength: the most visually dominant attribute wins and spreads.

FLUX's failures are the predicted failures, arriving on time. It got both tools into the correct paws — the hard part — then grew a third arm to hold everything. In every case the written prediction named the failure before the image existed. A model that fails where you expect is far more useful than one that fails at random.

And the prettiest image of the entire run is an SDXL failure: a genuinely beautiful illuminated manuscript, gold leaf and all, featuring two knights, no rubber duck, and an illegible caption. It answers none of the prompt. Grading on beauty would have ranked it first.

What I'd tell a team

Write the expectation before you generate. For text output you can get away with judging after the fact, because a wrong answer usually looks wrong. Image output doesn't work that way — it looks good while being wrong, and your judgement adapts to whatever appeared.

And keep the control set even when it looks redundant. Mine was the difference between an assertion and a comparison, and I'd already deleted it once.

What's next

The same method, applied to video, where a model can render a completely convincing scene and simply decline to perform the action you asked for.