Does a harness make a cheap model as reliable as a frontier model? Harnessed GPT-6 Luna against bare Sol
Tools in this post
On one task, yes. In Naresh Jain's experiment on martinfowler.com, published 13 October 2026, cost-optimized GPT-6 Luna inside a behavioral harness built an acceptable Java service in 10 of 10 runs, the same as GPT-6 Sol with no harness, at an estimated $0.273 per run against $0.406. It also took 35 minutes per run against 6 minutes 51 seconds, and used 21 times the tokens. Harnessed Luna wins on cost per run, and bare Sol wins on latency and on not having to build anything first.
Does a harness make a cheap model as reliable as a frontier model?
A behavioral harness made GPT-6 Luna as reliable as GPT-6 Sol on Jain's one task: both passed all 10 runs, where bare Luna passed 3. Jain ran Codex at low reasoning effort, 10 paired runs per model, each in a fresh Docker container. A run counted only if the service started and passed all 178 hidden acceptance tests, which the agent never saw.
| Per run, 10-run average | Luna, no harness | Luna, harness | Sol, no harness | Sol, harness |
|---|---|---|---|---|
| Accepted runs | 3 of 10 | 10 of 10 | 10 of 10 | 10 of 10 |
| Estimated cost | $0.086 | $0.273 | $0.406 | $2.679 |
| Tokens | 5.86M | 22.93M | 1.08M | 11.33M |
| Wall time | 15m 56s | 35m 00s | 6m 51s | 19m 53s |
| Cost per accepted app | $0.287 | $0.273 | $0.406 | $2.679 |
| List price per million tokens, input / cached / output | $0.10 / $0.01 / $0.50 | same | $2.00 / $0.20 / $10.00 | same |
Run figures are from Jain's article. Cost per accepted app is my division of cost by pass rate. List prices are from OpenAI's pricing page, read 23 October 2026, for prompts up to 272,000 tokens; the article does not print the prices it used. The costs are "API-equivalent estimates based on recorded token use and model prices, not actual billed costs."
The task was a Spring Boot backend-for-frontend that calls an Order API and publishes to Kafka, with awkward rules such as returning 202 and a link to poll when a creation times out. Jain calls its written brief "a favorable starting point for an unaided agent".
A behavioral harness is guides and sensors around the agent
Jain's behavioral harness, borrowing Birgitta Böckeler's terms, combines guides that steer the agent before it acts and sensors that report back after it acts. The guides were OpenAPI and AsyncAPI specs, request and response examples covering failure cases and Kafka events, and an API-development skill. The sensors were contract checks and API tests, run against contract-driven HTTP and Kafka mocks, whose reports went back to the agent during the build.
Figure 4 from Are Harnesses Worth the Cost? by Naresh Jain, 2026, resized and reused with attribution as the martinfowler.com FAQ allows.
None of the checks used an LLM as a judge: "The behavioral checks produced computational feedback without an LLM judging correctness." The agent had to call the test command itself, so the feedback did not run after every edit, and it was told to fix the code rather than weaken the expectations. This is narrower than the harness in the broad sense, which is everything in an agent except the model. Here Codex stayed the same and the harness was task-specific.
Jain founded Specmatic, and the harness was built with it. He writes that "equivalent contract-testing and service virtualization tools could provide the same capabilities."
The harness lifted GPT-6 Luna from 3 to 10 and left GPT-6 Sol's pass rate alone
On GPT-6 Sol the behavioral harness changed only the bill: 10 of 10 either way, at 6.6 times the cost, 10.5 times the tokens and 2.9 times the wall time. On GPT-6 Luna it changed the outcome. Bare Luna still passed 94.8% of the functional checks, so the failures were a few missed rules, not broken apps. Jain's words: "In this sample, the harness did not unlock a capability Luna lacked; it turned a 3-in-10 baseline into 10-for-10 reliability."
Bare Luna's misses were at protocol and integration boundaries. In seven runs a timed-out creation returned 503, 400 or 429 instead of 202 with a monitor link. Two apps used Spring's JSON serializer, which added an unexpected __TypeId__ Kafka header. One read ORDER_API_API_KEY where the launch supplied ORDER_API_KEY. Jain reads the harness as "an external memory and attention aid, pulling dropped requirements back into focus whenever a sensor went red."
This matches what the harness explainer found in September's studies: on a strong model the harness moves the bill more than the pass rate, and on a weak one it can decide whether the task gets done. Sol had no headroom to show a gain, since it passed every run either way. Google's Harness2 raised Claude Opus 4.8 and Opus 5 by 2.5 to 5.5 points when they edited their own harnesses, on benchmarks they had not saturated (Harness2 post), and Jain writes: "A more demanding architecture or a trickier resiliency scenario might expose gaps in Sol that require a harness."
Harnessed Luna costs a third less than bare Sol and takes five times as long
Harnessed GPT-6 Luna cost $0.273 per run against $0.406 for bare GPT-6 Sol, 33% less, while Luna's list prices are a twentieth of Sol's. The gap closes because the harness multiplies tokens: 22.9 million per run against Sol's 1.1 million, 21 times as many, and 35 minutes of wall time against under 7. On Luna the harness itself added 17.1 million tokens, 19 minutes and $0.187 per run.
Dividing each estimate by its token count gives about $0.012 per million tokens for harnessed Luna and $0.38 for bare Sol, both far below the uncached input prices of $0.10 and $2.00. That only works out if most tokens were cache reads. Jain did not publish the split, so a run with a cold cache could cost several times more.
Token volume also makes the long-context tier matter. Luna's input price doubles above 272,000 tokens in one request. Swap in Claude Haiku 5.5, which lists at the same $0.10 and $0.50, and every request with a prompt over 100,000 tokens bills at five times the rate (Haiku 5.5 vs Luna). Jain did not test Haiku, so how a harness this chatty would bill there is unmeasured.
Retrying bare Luna costs about the same as harnessing it
Bare GPT-6 Luna at 3 of 10 costs $0.086 per run, so getting one accepted app by retrying costs about $0.287 on average, against $0.273 for harnessed Luna. That is my arithmetic on Jain's numbers, not something he ran. It also takes about 53 minutes of expected wall time per accepted app, and it assumes each retry fails independently.
Retrying only works if something tells a good run from a bad one. In Jain's setup that is the hidden 178-test suite, run by an independent evaluator with its own fixtures. Without such a suite, bare Luna ships a failing service seven times in ten and nothing says which seven. A team that already has a strong acceptance suite has the gate that retries need; the harness adds the specs, examples and mocks the agent works against during the build.
Nobody measured what the harness costs to build
Jain's estimates leave out "the upfront effort of authoring, reviewing, and maintaining the contracts, examples, assertions, and harness", and the time spent diagnosing failed runs. Harnessed GPT-6 Luna saves $0.133 per run against bare Sol, so a harness that took a day of engineering at $100 an hour, $800, pays back after about 6,000 runs of the same task. That figure is an illustration, not a measurement.
The sample is small in both directions. 3 of 10 against 10 of 10 is unlikely to be chance (a one-sided Fisher exact test gives p ≈ 0.0015), but 10 of 10 is still consistent with a true pass rate as low as 69% at 95% confidence. Jain says the same: "This small sample cannot tell us the long-run failure rate or the total cost of achieving an acceptable application." One greenfield Java service, one effort level and one unusually complete brief cannot show how harnessed Luna does on legacy code.
A Carnegie Mellon paper (arXiv 2607.08938, July 2026) had Gemini 3.1 Pro rewrite the harness for small models on seven business tasks. Its best small model reached 80.2% accuracy at $0.071 per instance against the frontier model's 89.7% at $1.735, and the $20 tuning budget paid back after about 13 runs. That harness moved work out of the model and made two of three small models cheaper and all three faster. Jain's harness added verification loops and made Luna slower and dearer per run.
Test harness against model on your own task
Jain's setup is a recipe a team can copy before changing its default model: one frozen task, a hidden acceptance suite, and the same agent at a fixed effort level in fresh containers. The model routing post collects the prices developers measured for splitting work between models in September.
- Freeze one representative task with a written brief, and write acceptance tests the agent never sees.
- Run each model bare and harnessed, 10 times each, in isolated containers with no shared memory, sessions or skills.
- Record accepted runs, tokens, wall time and cost from your own bills, then divide cost by pass rate.
- Add the arm Jain proposed and did not measure: the frontier model with the tests as a final gate only. For frontier models he suggests running "the automated tests once as a deterministic final verification gate before acceptance".
- Log the hours spent writing the harness next to the per-run saving.
The agents and models people pair for this kind of split are on the page of AI tools developers list in their stacks.
Pick harnessed Luna if, pick bare Sol if
Pick harnessed GPT-6 Luna if:
- the same kind of task runs hundreds or thousands of times, so the harness pays back
- you already have specs, contract mocks or an acceptance suite the agent can call
- the work runs in the background and 35 minutes is fine
- prompts stay under the cheap model's long-context price step
Pick bare GPT-6 Sol if:
- the task is one-off or changes shape each time
- someone is waiting on the result, since Sol took a fifth of the time
- nobody has time to write and maintain contracts, examples and mocks
- the frontier model already passes the task unaided, where Jain suggests the tests only as a final gate
New posts by email
About one email a week with what we published. No account needed.
One email to confirm first. Every email has a one-click unsubscribe.
Which of these do you use?
Nobody on Stackness runs GPT yet. Tick what you use and be the first.
