Is xhigh reasoning effort worth it for a coding agent? What SlopCodeBench 10/2026 and an 85-task Auto run measured

For most coding work, the public numbers say xhigh reasoning effort does not pay for itself on current models. In HumanLayer's SlopCodeBench run, published on 15 October 2026, Claude Opus 5.5 passed 28 of 196 checkpoints at medium effort and 25 at xhigh. The xhigh run cost 4.22 times as much: $817.04 against $193.47. Across the three models run at both levels, xhigh cost 2.11 to 4.22 times medium. Strict passes rose for one model and fell for two. Each setting ran once, so these are single measurements, not settled differences.

Xhigh cost 2.1 to 4.2 times medium and bought no reliable gain

SlopCodeBench 10/2026 ran three models at both medium and xhigh effort on the full benchmark: 36 problems and 196 checkpoints, where each checkpoint extends code the agent wrote at the previous one. A strict pass means every test category passes, regression tests included. The table is HumanLayer's, with cost per strict pass derived by me from its estimated cost.

The "xhigh considered harmful" headline appears only in the post's title. The body is more careful: "This doesn't prove that more reasoning hurts coding. It says higher effort was not a reliable way to buy more strict passes in these runs." The costs are recorded inference cost, not invoices, and HumanLayer published no token counts or run times. GPT-6 Astra, OpenAI's top model, led the full table at 32 strict passes.

What does the reasoning effort setting change in a coding agent?

Reasoning effort tells the model how many output tokens to spend on a task: thinking, tool calls and the text it writes. Anthropic's effort docs call it "a behavioral signal, not a strict token budget". At higher effort, Claude may make more tool calls, explain its plan and write longer summaries. Claude offers low, medium, high, xhigh and max. Opus 5.5 defaults to medium in the API and in Claude Code.

OpenAI's reasoning docs list levels from none to max, depending on the model, and say of xhigh: "Only use when your evals show a clear benefit that justifies the extra latency and cost." Codex reads the setting from model_reasoning_effort. Anthropic's Opus 5.5 guide says the same thing: "Reserve xhigh and max for work where you've measured a quality gain." Thinking tokens are billed as output tokens: $20 per million on Opus 5.5 and $50 on GPT-6 Astra for prompts up to 272,000 tokens.

Opus 5.5 passed fewer checkpoints at xhigh than at medium

Opus 5.5 at xhigh passed three fewer strict checkpoints than at medium, 25 against 28, and spent $623.57 more to do it. Per checkpoint, it cost $0.99 at medium and $4.17 at xhigh. Its core pass rate, which counts only the main tests, also fell, from 73.98% to 71.94%.

Xhigh did produce cleaner code by one measure. On 28 matched problems, 77.75% of the xhigh run's lines were flagged by at least one of the benchmark's slop checks, against 94.25% for medium. The post calls that "close to ~5x cleaner". Neither the ratio of flagged lines (1.21) nor the ratio of unflagged lines (3.9) gives 5, and the post shows no metric that does. Commenters on Hacker News raised the noise: "When scores are ~15%, it's unclear how much is just in the noise."

A strict pass cost $6 to $9 at medium and $15 to $60 at xhigh

Across all ten campaigns in the SlopCodeBench table, every medium run cost between $6.16 and $9.00 per strict pass, and every xhigh run cost between $15.21 and $59.62. The cheapest was GPT-6.1 Sol at medium, with 29 passes for $178.51. The most expensive was Anthropic's Claude Fable 5.1 at xhigh, with 26 passes for $1,550.07; the first version of the post noted that Fable's cost metering had gaps.

GPT-6 Astra was the only model where xhigh bought more passes, and each extra pass cost $165.86. For the three models run at both levels, the price of a pass varied 1.3 to 1.6 times between models and 2.4 to 4.7 times between effort levels.

Caveman's Auto saved 54% mostly by running Opus at medium

Caveman's README claims Auto is "54% cheaper with Auto, same score" across "85 coding tasks vs Opus 5.5 at xhigh". The benchmark doc behind it is an interim run dated 3 October 2026, paused at 85 of 169 paired tasks from SWE-Interact and SWE-Together, one try per task in Claude Code. Auto resolved 43.7% against 44.3% for Opus 5.5 at xhigh, at $3.59 against $7.83 per attempt.

The doc explains the saving itself. Auto ran Opus 5.5 at medium on 87% of requests, Opus at high on 8% and Claude Sonnet 5.5 on 5%. In its words: "Most of the saving comes from lower effort (Opus xhigh to Opus medium or high), not from cheaper models." The score gap of 0.5 points has a 95% interval of −6.8 to +5.7 points, and median time fell from 35.2 to 15.9 minutes. Caveman sells the cloud service Auto runs on, ran the test itself and has not published the raw runs. The baseline it undercut on cost, xhigh on every request, is two levels above Opus 5.5's default.

unset CLAUDE_CODE_EFFORT_LEVEL
claude -p "$(cat task.md)" --effort medium --output-format json > medium-1.json
claude -p "$(cat task.md)" --effort xhigh --output-format json > xhigh-1.json
jq '.total_cost_usd' medium-1.json xhigh-1.json

codex exec -c model_reasoning_effort='"xhigh"' --json "$(cat task.md)" > xhigh-1.jsonl