Multi-model agent team vs a single frontier model: what the Memdoor claim measures, and the cost per task
Tools in this post
On the 100 newest hard problems in LiveCodeBench, a manager-and-worker loop of Qwen3.8-27B calls scored 92.4% against 90.4% for one call to Claude Fable 5, at $0.48 a problem against $0.61. The 2-point lead is not significant, and the team wrote about 15 times as many output tokens. Those are the numbers from version 2 of the GVS5H paper, dated 21 September 2026. Memdoor's "beats Fable 5" post of 31 October replicated that loop on 10 problems, and its own README calls the result parity. A team of small models pays off on problems a test can check, run on your own hardware; one frontier call wins on latency, on repo-level work and wherever there are no tests.
Multi-model agent team vs single frontier model, side by side
The deciding facts between a team of small open models and a single frontier call are score, cost per task, token volume, latency and where the model runs. Scores, costs and token counts below come from the GVS5H paper's five-pass runs on LiveCodeBench release_v6; calls, time and the first-ten scores from Memdoor's run files; prices from OpenRouter and Anthropic on 1 November. The Fable 5 column was measured by the paper's authors, not by Memdoor.
| Qwen3.8-27B team (GVS5H loop) | Fable 5, one call | |
|---|---|---|
| Score, 100 hard problems, 5 passes | 92.4% ± 2.3 | 90.4% ± 1.4 |
| Score, first 10 problems | 9.8 (Memdoor), 9.6 (paper's harness) | 8.0 |
| Cost per problem, list price | $0.48 | $0.61 |
| Output tokens per 100 problems | 18.6 million | 1.2 million |
| Calls per problem | 5 to 22 in Memdoor's runs | 1 |
| Time per problem | Median 40 min in Memdoor's runs, longest 4 h | Not published |
| Runs on | Your GPU or any of 22 OpenRouter hosts | Anthropic's API |
| Licence | Apache-2.0 weights | Proprietary |
| Price per million tokens, in / out | $0.21 / $2.55 in the paper; $0.15 / $1.875 on OpenRouter today | $10 / $50 |
Price and performance chart from "Zero-Shot Self-Orchestration with Ledger-Based Control Improves Coding in Language Models" (GVS5H) by Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi, Xihao Sun, Juhyun Lee and Simon Lee, Persis Capital, 2026, CC BY 4.0.
The team of five is one Qwen3.8-27B model playing six roles
The "team" in both GVS5H and Memdoor is a single model, Qwen3.8-27B, called again and again in a fresh context, each time in a different role. A manager writes a plan and seed tasks, a worker brainstorms approaches into notes.md, the manager picks one next task, a fresh worker rewrites solution.py, and a script runs the public sample tests. A finalizer writes the answer when the loop ends without a clean pass.
There is no voting and no second model. The paper says "every role but the verifier is the same underlying model". The loop stops when the tests pass and the manager says done, or after 10 manager-worker rounds. The calls run one after another, not as five models side by side.
Memdoor's team-solve README shows a different workflow from the one that scored 9.8. The eight files in steps/ allow one repair round, a shape Memdoor's earlier 100-problem bench scored at 47 of 100 with thinking off, against 35 for a single call. The 9.8 came from a generated workflow, bench/results/parity-5pass/workflow/gen.py, that runs the paper's 10-round loop.
Memdoor ran 10 problems and copied Fable 5's column
Memdoor measured 10 problems in 5 passes, 50 graded runs on LiveCodeBench release_v6, contests from spring 2025. It scored 9.8 out of 10 against 9.6 for the paper's own harness on the same problems. The Fable 5 score of 8.0 came from the paper's result files, one call per problem at effort: high per the authors' run config; the README says it "was not run in this harness".
The README is plainer than the Hacker News title: "On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it." The 50 runs cost $35.79 on OpenRouter, and the one lost run was a correct program that the manager talked the team into rewriting.
The full-set result depends on fixes the paper made to the grading. LiveCodeBench's stock grader mis-scored programs reading input with sys.stdin.buffer.readline(), which Qwen uses and Fable 5 does not, and the authors also corrected some answer keys. With the stock grader and the original keys, the team scores 84.4 and Fable 5 87.4. Contamination cannot be ruled out for either side: the problems date from December 2024 to April 2025, Fable 5 came out in June 2026, and Qwen3.8-27B in August 2026.
The Qwen3.8-27B team writes 15 times the tokens for a 22% smaller bill
The Qwen3.8-27B loop is cheaper per problem than Fable 5 only because Qwen's output tokens cost about a twentieth as much at the paper's prices. The paper prices one pass over 100 problems at $47.82 for the team and $61.11 for Fable 5; at OpenRouter's price today the team's tokens would cost about $35. The team wrote 18.6 million output tokens to Fable 5's 1.2 million, and used 1.5 million input tokens to Fable 5's 90,000.
That ratio decides what the team costs you outside a benchmark. On a subscription with rate limits, on a host that caps output at 32,768 tokens like Cerebras, or on one local GPU, the team's volume is the bill. Memdoor's docs put the floor at "a 27B at Q5 or above on a 24 GB card", and a hard problem at about 300,000 output tokens a run; at 50 tokens a second, my arithmetic gives about 100 minutes. The price per task also moves with the host: the paper priced Qwen at $0.21 / $2.55 per million, while Memdoor paid $2.30 to $3.00 per million output tokens on Modal, Wafer and CoreWeave.
The same loop on a stronger small model is the better buy. Qwen3.8-Flash-Next with the manager scored 93.0% for $5.76 per 100 problems, a tenth of Fable 5's cost. Against Claude Opus 5.5, $4 / $20 per million tokens and the model Anthropic says to start with "for most workloads", nobody has run the comparison.
The loop beats best-of-five sampling on problems a test can check
The Qwen loop's gain is not only extra sampling. I computed pass@k from the paper's stored run files, with an oracle that picks a passing answer whenever one exists: two or three independent Qwen single calls, at about the loop's budget, reach 81.6% to 87.1%. Five calls with a perfect picker reach 90.0% at $0.95 a problem, twice the loop's cost and still below its 92.4%.
The paper finds the gain largest where the base model cannot organise its own reasoning: +25.6 points for Qwen3.8-27B, whose single calls ran past the 128,000-token cap in 150 of 500 attempts, against +3.6 for DeepSeek-V4.1-Flash. Some models gained nothing. The only outside signal is the public sample tests, so the 27 LeetCode problems, whose call-based tests the verifier does not run, got no feedback at all.
Outside contest problems the evidence is mixed. Cohere's study of 14 October found teams gained up to about 10 points on ProgramBench at three to seven times the cost, and that cheap workers need a strong lead: "A Terra lead with Luna workers still fell short of a Sol agent working alone (51.6% against 57.7%)". On the Artificial Analysis Intelligence Index v5, a composite with 60% private test sets, Qwen3.8-27B scores 20 against 58 for Opus 5.5, and no team result exists there. The YC Labs game swarms of 29 October ran frontier models only, so they say nothing about small against large.
How do you verify a self-reported agent team benchmark?
Verifying an agent team benchmark takes four checks, and the Memdoor case fails the first two as published. Ask who measured the frontier baseline and at what effort; compare against a compute-matched baseline, not one call; check that the problems postdate both models; and confirm that the workflow you would copy is the one that was scored. The GVS5H repository stores every run, so most of this takes minutes.
- Regrade a stored answer. Memdoor's
bench/results/parity-5pass/holds each run's prompts, replies, per-call costs andsolution.py;bench/grade.py --private <id> solution.pyreruns the hidden tests. - Run the compute-matched baseline. The paper says "We do not attempt an equal-token comparison across the same model". Best-of-N on the single model at the same spend is the number to beat, as in the section above.
- Use problems newer than both models. LiveCodeBench's last release ends in April 2025, so no date filter works here. AtCoder or LeetCode contests after August 2026, with your own tests, do.
- Put the frontier model in the same loop. The paper ran Fable 5 single-only "due to its high cost", so how much the loop would add to Fable 5 is unknown.
The rerun command for the paper's own harness, from the GVS5H README:
cd codebase/v2-current
LCB_RELEASE=release_v6 ESCALATION_CLOUD_MAX_TOKENS=128000 ESCALATION_CLOUD_TIMEOUT=7200 \
MULTIAGENT_MODEL=openai:gpt-5.6-terra \
uv run --no-project --python 3.12 --with 'datasets<4' --with numpy --with anthropic \
python escalation/run_bench.py --engine multiagent --only lcb --lcb 100 --parallel 8Switching between a team and one call is a test-suite problem
Moving from one frontier call to a team of small models is cheap in code and expensive in inputs: the loop needs a task file and paired test cases, because the sample tests are its only ground truth. Memdoor installs as one Go binary and connects to OpenRouter, Anthropic, OpenAI, Gemini or a local runtime such as Ollama, whose default 4,096-token window has to be raised to 32,000 or more first.
Moving back is one API call with adaptive thinking. Fable 5 is a legacy model, still active with no retirement before 9 June 2027, at the same $10 / $50 as Fable 5.1. The paper's code, GVS5H, is a Python harness for contest-style problems, not a coding agent, so neither direction touches the agent you use for repository work. For that, the harness and model pairing question is the closer one, among the language models developers list. On Stackness, as of 1 November 2026, Claude Opus is on 4 real public stacks and Qwen on none (how I count).
Pick a team if, pick a single frontier call if
Pick a team of small open models if:
- The task has a machine check: sample tests, a checker or a type system.
- You need open weights or your own hardware; Qwen3.8-27B is Apache-2.0 and needs about 24 GB of VRAM at Q5.
- Tens of minutes per task is acceptable.
- Your base model rambles past its output cap or never plans; that is where the loop added 25.6 points.
- You can pick the small model too: Qwen3.8-Flash-Next in the loop scored 93.0% for a tenth of Fable 5's cost.
Pick a single frontier call if:
- You need the answer in one round trip.
- The work is repository-level or agentic, where no team result exists and Qwen3.8-27B scores 20 to Opus 5.5's 58 on the Artificial Analysis index.
- There are no tests to drive a loop.
- Token volume costs you something: rate limits, logs, review time, a 32K output cap.
- You want one measured baseline before adding a harness around it.
New posts by email
About one email a week with what we published. No account needed.
One email to confirm first. Every email has a one-click unsubscribe.
Which of these do you use?
Nobody on Stackness runs Qwen yet. Tick what you use and be the first.
