Property testing with AI agents: the swarm run behind 13 merged fixes in uv, Prometheus and rv
Tools in this post
Property testing with AI agents means the agents write the test machinery, not the test cases: a simple reference implementation, generators of random operation sequences and assertions comparing the two, which a property-testing library then runs for CPU time instead of tokens. Inanna Malick ran this on six open source projects between 5 and 8 October 2026. By her count that run, plus an earlier one on Apollo Router, produced more than 100 correctness findings. By 28 October, 13 of her fix PRs from the six-repo run had merged: 4 in uv, 3 in Prometheus and 6 in rv. She published no token or dollar figures. This is her run, not mine; I rebuilt the record from her post and the GitHub threads.
What is property testing with an agent swarm?
Property testing with an agent swarm splits the work in two. Coding agents write the slow, careful part: a reference model, generators and assertions for each complicated data structure or algorithm. A property-testing library such as proptest then generates and checks cases, and shrinks each failure to a small reproduction. Malick's one-line version: "Have agents build the machinery to find bugs in your repo, then let that machinery generate and check cases without spending tokens on each one."
The reference model can be crude. In her words: "A Vec and some O(n²) loops may be enough to check a heavily optimized implementation." Once the suite exists, it "can check as many histories as you’re willing to run". One data point on scale: the discovery suite behind rv PR #808 "passed 600,000 generated cases" before the fix went up.
The swarm part is the division of labour: a planner picks targets, an orchestrator hands out work, and workers write property tests in their own git worktrees. Only a regression test and a minimal fix per bug go upstream.
The result: 13 merged fixes, and a lot still open
The run touched OpenAI Codex, Jujutsu, uv, Prometheus, Babel and rv. Her post says twelve fixes had merged as of 13 October. That count matches the merge timestamps on GitHub, and a 13th, rv #808, merged on 27 October. Most of what she filed is still open, mainly because nobody has reviewed it yet.
| Repo | Filed | Merged by 28 Oct | Closed unmerged | Open |
|---|---|---|---|---|
| uv | 11 PRs, 1 issue | 4 | 0 | 8 |
| Prometheus | 25 PRs, 1 issue | 3 | 1 | 21 |
| rv | 11 PRs | 6 | 4 | 1 |
| Babel | 9 PRs | 0 | 8 | 1 |
| Jujutsu | 4 PRs | 0 | 0 | 4 |
| OpenAI Codex | 14 issues | 0 PRs; 1 issue fixed by OpenAI | n/a | 13 issues |
Counted from the GitHub API on 28 October 2026. The Prometheus issue was fixed by another contributor, and the uv issue is still open.
The merged fixes are real but small. In uv, overriding leaf==1; extra == 'a' or extra == 'b' with leaf==2 could install leaf when neither extra was requested (#22237). In rv, ~> 1.0 accepted the prerelease 2.0.a, unlike RubyGems (#814). In Prometheus, BucketQuantile panicked on empty input where its docs promise NaN (#19927), though the PR itself says current PromQL call sites guard against that. For Codex, OpenAI closed issue #51878 and wrote that the fix will ship in "either 0.163.0 or 0.164.0".
The stack: OpenAI models, proptest and a skill file
The stack behind Malick's agent swarm is three OpenAI models in the OpenAI Codex harness, a property-testing library per language and one long skill file of instructions. The post names the models and the Rust library. The pull requests name the harness and the JavaScript library. How many agents ran at once is not published anywhere.
- Planner: GPT-6 Astra ("Astra for planning").
- Orchestrator and workers: GPT-6.1 Sol, plus "Lunas", in parallel git worktrees. The uv, Jujutsu and Babel PRs carry
Assisted-by: GPT-6.1 Sollines; the Prometheus and rv PRs name no model. That "Lunas" means GPT-6 Luna is my reading; no PR names it. All three are OpenAI GPT models. - Harness: OpenAI Codex. The post does not say so, but her Prometheus PRs read "Developed with AI assistance by OpenAI Codex", and her skill installs into
~/.codex/skills/. - Property-testing library: proptest and
proptest-state-machinefor Rust, and fast-check for Babel, whose PRs say "The discovery generators and fast-check dependency remain off this PR branch." The skill says to use "the repository's equivalent framework for other languages", which in Python means Hypothesis. The Go library for Prometheus is not named. - Instructions: proptest-praxis, an MIT-licensed skill of about 20 KB.
The run, stage by stage
The run follows four stages that Malick's post and the proptest-praxis skill describe: ask the maintainers where bugs hide, let a planner pick targets with a simple model, have workers build generators that are checked against a planted bug, and turn each confirmed bug into one small pull request. Only the last stage produces anything a maintainer sees.
Ask the maintainers what worries them
The run starts with people, not agents. "Pick a complex repo at work. Ask the people who know it well which parts worry them." The rv run was requested by one of its maintainers, according to her comment on rv #805. The planner gets those leads and is told to audit beyond them.
The planner picks targets with a simple way to check them
Good targets are "custom data structures, query planners, graph algorithms: complicated behavior with a simple way to check correctness". The skill lists the matching checks: a Vec and linear scans for an optimized collection, full recomputation for a cache or incremental summary, a fresh traversal for an incremental graph algorithm, a small interpreter for a query planner, round trips for parsers and serializers. My reading of the skill is that code with no simpler model to compare against is a poor target, and so is behaviour nobody can state as a contract; the skill tells agents to "Surface genuine ambiguity for discussion" instead of inventing a specification.
Workers build generators, and the run checks they can fail
Generators decide whether the suite finds anything. "Use a small key pool: fresh random keys mostly produce unrelated inserts and missing-key lookups." And: "Inspect sample traces: a million sequences that barely touch the same key aren’t buying you much." When a target produces nothing, the suite gets a planted bug first: "Temporarily remove a cache invalidation: the tests should catch stale results. If they pass, improve the generators or assertions."
Each confirmed bug becomes one small pull request
Each confirmed bug gets a branch from main with only a regression test and the fix, checked to fail before and pass after. "Nobody wants a giant agent-generated PR full of test machinery." The reviewer reads a plain unit test and never has to trust the generator or the reference model.
What one run costs is unpublished
No token count, dollar figure, number of parallel agents or hardware spec tied to this run has been published, in the post, the skill, the PRs or the author's Bluesky posts. The time figures are loose and self-reported: "about three days of agents running in the background" for Apollo Router, an afternoon for Codex, two hours for five bugs in the Swift library Boutique. The PR dates show the six-repo filings took about two days, from the evening of 5 October to the start of 8 October UTC, with the agents running "mostly in the background alongside regular work".
Two published studies of agents writing property tests do give costs, for a different method:
| Run | Model and harness | Scope | Cost and time |
|---|---|---|---|
| Malick, October 2026 | GPT-6 Astra, GPT-6.1 Sol, OpenAI Codex | 6 repos, 13 merged fixes | Not published |
| Maaz et al., October 2025 | Claude Opus 4.1, custom Claude Code command, Hypothesis | 100 Python packages, 984 bug reports | $5,474.20 in total, $9.93 per valid bug, 136.6 agent-hours, 2.21 billion tokens |
| PBT-Bench, 2026 | Eight models on OpenHands | 100 problems, 365 injected bugs | $2,288 across eight models, about 200 node-hours |
The difference in method is what makes the machinery cheaper. In the Maaz study, every module gets its own agent run of about 110 turns, which cost $5.81 at the median whether or not it found anything. In Malick's method the tokens go into building a suite once, and every case after that costs CPU only. No source measures the two side by side, so there is no break-even number to give. Nothing shows what she paid, since Sol also runs inside Codex on ChatGPT plans. The API list prices are $10 in and $50 out per million tokens for Astra, and $2 in and $10 out for Sol.
Where the run broke
Malick's run broke mostly at review, after the generators had done their part. Babel closed all eight fix PRs within about two hours under its AI policy, which says "You are not allowed to post LLM-generated PR descriptions". The rejection was about the descriptions, not whether the bugs were real. A PR with 17 failing tests is still open. The post does not mention it.
Maintainers also judged some findings not worth fixing. In rv, one fix was "incorrect (at least incompatible with RubyGems)", another "complicates version comparison just to support a very rare edge case". A Prometheus maintainer asked "have you run into this in practice?", and the PR was closed when the answer was no. Many of the findings are library-contract violations with no user-visible failure, and maintainers split on whether those are worth a merge. Volume was its own problem: "you seem to have opened ~25 PRs in quick succession," a Prometheus maintainer wrote on #19917.
OpenAI's cybersecurity filter also got in the way: an occasional "This content can’t be shown" notice, which Malick got past by talking the agent down or compacting the context.
A property suite catches what an AI reviewer cannot, and the reverse
A property suite and an AI code reviewer check different things, and uv #22237 shows both. The swarm found the extras bug and wrote the fix. GitHub Copilot's review bot then flagged that the same guard had to apply in Constraints::apply, or the constraint would still request the package. uv maintainer Charlie Marsh wrote that change himself, pushed it to her branch and merged the PR the same day. On rv, Copilot recommended approving every PR it reviewed, including two the maintainer objected to: he closed one, and Malick withdrew the other.
Screenshot of astral-sh/uv pull request #22237 by Inanna Malick, with a review by GitHub Copilot, 28 October 2026.
Every fix lands as an ordinary unit test, so the suite adds a layer next to the unit tests rather than replacing them. Apollo Router was "already backed by thousands of unit and integration tests plus extensive snapshot testing" before this method found more than 30 bugs in it, by Malick's count. That layer sits next to the six verification layers for agent-written pull requests I compared earlier, and its evidence is merged upstream fixes, not a benchmark score.
On Stackness, as of 28 October 2026, no public stack lists proptest, Hypothesis or fast-check, and uv sits on one, mine (how I count). The agents sit among the AI tools developers list.
Run it yourself
Running Malick's method yourself starts from the one piece she published: the proptest-praxis skill, MIT-licensed, in her agent-skills repository. No orchestration scripts or discovery suites for the six repos are public, so the swarm setup, the number of agents and the model mix are yours to choose. The install below is for OpenAI Codex, the harness her README names.
git clone https://github.com/inanna-malick/agent-skills
mkdir -p ~/.codex/skills
ln -s "$PWD/agent-skills/skills/proptest-praxis" ~/.codex/skills/proptest-praxisThen, in the agent, invoke $proptest-praxis and point it at the repository with the maintainers' list of worries. For other harnesses, give the agent the SKILL.md directly. Before trusting a quiet run, plant a bug and check the suite catches it. Each pull request carries one regression test and one fix, with the description in your own words.
New posts by email
About one email a week with what we published. No account needed.
One email to confirm first. Every email has a one-click unsubscribe.
Which of these do you use?
Nobody on Stackness runs rv yet. Tick what you use and be the first.
