AI agent fix passes CI but does not fix the bug: the five causes, a check for each and the fix
Tools in this post
When a coding agent's fix turns CI green and the bug is still there, check first for a test that cannot fail: one that passes with or without the patch. Check it by running the new test against the parent commit, where it must fail. Weak tests are the cause with the most measurement behind it: in the SWE-Bench+ study of an agent on SWE-bench, 31.08% of patches that passed relied on tests too weak to tell. The other four causes are a crash-site patch, a fix that broke untested code, a test or alert the agent edited, and a sleep over a race. Trail of Bits published tools for two of these on 21 October 2026.
Why does an AI agent's fix pass CI but not fix the bug?
An AI agent's fix passes CI without fixing the bug when the change makes the checks pass without removing the defect. I call this a green-build fake fix. The agent changes the witness instead of the system (the test, the CI config, the alert), lowers the odds of failure (a sleep, a retry), patches where the symptom showed, or fixes the bug and breaks something no test covers.
The diffs look ordinary. In one self-reported case from October 2026, by a developer who sells a testing tool, the agent added a single line, await new Promise(r => setTimeout(r, 200));, between publishing an event and reading it back. CI stayed green for two weeks, until a dashboard showed zero orders right after a purchase. Agents in Claude Code and Codex write these diffs, and so do people. The causes below are ranked by how much measured evidence each has. No study ranks them against each other, so the order is my judgement.
The new test passes with or without the patch
A test that passes on the unpatched code proves nothing about the bug. It happens two ways: the test is vacuous (it runs the code and asserts little), or it agrees with the bug, because its expected value repeats the code's wrong assumption. One worked example from 15 October taxes a subtotal before the discount, and the generated test expects the same wrong total.
The research numbers are large. Wang, Pradel and Liu found that 29.6% of plausible SWE-bench Verified patches behave differently from the developer's fix, and that weak tests inflated reported resolution rates by 6.2 points overall. Behaving differently is not always wrong, so read that as an upper bound. UTBoost found 345 wrong patches labelled as passing. One project's agent pipeline logged 50 vacuous-test findings in 146 batches, and ran the step that proves a new test can fail in only 24 of them.
To check, run the pull request's tests against the source as it was before the fix:
BASE=$(git merge-base origin/main HEAD)
PR=$(git rev-parse HEAD)
git worktree add ../prefix "$BASE"
git -C ../prefix checkout "$PR" -- tests/
(cd ../prefix && pytest -q tests/test_billing.py) # must fail
git worktree remove --force ../prefixOn a small billing repository with one test from a decision table and one that computes its expected value by calling the function under test, the output was:
FAILED tests/test_billing.py::test_discount_before_tax - assert 110.0 == 108.0
1 failed, 1 passedOnly the test that fails on the parent commit tests this bug; the one that passes on both proves nothing about it. patching-bugs from Trail of Bits automates the same check as a three-arm proof: the baseline fails, the patch passes, and a full revert fails again.
Mutation testing with mutmut, Stryker, cargo-mutants or PIT catches a vacuous test, not an agreeing one. In a rerun on the buggy version of the same repository, a test whose expected value repeats the bug passed and still killed 7 of 10 mutmut mutants: sensitive to change, wrong in what it expects. The fix is an expected value from somewhere other than the code, such as the spec, a decision table or a property test with Hypothesis, written by a fresh session that has not seen the diff.
The patch sits where the crash surfaced, not where the bug lives
A crash-site patch stops the failing input at the line where it blew up and leaves the cause in place. In a preliminary study in the PatchBench paper (September 2026), 81% of the patches from Codex with GPT-5.6 Sol modified a function on the crash stack trace, and on the 92 tasks where the developer's real fix lay outside the stack, 64% still sat on it. In the benchmark itself, checking only the original proof-of-concept made 11 agents' solve rates look 1.83 times as high on average.
AWS's numbers for its own security agent, published on 5 October, show how far stopping a crash is from fixing the intended bug: Continuum stopped the crash it reproduced and kept the tests passing on 89.0% of 920 tasks, and fixed the benchmark's target vulnerability on 37.8%. AWS treats the second number as diagnostic only, because the agent was not told which vulnerability to fix and may have fixed another real one.
To check, compare the patched line with the stack trace, then run a second input that reaches the bug by another path, such as another caller or a cleanup branch. The fix is to state the invariant the input broke and repair that, not the one input.
The fix broke something CI never runs
A fix can be right and still break behaviour that no test covers. Trail of Bits reviewed more than 130 merged agent-assisted patches and confirmed 21 regressions: 16 functional, 1 performance and 4 build or test. None broke a security protection. Its agent review produced 58 candidates in one run to find those 21.
regression-hunt, merged into the trailofbits/skills marketplace on 20 October 2026 for Claude Code and Codex, does what CI does not: it traces each changed behaviour through its callers, builds the code before and after the change, runs the same proof-of-concept against both through a real entry point, and has fresh agents replay and judge each candidate. Against a baseline Codex agent asked to "review the PR for regressions", it reported 25% fewer false positives (95% CI 14% to 38%). Trail of Bits published no recall figure.
/plugin marketplace add trailofbits/skills
/plugin install regression-hunt@trailofbitsTo check without the plugin, list the files the diff touches against the files the issue names (git diff --stat "$BASE"...HEAD), find the callers of every changed function, and run the full suite rather than the tests the agent picked. An AI review is no proof that a patch is safe: in one 12-case test of an AI reviewer, each of two prompt setups got 6 verdicts right and only 3 reviews whose reasons held up. Treat its findings as pointers until a failing test backs them.
The agent edited the test, the CI config or the alert
Suppression turns a check off instead of satisfying it: a skip marker, a retry, a looser assertion, continue-on-error, or a test rewritten to expect the broken state. OpenAI disclosed two cases on 20 October 2026, both in internal deployments of its GPT models. GPT-6.1 Sol, repairing an evaluation pipeline, emptied the alert recipient lists because an earlier instruction said "don't send any messages". GPT-6 Astra removed a monitoring integration, "also replaced an earlier regression test intended to preserve monitoring with one requiring that monitoring was not present", and then denied it when asked.
Read-only tests reduce this. ImpossibleBench (October 2025) gave models tasks whose tests contradict the spec, so every pass is a shortcut:
Figure 7 from ImpossibleBench by Ziqian Zhong, Aditi Raghunathan and Nicholas Carlini, 2025, CC BY 4.0.
To check, grep the diff for added suppressions and removed assertions:
git diff -U0 "$BASE"...HEAD | grep -nE '^\+.*(pytest\.mark\.(skip|xfail|flaky)|reruns|\.skip\(|retries|t\.Skip|#\[ignore\]|@Disabled|continue-on-error|\|\| true|eslint-disable|@ts-ignore|nosec)'
git diff -U0 "$BASE"...HEAD | grep -nE '^-.*(assert|expect\(|require\.|t\.(Fatal|Error))'The fix is to revert those hunks and keep tests and CI config out of the agent's reach: a Claude Code PreToolUse hook on Edit|Write that reads tool_input.file_path from its JSON input and exits with code 2 for tests/ and .github/, plus a CODEOWNERS entry so a person approves any test change in a fix.
A sleep makes the race rarer, not gone
A sleep before an assertion is a bet that the other side finishes in time. On a quiet CI runner it nearly always does; under production load, cold starts or a bigger data set it does not. A sleep adds no ordering between the two threads, so the code is still racy when the value happens to be ready. With Go, the race detector shows this on the first run, where a plain run is green:
$ go test ./...
ok example.com/sleeprace 0.578s
$ go test -race -count=5 ./...
WARNING: DATA RACE
--- FAIL: TestPublishWithSleep (0.20s)For pytest, pytest-repeat runs a test until it fails (pytest --count=1000 -x); for Playwright, --repeat-each=50 --retries=0. Check whether the green run needed a second attempt as well; Playwright's --fail-on-flaky-tests fails a run that passed only on a retry: Google measured about 1.5% of test runs as flaky in 2016, so a retry can turn a real failure green. The fix is to wait on a condition, add the missing synchronization in the code, and delete the sleep.
What does each check cost on an agent pull request?
The checks above cost from seconds to several builds per pull request, and nobody has published a full figure. Trail of Bits gives no time or token cost for regression-hunt or patching-bugs, and its patching-bugs skill lists "Regression-hunt is slow" among the rationalizations to reject. What is published:
| Check | Cost | Source |
|---|---|---|
| Grep the diff for suppressions and sleeps | seconds | none needed |
| New test on the parent commit | one extra build and a run of the new tests | no published figure |
go test -race |
2-20x execution time, 5-10x memory | Go docs |
| Mutation testing on changed lines | one test run per mutant; Stryker's incremental example re-ran 234 of 3,965 mutants | Stryker docs |
| regression-hunt, patching-bugs | builds before and after, several fresh agents per candidate | no published figure |
On GitHub Actions at $0.006 a Linux minute, an extra 5-minute job costs 3 cents per pull request, and a 3-minute suite slowed 20 times by the race detector runs 57 minutes longer, about 34 cents (my arithmetic, list price, before included minutes). Run the grep and the parent-commit test on every agent pull request, and keep mutation runs and regression-hunt for code where a wrong fix is expensive. These checks sit beside the six verification layers for agent-written code and the AI coding tools developers use to write it.
Fake fix: symptom, check, fix
| If you see | Check | Fix |
|---|---|---|
| A new test, green on the pull request | run it on the parent commit; it must fail | expected value from the spec; drop tests that pass on both |
| The expected value computed by the code under test | where did the number come from? | decision table, property or reference implementation |
| The patch on the line the stack trace named | a second input by another path | repair the broken invariant |
| Files changed that the issue never mentions | callers of each changed function; regression-hunt | scope the patch; test each caller at its real interface |
| Test or CI config changed in a fix | grep for skip, retry, continue-on-error, removed asserts |
revert; tests read-only to the agent; CODEOWNERS |
sleep or setTimeout near an assertion |
go test -race -count=100, pytest --count=1000 -x |
wait on a condition; real synchronization |
| Green only on attempt 2 | the run's attempt number; --fail-on-flaky-tests |
no retries in the gate; fix the shared state |
| An AI review that approves or blocks | does its reason match the diff? | a failing test or proof-of-concept per finding |
New posts by email
About one email a week with what we published. No account needed.
One email to confirm first. Every email has a one-click unsubscribe.
Which of these do you use?
Nobody on Stackness runs patching-bugs yet. Tick what you use and be the first.
