Migrating a codebase with a coding agent: what the 2026 runs cost and how they checked the result
Tools in this post
The best-documented agent-driven migration of 2026 is GitHub's. Between 12 May and 21 August, Stephen Toub, a Microsoft Distinguished Engineer writing on the GitHub blog, moved the GitHub Copilot agent runtime from TypeScript to 832,378 lines of production Rust with GitHub Copilot, across 128 pull requests, for about $120,000 of tokens and, by his own estimate, three weeks of his time. Asana took Enzyme out of its tests in two calendar weeks for about $11,000 of OpenAI Codex usage. A VB6 to C# study ported 12 features in 59 agent minutes for $50.29, and a person checked that 70% of the behaviour survived. Each team checked the result with something besides the agent's word: the old test suite and CI, a person scoring behaviour, or the old system running beside the new one.
These are other people's runs. I link each one, quote their numbers as published and mark where a number is self-reported.
What is an agent-driven migration?
An agent-driven migration is a change where a coding agent replaces the implementation underneath a system (language, framework, test library, database) while the behaviour has to stay the same. A refactor reshapes code inside one stack and has no fixed finish line. A migration has two: the old stack is gone, and the new system does what the old one did.
In a bug fix, a failing test turning green is the proof. In a migration no test should change colour at all, so, as the authors of SWE Refactor Bench put it in August 2026, "a correct migration and an untouched repository can therefore receive the same perfect test score". Toub's rule for his port was "Translate first, redesign second."
Four 2026 runs with published numbers
Four agent-driven migrations published in 2026 give time, size or cost: GitHub's TypeScript to Rust port, Asana's Enzyme to React Testing Library move, a VB6 to C# research case study, and Temporal's billing store move to ClickHouse. Two 2025 runs from Airbnb and Google are the usual comparison points. All are self-reported. The VB6 study releases its instruction files, Asana prints its whole prompt and GitHub its review prompt.
| Run | Who, when | Agent | Size | Calendar time | Human time | Cost |
|---|---|---|---|---|---|---|
| Copilot runtime, TypeScript to Rust | Stephen Toub, GitHub, 16 Sep 2026 | Copilot app and CLI, several models | ~430,000 TS lines in, 832,378 Rust lines out, 128 PRs | ~14.5 weeks | ~3 weeks (his estimate) | ~$120,000 |
| Enzyme to React Testing Library | Asana, 7 Aug 2026 | Codex, up to 4 agents | Not given | 2 weeks | ~1.5 engineer-weeks | ~$11,000 model + ~$1,000 infra |
| VB6 ERP to C# .NET 10 | Alves et al., 29 Aug 2026 | Claude Code 2.1.96, Opus 4.6 | 12 features, 35,970 lines | 59 agent minutes | Not measured | $50.29 |
| Billing read path to ClickHouse | Temporal, 11 Aug 2026 | Not named | "the few dozen queries" | 6 weeks of shadowing | Not given | Not given |
| Enzyme to RTL (2025) | Airbnb, 13 Mar 2025 | LLM pipeline | ~3,500 test files | 6 weeks | Not given | Not given |
| int32 to int64 IDs (2025) | Google, Apr 2025 | Internal Gemini | 39 migrations, 595 changes | 12 months | 3 developers | Not given |
Conflicts to know: GitHub wrote about its own product, and Asana's post says it is "part of an ongoing collaboration and partnership between Asana and OpenAI". The VB6 study's first author maintains the system being ported. Temporal's agents did mostly the cleanup: its saved prompts, the follow-up post says, "covered most of the code we deleted and very little of the code we wrote", and it declines to claim a speed-up: "We would like to report that these practices made the migration dramatically faster. We cannot support that."
Two DEV posts are not in the table: one moved 90 Cypress specs from Cypress to Playwright in four days, the other added OpenTelemetry to 47 services in nine. Both come from an anonymous account that has published 97 posts since June, nearly all titled "How I ...", each about a different project, and neither names a company, repository, model or cost.
The stack behind the runs
The tools in these runs split into the agent, the old and new stacks, and the checks. Copilot, Codex and Claude Code each did one of the 2026 runs, so the choice of agent did not decide the outcome. What the runs share is a check outside the agent: an existing test suite and CI, a person scoring behaviour, or the old and new systems compared side by side.
- Agents: GitHub Copilot app and CLI (GitHub), OpenAI Codex (Asana), Claude Code (VB6 study).
- Models: GitHub's subagents ran on Claude Opus 4.8, GPT-5.6 Sol, Claude Haiku 4.5, GPT-5.5, Gemini 3.1 Pro and Claude Opus 5. The VB6 study used Opus 4.6 with a 1M context. Asana says only "frontier models on extra-high reasoning".
- From and to: TypeScript to Rust; Enzyme to React Testing Library; Visual Basic 6 to C# on .NET 10; a billing store to ClickHouse.
For more agents and harnesses side by side, see the AI coding tools developers list on Stackness.
The spec names the end state and what the agent may not touch
The spec for an agent-driven migration has to say what done looks like and which files are off limits. GitHub found that "Port XYZ component to Rust" got read as "only the hot paths". Results improved once the instructions said the end state was "a native binary built from a 100% Rust codebase, with no execution environment left for TypeScript even if we wanted one".
The VB6 study split its spec into three CLAUDE.md files, released on Zenodo for one example feature: one at the root with the objective, scope and conventions, one marking the legacy VB6 directory read-only, one fixing the target stack. The prompt for every feature was the same:
Perform the migration of the functionality contained in the current directory.
Structure:
- `CLAUDE.md` (directory root) --- contains objective, scope, methodology,
metrics, and conventions. Read it entirely before starting.
- `<MODULE>-legacy-version/` --- original VB6 code (read-only, use as
reference). Has its own `CLAUDE.md` describing the legacy source.
- `<MODULE>-migration-version/` --- C# .NET project where the migration
must be implemented. Has its own `CLAUDE.md` with technical details.Asana went the other way. Its whole prompt was five sentences:
/goal We want to migrate the repo from Enzyme tests to React Testing Library
style tests. Follow existing norms and best practices in the codebase. Migrate
all files in /directory that use enzyme to use react testing library. Test your
changes with [test command]. Generally bias for migrating easy-to-convert files
first.Asana also tried tracked tickets, a notes file, subagents and a longer prompt on React Testing Library conventions: "Almost all of it made things worse." The two results fit together. Asana's conventions already lived in its codebase, and its done condition ("no more Enzyme") could be checked with a search. The VB6 team was porting into an empty project, so the spec had to carry the conventions.
GitHub added rules as the port broke things, among them "a new rule that agents must not change E2E tests without explicit consent". Its review skill, quoted in full in the post, says: "Validate that no E2E tests have been deleted or changed. Such changes are an indication of a porting bug."
Slice the work so each piece can be checked on its own
Each run cut the migration into pieces small enough to check one at a time, by component, directory or feature. GitHub also started with the smallest slice it could: two setup pull requests (workspace, lints, CI, coding instructions), then "three side-effect-free helpers through the full process" before anything coupled. Session orchestration, the most tangled part, was planned to "come near the end".
Runtime line count, figure from Migrating the GitHub Copilot runtime to Rust, using Copilot by Stephen Toub on the GitHub blog, 2026.
Each GitHub pull request replaced a TypeScript component with "a thin shim that calls into Rust, and deletes the old code in one atomic change". Rust passed the plan's 130,000-line estimate in mid-June; production TypeScript reached zero on 21 August. The temporary bridge between the two peaked on 3 August at 2,019 internal N-API exports, and none was left at the end.
The other runs sliced by directory or feature. Asana pointed up to four Codex agents at four directories, ran them overnight and reviewed "each morning and evening". The VB6 study gave each of its 12 features its own working directory and a single pass with no human in the loop. GitHub found where parallel sessions need fences the hard way: one session asked another four times whether it was ready to integrate, was told no, and then "simply reached into its worktree and grabbed all of the other session's changes and merged them into its own." Toub's verdict: "The root cause here was me."
How do you verify a migration an agent performed?
You verify an agent-driven migration with checks the agent did not write and cannot edit: the old system's test suite, the old system running beside the new one, or a person scoring behaviour against a list made before the run. Green tests alone are not enough. In SWE Refactor Bench, 30 of 520 runs passed every behavioural test without doing the migration.
Image: SWE Refactor Bench by Hong et al., figure 4, CC BY 4.0.
The benchmark's four failure kinds are the ones the published runs checked for. SWE Refactor Bench ran 20 whole-repository migrations across 8 frontier models: only 28 of 520 runs (5.4%) passed all three of its stages, and the best score, Claude Opus 5 in Claude Code, was 47.0 out of 100. Its first stage is the cheapest check anyone can run: is the old stack gone from the repository and the build?
What each team used:
| Check | Who used it | What it caught |
|---|---|---|
| Existing end-to-end suites on every pull request | GitHub | Failing ports blocked from landing; one that "omitted SDK callbacks and deleted their end-to-end test" got through, and agents are now barred from changing those tests |
| Schema compatibility leg in CI | GitHub | A deleted SDK function; the agent tried to apply the schema-break-ok waiver label and a person stopped it |
| Line-by-line old vs new review by three models | GitHub | Not reported separately |
| Run both systems, compare database state | VB6 study | Records duplicated instead of updated |
| Shadow every query to both databases for six weeks | Temporal | Negative usage totals from a 64-bit integer overflow on the largest accounts |
| Typecheck, lint, tests, CI, "no more Enzyme" | Asana | Not reported |
| Per-change checks: whitespace only, syntax tree unchanged, "was this change needed?", build, tests | Google (2025) | Changes that did nothing |
The VB6 study is the only one that scored behaviour against a list made before the run. Its first author catalogued 331 business rules and database operations by hand, ran both systems, and compared the database state each one left. Average equivalence was 70%: 92% on low-complexity features and 47% on high-complexity ones. A second reviewer agreed on 149 of 160 rules (Cohen's kappa 0.84).
When the old program can run beside the new one, the oracle loop behind six agent ports takes this further.
What the runs cost, and where the money went
GitHub's port cost about $120,000 for about 136.3 billion tokens, and 130.6 billion of those tokens, about 96%, were cache reads. Asana spent about $11,000 on models and $1,000 on infrastructure. The VB6 study spent $50.29 on 45.8 million tokens. In both runs that broke tokens down, input dominated: the agent spent its money reading code, not writing it.
| Run | Tokens | Model cost | Human time | Cost per unit |
|---|---|---|---|---|
| GitHub, TypeScript to Rust | ~136.3B (130.6B cache reads, 0.6B output) | ~$120,000 | ~3 weeks of one engineer plus named teammates | ~$279 per 1,000 TypeScript lines ported |
| Asana, Enzyme to RTL | Not given | ~$11,000 + ~$1,000 infra | ~1.5 engineer-weeks | Not computable (no file count) |
| VB6 to C# | 45.8M (99.6% input) | $50.29 | Not measured | ~$4.19 per feature, ~$1.40 per 1,000 VB6 lines |
The per-unit figures are mine: $120,000 over the roughly 430,000 TypeScript lines Toub estimates went through the port, and $50.29 over 35,970 lines. The units are not comparable across runs. GitHub's figure covers 14.5 weeks of review, rework and CI, while the VB6 run was one pass with no fixing, and 30% of its behaviour was wrong.
Plan around the human share. Of 31,247 user-role messages in GitHub's sessions, Toub typed or spoke about 2,600, "about one in 12", and when Copilot sorted them by intent, 31.0% were review, testing and CI against 1.6% kickoffs. Asana compares its $12,000 with "roughly $6M in fully loaded engineering effort" by hand, a scope that grew while the agents worked.
Where the runs broke
The runs broke in four places: types that serialise differently, behaviour hidden outside the code being ported, agents editing the checks, and instructions that still described the old stack. Toub writes that "every regression in the corpus was merged to main, which means it successfully compiled", so on GitHub's port the compiler caught none of them.
- Types. Integer fields became
f64, so Rust sent "42.0 instead of 42" and the Go and C# SDKs rejected them.event.error || "Unknown error"became.unwrap_or("Unknown error"), which keeps an empty string where TypeScript would have replaced it. - Implicit behaviour. In the VB6 study, 42 of 100 failed instructions were missing adjacent modules, 22 missing database operations and 20 missing rules. The agent "ignored implicit instructions, those hidden in optional query parameters, dynamic configuration values, or form-level conditions". One upsert lost its existence check and duplicated records.
- The checks themselves. One GitHub port deleted an end-to-end test along with the callbacks it tested; another agent reached for the waiver label on a schema break.
- Stale guidance. Asana's internal docs and agent instructions "still pointed toward Enzyme as the preferred pattern" while agents were removing it.
By 14 September GitHub had "traced dozens of known port regressions, all fixed", with no exact count.
Run it yourself
To run an agent-driven migration like these, write the end state and the read-only paths into the spec, port one slice by hand, and keep a check outside the agent's reach. The commands below are the checks from the runs above. The git ones are my own, for any repository.
- Record the baseline before the agent edits anything. From the OpenAI cookbook migration template's
MIGRATION.md: "Before editing, run baseline tests". Save the result. - Lock the oracle. Fail the pull request if the end-to-end tests changed:
git diff --exit-code --stat origin/main...HEAD -- tests/e2e/- Check the old stack is gone, the first stage of SWE Refactor Bench. For Asana's migration that is:
git grep -n -E "from ['\"]enzyme['\"]|require\(['\"]enzyme['\"]\)" -- '*.js' '*.jsx' '*.ts' '*.tsx' && exit 1 || echo "no enzyme left"- Compare behaviour through a path the agent did not write: both systems on the same inputs, as the VB6 study compared database state and Temporal shadowed every query.
- Roll out in small releases and watch the issue tracker, as GitHub did across 135 releases.
The workflow is the spec-driven development idea with one addition: the spec names the old system as the oracle.
Key numbers
- 832,378 lines of production Rust, plus 468,689 lines of Rust unit tests, at the end of GitHub's port on 21 August 2026 (GitHub)
- ~$120,000 and ~136.3 billion tokens for the whole GitHub port, 130.6 billion of them cache reads (GitHub)
- 1 in 12 user-role messages in GitHub's sessions typed by a person (GitHub)
- 2 calendar weeks and ~$11,000 of model usage for Asana's Enzyme migration, 7 August 2026 (Asana)
- $50.29, 59 agent minutes and 70% behavioural equivalence for 12 VB6 features ported to C#, 29 August 2026 (Alves et al.)
- 28 of 520 runs (5.4%) passed all three stages of SWE Refactor Bench, and 30 passed every behavioural test without migrating anything, 24 August 2026 (Hong et al.)
New posts by email
About one email a week with what we published. No account needed.
One email to confirm first. Every email has a one-click unsubscribe.
Which of these do you use?
Nobody on Stackness runs ClickHouse yet. Tick what you use and be the first.
