SWE Refactor Bench

The job

Replacing the stack underneath a system that still has to work — C to Rust, Maven to Gradle, POSIX to WebAssembly — costs senior engineers weeks, and most of those weeks go into proving that nothing changed. It is long-horizon work: the unit is a repository, not a function, and one attempt runs for hours or days.

The leaderboard

All 26 configurations — 8 models, each swept over the reasoning-effort levels it exposes — on the same 20 tasks. The best of them scores 47.0 of 100, which is further from a finished migration than it is from the bottom of the strip.

The rest of this page is why those numbers are so low, and why a ranking alone would not have found it.

A passing suite cannot see this work

Bug fixing starts red and turns green, and that jump is the evidence. Migration starts green. Hand the original repository back untouched and it still scores full marks — so the maximum of the reward sits on a submission with no work in it, and more tests cannot move it.

We call it blindness: the suite is not incomplete, it is answering a different question — was anything broken, not was anything changed.

Four kinds of debt

20 migrations of load-bearing open-source infrastructure — SQLite, zlib, libsodium, GraphHopper and others. Each moves exactly one layer of the stack, keeps the interface, and gets 6 to 30 hours of autonomous work, offline.

Passing the gate and being accepted are different things: build toolchain clears the gate most often and still loses most of it at stage III, while framework rewrite takes half of all acceptances.

Three stages in series

The agent sees a repository, an instruction and an offline image. It never sees any of this.

What the models did

26 configurations, 520 graded runs, 8 models — and 28 of those runs cleared all three stages. The configuration at the top of the board, claude-opus-5 at xhigh, spent $75 a task to get there.

Ranking them is the easy part. The two findings below are what the ranking cannot say.

Two abilities, not one

Migration completeness and behavioural correctness are distinct abilities, and the failures divide along that line. A few runs preserve the behaviour by skipping the migration, and only stage I stops them. Most attempt the migration and break the behaviour doing it, and only stage II stops those. Neither stage can stand in for the other: stage II alone would reward doing nothing, stage I alone would reward doing damage.

The last mile

Once the migration is real, most runs get behaviour back above 99%. Almost none finish it: a quarter make no error at all, and two thirds of those still lose to a counterexample inside the hour.

A stronger verifier is simply a higher bar: retire the two strongest and the remaining four would accept 46 submissions instead of 28.

What we release

Citation

@misc{hong2026swerefactorbenchcoding,
  title   = {SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon,
             Whole-Repository Stack Migration?},
  author  = {Deyao Hong and Yizhe Chi and Wenyi Li and Xiaoqiu Wang and
             Mingju Gao and Kaisen Yang and Bingxiang He and Youjie Zheng and
             Calvin Xiao and Qinhuai Na},
  journal = {arXiv preprint arXiv:2608.23564},
  year    = {2026}
}