Navers lab
← SWE Refactor Bench

Tasks

20 real repositories — 803 to 358,006 lines — each moved across a boundary a maintainer would actually cross, and each graded by an instrument frozen before any run.

Migrations
20

whole repositories

Lines of code
867,062

10,594 tracked files

Frozen checks
130,118

136 acceptance gates

Behavioral modules
264

ported test suites

Budget hours
262

wall clock, per model

Where all 520 runs stopped

20 tasks × 26 configurations
150
Never migrated
252
Migrated, behavior lost
30
Perfect suite, gate rejected
60
Verifier found a difference
28
Accepted

The hatched band is the interesting failure: perfect on every frozen check, and never migrated. Each task below carries the same bar.

Build-toolchain rewrite

3 tasks · 78 runs · mean 31.41

What produces the artifact changes, and the package is the observable.

Task Project Lines Checks Budget Runs Best
build01 Autotools → CMake libsodium 1.0.20 73,228 4,720 6h
60
build02 Maven → Gradle Gson 2.10.1 19,015 2,669 6h
70
build03 setuptools → Meson pycryptodome 3.20.0 78,839 381 6h
100

Platform port

3 tasks · 78 runs · mean 17.18

The host the code assumes changes.

Task Project Lines Checks Budget Runs Best
pf01 POSIX → wasm32-wasi SQLite 3.31.1 358,006 2,668 10h
60
pf02 CommonJS → V8 realm Stylus 0.63.0 16,012 2,570 10h
—
pf03 x86-64 → tri-arch QuickJS 89,429 1,487 6h
100

Framework rewrite

7 tasks · 182 runs · mean 11.98

The language stays, the framework it is organized around is replaced.

Task Project Lines Checks Budget Runs Best
fw01 Flask → Starlette httpbin 2,455 2,830 9h
—
fw02 Express → Fastify json-server 2,992 6,760 9h
—
fw03 Vue → React RealWorld Conduit 2,261 21,769 11h
100
fw04 Gin → chi ChartMuseum 5,401 11,429 12h
100
fw05 actix-web → axum miniserve 3,484 12,466 8h
60
fw06 gorilla/mux → net/http go-simple-upload-server 803 485 6h
100
fw07 Dropwizard → Spring Boot GraphHopper 11.0 94,766 113 16h
—

Language rewrite

7 tasks · 182 runs · mean 5.60

The implementation language moves, the artifact does not.

Task Project Lines Checks Budget Runs Best
lang01 C → Rust cmark 0.31.1 22,619 4,184 30h
—
lang02 C → Java zlib 1.3.1 22,732 4,162 20h
90
lang03 Python → Go sqlparse 0.5.3 4,393 8,530 15h
—
lang04 JavaScript → Rust acorn 8.14.0 9,571 16,039 30h
—
lang05 Go → Zig go-yaml v3.0.1 11,967 10,271 20h
100
lang06 C++ → C# jsonnet 0.20.0 39,842 2,608 20h
60
lang07 JavaScript → TypeScript JSONata 2.0.6 9,247 13,977 12h
100
← SWE Refactor Bench Leaderboard → All runs →