Navers lab
← SWE Refactor Bench

Leaderboard

Every figure here is computed from the released score table at build time. The best configuration in the sweep accepts 5 of 20.

All 20 tasks and 520 trajectories →
Graded runs
520

26 configurations

Passed the gate
65.4%

340 of 520

Reached a verifier
88

gate pass and perfect suite

Then broken
68.2%

60 of 88 admitted

Mean composite
13.44

of 100, over all runs

The board

One row per configuration — a model at one reasoning-effort level — and every one of the 26 ran all 20 tasks, with what one task cost beside each score.

Score (out of 100) Cost per task
claude-opus-5 Anthropic best 47.0 · 16 accepted
low
20.5
$34.8 2 accepted, 3 broken by a counterexample, 0 blind, of 20 runs
medium
28.5
$38.4 2 accepted, 5 broken by a counterexample, 2 blind, of 20 runs
high
34.5
$55.7 4 accepted, 4 broken by a counterexample, 2 blind, of 20 runs
xhigh
47.0
$74.9 5 accepted, 6 broken by a counterexample, 1 blind, of 20 runs
max
31.0
$72.4 3 accepted, 4 broken by a counterexample, 2 blind, of 20 runs
gpt-5.6-sol OpenAI best 28.5 · 5 accepted
none
4.0
$2.9 0 accepted, 1 broken by a counterexample, 0 blind, of 20 runs
low
7.0
$5.9 0 accepted, 2 broken by a counterexample, 0 blind, of 20 runs
medium
6.5
$6.0 0 accepted, 2 broken by a counterexample, 3 blind, of 20 runs
high
19.0
$7.7 1 accepted, 4 broken by a counterexample, 0 blind, of 20 runs
xhigh
9.5
$19.1 0 accepted, 3 broken by a counterexample, 1 blind, of 20 runs
max
28.5
$143.5 4 accepted, 3 broken by a counterexample, 0 blind, of 20 runs
kimi-k3 Moonshot best 19.5 · 2 accepted
max
19.5
$28.9 2 accepted, 3 broken by a counterexample, 0 blind, of 20 runs
claude-sonnet-5 Anthropic best 15.0 · 3 accepted
low
4.0
$4.4 0 accepted, 1 broken by a counterexample, 1 blind, of 20 runs
medium
15.0
$11.9 1 accepted, 3 broken by a counterexample, 0 blind, of 20 runs
high
6.0
$24.6 0 accepted, 2 broken by a counterexample, 2 blind, of 20 runs
xhigh
9.0
$27.0 1 accepted, 1 broken by a counterexample, 1 blind, of 20 runs
max
8.5
$27.5 1 accepted, 1 broken by a counterexample, 1 blind, of 20 runs
gpt-5.6-luna OpenAI best 10.5 · 0 accepted
none
4.0
$1.6 0 accepted, 1 broken by a counterexample, 1 blind, of 20 runs
low
4.0
$1.7 0 accepted, 1 broken by a counterexample, 1 blind, of 20 runs
medium
0.0
$1.7 0 accepted, 0 broken by a counterexample, 1 blind, of 20 runs
high
4.0
$1.8 0 accepted, 1 broken by a counterexample, 2 blind, of 20 runs
xhigh
5.5
$2.9 0 accepted, 2 broken by a counterexample, 2 blind, of 20 runs
max
10.5
$2.8 0 accepted, 3 broken by a counterexample, 1 blind, of 20 runs
qwen3.8-max Alibaba best 10.0 · 2 accepted
max
10.0
$14.5 2 accepted, 0 broken by a counterexample, 4 blind, of 20 runs
dsv4-flash DeepSeek best 7.0 · 0 accepted
max
7.0
$4.3 0 accepted, 2 broken by a counterexample, 1 blind, of 20 runs
glm-5.2 Z.ai best 6.5 · 0 accepted
max
6.5
$17.5 0 accepted, 2 broken by a counterexample, 1 blind, of 20 runs
One bar per configuration, 26 in all; 20 tasks each. Colour is the model, depth is the effort level. Cost is dollars of API spend for one task.

What the verifiers caught

88 submissions reached the third stage with a perfect suite behind them; 60 of them lost to a counterexample inside the hour. The two claude-opus-5 slots break more than half of what they see, the other four about a quarter — so an accepted run is one this panel could not break, not one proven correct.

Break rate of each of the six verifiers: the share of its rounds that produced an executable counterexample, coloured by the model behind it and annotated with its reasoning effort.

Task × model

Best composite any run of that model reached on that task. Hatched cells are the interesting failure: perfect on every frozen check and still nothing, because the gate or a verifier caught them. The 20 tasks fall into three bands with nothing in between — 7 somebody solved outright, 6 where the best run reached a verifier and was broken, and 7 where no run scored at all. Sizes, check counts and trajectories are on the tasks index.

Task
claude-opus-5
gpt-5.6-sol
claude-sonnet-5
gpt-5.6-luna
kimi-k3
qwen3.8-max
dsv4-flash
glm-5.2
Best
lang01 C → Rust
—
lang02 C → Java
90
90
90
lang03 Python → Go
—
lang04 JavaScript → Rust
—
lang05 Go → Zig
100
100
lang06 C++ → C#
60
60
lang07 JavaScript → TypeScript
100
100
100
fw01 Flask → Starlette
—
fw02 Express → Fastify
—
fw03 Vue → React
100
100
fw04 Gin → chi
100
100
80
100
fw05 actix-web → axum
60
60
fw06 gorilla/mux → net/http
100
100
100
70
100
100
100
fw07 Dropwizard → Spring Boot
—
pf01 POSIX → wasm32-wasi
60
60
pf02 CommonJS → V8 realm
—
pf03 x86-64 → tri-arch
100
100
80
80
100
60
70
100
build01 Autotools → CMake
50
60
60
build02 Maven → Gradle
60
70
60
40
40
60
70
build03 setuptools → Meson
100
90
100
80
100
100
0
100 best composite
reached the behavioral ceiling, scored nothing

Pooled by model

Every run of a model at once — a different question from the board, and not comparable across the two blocks below: a six-level sweep and a single configuration have different denominators. This ranks models as deployed, not weights.

Proprietary models swept over reasoning effort

4 models
# Model Client Cfg Runs Gate Ceil. Blind Adm. Broken Acc. Score
1 claude-opus-5 Claude Code 5 100 88 45 7 38 22 58% 16
32.30
2 gpt-5.6-sol Codex CLI 6 120 68 24 4 20 15 75% 5
12.42
3 claude-sonnet-5 Claude Code 5 100 69 16 5 11 8 73% 3
8.50
4 gpt-5.6-luna Codex CLI 6 120 69 16 8 8 8 100% 0
4.67

Open weights at a single configuration

4 models
# Model Client Cfg Runs Gate Ceil. Blind Adm. Broken Acc. Score
1 kimi-k3 Claude Code 1 20 15 5 0 5 3 60% 2
19.50
2 qwen3.8-max Claude Code 1 20 8 6 4 2 0 0% 2
10.00
3 dsv4-flash Claude Code 1 20 13 3 1 2 2 100% 0
7.00
4 glm-5.2 Claude Code 1 20 10 3 1 2 2 100% 0
6.50

Blind counts runs perfect on every frozen check that the gate rejected anyway — what a behavioral-only report would have credited as clean. Broken is a share of admitted runs, not of all runs.

← SWE Refactor Bench Tasks →