Navers lab
← All tasks
build01 Build-toolchain rewrite Task 18 / 20

Autotools → CMake

libsodium 1.0.20 Autotools CMake 3.20+

What produces the artifact changes, and the package is the observable. 73,228 lines, 6 hours per model, and 4,720 checks across 15 modules.

Runs
26

one per configuration

Past rung 1
19

of 26 — the migration happened

Past rung 2
2

of those 19 — every check perfect

Past rung 3
0

of those 2 — accepted

Best composite
60

highest score on this task

The instrument

Acceptance gates

6

decide whether the migration happened

Behavioral modules

15

ported suites, run against the new tree

Frozen checks

4,720

all must pass to open rung three

Verifiers

6

hostile programs, written against the spec

Where the 26 runs stopped

7
Never migrated
17
Migrated, behavior lost
0
Perfect suite, gate rejected
2
Verifier found a difference
0
Accepted

2 runs were perfect on all 4,720 checks, and all passed the gate. Of the 2 that reached rung three, 0 survived all six verifiers.

By model

Model Runs Gate Ceil. Where they stopped Mean Best
kimi-k3 1 1 1
60.0 60
gpt-5.6-sol 6 4 1
8.3 50
claude-opus-5 5 5 0
0.0 —
claude-sonnet-5 5 5 0
0.0 —
dsv4-flash 1 0 0
0.0 —
glm-5.2 1 0 0
0.0 —
gpt-5.6-luna 6 4 0
0.0 —
qwen3.8-max 1 0 0
0.0 —

Every run

best composite first

One row per configuration. Shape is the run's tool sequence in 24 slices — pale is shell, dark is an edit — and each log is the full session as recorded.

ModelEffortOutcomeGateChecksVerif.ScoreShape
kimi-k3 max broken 6/6 100.0% 2/6 60
gpt-5.6-sol max broken 6/6 100.0% 1/6 50
gpt-5.6-luna max partial 6/6 100.0% — 0
gpt-5.6-luna xhigh partial 6/6 99.9% — 0
gpt-5.6-luna high partial 6/6 99.8% — 0
glm-5.2 max failed 5/6 99.7% — 0
claude-opus-5 medium partial 6/6 99.7% — 0
gpt-5.6-sol xhigh partial 6/6 99.6% — 0
claude-opus-5 low partial 6/6 99.6% — 0
claude-opus-5 high partial 6/6 99.6% — 0
claude-opus-5 xhigh partial 6/6 99.6% — 0
claude-opus-5 max partial 6/6 99.5% — 0
claude-sonnet-5 medium partial 6/6 98.9% — 0
claude-sonnet-5 max partial 6/6 98.7% — 0
gpt-5.6-sol low partial 6/6 98.3% — 0
dsv4-flash max failed 4/6 98.1% — 0
gpt-5.6-sol none partial 6/6 97.3% — 0
gpt-5.6-sol high failed 4/6 97.1% — 0
claude-sonnet-5 low partial 6/6 96.7% — 0
claude-sonnet-5 high partial 6/6 96.4% — 0
claude-sonnet-5 xhigh partial 6/6 95.1% — 0
gpt-5.6-luna none partial 6/6 92.8% — 0
gpt-5.6-luna medium failed 5/6 78.1% — 0
gpt-5.6-luna low failed 4/6 76.6% — 0
gpt-5.6-sol medium failed 4/6 68.3% — 0
qwen3.8-max max failed 5/6 0.4% — 0