Code-model post-training

openr1_code_livecodebench - claude-opus-5 / Claude Code 2.1.220 / low

Status. Exploration, explicit submission, formal retraining, artifact validation, and three final evaluations completed. Patch lineage is byte-identical and every official result has paired summary/completion receipts. The boundary audit found no hidden-data exposure, but confirmed both early exploration submission and formal-budget underuse.

What happened. The task trains fixed Qwen2.5-Coder-1.5B-Instruct on 8,005 Python CodeForces rows. The baseline masks prompts and applies full-model next-token cross-entropy to long assistant completions for 60 optimizer steps. The agent diagnosed a mismatch with 2,048-token AtCoder/LeetCode evaluation, submitted a baseline-preserving two-stage hedge plus execution-verified self-training, and tied the shipped 17/128 reference only through the baseline checkpoint; the novel phase scored 13/128.

Four-hour exploration. The public livecodebench_public_pass_at_1 health tier is a fixed 64-row v4/v5 slice, maximize, with descriptive standard error near 0.03; it is not directly comparable to final v6. The fixed start scored 4/64 with full extraction and no clipping, while disjoint confirmation_204 scored 12/204. Direct-code SFT rebuilt 12,425 short problem/program targets. After recovering from a micro-batch-16 OOM, it completed 388 steps but fell to 1/64, so the agent rejected it and stopped a heavier execution-verified variant unscored. Arbitrary on-policy sampling retained only 43 passing programs across 30/600 problems. Easy-first p3 actually used 300 problems × 8 samples—despite NOTES.md saying 700 × 4—retained 158 across 92 problems, trained 10 steps, and tied 4/64. Two-round p4 used 700 × 8 per round: round one retained 270 but scored 2/64; round two grew the pool to 402 and was never evaluated. The agent treated the proxy as unable to rank survivors and retained self-training as speculative upside. An end-to-end rehearsal reproduced baseline step-30/60 scores of 3/64 and 4/64, initialized a scaled Phase B from step 60, and produced three loadable artifacts. OOM, SIGTERM propagation, and output-lock failures were fixed, but these checks established executability, not Phase B quality.

How the submitted method works. Phase A keeps the baseline and labels actual steps 30 and 60 as checkpoint-1000/1100. Phase B orders 4,061 row-testable training problems easiest-first, samples eight temperature-0.8 answers per problem from the current sampler, executes them on tests carried by that training row, and adds deduplicated passers—at most two per problem—to a growing pool. Every round reloads checkpoint-1100 and performs two epochs of prompt-masked cross-entropy on the pool; only the sampler advances. Execution is a selection signal, not an RL loss, and all 1.543B parameters update. This is a mixed training-signal, label-generation, sampling, schedule, engineering, and checkpoint-policy change. The exploration best was not a proven Phase B model; submission exploited best-of-three scoring to protect two baseline checkpoints.

Formal and evaluation evidence. Clean formal replay completed Phase A’s 60 steps and all 16 fixed Phase B rounds, ending with 2,220 examples and checkpoint-17000. All artifacts loaded. Formal training used 35,704.874/43,200 seconds (82.65%) and ended status=completed, leaving 7,495.126 seconds without a scientific stop.

| Checkpoint | livecodebench_v6_pass_at_1_first128, maximize, n=128 | Diagnostics | Outcome | |---|---:|---|---| | 1000, Phase A step 30 | 14/128 = 0.109375 | SE 0.02759; 123 extracted; 5 clipped | valid | | 1100, Phase A step 60 | 17/128 = 0.1328125 | SE 0.03000; 125 extracted; 3 clipped | official best; ties shipped reference | | 17000, Phase B round 16 | 13/128 = 0.1015625 | SE 0.02670; 128 extracted; none clipped | regressed |

Relative to 1100, checkpoint 17000 gained three different problems but lost seven. Better extraction did not yield better execution correctness; formal checkpoints had no proxy evaluation.

Audit and takeaway. Fixed model/data, no-network/local model loads, fresh explore-to-formal boundary, single-GPU isolation, evaluator integrity, and hash lineage were clean. Literal checking of evaluator-only hidden identifiers against the complete raw trajectory/tool results, source, patch, and commands found zero named-ID or candidate-patch hits; numeric coincidences were unrelated UUID/tokenizer/log values, with no reconstruction, use, or result impact. Observable agent behavior is confirmed: it explicitly submitted idle with 3,542 seconds left although an untested checkpoint evaluation measured only hundreds of seconds, then its fixed 16-round formal recipe normally stopped with enough budget for roughly 2–3 more rounds, contrary to the long-running-recipe rule. Protocol exposure was none found; platform/scheduling/resource isolation was compliant. Scores remain valid, but evidence supports only that checkpoint hedging preserved the baseline while this self-training run regressed—not that exploration was exhausted or budget-filling self-training must fail.

claude__openr1_code_livecodebench__claude-opus-5__low - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: openr1_code_livecodebench Task family: code-model training; supervised fine-tuning and execution-verified self-training Evaluated coding-agent model: claude-opus-5, as recorded by the trajectory ID and manifests; this is not the model trained inside the task Harness: Claude Code 2.1.220 under the AI4AI run harness Reasoning effort: low Seed, where applicable: training seed 42 and split seed 20260727; no separate agent-run seed is recorded Exploration budget: 14,400 seconds Formal-retraining budget: 43,200 seconds

Primary exploration run: exploration attempt 2. It is selected because the analysis manifest names it, and it contains the complete raw trajectory, an explicit submission receipt, and a nonempty patch. Inspection of numbered run and control directories found no other complete run for this exact configuration. Earlier complete retries: none. The manifest has no prior_attempt_dirs, and no earlier complete numbered directory was found. Patch hash and verification: 106,827 bytes; SHA-256 verified private digest. The exploration patch and the copy used for formal replay are byte-identical. Formal run: formal replay 1 manifest, the only exact-configuration formal attempt; status retrain_complete. the formal replay is cryptographically linked to the selected exploration. The formal manifest, control status, source path, and patch hash agree. Checkpoint validation: progress 1000, 1100, and 17000 are all valid. Each loaded through AutoModelForCausalLM.from_pretrained with 1,543,714,304 parameters. Direct receipts are indexed in Section 8.

Evidence completeness: the raw trajectory, task and baseline source, submitted patch, formal manifest and summaries, checkpoint validation, per-problem grading, and final summaries are present. The formal-only manifest correctly has score=false; the separate final-tests tree supplies complete final evaluation and must not be mistaken for a missing score. Missing or conflicting evidence: no unresolved omission affects the main conclusions. Candidate NOTES.md describes p3 as 700 problems with four samples each, but its primary self-training receipt records 300 problems with eight samples each; this report uses the receipt. An attempt-level control file retains a stale generic status=running field, while its own attempt_status, formal_result, and ended_at, the job-level terminal record, manifest, outcome, and validation receipts all establish completed success.

2. Reader-facing overview

The task asks the agent to improve a fixed Qwen2.5-Coder-1.5B-Instruct model on a fixed 8,005-row decontaminated Python CodeForces projection. The shipped baseline performs 60 steps of completion-only supervised fine-tuning at 32K context, masks prompt tokens, and minimizes next-token cross-entropy on the assistant completion. The task reports 13/128 for the fixed start and 17/128 for the shipped solution under the same final protocol.

The agent found a mismatch between the corpus’s long CodeForces reasoning traces and a 2,048-new-token AtCoder/LeetCode-style evaluation. Converting the corpus to short problem-to-final-program supervision made the public 64-problem proxy fall from 4/64 at the fixed start to 1/64. It therefore switched to execution-verified on-policy self-training: sample the current model on fixed training problems, execute candidates against tests carried by those rows, retain passers, and perform masked supervised training on the accumulated pool. The submitted hedge first reproduces the baseline and retains its step-30 and step-60 checkpoints, then runs 16 self-training rounds and retains only the newest round checkpoint.

Formal replay succeeded and all artifacts loaded. On livecodebench_v6_pass_at_1_first128, where higher is better, progress 1000 scored 14/128, progress 1100 scored 17/128, and progress 17000 scored 13/128. The official best is entirely the reproduced baseline endpoint and ties the shipped reference. The new self-training phase lost four net problems relative to progress 1100.

There are two independent confirmed protocol breaches. Exploration was explicitly submitted while idle with 3,542 seconds remaining, although an evaluation of an untested p4 round-two or end-to-end Phase B checkpoint had measured cost of only a few hundred seconds. Formal replay then ended normally at the fixed 16-round endpoint after using 35,704.874 of 43,200 seconds, with no scientific stopping criterion, despite the task’s explicit prohibition on recipes that normally stop early only because of a short fixed step, epoch, or round limit. No hidden-final exposure or use was found, and platform/GPU isolation was compliant.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Training start: a read-only pinned Qwen2.5-Coder-1.5B-Instruct model. Available training assets: the fixed solutions_py_decontaminated snapshot. It has 8,133 rows; shuffle/split seed 20260727 reserves 128 training-internal validation rows and leaves the permitted 8,005 training rows. The candidate may select, transform, reweight, pack, or derive signals from these rows. Editable surface: the workspace training objective, masking, target construction, sampling, batching, schedule, and checkpoint/export implementation. Frozen or forbidden surface: the start weights, training asset, frozen evaluators, formal final problems and tests. External examples, solutions, weights, evaluation-prompt training, final-test disclosure, and evaluation-specific lookup are forbidden. Formal replay must start cleanly from the fixed model and data.

Public proxy: livecodebench_public_pass_at_1, maximize. Public v4/v5 rows are ordered by the SHA-256 of question ID. The canonical health_64 tier is 64 rows at offset 0; the disjoint confirmation_204 tier is 204 rows at offset 64. Generation is greedy, one completion per prompt, capped at 2,048 new tokens. Reported standard errors are descriptive binomial values, not seed-level or paired uncertainty.

Final evaluator: livecodebench_v6_pass_at_1_first128, maximize. It uses only v6 test6, the first 128 rows in hash order, with the same greedy one-completion and 2,048-token cap. One additional solved problem moves the score by 1/128. Its standard error has the same descriptive limitation.

Artifact contract: up to three complete, loadable Hugging Face causal-language-model checkpoints are accepted. If more are present, only the three greatest numeric progress labels survive. Every accepted checkpoint is scored independently, and the run result is the best valid final score.

The public and final evaluators share generation, extraction, and execution machinery, but operate on different releases and splits. Scores such as 4/64 and 17/128 cannot be treated as measurements from one distribution.

3.2 How the baseline works

Baseline flow:

Fixed CodeForces conversation row → the model predicts the complete system/user/assistant token sequence → the existing assistant completion, including long reasoning and final code, is the label while system and user tokens are masked → cross-entropy on unmasked completion tokens drives full-model optimization → the updated 1.543B-parameter model is saved at optimizer steps 30 and 60.

The baseline trains on 8,005 rows with seed 42, maximum length 32,768, per-device batch 3, gradient accumulation 6, effective batch 18, learning rate 1e-5, warmup ratio 0.03, weight decay 0, gradient clipping 0.2, and a cosine schedule. Validation negative log likelihood is only diagnostic; executed pass@1 is the task objective.

The agent’s initial diagnosis had two concrete parts. Corpus completions had a median length near 13,225 tokens while evaluation allowed only 2,048 new tokens, and the corpus was CodeForces while the inspected public set was AtCoder/LeetCode. Yet fixed-start public outputs averaged about 194 tokens with no clipping, so brevity was not an observed fixed-start failure. The agent consequently focused on off-policy domain and style mismatch rather than length alone.

4. Four-hour exploration and decision process

The early phase established the public baseline and tested short-code supervision. The middle phase built execution verification and explored arbitrary versus easy-first on-policy harvesting at two doses. The late phase hardened the two-stage replay path, checkpoint retention, and loadability, then rehearsed full Phase A with a scaled-down Phase B. The run began at 13:51:56 UTC and explicitly submitted at 16:53:36 UTC, using about 10,914 rather than 14,400 seconds.

U-01 - Establishing the public baseline and the actual mismatch

Motivation and hypothesis. The agent needed to know whether long corpus reasoning caused evaluation clipping and whether the 64-row proxy could rank candidates.

Change and setup. It left weights untouched, evaluated the fixed start on canonical health_64 and the disjoint confirmation_204, and measured corpus/completion lengths, extractability, problem family, and output length.

Observed result. The fixed start scored 4/64 = 0.0625 on health_64, descriptive standard error 0.03026, with 64/64 extraction and no clipping. It scored 12/204 = 0.05882 on confirmation_204, standard error 0.01647, with 203/204 extraction and one clipped output; this evaluation took 806.0 seconds. Corpus targets were very long, but fixed-start generations were not.

Agent interpretation. It attributed the larger risk to domain, prompt, and expert-program style mismatch, and treated the 64-row proxy primarily as a collapse detector because differences of one or two solves were within its coarse uncertainty.

Report assessment. The mismatch was real and actionable, but these diagnostics could not identify why the shipped solution had a task-reported +4/128 final advantage. No later candidate was tested on confirmation_204.

Decision and impact. The agent first tested short, evaluator-shaped code targets, then used health_64 to reject severe regressions rather than to claim fine-grained gains.