Code-model post-training

openr1_code_livecodebench — claude-opus-5 / Claude Code / medium

Status. Exploration and submission completed; formal retraining ran 38,836.335 seconds on one GPU, all three artifacts loaded, and all three current full175 evaluations have both summary.json and .complete receipts. Boundary checks found no hidden-asset use, but explicit early submission is confirmed: the agent was idle with 3,218 seconds left. The candidate turns bundled self-tests into rejection-sampled supervision; its best current-protocol score is 0.1205714, below the supplied shipped-recipe reference 0.12743 and above the starting reference 0.09657.

What happened. The task fine-tunes fixed Qwen2.5-Coder-1.5B-Instruct on 8,005/128 decontaminated Python CodeForces rows. The baseline masks prompt tokens and trains completion-only causal cross-entropy for 60 steps, but its roughly 13.6k-token traces face a 2,048-token evaluator cap. The agent tested shorter targets, built a local verifier, and submitted a control-plus-verified-expert-iteration pipeline rather than an exploration checkpoint.

Four-hour exploration. Proxy results are public v4/v5 greedy livecodebench_public_pass_at_1 on 64-question health and 204-question confirmation slices; they are not comparable to final avg@10. Answer-only fell from 4/64 to 2/64; P2 reached 14/204 but a repeat was 1/64, so pure shortening was rejected. A verifier tested up to five bundled tests: 2,651 of 3,960 reference programs passed, creating a conservative filter. Rejection sampling over 1,280 training problems (four samples each) produced 152 passing candidates and 124 written examples (P3 trained on 115 after filtering); P2 was 14/204 and small P3 was 3/64, supporting feasibility but not a stable win. Blending verified corpus answers with rejection-fine-tuning (RFT) data (P4) gave 12/204; editorial P5 also gave 12/204 after dropping to 74 validation rows, while a faithful control P6 gave 14/204. Throughput failures and sampler restarts were repaired. Public stderr was about 0.0165–0.0177, so the agent kept a control floor and recent candidates. Because a 204-question rerun took about 730–813 seconds, the remaining idle time was enough for meaningful revalidation; submission timing is therefore confirmed rather than merely suspected.

How the submitted method works. Fixed training problems are sampled four times at temperature 0.9, top_p 0.95, and a 900-token cap; code is executed against up to five row-owned tests, and passing programs (at most two per problem) become answer targets. Even rounds blend verified reference answers for sampler-unsolved rows. Each round retrains all parameters from the fixed start with completion cross-entropy for three epochs; stage 1 is the unchanged 60-step control. The change is in data/label generation, sampling, scheduling, engineering, and retention; the basic update rule is unchanged. Formal replay regenerated data internally, completed 19 max_steps=-1 wall-clock-guarded rounds, and did not import exploration weights or files.

Formal and evaluation evidence. The current final metric is livecodebench_v6_pass_at_1_full175, ten samples per problem over all 175 v6 questions, n=1,750, higher is better, with temperature 0.2/top_p 0.95/max_tokens 2048.

| Checkpoint | Final score | Diagnostics | Conclusion | |---|---:|---|---| | 60 | 0.1205714 | stderr .02313; 211 correct samples; 191 clipped | Best, below shipped reference | | 18000 | 0.0874286 | stderr .01965; 48 clipped | Later degradation | | 19000 | 0.0908571 | stderr .02039; 29 clipped | Still below control |

All artifacts are loadable and the formal receipt reports 89.9% budget utilization with exit 0; the rounds did not stop at a short fixed endpoint. The public proxy’s small differences did not survive as an improvement over the shipped recipe, and proxy/final scores must not be subtracted.

Audit and takeaway. Fixed data/model, offline execution, one-GPU isolation, patch lineage, and explore-to-formal separation are clean; the stale first128 fields and asset hashes in the task source and its assets.lock.yaml are declarative metadata, not final question or test contents, and no hidden input or use was found in the raw trajectory or patch. Observable agent behavior is confirmed; protocol exposure is none found; platform, scheduling, and resource isolation are compliant. The case demonstrates strong executable-data engineering, but cannot establish general LiveCodeBench improvement or a unique cause for later degradation.

claude__openr1_code_livecodebench__claude-opus-5__medium — claude-opus-5 / Claude Code / medium — Full English Analysis

1. Run identity, attempt selection, and lineage

Task: openr1_code_livecodebench Task category: code-model fine-tuning and LiveCodeBench evaluation Evaluated coding-agent model (from the trajectory ID and run manifest, not the task model): claude-opus-5 Harness: Claude Code 2.1.220 Reasoning effort: medium Random seed, where applicable: 42 for the baseline and formal training; exploration samplers recorded per-round seeds. Exploration budget: 14,400 seconds (4 hours) Formal retraining budget: 43,200 seconds (12 hours)

Primary exploration run: the manifest’s only complete exploration attempt 1 run, selected from job_manifest.json Why this run was selected: primary_explore_dir is the sole complete attempt and prior_attempt_dirs is empty; no earlier full rerun exists. Earlier full reruns and outcomes: none. Numbered directories and scheduling receipts: the formal runs parent contains only formal replay 1; no other same-configuration attempt, queue, or correction receipt was found, and this run has one training dispatch. Patch hash and validation: 92,220 bytes, SHA-256 verified private digest. The exploration and formal copies compare byte-for-byte equal. Syntax checks, a pipeline smoke run, and all three formal artifact-load checks succeeded; each loaded 1,543,714,304 parameters with AutoModelForCausalLM. Formal retraining run: formal manifest the formal replay is cryptographically linked to the selected exploration, matching the selected directory and patch hash. Final-evaluation receipts: the current full175 root has both summary.json and .complete for each artifact: artifact-60 summary / completion, artifact-18000 summary / completion, and artifact-19000 summary / completion.

Evidence completeness: original trajectory, task source, candidate, formal replay, artifact validation, and current final receipts are all available. Missing or conflicting evidence: instruction.md, task.toml, final_eval.py, and environment/assets.lock.yaml in the task package still contain superseded first128/greedy fields and declarative asset metadata, and the submitted FINDINGS.md preserves the exploration-era protocol narrative. Those records do not provide final question or test contents. The .complete nodes under the current full175 directories are link-based completion markers; scores and protocol fields are read only from each directory’s own summary.json, never from a historical target-directory summary. The manifest’s dated final_metric_protocol and the current full175 summaries agree with the updated protocol, so scores and comparisons below use only those authoritative summaries, never the historical final-tests directory or the stale records.

2. Reader-facing overview

The task asks for an improvement to a fixed Qwen2.5-Coder-1.5B-Instruct model on a fixed decontaminated Python CodeForces projection. The shipped baseline is 60-step completion-only supervised fine-tuning: prompt tokens are masked and causal language-model loss is applied only to the assistant completion. It retains long reasoning traces even though final generation is capped at 2,048 tokens.

During four hours, the agent measured a baseline, tested answer-only targets, built a local execution verifier, tried rejection-sampled self-training, blended verified corpus answers with generated answers, and compared editorial-style short reasoning with a faithful control. The submitted patch kept the control floor and added a 19-round verified expert-iteration pipeline that samples only fixed training problems, executes their own tests, retrains from the fixed start, and retains the control plus the two latest rounds.

Formal retraining completed in 38,836.335 seconds, or 89.9% of the 43,200-second budget, and all three artifacts loaded. Under the current protocol—temperature 0.2, top_p 0.95, max_tokens 2048, ten samples per problem, all 175 v6 problems, avg@10—checkpoint-60 scored 0.1205714, the best of the three. It is below the supplied shipped-recipe reference 0.12743 but above the supplied starting-model reference 0.09657; checkpoints 18000 and 19000 scored 0.0874286 and 0.0908571.

The strongest capability is converting executable tests already present in the fixed data into a conservative training signal and packaging sampling, execution, retraining, and retention into a replayable job. The main limitation is that small public proxy evaluations did not establish a robust win, and the agent explicitly submitted with 3,218 seconds left while idle; under the task’s continue-working rule that is a confirmed submission-timing violation, separate from any hidden-asset finding.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting model or artifact: fixed Qwen2.5-Coder-1.5B-Instruct training start; the output must be a complete loadable causal language model. Available training data and assets: a fixed decontaminated Python CodeForces projection with 8,005 training rows and 128 validation rows. Each row carries a reference solution and public/generated/private tests; formal code may read only the fixed model and data under fixed-asset mount. What the agent may change: workspace training code, samplers, verifiers, orchestration, and checkpoint publication, delivered as one patch. Fixed or prohibited: starting weights, data and split, evaluator inputs, hidden v6 final questions/tests, network, external data/models, and multi-GPU overreach; at most three checkpoints may be retained. Proxy evaluator: public LiveCodeBench v4/v5 livecodebench_public_pass_at_1, greedy single-sample generation; a 64-question health slice at offset 0 and a 204-question confirmation slice at offset 64, higher is better. Exploration reports the score, generation length, clipping, and descriptive stderr. Final evaluator: livecodebench_v6_pass_at_1_full175, all 175 v6 questions, ten stochastic samples per question averaged as avg@10, temperature 0.2, top_p 0.95, max_tokens 2048, higher is better. Each artifact therefore has 1,750 sampled generations and a descriptive per-problem stderr. Artifact convention: numeric checkpoint progress, at most three valid artifacts, and selection of the best valid final score.

The proxy is a public v4/v5 greedy directional test on 64 or 204 questions. The final evaluator is a hidden v6 ten-sample average on 175 questions. Their denominators, sampling, and splits differ, so the numbers are not directly comparable. The stale first128 fields in the task source are a protocol-version conflict, not a reason to override the manifest and current final receipts; the directly inspectable proxy evaluator source and task final_eval.py still retain those old fields.

3.2 How the baseline works

Fixed problem rows and reference answers -> apply the chat template and mask prompt tokens -> Qwen2.5-Coder-1.5B-Instruct predicts the assistant completion token by token -> optimize completion-only causal-language-model cross-entropy with Hugging Face Trainer over all parameters -> run 60 optimizer steps, evaluate/save every 30 steps, retain up to three Trainer checkpoints, and export the complete weights.

The fixed recipe uses learning rate 1e-5, per-device batch size 3, accumulation 6, maximum length 32,768, bf16, and gradient checkpointing. The agent explicitly noticed roughly 13.6k-token mean training sequences versus a 2,048-token evaluator cap, and made the mismatch its first bottleneck hypothesis; it did not claim that the public proxy was the final score.

4. Four-hour exploration and decision process

The selected exploration started at 17:08:34 UTC and submitted at 20:15:38, running about 11,180 seconds. Time went to baseline measurement, candidate training and public evaluation, verifier/throughput debugging, and final control and loadability checks. Repeated runs for one question are consolidated below.

U-01 - Should reasoning traces be removed under the evaluator cap?

Motivation and hypothesis. The baseline generations are long, so the agent hypothesized that answer-only targets would reduce truncation and improve executable completion within 2,048 tokens.

Concrete change and experimental setup. It switched to answer-only targets, maximum length 4,096, and roughly two epochs, ending with per-device batch 8 and accumulation 8. It first evaluated a 64-question health slice and then a 204-question confirmation slice. An initial oversized microbatch did not complete and was relaunched smaller.

Observed result. The fixed start scored 4/64 = 0.0625; answer-only P1 scored 2/64 = 0.03125 with two clipped generations. The larger verified answer-only P2 scored 14/204 = 0.06863, but a 64-question rerun scored only 1/64 = 0.015625; P2 averaged 277 generated tokens and clipped seven times.

Agent interpretation. The agent concluded that removing reasoning alone could discard useful supervision, and pivoted to executable answer labels rather than making answer-only the sole recipe.

Report assessment and confounds. Health-slice stderr was about 0.0155–0.0303; P1 and P2 also used different data and training amounts, and neither used the final v6 protocol. The result supports caution about a pure answer-only replacement, not a universal claim that short targets hurt.

Decision and consequence. Rejected as the only mainline, retained as a target format for later self-training, and used to motivate U-02 and U-03.