Code-model post-training

claude__openr1_code_livecodebench__claude-sonnet-5__high - claude-sonnet-5 / Claude Code 2.1.220 / high

Status. Exploration and submission completed. formal replay 1 stopped before recipe execution because the control layer classified a competing-GPU reservation as infrastructure failure; formal replay 2 retrained, validated three loadable artifacts, and completed the current full175 evaluation. Boundary and lineage checks are clean, but the agent's early submission is confirmed execution-protocol noncompliance.

What happened. The task improves a pinned Qwen2.5-Coder-1.5B-Instruct model with 8,133 decontaminated Python CodeForces rows. The baseline masks prompt labels and trains assistant-completion tokens with full-parameter cross-entropy, but its 60-step endpoint is short for 32K examples and the corpus contains unfinished or self-test-failing supervision. The candidate filters such rows, adds wall-clock learning-rate and stopping control, and exports complete checkpoints without changing the completion-only update rule. Formal replay used the fixed start, not exploratory weights.

Four-hour exploration. Public v4/v5 proxy evaluation was greedy n=1 on a 64-item health tier and a non-overlapping 204-item confirm tier; these scores are not comparable with final full175 scores. First, baseline measurements were 4/64 health and 12/204 confirm; an eight-step unfiltered control and row diagnostics motivated, but did not prove, a quality-noise hypothesis. Second, the quality gate removed 357 incomplete and 859 self-test-failing rows, leaving 6,789 training rows. Probe3 checkpoint-54 reached 15/204 confirm (stderr 0.01827) and probe4 checkpoint-66 reached 14/204 (stderr 0.01770), so the agent retained filtering, while checkpoint-45's 2/64 health result and the confounding longer runtime left its isolated effect inconclusive. Third, a 100,000-step safety ceiling, deadline callback, and wall-clock warmup/cosine schedule were tested; probe2's 40-second timeout experiment produced no performance result, but the mechanism worked formally. Fourth, checkpoints loaded and the agent chose the source patch rather than checkpoint-54 weights. It did not run a second seed or a clean ablation. The submission receipt showed 1,710 seconds idle; measured runtimes of 344.5 seconds for a short probe and 723.2 seconds for confirm evaluation show that meaningful work could still fit, establishing the confirmed timing violation.

How the submitted method works. Fixed conversations are split, filtered only with their own finish_reason and public_tests, rendered with the same chat template, and trained with prompt-masked completion NLL over all Qwen parameters. The changes are data filtering, scheduling, runtime engineering, and checkpoint policy; the objective and optimizer update remain the baseline's. Formal orchestration overrides source fallbacks with a 42,600-second wall clock and 600-second reserve.

Formal and evaluation evidence. The formal phase lasted 42,354.237 seconds in the manifest; its trainer summary records wall_seconds 41,997.5 and 974 completed steps before the wall-clock callback. The manifest utilization is 98.04% of the 43,200-second budget. Three artifacts were cumulatively published and all three were retained under the limit; each passed loading validation. Current-protocol results are:

| Progress | livecodebench_v6_pass_at_1_full175 | Diagnostics | |---|---:|---| | 920 | 0.1125714286 (197/1,750), best | stderr 0.0216498; extraction 0.8114; clipped 318 | | 960 | 0.1108571429 (194/1,750) | stderr 0.0215716; extraction 0.8400; clipped 322 | | 974 | 0.1097142857 (192/1,750) | stderr 0.0212372; extraction 0.8286; clipped 332 |

Each uses v6's 175 problems, ten samples per problem, temperature 0.2, top_p 0.95, and max_tokens 2048 for the maximize-direction avg@10 estimate; each summary has a matching .complete receipt. checkpoint-920 is above the 0.09657 start reference and below the 0.12743 shipped-recipe reference. Later decline means the proxy/NLL trend did not establish that longer training helps.

Audit and takeaway. The formal mount contained only the fixed model, corpus, and patch. Stale first128 fields in the task package were protocol metadata, not hidden values. Literal searches of the complete trajectory, tool results, patch, and commands found no v6 rows, test6 data, or livecodebench_final; private_tests matches were allowed training-row fields. No external data, evaluator modification, or selected-GPU sharing was found, and the infrastructure retry was not scored. Observable agent behavior is confirmed because of early submission; no hidden value reached the trajectory, no reconstruction or use is observed, and none affected the candidate or score; platform, scheduling, and resource isolation were compliant. The run demonstrates reproducible quality-gated, wall-clock training, but no causal component gain, superiority to the shipped recipe, or global optimum can be claimed.

claude__openr1_code_livecodebench__claude-sonnet-5__high - claude-sonnet-5 / Claude Code 2.1.220 / high - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: openr1_code_livecodebench. Task family: supervised fine-tuning improvement for a fixed code model. Evaluated coding-agent model: claude-sonnet-5, as encoded by the trajectory ID and run manifest; this is distinct from the model being trained. Harness: Claude Code 2.1.220. Reasoning effort: high. Seed: training seed 42; split seed 20260727. Exploration budget: 14,400 seconds. Formal budget: 43,200 seconds.

The selected primary explore run is the manifest-linked, complete run exploration attempt 1. It is the only complete attempt and has no prior full-run attempt. It ran from 2026-08-13 10:07:13 UTC to 13:39:46 UTC, about 3 hours 32 minutes, and the submission receipt records 1,710 seconds remaining. The run has the raw trajectory, submit receipt, lifecycle record, and .explore.complete receipt.

There is one formal retry that must remain in the execution history but must not be treated as a scientific zero. formal replay 1 exited during launcher startup after a newly competing container reserved GPU 4. Its control status and correction receipt classify it as terminal_infrastructure and state that no recipe code ran. It produced no model or score. The authorized retry produced formal replay 2, which completed retraining and validation.

The candidate patch is candidate output, 42,677 bytes, with SHA-256 verified private digest. The formal copy has the same hash. formal replay 2 records the selected explore run in selected-exploration lineage and records source commit pinned private revision. The difference from the frozen source hash is covered by an authorized orchestrator-only source-commit waiver; the correction describes a Claude transport-retry hotfix and says the task tree was unchanged.

Retraining completed, and checkpoint-920, checkpoint-960, and checkpoint-974 all passed AutoModelForCausalLM.from_pretrained loading validation with 1,543,714,304 parameters. The manifest-linked current full175 evaluator produced a summary and matching .complete receipt for each of the three artifacts, so these are official final results.

Evidence is sufficient for the conclusions below. The task package still contains historical first128 fields and an old final-evaluation script, while the manifest, the user-specified protocol change, and the final-tests-full175 receipts specify the current full175 protocol. This report uses the current receipts and does not use the superseded final-tests directory. Formal proxy receipts are not available because the formal stage did not run the public fast evaluator; this is not a score of zero.

2. Reader-facing overview

The task asks the agent to improve a fixed Qwen2.5-Coder-1.5B-Instruct model using a decontaminated Python CodeForces corpus. The baseline applies chat-template tokenization, masks prompt labels, and trains the assistant-completion tokens with ordinary full-parameter cross-entropy. Its short 60-step recipe is slow on 32K examples, and the corpus contains unfinished generations and some diff answers that fail their own public examples.

During exploration the agent measured public LiveCodeBench proxy tiers, diagnosed those row-level issues, implemented a quality gate, and replaced a short fixed-step schedule with a wall-clock-aware warmup/cosine schedule and a large safety step ceiling. It also added deadline-aware stopping and complete-checkpoint export. The formal job retrained from the fixed start rather than from an exploration checkpoint.

The formal retrain phase lasted 42,354.237 seconds in the manifest. Its trainer summary records wall_seconds 41,997.5, 974 completed optimization steps, and wall-clock stopping under the framework-injected 42,600-second training budget. It published three loadable artifacts. Under the current protocol, checkpoint-920 is best at livecodebench_v6_pass_at_1_full175 = 0.1125714286; checkpoint-960 and checkpoint-974 are lower at 0.1108571429 and 0.1097142857. The best result is above the current start-model reference 0.09657 but below the shipped-recipe reference 0.12743.

The strongest capability is a reproducible data-and-runtime intervention with a complete lineage and validation chain. The main limitation is scientific isolation: filtering and scheduling changed together, no second seed or clean ablation was run, and the agent explicitly submitted while 1,710 seconds remained. Given measured runtimes for a further short training probe and confirm evaluation, that submission timing is confirmed execution-protocol noncompliance. No hidden v6 asset exposure or GPU-sharing contamination was found.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

| Item | Direct evidence and meaning | |---|---| | Starting artifact / model | Read-only pinned Qwen2.5-Coder-1.5B-Instruct revision; validation reports 1,543,714,304 parameters. | | Available training data and assets | The complete solutions_py_decontaminated subset has 8,133 rows. The fixed split provides 8,005 training rows and 128 validation rows. Exploration also mounts public v4/v5 exports. | | Agent-editable surface | Data selection, label masking, targets, batching, training schedule, checkpoint export, and ordinary training code in the work tree. | | Fixed or forbidden components | Starting weights and training mounts are read-only. Formal retraining has no LiveCodeBench rows, external weights, network access, or hidden-evaluation lookup. | | Proxy evaluator | livecodebench_public_pass_at_1, maximize. It uses public v4/v5 data: a 64-item health tier and a non-overlapping 204-item confirm tier, greedy n=1. The receipts report descriptive standard errors. | | Final evaluator | livecodebench_v6_pass_at_1_full175, maximize. It uses all 175 v6 problems, ten samples per problem, and averages pass@1 over 1,750 samples (avg@10). The authoritative settings are temperature 0.2, top_p 0.95, max_tokens 2048, seed 0. The reported standard error is descriptive across problems, not seed or replay uncertainty. | | Artifact contract | Complete loadable Hugging Face causal-language-model checkpoints, with at most three retained/published artifacts selected by numeric progress. |

The proxy measures public v4/v5 code-generation behavior and was used for quick health and relative-order checks. The final evaluator uses the score-only v6 asset and a different sampling protocol, denominator, and split. Their numerical scores are therefore not directly comparable. The old source text and old final_eval.py still mention first128 greedy evaluation; those are package-history fields, not the current final contract.

3.2 How the baseline works

The baseline reads four parquet shards, shuffles with split seed 20260727, and takes 8,005 rows for training and 128 for validation. Each conversation is rendered with the Qwen chat template. Labels before the assistant completion are set to -100, so the loss is computed only on assistant-completion tokens; the existing 32,768-token truncation behavior is retained. The full Qwen model is updated with bf16, gradient checkpointing, per-device batch size 3, gradient accumulation 6 (effective batch 18), learning rate 1e-5, gradient clip 0.2, and cosine decay to 0.1 of the base rate. Hugging Face Trainer evaluates and saves every 30 steps, retains at most three checkpoints, and defaults to 60 optimization steps.

The data flow is: fixed CodeForces conversation -> Qwen completion tokens -> labels from the recorded assistant answer with the prompt masked -> completion-only cross-entropy update -> changed full Qwen weights and loadable checkpoints. There is no reward model, generated label, search service, or external data. The agent's initial diagnosis was that some rows were not reliable supervision because they did not finish normally or their diff answers failed self-contained public tests; it also suspected that the short step budget left the schedule in warmup. The records do not establish either cause independently.

4. Four-hour exploration and decision process

The agent first read the task, baseline, data fields, and public evaluator, then measured health and confirm baselines. It used a short unfiltered control, inspected row-level quality, debugged deadline handling, ran a roughly 45-minute and a roughly 50-minute filtered training probe, evaluated selected checkpoints on both public tiers, checked loading and source lineage, and then submitted. The following units group repeated polling and related parameter variants by research question.

U-01 - Establish the baseline and identify row-level supervision risks

Motivation and hypothesis. The agent wanted a public reference at two tiers and a diagnosis of whether the fixed corpus contained cheaply detectable noisy supervision.

Concrete change and experimental setup. The fixed starting model scored 4/64 = 0.0625 on the health tier, with descriptive standard error 0.03026, and 12/204 = 0.0588235 on the offset-64 confirm tier, with standard error 0.01647. An eight-step unfiltered baseline probe took 502.6848 seconds, reached training loss 1.16693 and validation completion NLL 1.12407, and showed one 32K truncation among 8,005 training rows. The agent also inspected finish_reason, problem_type, public/private/generated-test fields and sampled self-test outcomes.

Observed result. The corpus contains 8,133 rows. The small diagnostic sample included passes, failures, errors, timeouts, no-code cases, and rows without tests. These are fields of the allowed CodeForces training corpus, not v6 evaluation inputs.

Agent interpretation. The agent treated non-stop rows and diff rows that could be falsified by their own public tests as “provably unreliable” supervision and expected filtering to improve the signal.

Report assessment and confounds. The diagnosis is plausible but not causal evidence. The public tiers are small, and the later experiments also changed elapsed training and scheduling. No matched, same-wall-clock filter-only control was run.

Decision and consequence. The quality-filter hypothesis was retained for implementation; the data inspection itself was not treated as a performance result.