Code-model post-training

claude__openr1_code_livecodebench__claude-sonnet-5__medium - Claude Code 2.1.220 / medium

Status. Exploration and submission completed. Formal replay applied the patch from the fixed start but failed at optimizer step 444 with CUDA OOM; no accepted artifact or checkpoint-validation receipt exists, and final scoring never started. The effective final contract is livecodebench_v6_pass_at_1_full175: all 175 v6 problems, ten samples per problem, temperature 0.2, top_p 0.95, max_tokens 2048, averaged as avg@10. Its candidate score is not available. Historical final-tests/ records are excluded. The submitted method keeps completion-only cross-entropy but adds length grouping, wall-clock control, and a time-based cosine learning rate; it did not beat the public confirmation control before formal failure.

What happened. The task fixes Qwen2.5-Coder-1.5B-Instruct and an 8,005/128 split of solutions_py_decontaminated. The baseline masks prompt labels and updates all parameters on assistant-completion cross-entropy with effective batch 18. The agent diagnosed long, variable sequences and padding as the main practical bottleneck. A 90-step group_by_length candidate had lower training loss/NLL and a higher measured step rate than the 60-step control, but both scored 14/204 on the disjoint v4/v5 confirmation tier. The patch then added a pre-load wall clock, time-fraction warmup/cosine LR, and a callback-order fix.

Four-hour exploration. The 8-step diagnostic took 532.03 s and exposed roughly 13.6K mean sequence tokens; the 60-step control took 2,992.27 s at 49.87 s/step. Candidate A took 3,192.23 s at 35.47 s/step, with peak memory rising from about 193.6 GB to 228.3 GB. Health_64 was 3/64 for candidate versus 2/64 for control, but confirmation_204 was tied at 14/204 (both stderr 0.01770), so the capability conclusion remained inconclusive. Wall-clock integration stopped and reloaded checkpoints; the first 300-row test revealed HF scheduler LR logging being clobbered, and the second test fixed it (step 5, 244.06 s, final LR 4.4872e-6). No objective, data-source, or masking alternative was tested.

How the submitted method works. Fixed rows still produce assistant targets with prompt labels set to -100, and Qwen still receives the same full-parameter cross-entropy update. group_by_length changes batching; MAX_STEPS=20000 is only a backstop; formal orchestration actually injected MAX_WALL_TIME_SECONDS=42600 and DEADLINE_RESERVE_SECONDS=600; saves/evals move to every 100 steps with three retained. The exploratory checkpoint-90 was not reused: formal replay restarted from the fixed model.

Formal and evaluation evidence. Although manifest formal_run_dirs is empty, its unlinked_or_partial_formal_run_dirs target did execute: the command ran for about 13,781.6 s, entered step 444/20,000, and the forward cross-entropy at that step attempted a 55.64-GiB allocation before raising torch.OutOfMemoryError. The last written Trainer checkpoint was step 400; directories at 200/300/400 were not promoted to run output area and have no validation summary or .complete, so there is no official checkpoint row and no final metric. The current-protocol fixed-start and shipped-recipe references are 0.09657 and 0.12743, but no candidate comparison is valid. This is a crash, not a normal short fixed-step endpoint.

Audit and takeaway. Both phases exposed one GPU, formal used the fixed model/data in a fresh container, and no external network, public/final rows, exploration weights, or hidden test content entered the patch; after identifying score-only declarations in evaluator source/lock files, literal searches found no test6, livecodebench_final, matching path/hash, or hidden-row content. No hidden value reached the trajectory, no reconstruction or use is visible, and therefore no candidate result could depend on it. The same-config numbered records and queue/correction checks show no second medium formal attempt. Observable agent behavior is confirmed because submit.json recorded an explicit idle submission with 2,140 s remaining, while this trajectory's confirmation evaluation took 910.54–930.61 s and additional candidate checkpoints were unscored. Protocol/evaluation-boundary exposure: none found. Platform, scheduling, and resource isolation: compliant; L20D is accepted. The patch demonstrates useful throughput and callback debugging, but the tied proxy result and formal OOM leave current final ability and ranking unestablished.

claude__openr1_code_livecodebench__claude-sonnet-5__medium — Claude Code 2.1.220 / medium — Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks the agent to improve a fixed Qwen2.5-Coder-1.5B-Instruct on a fixed Python CodeForces projection. The baseline is completion-only supervised fine-tuning: prompt tokens are masked and cross-entropy is computed only on the assistant completion while all model parameters are updated. The agent measured highly variable long sequences and treated padding and wall-clock utilization as the practical bottlenecks. It submitted a recipe with length-grouped batches, a wall-clock stop, and a time-fraction cosine learning-rate schedule; the training signal and update rule stayed unchanged.

During exploration, the 90-step candidate reduced training loss and validation completion NLL and increased the measured optimizer-step rate relative to a 60-step control. However, candidate and control both solved 14/204 on the disjoint public confirmation slice, so the proxy evidence did not establish a code-capability gain. Formal replay applied the patch in a fresh container from the fixed start, but failed at optimizer step 444 with a CUDA out-of-memory error during cross-entropy. No accepted model artifact was published.

There is therefore no candidate score under the effective livecodebench_v6_pass_at_1_full175 protocol. The fixed-start and shipped-recipe reference values under that protocol are 0.09657 and 0.12743, respectively; neither is this trajectory's score. The strongest positive evidence is careful instrumentation and successful callback plumbing. The main failures are an explicit early submission with enough time for another measured experiment and an unmitigated formal-memory risk. The OOM was a recipe failure, not a short fixed-step completion or demonstrated platform contamination.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: Qwen/Qwen2.5-Coder-1.5B-Instruct, revision pinned private revision. Available training data and assets: open-r1/codeforces-cots, solutions_py_decontaminated; the lock contains 8,133 rows, with the default split using 8,005 training and 128 validation rows. Exploration also mounts a read-only LiveCodeBench v4/v5 public export. Agent-editable surface: editable workspace training objective/masking, row use and ordering, batching, length handling, schedule, and checkpoint/export code. Fixed or forbidden components: the starting weights, fixed data boundary, and evaluator; formal execution has no network and one GPU; external examples, answers, weights, evaluator prompts, and evaluation-specific lookup are forbidden. Proxy evaluator: livecodebench_public_pass_at_1, maximize; v4/v5 health_64 at offset 0 and disjoint confirmation_204 at offset 64; greedy n=1 with max_new_tokens=2048. Receipts report descriptive binomial standard errors, not paired or seed uncertainty. Final evaluator: livecodebench_v6_pass_at_1_full175, maximize; effective protocol uses all 175 v6 problems, 10 samples per problem, temperature 0.2, top_p 0.95, max_tokens 2048, and avg@10. No execution receipt exists. Artifact contract: complete Hugging Face causal-LM exports under run output area>; at most the three highest-progress valid checkpoints are accepted, each with validation/completion evidence. ``

The proxy measures one greedy generation on public v4/v5 rows; the current final measures ten sampled generations per problem over all v6 rows. Their data, sampling, and aggregation differ, so proxy values cannot be subtracted from the current final references. The checked task package still contains the old first128 greedy evaluator and metadata. That conflicts with the manifest/current instruction at the implementation level, which is inconclusive without a final receipt; the current manifest/task instruction is the authoritative final contract here. Historical final-tests/ records are excluded from scores and comparisons.

3.2 How the baseline works

``text fixed parquet rows -> shuffle with split seed 20260727, take 128 validation and 8,005 training rows, and apply the Qwen chat template -> use the original assistant completion as the target and set prompt labels to -100; no external reward or generated label is used -> run full-parameter bf16 causal-LM cross-entropy with Hugging Face Trainer, AdamW, gradient checkpointing, learning rate 1e-5, per-device batch 3, and accumulation 6 (effective batch 18) -> export a complete causal-LM checkpoint, normally evaluating/saving every 30 steps and retaining at most three ``

The agent's concrete diagnosis was a wide length distribution: the exploration summaries report about 13,590.6 mean full-sequence tokens and 12,791.3 mean supervised tokens, with a 32K cap. One training row was truncated and no row lacked a supervised target in the 8-step diagnostic. The agent did not execute a data-reweighting, objective, or external-label alternative; it focused on throughput and predictable use of an unknown formal hardware budget.

4. Four-hour exploration and decision process

The agent first inspected the trainer, data, evaluator boundary, and GPU, then ran an 8-step baseline smoke. It used the middle of the budget for a 60-step baseline, a 90-step grouped-length candidate, and four public proxy evaluations. The final part was spent on wall-clock stopping and learning-rate callback integration, checkpoint reload, patch preview, and source review. Repetitive polling is consolidated below. The run lasted about 12,314 seconds from 2026-08-13 18:08:32 to 21:33:46, leaving 2,140 seconds at submission.

U-01 - Establish the bottleneck and a comparable baseline

Motivation and hypothesis. The agent needed fixed-input, token-length, step-cost, and validation measurements before deciding whether to change the objective or simply increase useful training per wall-clock second.

Concrete change and experimental setup. It ran an unchanged 8-step diagnostic and then an unchanged 60-step control on the fixed 8,005/128 split, seed 42, effective batch 18, max length 32,768, with wall-clock stopping disabled and 30-step evaluation/saving.

Observed result. The 8-step run took 532.03 seconds (66.50 seconds/step), with training loss 1.16694 and validation NLL 1.12408. The 60-step run took 2,992.27 seconds (49.87 seconds/step), with training loss 1.02321 and validation NLL 0.98986. Both used one visible L20D GPU and the fixed rows; the control's peak memory was about 193.6 GB. Its checkpoint-60 was loadable for exploration, not a formal artifact.

Agent interpretation. The agent called the 60-step control “smoke-sized” and identified mixed long/short sequences and padding as the actionable bottleneck. It also concluded that a formal run should be wall-clock-governed rather than end at a short step count.

Report assessment and confounds. These are training diagnostics, not LiveCodeBench scores, and the 8- and 60-step endpoints do not establish final capability. The length measurements support the padding hypothesis, but there was no isolated per-token compute ablation.

Decision and consequence. The 60-step control became the later proxy A/B reference. The agent pursued length grouping and a long-running recipe without changing the data or objective.