Code-model post-training

openr1_code_livecodebench - claude-opus-5 / Claude Code / max

Status. Exploration and submission completed normally; formal replay ran to completion, all three published artifacts loaded, and the current full175 final evaluation has a summary plus matching .complete for each artifact. Boundary audit: observable behavior clean, no protocol or evaluation-boundary exposure found, and platform/resource isolation compliant.

What happened. The task fine-tunes fixed Qwen2.5-Coder-1.5B-Instruct on 8,133 fixed CodeForces-Python rows. The baseline masks the prompt and trains all parameters on assistant-completion causal-language-model negative log likelihood (NLL). The agent measured a median raw completion of about 13.2k tokens, only 4.9% within the 2,048-token generation cap, and found 40.5% of testable teacher programs failing their own tests. It submitted a sweep that rewrites targets, verifies teachers with row-owned tests, ranks branches on a fixed internal holdout, and exports three checkpoints. The best current-protocol score is 0.124571, above the starting reference 0.09657 but below the shipped reference 0.12743.

Four-hour exploration. The agent first diagnosed length and teacher quality, then tested code-only and brief chain-of-thought (COT) targets. Code-only at 958 steps scored 2/64; brief-COT at 479 and 958 steps scored 1/64 and 0/64; verified brief-COT at 125 steps scored 3/64. It interpreted heavier short-target supervised fine-tuning (SFT) as damaging, while the small 64-row tier and changing retained rows make this directional rather than definitive. A verified full-trace candidate trained for 60 steps in 1,726 seconds and scored 3/64 health and 14/204 confirmation versus the fixed start's 4/64 and 12/204; the agent retained the long family, but the two-row proxy difference is noisy. The prior no-think SFT in exploration attempt 1 also reached 3/64 health and 14/204 confirmation with ten clips, while its rejection-sampling/RFT continuation reached 11/204; neither was submitted. A real out-of-memory (OOM) event motivated a memory ladder; a failed sweep, scoring segmentation faults, and an end-to-end plumbing check were rerun or kept separate from performance. The prior attempt was host-captured with active work. A final 204-row short-candidate evaluation was killed unread when the objective timer showed 120 seconds remaining.

How the submitted method works. Fixed rows become full-trace, extracted-code, or budgeted brief targets; an executor filters clearly wrong teacher programs; full-parameter causal-LM NLL with AdamW still performs the update; an internal evaluator greedily generates and runs code on fixed holdout rows, ranking by passes and cleared-test fraction. Thus the change is mixed target/data generation, filtering, schedule, engineering, and checkpoint policy, not a new optimizer. Formal replay reconstructed everything from the fixed start with seed 42 and split seed 20260727; exploration weights and caches were excluded.

Formal and evaluation evidence. The formal sweep completed 19 branches in 39,510.623 of 43,200 seconds (91.46%), without wall-clock salvage. Publication progress is not optimizer-step count:

| Checkpoint | Current metric and diagnostics | Finding | |---|---|---| | 1901, brief2400+4800 verified, 250 steps | 0.109714 = 192/1,750; stderr 0.021544; extraction 0.7486; 396 clipped | valid, lowest | | 1902, long verified, 240 steps | 0.124571 = 218/1,750; stderr 0.022710; extraction 0.8629; 243 clipped | valid, selected best | | 1903, long verified, 120 steps | 0.121143 = 212/1,750; stderr 0.022336; extraction 0.8743; 232 clipped | valid, below 1902 |

The metric is livecodebench_v6_pass_at_1_full175: temperature 0.2, top_p 0.95, max_tokens 2048, ten samples per each of 175 v6 problems, avg@10. The 1902–1903 gap is smaller than descriptive stderr, so it is a receipt-level selection, not stable superiority. The 141-row internal proxy and final full175 score are different protocols; the long proxy direction only partly survived.

Audit and takeaway. The fixed data/model, offline boundary, evaluator, patch hash, source run, validation, and final receipts agree; generic formal control status is running beside formal_result=succeeded and exit 0, so its meaning is inconclusive but completed receipts prevail. No network acquisition, extra model, hidden question ID, prompt, test, or answer was found in the complete visible trajectory, patch, or commands. The old first128 evaluator metadata is a task-package defect, not hidden-value exposure. One GPU was used per phase with no recorded external sharing; 120 seconds could not complete the previously measured 562–694-second confirmation run, and formal replay used 91.46% of its budget without a short endpoint. The agent demonstrates disciplined target and resource engineering, but the evidence cannot show that it beats the shipped recipe or is seed-stable.

claude__openr1_code_livecodebench__claude-opus-5__max - Claude Code / max - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks the agent to improve a fixed Qwen2.5-Coder-1.5B-Instruct on a Python CodeForces projection so that generated programs pass execution tests. The baseline is completion-only supervised fine-tuning: prompt labels are masked, causal language-model negative log likelihood is computed only on the assistant completion, and all base-model parameters are updated. The agent identified a concrete mismatch: raw assistant traces have a median completion of about 13.2k tokens and only about 4.9% finish within the final 2,048-token generation cap; among testable teacher programs, about 40.5% fail their own tests.

During four hours it compared code-only targets, clipped brief reasoning, full long traces, and execution-based teacher filtering. It recovered from a real out-of-memory event, a failed long sweep, and scoring segmentation faults. The submitted artifact is not one hand-picked weight: it is a budget-aware sweep that rewrites targets, filters teachers with fixed-row tests, ranks candidates on held-out corpus rows, and publishes three loadable checkpoints.

Formal replay completed 39,510.6 of 43,200 seconds (91.46%), scored 19 branches, and validated three artifacts. Under the current livecodebench_v6_pass_at_1_full175 protocol, checkpoint-1902 is best at 0.124571 (218 correct samples out of 1,750). Every checkpoint exceeds the current starting reference 0.09657, but the best remains below the shipped-recipe reference 0.12743. The small proxy gain of the long family over the fixed start therefore did not establish an advantage over the shipped recipe.

The strongest capability is turning target length, teacher quality, runtime, and artifact integrity into an executable selection procedure. The main limitations are small public proxy samples, only 141 scorable rows in the formal internal proxy, and final extraction rates of roughly 0.75–0.87. Literal cross-source checks found no hidden question values in the visible trajectory or candidate; objective receipts support clean submission timing and formal-budget use.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

~~~ Starting artifact / model: Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397..., mounted as fixed-asset mount Available training data and assets: open-r1/codeforces-cots@39ac..., subset solutions_py_decontaminated, 8,133 fixed rows. The task source describes a shipped 8,005/128 split; formal sweep reallocated the fixed rows to 7,605 train plus 528 internal holdout rows. Agent-editable surface: training scripts, target construction, row selection, batching, schedule, and checkpoint export under editable workspace; no data or weights may be carried in the patch. Fixed or forbidden components: starting weights, fixed corpus, evaluator inputs, hidden/final assets, and offline/network boundary. Formal replay mounts fixed assets only and cannot train on public or v6 questions. Proxy evaluator: public fast-eval, greedy n=1 and max_new_tokens=2048; 64-row health and 204-row confirmation tiers; executed passes are higher-is-better, with extraction, clipping, and descriptive stderr diagnostics. Final evaluator: livecodebench_v6_pass_at_1_full175; temperature 0.2, top_p 0.95, max_tokens 2048, ten samples per problem, all 175 v6 problems, avg@10 over 1,750 samples; higher is better. Reported stderr is descriptive across problems, not replay or seed uncertainty. Artifact contract: publish complete HF checkpoints as checkpoint-<numeric>, retain at most the three highest numeric progresses, and pass frozen AutoModelForCausalLM loading validation. ~~~

The proxy measures one greedy execution against public or fixed-corpus tests; the final evaluator estimates pass@1 with ten stochastic samples on all v6 problems. The formal internal proxy requested 192 rows but only 141 had executable tests. These protocols have different splits, sampling, and denominators and must not be directly compared. Current references are 0.09657 for the start and 0.12743 for the shipped recipe; old first128 source values are protocol background only.

3.2 How the baseline works

The fixed dialogue is rendered with the chat template. The model predicts the assistant completion, while the common prompt prefix is assigned label -100 and contributes no loss. AdamW and a cosine-style schedule update the full Qwen parameter set, and complete HF checkpoints are saved by numeric progress:

~~~ [fixed CodeForces dialogue and assistant completion] -> [Qwen predicts assistant tokens] -> [dataset completion supplies labels; prompt is masked] -> [completion-only causal-LM NLL and full-parameter AdamW update] -> [loadable HF checkpoint] ~~~

The agent explicitly diagnosed output-length mismatch and unverified teacher errors; it did not claim that a lower training loss alone implied better code correctness.

4. Four-hour exploration and decision process

The agent first inspected the baseline, fixed assets, and evaluator boundary. It then used short probes for target statistics, memory, and health-level behavior, followed with target rewriting and teacher verification, and spent the remaining time on long-branch confirmation, an internal sweep, and checkpoint publication plumbing. OOM and concurrent scoring failures were handled by reducing forward microbatches while preserving per-step token and optimizer work, stopping runaway processes, and rerunning before drawing conclusions.

U-01 - Are target length and teacher correctness the primary bottlenecks?

Motivation and hypothesis. The final generation budget is 2,048 new tokens. If most training targets are long reasoning traces, the model may not emit code before truncation. The agent also needed to test whether teacher programs were executable.

Concrete change and experimental setup. It measured code-only, brief-COT budgets of 2,400 and 4,800 characters, and full traces on the fixed corpus, then ran each row's own public/private/generated tests while preserving row identity.

Observed result. In a 586-row length sample, code-only target p50/p90/p99/max was 267/739/1,692/2,742 tokens; brief-COT@2,400 was 863/1,310/2,238/3,410; @4,800 was 1,526/1,952/2,737/4,027. Of 5,491 testable teacher programs, 3,269 passed (59.5%); failures were 1,938 wrong answers, 184 runtime errors, and 100 timeouts. Another 2,642 rows had no executable tests and were retained.

The earlier captured attempt supplied a related audit: 3,958 rows were testable, with 2,771 passes and 1,187 failures, while 4,175 rows were unknown. Its refuted-row-filtered no-think probe reached 14/204 on the confirmation tier after 197 steps, but this was an exploratory public-proxy result rather than a formal checkpoint.

Agent interpretation. Long traces are the central output-cap mismatch; execution filtering can remove obvious errors but should not discard every untestable row.

Report assessment and confounds. Length values are sampled, and the pass rate applies only to testable rows. The two independent diagnostics nevertheless support changing target construction.

Decision and consequence. The agent kept two families: full long traces as a capability hedge and light short targets, with execution verification as an optional filter in the submitted patch.