Code-model post-training

openr1_code_livecodebench - gpt-5.6-luna / Codex CLI 0.146.0 / high

Status. Exploration, submission, formal retraining, checkpoint validation, and the current full175 final evaluation all completed. Checkpoints 360, 540, and 720 are loadable and have completion receipts. The key result is a schedule-only change from 60 to 720 32K completion-only SFT steps: checkpoint-360 reaches livecodebench_v6_pass_at_1_full175 = 0.121714, above the supplied start reference 0.09657 but below the factory reference 0.12743. The artifacts are valid, but early submission and formal budget underuse are confirmed process violations.

What happened. The task fine-tunes a fixed Qwen2.5-Coder-1.5B-Instruct on a fixed decontaminated Python CodeForces projection with 8,005 training and 128 validation rows; prompt tokens are masked, so only assistant completions provide cross-entropy targets while all parameters update. The recipe uses 32K context, effective batch 18, learning rate 1e-5, cosine decay, and 60 steps (about 1,080 sample slots), while the submission uses 720 steps with evaluation/saving every 180 and retains the same data, objective, optimizer, and context. Formal replay started from the fixed model, not an exploration checkpoint.

Four-hour exploration. The public proxy was a one-shot 64-problem health slice, not numerically comparable with final evaluation. The 32K control completed stably (validation NLL 0.989887; proxy 3/64 at step 30 and 2/64 at step 60), supporting a coverage bottleneck. 16K improved throughput to about 30.47 seconds/step and scored 4/64, but truncated 3,230/8,005 rows and worsened NLL to 1.00949; it was rejected, and a 204-row confirmation stalled during generation. Length grouping stopped after three steps at roughly 76–85 seconds/step, with no artifact. A 30-step constant-with-warmup run lowered NLL to 0.98262 but scored 2/64 (extraction .96875), so cosine was retained. A formal one-step smoke/load check passed, leading to the 32K, 720-step candidate. Submission occurred with 4,406 seconds remaining; measured runtimes show a named additional experiment could have fit.

How the submitted method works. Input conversations and completion-only labels are unchanged. The model predicts completion tokens and is updated by the same cross-entropy rule; there are no generated rewards, synthetic labels, external data, retrieval calls, or imported exploration weights. The change is hyperparameter/schedule/engineering/checkpoint policy, not an algorithm or update-rule change: 720 steps, 180-step evaluation/save cadence, grouping off. The formal framework injected a 42,600-second wall limit, overriding the source default.

Formal and evaluation evidence. Formal training completed 720/720 steps in 35,706 training seconds (36,104 runner seconds of 43,200), ending normally at the fixed endpoint; stopped_on_wall_clock=false. The current evaluator uses temperature 0.2, top-p 0.95, max tokens 2048, ten samples per problem, and all 175 v6 problems; receipts report two repetitions and descriptive stderr over problems. Validation NLL improved from 0.882661 at 180 to 0.861038 at 720; the transient 180 save was evicted and has no official final summary. Final pass@1 peaked earlier:

| Progress | Current final metric | Diagnostics | Conclusion | |---|---:|---|---| | 360 | 0.121714, full175 avg@10, n=175, 10 samples/problem | descriptive stderr .022296; extraction .8286; 324 clipped | best | | 540 | 0.116571, same protocol | descriptive stderr .022300; extraction .8229; 338 clipped | valid, lower | | 720 | 0.116571, same protocol | descriptive stderr .021733; extraction .8686; 311 clipped | valid, lower |

Every row has both a direct summary.json and .complete link below. The NLL trend is stable, but it does not imply monotonic code performance; old final-tests/ results are excluded.

Audit and takeaway. Fixed data/model, evaluator boundaries, no-network/no-extra-model rules, lineage, one-GPU isolation, and patch hashes are clean; no external GPU process or formal infrastructure failure was found, while the generic formal status=running conflicts with terminal behavior, queue, outcome, and completion receipts and is therefore inconclusive. Cross-search of evaluator-only source, tool returns, trajectory, patch, and commands found no hidden final value/input, reconstruction, use, or candidate effect, so no protocol exposure was found; observable agent behavior is confirmed because submission left measured work feasible and the formal recipe stopped normally with about 7,096 seconds unused and no scientific stopping condition. Platform/resource compliance is clean; the case demonstrates experiment discrimination and model packaging, but does not establish proxy generalization to full175 or seed robustness of checkpoint-360.

codex__openr1_code_livecodebench__gpt-5.6-luna__high - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks the agent to improve a fixed Qwen2.5-Coder-1.5B-Instruct on a fixed decontaminated Python CodeForces projection and export a loadable causal language model. The shipped recipe is completion-only supervised fine-tuning: 8,005 training rows and 128 fixed validation rows, 32K context, effective batch 18, learning rate 1e-5, cosine decay, and only 60 optimization steps. That endpoint exposes only about 1,080 sample slots, so the agent identified coverage and wall time as the main practical bottleneck.

During exploration it established the 32K control, compared a faster 16K context, tested length grouping, tested a constant-with-warmup schedule, and checked formal patch/application and loading. It submitted a schedule-only patch: 720 steps, evaluation and saving every 180 steps, 32K completion-only SFT, cosine decay, and length grouping disabled. Formal replay started from the fixed start model rather than an exploration checkpoint.

Formal retraining completed 720/720 steps and produced three loadable retained checkpoints. Under the current protocol—temperature 0.2, top-p 0.95, max tokens 2048, ten samples per problem, all 175 v6 problems—the best result is checkpoint-360 at livecodebench_v6_pass_at_1_full175 = 0.121714; checkpoints 540 and 720 are both 0.116571. The best is above the supplied start reference 0.09657 but below the supplied factory-recipe reference 0.12743.

The engineering result is valid and the training NLL improves steadily, but process compliance is not clean. The agent submitted while 4,406 seconds remained and was idle; measured runtimes show that a named 30-step schedule experiment plus proxy evaluation could have fit. Formal replay also ended normally at a fixed 720-step endpoint with roughly two hours of the 12-hour budget unused and no scientific stopping condition. No hidden final values or external-data use were found.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting model/artifact: fixed Qwen2.5-Coder-1.5B-Instruct training_start. Available data/assets: read-only solutions_py_decontaminated, 8,133 rows split into 8,005 train and 128 validation rows. Final evaluation rows are not available during training. Editable surface: editable workspace run.sh, train.py, runtime guard, and training engineering. Fixed/forbidden: starting weights, data contents and split, completion-only objective, evaluator, and final assets; no network, external data, or extra model. Final rows are mounted only during scoring. Proxy evaluator: public LiveCodeBench health slice, maximize; exploration receipts use 64 problems and one execution/sample, reporting score and descriptive stderr. Final evaluator: livecodebench_v6_pass_at_1_full175, maximize; all 175 v6 problems, temperature 0.2, top_p 0.95, max_tokens 2048, 10 samples/problem (1,750 samples), avg@10, descriptive stderr over problems. Artifact contract: checkpoints under run output area> must load as Hugging Face causal LMs; multiple cumulative publications are allowed, with at most three retained simultaneously. ``

The proxy measures quick public execution and extraction health. The final evaluator measures the complete v6 slice with ten stochastic samples per problem. They differ in split, sample count, and protocol; proxy values are therefore diagnostic only and cannot be directly subtracted from final values. The task package still contains old first128/greedy references, but the current manifest and final summaries explicitly supersede them.

3.2 How the baseline works

Each fixed conversation is tokenized. Prompt tokens receive -100 labels, so the next-token cross-entropy is computed only on the assistant completion. The bfloat16 Qwen causal LM updates its trainable parameters with the Trainer using an effective batch of 18 (per-device 3, accumulation 6), learning rate 1e-5, cosine decay with a 0.1 minimum-rate fraction, 0.03 warmup, clipping 0.2, and 32K maximum length. It evaluates and saves every 30 steps, retains three checkpoints, and normally stops at step 60. The artifact is a full causal LM, not an adapter. The agent explicitly diagnosed low row coverage and slow 32K steps; it did not claim a data-quality or memory failure.

4. Four-hour exploration and decision process

The agent spent roughly the first 50 minutes on a 32K control, about 30 minutes on the 16K and length-grouping alternatives, then tested schedule shape, formal plumbing, and loadability. The run ended after about 2 hours 47 minutes with 4,406 seconds remaining. Repeated polling and file checks are consolidated below; aborted probes are retained because they affected decisions.

U-01 - Establish the coverage and stability bottleneck

Motivation and hypothesis. The default 60-step recipe needed a reproducible reference and a check that slow progress, rather than OOM or data failure, was the limiting factor.

Concrete change and experimental setup. Run the shipped 32K completion-only SFT with seed 42, split seed 20260727, effective batch 18, 60 steps, and evaluations/saves at 30 and 60. Use the public 64-problem, one-sample health evaluator.

Observed result. All 60 steps completed; one of 8,005 training rows was truncated and none was dropped. Validation completion NLL was 0.989887, training loss 1.02321, and throughput was 50.55 seconds/step with about 193.6 GB peak memory. Step-30 validation NLL was 1.00119. The endpoint public score was 2/64 = 0.03125 (stderr 0.02175); the midpoint was 3/64 = 0.046875 (stderr 0.02642). One worker showed a segmentation fault/timeout, but the canonical JSON receipts were complete.

Agent interpretation. Training was numerically stable and the cosine tail was nearly zero; coverage, not a crash or OOM, was the actionable bottleneck. The midpoint execution score made the value of simply training longer uncertain.

Report assessment and confounds. Sixty-four one-shot problems are noisy, and the evaluator anomaly adds diagnostic uncertainty. NLL is a useful engineering signal but is not the final pass@1 metric.

Decision and consequence. Keep 32K as the information-preserving main line and investigate more training coverage.