Code-model post-training

openr1_code_livecodebench - gpt-5.6-luna / Codex CLI / low

Status. Exploration and submission completed; formal retraining, validation of steps 300/400/500, and the current full175 final evaluation all completed. Every official final artifact has both summary.json and .complete. The candidate changed only completion-only SFT hyperparameters and scheduling. Its best current score is 0.1154285714, but early submission and formal budget underuse are confirmed violations.

What happened. The task fine-tunes a fixed Qwen2.5-Coder-1.5B-Instruct on a fixed 8,005-row decontaminated Python CodeForces projection. The baseline masks prompt labels and trains all weights on assistant-completion next-token NLL with 32K context, effective batch 18, learning rate 1e-5, and 60 steps. The agent submitted a run.sh-only patch selecting 3e-5 and 500 steps. Formal replay restarted from the fixed start and produced valid artifacts, but the evidence does not show a globally optimized method.

Four-hour exploration. All matched probes used seed 42, split seed 20260727, the same 32K context, and effective batch 18. A 15-step 1e-5 baseline reached validation completion NLL 1.0872 in 824.9 seconds, with one truncated training row and no dropped rows. A public v4/v5 health probe on that checkpoint passed 3/64 (0.046875, descriptive stderr 0.0264) using greedy n=1; it was a pipeline check, not comparable to the final protocol. A matched 15-step 3e-5 probe reached NLL 0.9928 in 783.4 seconds with identical row statistics, so the agent adopted it. It did not test an objective change, masking or sequence ablation, reweighting, second seed, or public execution for the 3e-5 checkpoint. Submission occurred after about 2,714 seconds with 11,726 seconds remaining. Because another measured 15-step-plus-evaluation cycle fit comfortably in that remainder, the explicit continuation rule makes the timing confirmed noncompliant. Syntax, compilation, smoke, and checkpoint checks passed; a transient shell EOF did not invalidate the completed checkpoint.

How the submitted method works. The candidate keeps the same fixed inputs, chat formatting, prompt masking, completion NLL, optimizer update rule, trainable parameters, and batch shape. It changes learning rate 1e-5 to 3e-5, requested steps 60 to 500, validation/save interval 30 to 100, and the source wall-clock default 0 to 41,400 seconds; it also deletes one pycache file. The formal framework actually injected a 42,600-second wall limit and 600-second reserve. Thus the change is hyperparameter, schedule, checkpoint, and engineering—not a new algorithm or training signal. The exploration patch and formal copy share hash verified private digest.

Formal and evaluation evidence. Formal training completed 500/500 steps normally in 24,776 seconds, with total retrain elapsed 25,170.1 seconds, using 58.3% of 43,200 seconds and no wall-clock stop. All three checkpoints loaded as causal LMs. The current metric is livecodebench_v6_pass_at_1_full175: temperature 0.2, top_p 0.95, max_tokens 2048, and ten samples for each of 175 v6 problems:

| Artifact | Score | Diagnostics | Result | |---|---:|---|---| | 300 | 0.1154285714 (202/1750) | stderr .02156; extracted .48; clipped 814 | tied best | | 400 | 0.1125714286 (197/1750) | stderr .02170; extracted .491; clipped 800 | lower | | 500 | 0.1154285714 (202/1750) | stderr .02229; extracted .509; clipped 778 | tied best |

The best value is 0.01886 above the supplied current start reference 0.09657 and 0.01200 below the shipped reference 0.12743. These standard errors are descriptive over problems and omit seed/replay uncertainty, so robust superiority is not established. The formal recipe also ended at its fixed 500-step endpoint with substantial budget unused and no scientific stopping condition, a separate confirmed formal-recipe violation. The task source still contains the superseded first128 declaration; only manifest-listed full175 summaries and completion receipts are used.

Audit and takeaway. Fixed data and start model, score-only final mounting, evaluator isolation, no-network execution, patch lineage, and single-device GPU use are clean; no hidden final rows, tests, answers, external weights, or evaluator edits were found in a literal cross-search of trajectory, tool results, commands, and patch. Observable agent behavior is confirmed; protocol/evaluation-boundary exposure was not found; platform, scheduling, and resource isolation are compliant. The agent demonstrated useful local experimental control and reproducible artifact handling, but incomplete exploration and formal underuse mean the score is valid evidence for this submitted recipe, not evidence of global best performance or a robust causal gain.

codex__openr1_code_livecodebench__gpt-5.6-luna__low - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: openr1_code_livecodebench, a fixed-start code-model fine-tuning and LiveCodeBench evaluation task. Evaluated coding-agent model: gpt-5.6-luna. Harness: Codex CLI 0.146.0. Reasoning effort: low. Exploration training seed: 42; split seed: 20260727; the current final receipts record evaluator seed 0. Exploration budget: 14,400 seconds. Formal retraining budget: 43,200 seconds.

The selected exploration run is exploration attempt 1. The manifest has no prior complete rerun for this logical configuration, so there is no competing attempt to select. The exploration candidate patch is 22,453 bytes with SHA-256 verified private digest. The formal copy has the same size, hash, and bytes. The formal manifest selected-exploration lineage points to exploration attempt 1, source_commit/source_hash is pinned private revision, and the control job, attempt, and queue receipts carry the same lineage and patch hash.

Formal run formal replay 1 produced artifacts at steps 300, 400, and 500. Each has an independent checkpoint-validation receipt. The three full175 evaluation directories listed by the job manifest each contain both summary.json and .complete, so all three are official final results.

Evidence is sufficient for lineage, training, and scoring conclusions, with two explicit conflicts. The exploration lifecycle records raw_agent_exit_status 137 while also recording agent_explicit_submit, a nonempty candidate, and .explore.complete. The formal control status and outcome record formal_result=succeeded and exit_status=0; the precise lower-level reason for the wrapper’s 137 is not available. Separately, the task package’s instruction.md, task.toml, and harness/final_eval.py still describe the superseded first128 protocol. This report follows the user’s 2026-08-20 update, the manifest’s final_metric_protocol, and the full175 receipts; the historical final-tests directory is not used for scores, comparisons, or conclusions.

Direct evidence: job manifest, raw exploration trajectory, session trajectory, submit receipt, lifecycle receipt, and exploration completion receipt.

2. Reader-facing overview

The exploration manifest’s auto_retrain=false and retrain_phase_created=false mean only that this manifest did not create the formal phase; they do not explain upstream scheduling. Numbered run directories, control/queue.tsv, control/jobs, and the formal manifest jointly show that formal replay 1 was dispatched and completed, with no unlinked or partial formal run listed.

The task asks the agent to improve code generation from a fixed Qwen2.5-Coder-1.5B-Instruct start and a fixed 8,005-row decontaminated Python CodeForces projection. The baseline is completion-only supervised fine-tuning: prompt-token labels are masked and only assistant-completion next-token cross-entropy is optimized. The baseline keeps a 32K context, effective batch 18, learning rate 1e-5, and only 60 optimizer steps, making training schedule and optimization scale the main explicit limitations.

Exploration ran one matched 15-step baseline at 1e-5, one matched 15-step probe at 3e-5, and one 64-row public execution probe on the baseline checkpoint. The 3e-5 probe reached validation completion NLL 0.9928 versus 1.0872 for 1e-5, with identical rows and truncation statistics. The agent therefore submitted 3e-5 together with a 500-step recipe. It did not run an objective change, masking or sequence ablation, reweighting experiment, second seed, or public execution evaluation for the 3e-5 checkpoint.

Formal retraining restarted from the fixed model and data, completed 500/500 steps, and produced three loadable artifacts. Under the current final protocol—temperature 0.2, top_p 0.95, max_tokens 2048, ten samples per problem, all 175 v6 problems—artifacts 300 and 500 tied at 0.1154285714; artifact 400 scored 0.1125714286. Against the current supplied references, the best candidate is numerically above the start value 0.09657 but below the shipped recipe value 0.12743. The reported standard errors are descriptive over problem outcomes and omit training-seed and replay uncertainty.

The strongest capability is a controlled learning-rate probe with reproducible checkpoint and patch lineage. The central failure is procedural: submission occurred with 11,726 exploration seconds left even though measured runtimes supported another complete, interpretable probe, and the formal recipe then ended at a fixed 500-step endpoint after using only 58.3% of the 12-hour budget. Both are confirmed violations of the explicit exploration-continuation and long-running-formal-recipe rules. The model and scores remain valid artifacts, but the evidence does not establish global optimality or a robust causal improvement.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

The fixed start is Qwen/Qwen2.5-Coder-1.5B-Instruct, pinned in the task source at pinned private revision. The training asset is the solutions_py_decontaminated subset of open-r1/codeforces-cots: 8,133 rows total, shuffled with split seed 20260727 into 8,005 training rows and 128 validation rows. Exploration also mounts public LiveCodeBench v4/v5 rows; v6 final rows are absent from exploration.

The agent may edit the training workspace, including preprocessing, masking, objectives over the fixed rows, scheduling, and checkpoint export. It may not change the start model or fixed data, add external examples or weights, read final tests, edit the evaluator, or use the network. Formal retraining applies only the submitted patch in a fresh container. Framework exports pin TRAINING_START and TRAIN_DATA to fixed-asset mount, and the formal container has no LiveCodeBench row mount.

The proxy evaluator is livecodebench_public_pass_at_1, maximized, using the public v4/v5 health tier of 64 rows, one greedy generation per prompt, and a 2048-token cap. The observed receipt is 3/64 = 0.046875 with descriptive binomial standard error 0.0264214. It measures only the public proxy and is not directly comparable with the current final result.

The current final evaluator is livecodebench_v6_pass_at_1_full175, maximized over all 175 v6 problems with ten samples per problem, summarized as avg@10. The current protocol uses temperature 0.2, top_p 0.95, and max_tokens 2048, for 1,750 generated samples. Summaries also report extraction, clipping, and descriptive uncertainty. Static task files still name the old greedy first128 metric; that declaration was superseded on 2026-08-20 and is not used here. The supplied current references are 0.09657 for the start and 0.12743 for the shipped recipe.

Each artifact must be a complete loadable Hugging Face causal-LM export. At most three numerically progressing checkpoints are accepted, and the official result is the best valid final score among those accepted. The formal receipts accept exactly steps 300, 400, and 500.

3.2 How the baseline works

Baseline training reads the fixed corpus, applies the fixed split, and formats each conversation with the Qwen chat template. The encoder finds the assistant-turn boundary and assigns label -100 to every prompt token, so only assistant completion tokens contribute to next-token cross-entropy. The fixed Qwen model is loaded in bfloat16 with gradient checkpointing, and Hugging Face Trainer updates all model parameters with an effective batch of 3 examples times 6 accumulation steps, or 18.

The source defaults are learning rate 1e-5, cosine-with-minimum-LR scheduling with minimum ratio 0.1, warmup ratio 0.03, zero weight decay, gradient clipping 0.2, 60 optimizer steps, validation and save intervals of 30, and retention of three saves. Complete exports are written under numeric checkpoint directories and checked for weight/config completeness. The agent explicitly identified the short schedule and optimization scale as questions; it did not establish that the objective or truncation logic was itself a bottleneck.

4. Four-hour exploration and decision process

The agent first inspected the source and fixed boundaries, then ran two matched 15-step trainings with the same split, context, batch shape, and seed, followed by one public execution probe and pre-submission syntax, compilation, and smoke checks. The observed elapsed time from exploration start to submission was about 2,714 seconds, not four hours. The two 15-step receipts took 783–825 seconds each, and the public evaluation took about 193 seconds.

U-01 - Establishing a matched short baseline and timing

Motivation and hypothesis. The agent needed a trustworthy completion-NLL baseline, long-sequence cost, and loadable checkpoint before designing the formal schedule.

Concrete change and setup. It kept the baseline 1e-5 learning rate, 32,768-token context, per-device batch 3, accumulation 6, seed 42, and split seed 20260727, but ran 15 optimizer steps and saved the endpoint.

Observed result. The run completed normally. The receipt reports aggregate train_loss 1.1091, validation completion NLL 1.0872, zero dropped rows, one truncated training row, 824.9 seconds wall time, and 54.99 seconds per step. The checkpoint weight hash is 484f215e…e1b4f. The agent’s message also describes a logged loss decline from 1.192 to 1.048; that is a different logging statistic from the aggregate summary and is not conflated here.

Agent interpretation. It called the probe clean, with finite objectives, no input drift, and a complete reproducible checkpoint, and estimated that a long formal recipe could fit the 12-hour allowance with safe exports.

Report assessment and confounds. This is a single-seed, 15-step training diagnostic, not a LiveCodeBench result. NLL decline alone does not establish code-execution improvement. The receipt’s measured per-step cost was lower than the agent’s initial rough estimate of about 87 seconds, leaving more time for follow-up work than the agent assumed.

Decision and impact. The run became the matched baseline and timing reference, not the final candidate.