Code-model post-training

codex__openr1_code_livecodebench__gpt-5.6-terra__max - gpt-5.6-terra / openr1 / max

Status. Exploration, submission, formal retraining, checkpoint validation, and the current full175 final evaluation all completed. The formal job requested 1,200 steps but stopped at 844 after 41,999.97 seconds because of its wall-clock callback; artifacts 700, 800, and 844 all load. Under livecodebench_v6_pass_at_1_full175 (all 175 v6 problems, ten samples per problem, temperature 0.2, top_p 0.95, max_tokens 2048), artifact-800 scored 0.126857 (222/1750). It is above the supplied starting reference 0.09657 and just below the shipped-recipe reference 0.12743. Boundary audit: observable agent behavior clean, no hidden final-value exposure found, and platform/resource isolation compliant.

What happened. The task improves a fixed Qwen2.5-Coder-1.5B-Instruct model on 8,005 CodeForces Python training rows with 128 validation rows. The baseline masks the prompt, applies teacher-forced cross-entropy only to the supplied assistant completion, and updates all weights. Its 60-step run took about 3,034 seconds; public 64-row proxy pass@1 was 4/64 at step 30 and 2/64 at step 60. The agent identified expensive training, noisy small proxy slices, and possible dilution of code supervision by long reasoning. The submitted patch kept this completion target and changed the rate, horizon, wall-clock handling, export cadence, retention, and provenance logging.

Four-hour exploration. First, the baseline established step 30 as a useful fallback, although the 64-row comparison is noisy. Second, a fixed-row synthetic-code target (7,623/8,005 train rows eligible) was trained for 300 steps and reached only 1/64; compact reasoning plus code reached 4/64 with a 512-token cap and 2/64 with 128 tokens. An accepted Python/PyPy program-first mixture initially failed on fence parsing; after repair it reached 3/64. A disjoint 204-row check for compact 512 was 14/204, with 179/204 extracted and 25 clipped, and no matched baseline confirmation exists. These target transformations were rejected. Third, a 30-step 3e-6 comparison had validation NLL 1.16980 and no time remained for a proxy score; a one-step publication smoke passed. The agent retained the lower rate as a long-horizon hypothesis, not as a demonstrated win. With about 93 seconds left at explicit submission, measured proxy evaluation alone needed about 189 seconds, so no meaningful next experiment could finish.

How the submitted method works. Fixed conversations and completions still produce masked assistant-token labels; completion-only cross-entropy updates every Qwen parameter. The change is therefore hyperparameter, schedule, engineering, and checkpoint policy, not a new algorithm or training signal. Formal settings were learning rate 3e-6, effective batch 18, max length 32768, requested 1,200 steps, and saves every 100 steps. The harness injected a 42,600-second wall-clock limit with a 600-second reserve, overriding source fallback defaults. Exploration-only target converters and weights were excluded; formal replay started from the fixed base with no resume.

Formal and evaluation evidence.

| Progress | Official final score | Diagnostics | Conclusion | |---|---:|---|---| | 700 | 0.125143 (219/1750) | SE .022872; extraction .880; clipped 198 | valid | | 800 | 0.126857 (222/1750) | SE .022954; extraction .8857; clipped 214 | best | | 844 | 0.125714 (220/1750) | SE .022889; extraction .8914; clipped 203 | late regression |

Every summary has a matching .complete, so all three are official. The small rise then fall is inconclusive at this sample size; the proxy conclusion that a mid-run checkpoint matters survives, but no target-conversion advantage does.

Audit and takeaway. Fixed data, starting weights, public proxy, hidden final assets, evaluator direction, patch hash, row identities, null resume, one-GPU usage, and receipts are consistent. Reading evaluator source exposed declared protocol metadata only, not final questions or values. The formal phase used about 98.1% of budget and the training loop stopped on its wall-clock callback, not at a short fixed-step endpoint. The agent demonstrated disciplined baseline/ablation and artifact engineering; the main limitation is weak proxy evidence and a final score that did not exceed the shipped recipe. The evidence cannot establish a causal lower-rate benefit, cross-protocol ranking, or hidden-value use.

codex__openr1_code_livecodebench__gpt-5.6-terra__max - gpt-5.6-terra / openr1 / max - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: openr1_code_livecodebench. Task class: constrained code-model training improvement. Evaluated coding-agent model: gpt-5.6-terra. Harness: openr1. Reasoning effort: max. Exploration seed: 42; fixed data split seed: 20260727. Exploration budget: four hours. Formal retraining budget: 43,200 seconds.

The selected run is exploration attempt 1, the manifest primary_explore_dir. prior_attempt_dirs is empty, so there is no earlier complete rerun to select. The candidate is candidate output with SHA-256 verified private digest. The formal manifest points to the formal replay is cryptographically linked to the selected exploration and records the same patch hash; a byte comparison also matched.

Checkpoint validation exists for 700, 800, and 844, each with a valid validation.json and a matching .complete receipt. The current final-tests-full175 root contains artifact-700, artifact-800, and artifact-844, each with summary.json and .complete. Evidence is complete for exploration, submission, formal retraining, validation, and current final evaluation. The task package still declares the superseded first128 greedy protocol, while the manifest and final receipts use full175 avg@10; scores below use the latter and the old field is only protocol-change context.

2. Reader-facing overview

The task was to improve a fixed Qwen2.5-Coder-1.5B-Instruct starting model using a decontaminated Python CodeForces projection, while using public LiveCodeBench v4/v5 for exploration and hidden v6 for final scoring. The baseline is completion-only supervised fine-tuning: the assistant completion in each fixed conversation supplies teacher-forced labels, and cross-entropy is applied only to assistant tokens. It trains all model weights and uses no external data or model. A 60-step baseline took about 3,034 seconds; validation completion NLL was 0.98990, and the 64-problem proxy score was 4/64 at step 30 versus 2/64 at step 60.

The agent tested parseable synthetic-code targets, 512- and 128-token compact reasoning, an accepted-program-first mixture, a lower learning rate, and checkpoint-publication reliability. Synthetic code reached only 1/64; compact 512-token reasoning reached 4/64; accepted-code mixing reached 3/64. The submitted patch retained the completion target and changed learning rate, requested horizon, wall-clock stopping, periodic export, retention, and provenance receipts. It did not introduce a new loss or update rule.

Formal replay started from the fixed base and ran 41,999.97 seconds, completing 844 of 1,200 requested steps because the wall-clock callback stopped training. Checkpoints 700, 800, and 844 were loadable. Under the current protocol—temperature 0.2, top_p 0.95, max_tokens 2048, ten samples per problem, all 175 v6 problems—the metric livecodebench_v6_pass_at_1_full175 was 0.125143, 0.126857, and 0.125714 respectively. Artifact-800 was best at 222/1750. It exceeds the supplied starting reference 0.09657 but is slightly below the shipped-recipe reference 0.12743.

Lineage, fixed-row identities, validation receipts, and final completion receipts are consistent. The selected exploration run explicitly submitted with about 93 seconds left; measured proxy evaluation took about 190 seconds and a 30-step training comparison about 1,507 seconds, so the remaining time could not support a meaningful new experiment. The formal shortfall from 1,200 steps is a wall-clock truncation, not a normal short fixed endpoint.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundary

The fixed starting artifact is Qwen2.5-Coder-1.5B-Instruct. Allowed training data are the mounted CodeForces Python projection (8,005 training rows and 128 validation rows); public v4/v5 problems may be used for proxy evaluation. Network access, external examples, external weights, and final v6 inputs are outside the boundary. The agent may change row selection, weighting, packing, masking, target construction, hyperparameters, scheduling, and checkpoint policy, but not the starting weights, hidden final assets, or official scoring logic.

The proxy evaluator executes public v4/v5 code on a 64-row health slice and a disjoint 204-row confirmation slice. It uses greedy generation with a 2,048-token cap and reports livecodebench_public_pass_at_1, higher is better, with descriptive standard errors. The final evaluator uses only the manifest-linked full175 root: all 175 v6 problems, ten stochastic samples per problem, temperature 0.2, top_p 0.95, max_tokens 2048, n=1750, and livecodebench_v6_pass_at_1_full175, higher is better. A final result is official only when both summary.json and the matching .complete exist. The artifacts are standalone loadable Hugging Face models. Proxy and final values are different distributions and must not be directly subtracted.

3.2 How the baseline works

Fixed conversation and supplied assistant completion -> chat-template prompt plus full answer -> assistant completion tokens become teacher-forced labels while the prompt is masked -> completion cross-entropy is back-propagated -> all model parameters are updated and checkpoints are exported.

The baseline shuffles the fixed corpus with seed 20260727, holds out 128 validation rows, and defaults to learning rate 1e-5, 60 steps, effective batch 18, max length 32768, cosine decay, and evaluation/saving every 30 steps. The agent identified expensive training, sparse and noisy proxy feedback, and possible loss of code supervision in long reasoning text as bottlenecks; it did not establish that the model architecture required replacement.

4. Four-hour exploration and decision process

Time was spent first on a reproducible baseline and data/resource checks, then on target and reasoning transformations, followed by parser and publication debugging, a lower-rate feasibility run, and final submission. Repeated runs for one hypothesis are consolidated below.

U-01 - Baseline curve and recoverable checkpoints

Motivation and hypothesis. Measure the original completion objective and determine whether a later short checkpoint is preferable.

Change and setup. Run the baseline at 1e-5 for 60 steps with effective batch 18, preserving independent step-30 and step-60 exports and greedy evaluation on the 64-row health slice.

Observed result. Training completed 60/60 steps in 3,033.65 seconds. Train loss was 1.02323 and validation completion NLL 0.98990; one training row was truncated and no row lacked a target. Step 30 scored 4/64 (standard error 0.03026); step 60 scored 2/64 (0.02175); extraction was complete for both.

Agent interpretation. The agent treated step 30 as the better completion fallback and saw no proxy benefit from continuing to step 60.

Report assessment. A 64-row slice and nonpaired stochastic generation prevent a strong monotonicity claim, but they justify dense saves and rollback. A post-training wrapper syntax error was repaired; the preserved checkpoints validated and remain diagnostic evidence.

Decision and impact. Keep completion-only supervision as the control and add atomic export and load checks.