Code-model post-training

codex__openr1_code_livecodebench__gpt-5.6-terra__medium - gpt-5.6-terra / Codex / medium

Status. Exploration and submission completed; formal replay completed 720 requested and completed steps; checkpoint validation accepted artifacts 60, 120, and 720; all three current full175 evaluations passed with both summary.json and .complete. Fixed inputs, hidden assets, evaluator boundary, and resource isolation are auditable. The overall observable agent behavior is confirmed, because explicit early submission and formal fixed-endpoint underuse are both confirmed violations; the score receipts themselves remain valid.

What happened. The task fine-tunes a fixed Qwen2.5-Coder-1.5B-Instruct on 8133 available decontaminated CodeForces rows, normally split 8005/128. The baseline masks prompt labels and optimizes assistant-completion next-token cross-entropy with full-parameter HuggingFace Trainer updates for 60 steps. The agent found 362 length-truncated responses and tested a finish_reason=stop filter together with 120 steps. That public v4/v5 confirmation probe scored 15/204 (0.07352941176470588), but the agent correctly noted that filtering and longer training were confounded. The submitted patch kept the objective and model, made the filter optional, changed the formal default to 720 steps, and exported progress 60, 120, and 720.

Four-hour exploration. First, baseline and corpus inspection identified short training and truncated targets as possible bottlenecks; no independent current-protocol baseline run was available. Second, the filter/120-step probe used 7648/123 rows, took 5600.2105 seconds, and reached validation NLL 0.9428384900. Its public score was a diagnostic only: the JSON existed but no completion receipt accompanied it, and the protocol differs from final. The agent rejected it as evidence of improvement and retained an unfiltered fallback. Third, it encoded the longer schedule and explicit checkpoint callback, passed syntax/startup checks, and submitted without a second independent public control. A prior numbered attempt failed only the GPU gate and produced no scientific artifact. The selected explore run reported agent-explicit submission while idle with 7036 seconds remaining; because a same-trajectory 120-step experiment demonstrably fit in that time, early submission is confirmed.

How the submitted method works. Rows are chat-templated; prompt labels remain masked, and the model minimizes the same completion-only causal-LM loss. The optional finish-reason filter runs before tokenization, but formal replay passed an empty value and therefore used unfiltered 8005/128 data. The patch changes data-selection availability, hyperparameters, schedule, engineering, and checkpoint policy—not the training signal, update rule, architecture, reward, or external inputs. Formal replay starts from the fixed model/data rather than an explore checkpoint; the formal copy has the exact candidate hash verified private digest.

Formal and evaluation evidence. Retraining took 35998.068 of 43200 seconds, stopped_on_wall_clock=false, and ended normally at fixed step 720; validation NLL fell from 0.8791606426 at step 200 to 0.8610178232 at step 720. This leaves substantial budget unused without a scientific stopping condition, so formal-recipe underuse is independently confirmed. Under the current metric livecodebench_v6_pass_at_1_full175 (temperature 0.2, top_p 0.95, max_tokens 2048, 10 samples/problem, all 175 v6 problems), artifact-60 scored 0.1262857142857143 (221/1750, stderr 0.0224735464), artifact-120 scored 0.1251428571428571 (219/1750, stderr 0.0228720805), and artifact-720 scored 0.11828571428571429 (207/1750, stderr 0.0219728790). Artifact-60 is best. All exceed the supplied start reference 0.09657 but remain below shipped-recipe reference 0.12743; the stderr is descriptive over problems, not seed/replay uncertainty, so significance is not established. The proxy result cannot be compared directly with full175.

Audit and takeaway. The score-only v6 test6 inputs were identified from evaluator declarations, then literal searches over the complete raw trajectory including tool results, candidate patch, and experiment commands found no hidden rows, answers, or final values; exposure and use are clean. Fixed model/data, no-network execution, explore-to-formal isolation, one accepted L20D per run, and patch lineage are clean. Stale running fields and the raw 137 process state conflict with completion receipts and are inconclusive only for operational metadata. The agent showed good diagnostic and artifact engineering, but incomplete causal experimentation and two confirmed timing/recipe violations mean the evidence does not establish a gain over the shipped recipe or a causal benefit from filtering.

codex__openr1_code_livecodebench__gpt-5.6-terra__medium - gpt-5.6-terra / Codex / medium

1. Run identity, attempt selection, and lineage

|Field|Verified result| |---|---| |Task|openr1_code_livecodebench: improve a fixed code model and evaluate it on LiveCodeBench.| |Task family|Code-generation model training.| |Evaluated coding-agent model|gpt-5.6-terra, run through the Codex harness at medium reasoning effort. The trainable task model is Qwen2.5-Coder-1.5B-Instruct and is discussed separately below.| |Seeds|Training seed 42; split seed 20260727.| |Exploration budget|14400 seconds; selected attempt elapsed 7405 seconds.| |Formal budget|43200 seconds; retraining receipt reports 35998.068 seconds.| |Primary explore run|exploration attempt 2, started 2026-08-10 20:56:22.992 UTC and ended 22:59:48.200 UTC; 7036 seconds remained at submission.| |Selection|The job manifest has no prior attempt directories. exploration attempt 1 stopped at the GPU idle/ownership gate with infrastructure classification and no usable trajectory; exploration attempt 2 is the only complete scientific explore run.| |Candidate patch|421 lines, 25954 bytes, SHA-256 verified private digest.| |Formal run|formal replay 1; the control receipt says formal_result=succeeded, with retraining, checkpoint validation, and completion receipts.| |Lineage|The formal control receipt names exploration attempt 2 as selected-exploration lineage and reports the same patch bytes and hash.| |Checkpoint validation|Progress 60, 120, and 720 were each loaded with AutoModelForCausalLM and marked valid; each has 1,543,714,304 parameters.| |Final evaluation|The manifest points to artifact-120, artifact-60, and artifact-720 under final-tests-full175; each has both summary.json and .complete.|

The source identity closes at commit pinned private revision and source SHA-256 verified private digest. The clean task checkout contains task.toml (SHA-256 verified private digest) and declaration.py (SHA-256 verified private digest). The lineage is therefore fixed source -> exploration attempt 2 -> candidate.patch -> formal replay 1 -> three full175 artifacts.

There are two operational metadata conflicts. The exploration attempt 2 and formal control files still contain a status value of running even though their terminal classifications, timestamps, exit status, completion markers, outcomes, and artifacts show closure. The explore lifecycle also records a raw agent failed/137 state, while its outer submission exited 0 and the process snapshot places the 137 state during submission. I classify those process fields as inconclusive, not as training or artifact failure. The task source, baseline, patch, formal run, and final receipts are available; the historical evaluator fields that still mention first128 are superseded by the current job-manifest protocol.

Direct evidence: job manifest, explore manifest, submit receipt, lifecycle receipt, explore complete, formal manifest, and formal control status.

2. Reader-facing overview

The task asks the agent to improve code generation from a fixed Qwen2.5-Coder-1.5B-Instruct start and a decontaminated Python CodeForces corpus. The baseline masks prompt tokens and trains only on assistant-completion next-token cross-entropy with full-parameter updates, using a short 60-step run. During exploration the agent found 362 length-truncated responses, about 4.5% of the available corpus, and proposed filtering to finish_reason=stop. Its only performance probe combined that filter with a 120-step run.

That probe scored 15/204 on the public v4/v5 confirmation slice, livecodebench_public_pass_at_1=0.07352941176470588. Because filtering and longer training were confounded, the agent explicitly declined to treat it as evidence against the shipped baseline. The submitted patch kept the same objective and model, added an optional finish-reason filter, changed the formal default to 720 optimizer steps, and exported progress 60, 120, and 720. Formal replay passed an empty filter argument, so it actually trained on the unfiltered 8005/128 split.

Under the current full175 protocol—temperature 0.2, top_p 0.95, max_tokens 2048, ten samples per problem, all 175 v6 problems—the official scores are 0.1262857142857143, 0.1251428571428571, and 0.11828571428571429 for artifacts 60, 120, and 720. All exceed the supplied start reference 0.09657, but all are below the supplied shipped-recipe reference 0.12743. The descriptive stderr is not seed or replay uncertainty, so no significance claim is justified.

The boundary audit found no hidden-asset exposure, external data, network use, or GPU contamination. Process compliance is separate and not clean: the agent explicitly submitted with 7036 seconds left even though its measured 120-step run took 5600.2105 seconds, and the formal recipe normally ended at a fixed 720-step endpoint after using 83.33% of the budget while validation NLL was still falling. Both are confirmed execution-protocol violations, while the model and final score receipts remain valid.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

The direct task sources are instruction.md, task.toml, declaration.py, and assets.lock.yaml. The fixed start is Qwen/Qwen2.5-Coder-1.5B-Instruct. The available training asset is the solutions_py_decontaminated subset of open-r1/codeforces-cots; 8133 rows are mounted and the default split uses 8005 training rows plus 128 validation rows. Exploration can read public v4/v5 rows. Formal retraining mounts only the fixed model, training data, and patch. The v6 test6 rows and execution tests are score-only assets. Network mode is no-network and at most three loadable HuggingFace causal-LM artifacts are accepted.

The proxy evaluator is livecodebench_public_pass_at_1, maximize, on the public v4/v5 confirmation slice of 204 problems with one deterministic generation per problem. The final evaluator used in this report is the job-manifest protocol livecodebench_v6_pass_at_1_full175: maximize, v6-only, 175 problems, ten samples per problem, temperature 0.2, top_p 0.95, and max_tokens 2048. Its summaries report n=175, n_samples=1750, samples_per_problem=10, and descriptive stderr over problems. Proxy and final scores are different protocols and are not directly comparable. The task source and explore instruction still contain the superseded first128/greedy contract; the 2026-08-20 manifest protocol is authoritative here, and the old final-tests directory is excluded.

3.2 How the baseline works

Each fixed row is assigned to train or validation after the seeded split and rendered with the model's chat template. The model receives the prompt and conversation prefix and predicts assistant tokens. Labels for the prefix are set to -100, so only assistant-completion tokens contribute to the next-token causal-LM loss. HuggingFace Trainer then performs bf16, SDPA, gradient-checkpointed, full-parameter optimization with learning rate 1e-5, effective batch 18, 3% warmup, zero weight decay, 0.2 gradient clipping, and cosine-with-min-lr. The baseline run uses 60 optimizer steps and evaluates/saves every 30 steps before exporting a loadable model. The agent's concrete initial diagnosis was short training plus potentially harmful truncated targets; it did not claim that the objective or architecture had to change.

4. Four-hour exploration and decision process

The agent first established the baseline and fixed boundaries, then analyzed row quality, ran one complete filter/longer-training probe, waited for its public execution result, and prepared a longer formal schedule with multiple exports. Repeated monitoring of the same process is grouped below.

U-01 - Baseline length and bottleneck inventory

Motivation and hypothesis. The agent observed that 60 optimizer updates represented only about 57 minutes in the reference timing and reasoned that a formal run needed a longer, checkpointed schedule. It wanted to decide whether data selection or schedule changes were more promising.

Concrete change and experimental setup. It inspected the shipped trainer, prompt masking, fixed split, batch shape, and artifact contract without changing the objective or model.

Observed result. The source defines 8133 available rows, the 8005/128 default split, and a 14400-second exploration budget. There was no independent baseline run under the current full175 protocol; that score is not available.

Agent interpretation. The agent said it would use targeted, interpretable probes while keeping a reproducible fallback.

Report assessment and confounds. This was boundary and baseline work, not a performance comparison; the supplied 0.09657 and 0.12743 references are not measurements from this explore run.

Decision and consequence. It proceeded to inspect response termination quality while retaining the original completion-only recipe as a fallback.