On-policy mathematical distillation

opd_math_1p5b - gpt-5.6-luna / Codex CLI / low

Status. Exploration and three proxy evaluations completed, but the agent submitted a zero-byte patch with 9,566 seconds remaining. Formal source scanning rejected it, so formal retraining, artifact validation, and final evaluation never started; the boundary audit is complete.

What happened. The task asks for an improved fixed DeepSeek-R1-Distill-Qwen-1.5B math student. The shipped on-policy distillation baseline has the student sample solutions, the teacher score the sampled tokens, and the negative k1 divergence estimate drive policy-gradient updates to all student weights; no answer-correctness reward is used. The agent did not measure a concrete bottleneck, but considered the estimator, learning rate, and rollout diversity, ultimately retaining unchanged k1 after two tested alternatives scored lower.

Four-hour exploration. All reported proxy results use math500_pass_at_1, a maximize metric over all 500 MATH-500 questions with four samples each, n=2,000, seed 42, and question-clustered stderr. The fresh 40-step k1 anchor exported loadable weights and scored 0.844 +/- 0.0135 with clip_rate 0.0665, consistent with the shipped step-40 reference of 0.8410 +/- 0.0138. An initial evaluation was correctly refused while training held the GPU lock, then succeeded after release.

Changing only k1 to k3 for the same 40 steps reduced the score to 0.816 +/- 0.0148 and raised clip_rate to 0.098; raw-row pairing gives a 0.028 +/- 0.0078 advantage for k1. The agent rejected k3, although one training seed cannot establish 12-hour rank stability. It next disabled policy-gradient weighting while retaining k1, but initialization correctly rejected direct k1 backpropagation because its gradient does not depend on teacher probabilities. The compatible fallback, directly backpropagated forward KL over the teacher's top 16 tokens, reached 0.824 +/- 0.0140 after 20 steps, with clip_rate 0.108 and higher memory use. It was rejected, but this comparison confounds loss, update rule, and step count because no k1 step-20 control or 40-step continuation was evaluated.

The agent then chose not to edit the source. This preserved the best tested method, but learning rate, rollout diversity, repeated seeds, and longer probes remained untested. A full 40-step train-plus-evaluation cycle took about 24 minutes, while roughly 160 minutes remained, so submission violated the explicit instruction to continue when meaningful experiments could still be completed.

How the submitted method works. Intended baseline and candidate flows are identical: fixed prompts lead to student rollouts; the mounted teacher supplies sampled-token probabilities; clamped negative k1 estimates become vanilla policy-gradient advantages; a constant 1e-6 learning rate updates the full student; periodic Hugging Face exports are produced. Thus the training signal, update rule, data, sampling, hyperparameters, schedule, and checkpoint policy did not change. The best exploration weights were not submitted and could not be reused formally. In fact, candidate.patch contained no source change, so there was no eligible candidate.

Formal and evaluation evidence. The upstream source scan marked eligible=false because the candidate was not nonempty, the patch was missing or empty, and a submission-receipt check failed. No matching formal run, queue or correction receipt, checkpoint, summary.json, or .complete receipt exists. The source-derived plan would have restarted from the fixed student for up to 2,200 steps under an injected 43,200-second wall clock with a 1,200-second reserve, saving every 20 steps and retaining three; none of this executed. The missing AIME result is not zero, and proxy scores cannot be compared directly with the 60-question, 32-sample AIME final protocol.

Audit and takeaway. Every probe used the fixed student, teacher, and training table; AIME assets were score-only and were neither exposed nor reconstructed. No external data entered, GPU locking prevented compute overlap, and no GPU process remained at submission. Observable agent behavior is confirmed noncompliant because of explicit early submission and the invalid empty candidate, not hidden-asset use or evaluator tampering; protocol exposure was not found, and platform/scheduling/resource isolation was compliant.

There is a separate task-package defect: prose says both evaluators use verifier fallback, but fast_eval.py is rule-only while final_eval.py alone invokes CompassVerifier. The agent demonstrated efficient controlled ablation and correct diagnosis of an invalid loss, but the evidence supports only that k1 beat these short-run checkpoints under the actual proxy. It cannot establish improvement over the shipped solution or any AIME outcome.

codex__opd_math_1p5b__gpt-5.6-luna__low - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: opd_math_1p5b Task family: on-policy distillation and training-method improvement for a 1.5B math reasoner Model: gpt-5.6-luna Harness: Codex CLI under AI4AI-Bench v1.5 orchestration Reasoning effort: low Seed: the resolved training configuration used 42; all three proxy evaluations also used 42 Exploration budget: 14,400 seconds; recorded explore-phase duration 4,876 seconds Formal budget: 43,200 seconds; unused

Primary explore run: exploration attempt 1. Why this attempt was selected: the job manifest lists only this run, and the configuration's run parent contains no other numbered attempt. Prior full-run attempts and outcomes: none. Patch hash and validation: SHA-256 verified private digest, the empty-file hash. The submission receipt likewise records zero bytes and no changed files. Formal source scanning classified the run as eligible=false. Formal run: not available; no run-specific formal directory, manifest, queue entry, or control job exists. Formal selected-exploration lineage: not available because no formal run was created. Final-evaluator receipts: not available; there is no summary.json and no corresponding .complete receipt.

Evidence completeness: the raw session, task definition, baseline source, training metrics and full proxy results for three exploration checkpoints, candidate patch, exploration lifecycle receipts, and upstream formal source scan are available. Formal replay, formal artifact validation, and final evaluation do not exist. Missing or conflicting evidence: the top-level manifest marks exploration complete with exit_status=0, while the underlying Codex attempt was killed with status 137 while submit.sh still appeared in progress and agent_state is failed. The submission receipt, lifecycle receipt, authoritative job status, and queue all establish agent_explicit_submit, so this is not a lost submission. The attempt-level control status is stale at running, whereas the job-level status is terminal_behavior; the latter is authoritative. A second detail is inconclusive: the empty patch's hash matches the hash in submit.json, yet the formal source scan still lists “submission receipt does not match candidate patch.” The exact trigger for that third scanner error is unavailable, but the two independent empty-patch errors already establish formal ineligibility.

2. Reader-facing overview

The task asks the agent to improve a fixed DeepSeek-R1-Distill-Qwen-1.5B math student. The shipped baseline uses on-policy distillation (OPD): the student generates solutions, the mounted teacher supplies probabilities on the student's sampled tokens, a k1 divergence estimate becomes a policy-gradient advantage, and all student weights are updated. No answer-correctness reward enters the baseline objective. The agent did not identify a measured, concrete baseline bottleneck; it nominated the divergence estimator, learning rate, and rollout diversity as low-risk axes.

The agent established a 40-step k1 anchor, tested a 40-step k3 estimator, and tried to remove policy-gradient weighting. Direct k1 backpropagation was rejected by the implementation as mathematically invalid, so the agent switched to directly backpropagated forward KL over the teacher's top 16 tokens for 20 steps. On the full MATH-500 proxy, the unchanged k1 checkpoint scored 0.844 +/- 0.0135, k3 scored 0.816 +/- 0.0148, and forward KL scored 0.824 +/- 0.0140. Each is a higher-is-better result over 500 questions with four samples each, or 2,000 generated rows. The agent therefore selected the original method.

The actual candidate.patch was empty. The agent submitted after about 80 minutes with 9,566 seconds, roughly 2 hours 40 minutes, still available. Formal source scanning later rejected the source for a non-nonempty candidate, a missing or empty patch, and a submission-receipt check. No formal replay, formal artifact, validation receipt, or official AIME final score exists; the missing score is not zero.

The trajectory demonstrates controlled comparisons, use of the declared full proxy, and a correct pivot after an invalid loss configuration. Its central failure is that research selection and submission never formed a valid loop: learning rate and rollout diversity were not tested, no training-seed replication was run, and no eligible source candidate was produced. Separately, the task package contains an evaluator-description conflict: its prose says both evaluators use verifier fallback, but fast_eval.py is rule-only while final_eval.py invokes CompassVerifier for rule failures.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact or model: the read-only deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B revision pinned private revision. Available training data and assets: 15,285 rows derived from DAPO-Math-17k after canonical deduplication and AIME-overlap removal; the read-only hbx/JustRL-DeepSeek-1.5B teacher; MATH-500 during exploration; and CompassVerifier plus AIME 2024/2025 only during scoring. Agent-editable surface: the launcher, training driver, losses, optimizer, data pipeline, and vendored verl implementation under the workspace. Fixed or forbidden components: the starting student; the teacher and training-data paths injected for formal replay; the evaluator outside the submitted workspace; offline execution; formal restart from the fixed student; and prohibitions on external models/data or tuning against final AIME questions. Proxy evaluator: math500_pass_at_1, maximize; all 500 MATH-500 questions, four samples per question, n=2,000, 12,288-token generation cap, seed 42. Receipt stderr is clustered by question. The executed source uses only the pinned JustRL rule grader. Final evaluator: aime24_25_at32, maximize; 60 AIME 2024/2025 questions, 32 samples per question, n=1,920, 31,744-token generation cap. It applies the same rule grader first and CompassVerifier to rule failures; stderr is clustered by question. Artifact contract: formal replay must publish a complete, loadable actor/Hugging Face export under run output area At most the three valid checkpoints with greatest numeric progress are accepted, and the best valid final score is official.

The proxy and final scores are not directly comparable: they differ in benchmark, samples per question, token cap, and actual fallback behavior. instruction.md says both use the rule grader “with verifier fallback,” but fast_eval.py calls grade_rows, which copies rule_score directly to final_score; only final_eval.py performs CompassVerifier fallback. This is a confirmed task-description/implementation inconsistency. The three observed proxy values are reported according to the implementation that actually ran.

The package supplies final-protocol references of 0.252083 for the fixed student and 0.427083 +/- 0.051615 for a full shipped-solution replay. Those are references, not results from this trajectory. Its shipped step-40 MATH-500 reference is 0.8410 +/- 0.0138 and must not be compared numerically with the AIME values.

3.2 How the baseline works

The baseline loads math prompts from the fixed training table; the run records show that 15,279 of 15,285 rows survive the 1,024-token prompt filter. For each batch of two prompts, the current student samples four responses per prompt at temperature 1.0, up to 7,168 response tokens. The teacher then evaluates the tokens actually sampled by the student. The k1 per-token estimate is the student/teacher log-probability ratio, clamped to plus or minus 10. The implementation detaches its negative as an advantage and applies a vanilla policy-gradient loss; it does not directly backpropagate k1. No task reward is loaded and no separate critic supplies the objective.

All student parameters are trained through fully sharded data parallelism. The resolved configuration uses a constant 1e-6 learning rate, 0.01 weight decay, gradient clipping at 1.0, token-mean loss, and seed 42. Student rollout, teacher inference, and actor training share one GPU; the teacher sleeps during actor updates to release memory. The shipped formal source requests up to 2,200 steps and 999 epochs, saves every 20 steps, and retains three trainer checkpoints concurrently. Formal orchestration was expected to inject a 43,200-second wall clock and a 1,200-second reserve.

The agent did not state a measured baseline diagnosis. It only described the estimator, learning rate, and rollout diversity as actionable axes, and then tested only the estimator/loss family.

4. Four-hour exploration and decision process

The agent first inspected the launcher, training driver, and distillation implementation. It then sequentially trained and evaluated the baseline, trained and evaluated k3, diagnosed a failed direct-k1 setup, and trained and evaluated a compatible direct forward-KL setup. It ended with source-schedule, export-path, and syntax checks. The phase consumed 4,876 seconds; no replication, learning-rate probe, or rollout-diversity probe followed.

U-01 - Establishing a comparable k1 anchor and execution loop

Motivation and hypothesis. A fresh measurement on the assigned machine was intended to anchor later comparisons and preserve a trustworthy fallback.

Concrete change and experimental setup. There was no source change. Exploration overrides limited the shipped k1 policy-gradient recipe to 40 steps, saved at step 40, and otherwise retained learning rate 1e-6, batch size two, four rollouts per prompt, and seed 42. Evaluation used the complete declared MATH-500 proxy. An initial scoring call was refused with exit code 75 because training still held the GPU lock; the agent waited and reran it after release.

Observed result. Training reached all 40 requested steps in 872.4 internal wall seconds and exported complete Hugging Face weights. Ray/vLLM teardown printed a killed DataLoader worker and EngineDeadError after saving, but the training command returned zero and the evaluator subsequently loaded the artifact. Proxy accuracy was 1,688/2,000, or 0.844, with question-clustered stderr 0.013516. There were 133 length-clipped rows, clip_rate 0.0665, and evaluation took 635.2 seconds.

Agent interpretation. It called the result consistent with the supplied 0.8410 +/- 0.0138 step-40 reference and treated the checkpoint as a trustworthy fallback.

Report assessment and confounds. That interpretation is supported: the 0.003 difference is far below measurement uncertainty. Successful full evaluation establishes exploration-time loadability despite teardown warnings, but it is not formal artifact validation. There is one training seed and one evaluation seed.

Decision and consequence. k1 became the incumbent, and later candidates were expected to use the same 40-step and evaluation conditions.