On-policy mathematical distillation

opd_math_1p5b - kimi-k3 / Claude Code / max

Status. Exploration, submission, formal retraining, validation of three artifacts, and final evaluation completed; every final summary has its .complete receipt. Observable agent behavior is clean, with no protocol exposure or resource-isolation defect found.

What happened. The task fixes a DeepSeek-R1-Distill-Qwen-1.5B student, 15,285 math prompts, and a teacher. The shipped method samples eight student solutions per step, uses the teacher probability of only each sampled token in a K1 estimate, then applies policy gradient. Diagnosing sparse signal and small batches, the agent submitted direct forward-KL matching to the teacher's top 16 tokens, batch 8, learning rate 1e-5, seeded shuffling, and 50-step saving.

Four-hour exploration. The higher-is-better proxy was math500_pass_at_1 on 500 MATH-500 questions × 4 (n=2,000; question-clustered stderr), not numerically comparable to AIME.

• The fixed student scored 0.8120 ± 0.0146. An earlier unmodified step-40 run scored 0.8435 ± 0.0139, matching the task's 0.8410 ± 0.0138 reference.
• Earlier top-64 forward KL at low rate tied the baseline (0.8430 ± 0.0135). Primary top-16 direct KL, no policy gradient, batch 2, and 5e-6 reached 0.8665 ± 0.0131.
• Raising batch and rate together to 8 and 1e-5 reached 0.8860 ± 0.0122, with 1.6% clipping and stable dynamics. The two knobs were confounded. An earlier alternative retained K1, changed only its rate to 2e-6, and reached 0.8790 ± 0.0126 at step 160 before a 429 prevented patching; it is statistically close, so dense KL was not uniquely established.
• Task-reward mixing, another top-16 run, and a batch-8 K1 comparator overlapped other jobs on the assigned GPU and failed during vLLM memory/sleep transitions. They are inconclusive; the agent switched to sequential execution.
• Raising the training response cap from 7,168 to 12,288 first exceeded the 8,192-token training limit. After correction, entropy remained 1.1–1.35, response lengths and clipping oscillated, and steps took 50–70 seconds. The agent stopped around step 31 without an artifact or proxy, making rejection an engineering-risk decision rather than proof of harm.
• A two-step smoke run resolved the submitted defaults and exported a loadable 1,777,088,000-parameter model. Only run.sh entered the 2,697-byte patch. The submit receipt recorded 1,085 seconds remaining; including post-submit closeout, exploration used 13,648/14,400 seconds.

How the submitted method works. Baseline is student states → teacher score for the sampled token → K1 advantage → policy-gradient update. Candidate is student states → online teacher top-16 distribution → direct truncated forward KL → constant-rate AdamW full-parameter update. Correctness rewards remain off; fixed assets, four rollouts, temperature, and the 7,168-token cap are unchanged. Explore weights were excluded, and formal replay restarted from the fixed student.

Formal and evaluation evidence. The harness injected fixed assets, a 42,600-second trainer wall clock, and 1,200-second reserve. Replay used 41,710.04/43,200 seconds and completed 1,125 of 2,200 requested steps. Twenty-three internal saves occurred, but only three complete states were retained, published, and validated.

| Progress | aime24_25_at32 (maximize; AIME24/25; n=1,920) | Stderr / diagnostic | |---:|---:|---| | 1050 | 0.389583 | 0.052683; 565 clipped | | 1100 | 0.405208 | 0.050944; 525 clipped | | 1125 | 0.400521 | 0.051593; 492 clipped |

Step 1100 is best, but checkpoint differences are below uncertainty. It is +0.153125 above the fixed student's 0.252083; that reference's stderr is not available. It is -0.021875 below shipped 0.427083 ± 0.051615, also below combined uncertainty, so superiority is inconclusive. Its 27.3% AIME clipping shows that low proxy clipping did not transfer.

Audit and takeaway. Explore had no AIME inputs; readable evaluator source exposed protocol and public-source hashes, not questions or answers. No reconstruction, external input, evaluator change, or explore-weight transfer appears. Explore/formal patches are byte-identical, and an authorized commit correction records an unchanged task tree. exploration attempt 7 was blocked by the GPU gate; formal GPU 6 passed three idle samples, while self-contention affected only excluded probes. Other numbered attempts ended on gateway, rate-limit, launcher, or ambiguous failures without another candidate.

The agent showed strong source diagnosis, efficient proxy testing, and reliable packaging. Mixed interventions without multi-seed or long-horizon confirmation are the main limitation: the model improved over the fixed start, but evidence cannot show it beats the shipped recipe or identify the causal change.

claude__opd_math_1p5b__kimi-k3__max - Full English Analysis

1. Run identity, attempt selection, and lineage

The numbered-run audit found more launches than the analysis manifest listed. Two runs under the older 28198eeb0066 root stopped at API-gateway preflight with HTTP 000 and exit 76 and produced no manifest. A subsequent 81c3a96fdfaa-20260815 run exited before trial classification. Under the 20260815r2 root, exploration attempt 1 is the only earlier run with a substantial research record: it produced five valid proxies, then ended on an infrastructure 429 with an empty patch. exploration attempt 2 is authoritatively terminal_review because exit 143 had no attributable termination; attempts 003 and 004 failed on rate limiting/429, and attempts 005 and 006 exited before classification. exploration attempt 7, absent from the analysis manifest, stopped only at the continuous GPU idle/ownership gate with exit 75 and no manifest. exploration attempt 8 produced the submitted patch. Stale top-level running fields do not override each status file's authoritative attempt_status.

Formal dispatch declared source commit 81c3a96f..., whereas the selected explore manifest recorded 9a52d57.... An authorized provenance-correction receipt limits the difference to orchestrator/agent.py and its test, records the identical task tree 0b29e63c... at both commits, and binds the original run, launch, image, and patch hashes. This resolves a bookkeeping drift rather than a task-source change.

2. Reader-facing overview

The task asks the agent to improve a fixed 1.5B mathematical reasoner using only a mounted teacher and 15,285 training problems. The shipped recipe has the student generate its own solutions, compares only each sampled token's teacher and student likelihood through a K1 estimator, and applies a policy-gradient update. With two prompts and four rollouts per prompt, each step uses only eight sequences and a relatively sparse, high-variance teacher signal.

The agent investigated dense teacher-distribution supervision, batch and learning-rate scaling, a higher-learning-rate version of the original K1 method, task-reward mixing, longer training responses, and the reliability of concurrent probes. It ultimately replaced K1 plus policy gradient with direct forward-KL matching to the teacher's top 16 tokens on student-induced states. It also raised prompt batch size from 2 to 8 and learning rate from 1e-6 to 1e-5, enabled seeded shuffling, and changed the save interval from 20 to 50 steps.

In the selected explore run, the fixed student scored 0.8120 ± 0.0146 on the MATH-500 proxy. A 40-step small-batch dense-loss probe scored 0.8665 ± 0.0131, and the selected large-batch probe scored 0.8860 ± 0.0122; all are higher-is-better math500_pass_at_1 measurements over 2,000 generations. Formal replay restarted from the fixed student, completed 1,125 of 2,200 requested steps under the wall clock, and yielded three loadable artifacts. The best AIME24/25 final result was step 1100 at 0.405208 ± 0.050944 (778/1,920). Its point estimate is above the fixed start's 0.252083 but below the shipped recipe's 0.427083 ± 0.051615; the latter difference is much smaller than the combined uncertainty, so the evidence establishes neither superiority nor inferiority to the shipped recipe.

The strongest capability was translating an implementation-level diagnosis into a testable, replayable recipe. The main scientific limitation is that the strong short MATH-500 result did not become a higher AIME point estimate than the shipped recipe, while simultaneous loss, batch, learning-rate, and ordering changes prevent causal attribution. Observable agent behavior is clean; no hidden AIME content exposure or use was found, and the lineage and isolation receipts show no contamination.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: read-only deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B@ad9f0ae... Available training data and assets: 15,285-row zero-overlap DAPO-Math-17k projection, hbx/JustRL-DeepSeek-1.5B teacher, CompassVerifier-3B, and MATH-500 during exploration Agent-editable surface: launcher, training driver, loss, optimizer, data pipeline, and vendored VERL under editable workspace Fixed or forbidden components: formal start, mounted teacher/data, offline execution; no external data/weights, final-set reconstruction, or tuning on AIME inputs Proxy evaluator: math500_pass_at_1 / maximize / all 500 MATH-500 questions × 4 / n=2,000 / question-clustered stderr Final evaluator: aime24_25_at32 / maximize / 60 AIME 2024+2025 questions × 32 / n=1,920 / question-clustered stderr Artifact contract: fresh formal replay from the fixed student; at most three highest-progress complete Hugging Face exports are validated, then the best valid final score is selected ``

Proxy generation permits 12,288 new tokens, while final generation permits 31,744. Both use the same rule grader with CompassVerifier fallback, but they differ in question set, samples per question, and generation ceiling. Their values are not numerically comparable as measurements from one distribution. The reported final stderr is computed across the 60 question-level means; the naive binomial error over 1,920 correlated generations is not the relevant uncertainty.

3.2 How the baseline works

``text training prompts -> the fixed student samples four solutions per prompt at temperature 1.0, capped at 7,168 response tokens -> the fixed teacher scores the sampled token at every student-visited state -> a sampled-token K1 estimate becomes a token advantage for a PPO-style policy-gradient update -> single-GPU FSDP updates and exports all 1,777,088,000 student parameters ``

The baseline uses two prompts per step, hence eight sequences, with constant 1e-6 learning rate. It does not use task-correctness rewards and does not shuffle the data. It requests 2,200 steps but relies on the formal wall clock to stop it. It saves every 20 steps, retains at most three complete trainer states at once, and publishes loadable student weights after training. Student rollout, teacher inference, and full-parameter training alternate on one colocated GPU.

The agent explicitly identified two bottlenecks. K1 consumes only the teacher probability of the token that happened to be sampled even though the path can return a top-token distribution, and eight sequences per step may limit gradient quality and data coverage. It also raised the 7,168-token training ceiling as a possible AIME-tail limitation, but later evidence did not justify adopting a longer ceiling.

4. Four-hour exploration and decision process

The primary run began with source, loss, data, and fixed-student inspection. Its middle phase attempted two candidates concurrently, diagnosed same-GPU interference, then completed sequential 40-step dense-distillation probes and proxy evaluations. The final phase tested a longer response cap, stopped the unstable run, reduced the workspace to one source change, and exercised the exact formal entry path with a two-step smoke run. A transient 429 interrupted the agent session, which resumed in the preserved workspace. The submission receipt recorded 1,085 seconds remaining and no active GPU work when submission began; including subsequent lifecycle closeout, the primary explore phase used 13,648/14,400 seconds.

U-01 - Establish the fixed start and shipped reference

Motivation and hypothesis. The agent needed an actual fixed-student proxy and a short shipped-recipe reference before interpreting modifications.

Concrete change and experimental setup. The primary run evaluated the untouched student. Earlier r2 exploration attempt 1 also evaluated the untouched student and an unmodified step-40 shipped checkpoint. All valid receipts used 500 MATH-500 questions, four samples each, and generation seed 42.

Observed result. The primary fixed student scored 0.8120 ± 0.0146 with 0.1155 clipping. The earlier fixed-student run scored 0.8010 ± 0.0150; its unmodified step-40 checkpoint scored 0.8435 ± 0.0139 with 0.0710 clipping. The task's independent shipped step-40 reference is 0.8410 ± 0.0138, clipping 0.0645.

Agent interpretation. It treated the student-to-step-40 gain as confirmation that the pipeline learned, and the two student measurements as ordinary sampling variation.

Report assessment and confounds. The student estimates differ by 0.011, less than their combined stderr. The primary run did not reproduce the exact shipped step-40 recipe, so its later comparison partly uses the task-provided reference; the earlier direct result supports that reference.

Decision and consequence. The untouched student became a learning sanity check, while 0.8410 became the primary run's short-horizon shipped comparison. Neither was substituted for the final AIME baseline.