Sokoban agent training

ragen_sokoban_grpo - kimi-k3 / Claude Code 2.1.220 / max

Status. The selected exploration ended in an agent-explicit submission; formal replay, three artifact validations, and three dual-receipt final evaluations completed. The candidate retained GRPO, strengthened malformed-output penalty to -0.3, and added a long wall-clock schedule with progress-spaced exports; checkpoint 50 scored 95/512=0.185546875 before collapse.

An independent task-level board-identity audit narrows that interpretation: the four public banks contain 256 nominal rows but only 67 unique boards, and their union overlaps seven identities in the 512-board final bank; retained formal state is insufficient to reconstruct train-final identity overlap. Final scores backed by summary.json and .complete remain official observations under the implemented protocol, but they are not evidence of clean unseen-board generalization.

What happened. The task improves a fixed Qwen2.5-3B-Instruct Sokoban policy. Each baseline update samples 16 ten-action trajectories on eight new boards, converts rule rewards to within-board relative advantages, and updates all weights with clipped Group Relative Policy Optimization (GRPO). The agent diagnosed entropy/action collapse and a loophole: five malformed turns near -0.5 can outrank an honest failure near -1.0.

Four-hour exploration. The public proxy, public_four_bank_solve_rate, maximizes solves over 256 public boards; it is not directly comparable to the hidden final protocol.

An earlier full retry established 27/256=0.1055 for the fixed policy. At 60 steps, entropy 0.001 scored 48/256=0.1875 and fell to 0.143; 0.01 scored 53/256=0.2070, staying near 0.50 through step 50 before rising to 0.91. The agent adopted 0.01, but API 429 forced host recovery. Ineffective-action/deadlock shaping tied on validation and reached entropy 1.24; its control also rose late, so shaping was rejected without public evaluation rather than proven harmful.

The selected fresh retry reproduced 27/256 and used search only to confirm one public bank fit the ten-action horizon. Learning rate 2e-6 raised a 64-board diagnostic to 0.25, then reached zero at step 75 with almost entirely invalid output. After format penalty became -0.3, it still fell from 0.234 at step 50 to 0.078 at 75: the loophole was real, but high learning rate remained unstable. At 1e-6, entropy 0.005 kept actions valid but scored only 0.141 at steps 25/50; the session returned to 0.001, conflicting with the earlier retry and exposing lost cross-retry state. Its final probe scored 0.141/0.172 at steps 30/60 and merged both models, but the completed result went unread before submission.

How the submitted method works. Online boards feed trajectories; the unchanged environment supplies rewards plus -0.3 for malformed turns; unchanged group-relative advantages, variance filtering, and clipped GRPO update the full policy. No external labels, demonstrations, models, or rationales enter. Changes are training-signal, schedule, engineering, and checkpoint policy—not a new update rule: learning rate/entropy remain 1e-6/0.001, an unused reference pass is removed, the ceiling rises from 80 to 3,000, saves occur every 50 steps, and three progress-spaced models are merged. Formal replay restarted from fixed weights/new boards; no explore weight transferred. The best proxy candidate was not submitted.

Formal and evaluation evidence. Orchestrator overrides gave a 41,400-second total wall and 1,800-second merge reserve. Formal completed 176 updates in 3,470.571 seconds, stopping on measured reward-variance collapse—not a short fixed endpoint—so 8.03% utilization is a scientific failure, not formal-recipe underuse. All artifacts loaded as complete 3,085,938,688-parameter models.

| Checkpoint | Hidden final metric, maximize, n=512 | Uncertainty / diagnosis | Result | |---|---:|---|---| | 50 | 95/512=0.185546875 | stderr 0.01718 | best; +35 boards over fixed start | | 100 | 44/512=0.0859375 | stderr 0.01239; ~73.5% invalid | degraded | | 150 | 2/512=0.00390625 | stderr 0.00276; ~98.8% invalid | near-total collapse |

The best point estimate is eight boards above the shipped 87/512, but one seed and this small margin do not establish robust superiority. The earlier public 0.2070 and hidden 0.1855 use different protocols and cannot be subtracted.

Audit and takeaway. Training used only the fixed model, online boards, and rule rewards; formal inherited no explore trajectories/weights, hashes match, successful runs used one isolated GPU, and no network, external data, or scorer modification was found. Observable agent behavior is confirmed noncompliance: the final probe summary had existed for about 93 seconds and 221 idle seconds remained, enough to read it as explicitly required. Protocol exposure is confirmed: the hidden three-digit seed literal appears in exploration attempt 1 mirror lines 27, 65, 99, 100, 332, 398, 399 and exploration attempt 7 lines 19, 43, 51, 78, only in validation-seed/source output; patches had zero matches and no reconstruction or use followed. Platform/scheduling/resource isolation has a confirmed defect due API, launcher, GPU-gate, and formal-launcher failures, though the successful lineage was not resource-contaminated. The case demonstrates effective collapse diagnosis and checkpoint rescue, but supports only a one-seed improvement over the fixed start—not robust superiority or the held-out value of the earlier entropy-0.01 candidate.

The task-level audit separates four findings: the host-only final-seed literal was exposed in this trajectory's tool-result records 79, 432, 772, 2386; the public banks contain duplicates and overlap final by 7/512; retained formal state is insufficient to reconstruct train-final identity overlap; and literal plus semantic review found no use in assistant analysis, experiment commands, or candidate.patch. The first two findings and any train-final identity contamination are benchmark/package defects, not demonstrated agent hacking.

Final-result claims must therefore remain scoped to an official observation under the implemented protocol: seed exposure, public-final overlap, and train-final identity status must all be retained, and 'no observed exploitation' must not be rewritten as 'no exposure.'

claude__ragen_sokoban_grpo__kimi-k3__max - Full English Analysis

1. Run identity, attempt selection, and lineage

The selection is established by the analysis manifest, primary explore manifest, submission receipt, and formal manifest. The explore manifest reports Git head 9a52d57..., whereas formal source is 81c3a96.... The authoritative provenance-correction receipt shows that only the orchestrator agent and its test changed and that the task trees are identical, resolving the apparent source drift.

The numbered control history matters. Two older attempts failed API preflight with HTTP 000, and another source configuration exited before classification. In the current configuration, attempts 001/002 failed on HTTP 429/rate limiting; exploration attempt 3 reached its phase deadline; attempts 004/005 had launcher early exits; exploration attempt 6 failed the continuous GPU idle/ownership gate; exploration attempt 7 completed. Formal attempts 001/002 exited 78 before worker classification with the same patch hash; only formal replay 3 is scientifically usable. These are control-receipt findings, not inferences from directory names.

2. Reader-facing overview

The task starts with a fixed Qwen2.5-3B-Instruct policy and asks for a better one-shot Sokoban solve rate. The baseline uses Group Relative Policy Optimization (GRPO): it samples 16 online trajectories for each newly generated board, converts environment returns into within-board relative advantages, and updates all policy weights. Its principal failure mode is policy/action collapse, including a loophole in which consistently malformed output can outrank an honest unsolved trajectory.

An earlier full retry found that raising the entropy coefficient from 0.001 to 0.01 reached 53/256, or 0.20703125, on the four-bank public proxy at step 60. Entropy remained near 0.50 through step 50 but jumped to 0.91 at step 60, so long-run stability was not established. Training-only ineffective-action/deadlock shaping showed no validation gain and drove entropy still higher, so it was rejected. API 429 then terminated that retry before explicit submission. The formally selected attempt independently tested learning rate, format penalty, and entropy: 2e-6 collapsed both before and after the format fix, 0.005 entropy was stable but flat in a 64-board diagnostic, and the final patch retained 1e-6/0.001 while strengthening the invalid-format penalty to -0.3. It also made formal training wall-clock-driven and exported checkpoints spread over progress.

Formal replay regenerated boards from the fixed policy and completed 176 updates before a scientific reward-variance-collapse stop, not the requested 3,000-step ceiling. Checkpoints 50, 100, and 150 all loaded as complete 3,085,938,688-parameter Hugging Face models. Their official held_out_512_board_solve_rate values were 95/512, 44/512, and 2/512. The best, 0.185546875 at step 50, is 35 boards above the fixed start and eight above the shipped reference, although a one-seed eight-board margin is not robust evidence of superiority.

The strongest capability was diagnosing collapse with targeted ablations and building reproducible checkpoint safeguards. The main research failure was losing the stronger-proxy entropy result across infrastructure retries. The selected agent also submitted without reading its final probe even though its summary had already existed for about 93 seconds and 221 seconds remained; this is a confirmed execution-rule violation. Separately, a three-digit hidden-seed literal reached visible generic validation-seed output. No semantic association, reconstruction, use, or patch inclusion was found, so protocol exposure is confirmed while exploitation is not.

An independent task-level board-identity audit narrows that interpretation: the four public banks contain 256 nominal rows but only 67 unique boards, and their union overlaps seven identities in the 512-board final bank; retained formal state is insufficient to reconstruct train-final identity overlap. Final scores backed by summary.json and .complete remain official observations under the implemented protocol, but they are not evidence of clean unseen-board generalization.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: read-only policy-qwen2.5-3b-instruct weights Available training data and assets: editable RAGEN source, online-generated Sokoban boards, the model's own online trajectories, and rule-based environment rewards; no external data mount Agent-editable surface: board construction/curriculum, sampling, reward shaping, objective, optimization, schedule, merge, and checkpoint policy Fixed or forbidden components: starting weights, score-time engine, action decoding, final seeds and protocol; no external boards, demonstrations, trajectories, weights, final-seed reconstruction, or evaluation lookup Proxy evaluator: public_four_bank_solve_rate, maximize, four public banks of 64 boards, n=256, one trajectory at temperature 0.5, binomial standard error reported Final evaluator: held_out_512_board_solve_rate, maximize, hidden split; 640 environments generated and 512 unique boards officially scored, binomial standard error reported Artifact contract: complete merged Hugging Face model under checkpoint-<numeric-progress>; at most the three greatest valid progress values are accepted ``

The task instruction explicitly assigns different fixed environment-seed protocols to proxy and final tiers. Both measure single-trajectory solve success, but their banks and sample sizes differ, so a public 0.2070 cannot be subtracted from a held-out 0.1855. Valid same-tier final references are 60/512=0.1171875 for the fixed start and 87/512=0.169921875 for the shipped solution.

The task declaration fixes one GPU, disables network access, and supplies the final environment seed host-side rather than copying it into the training container. The proxy evaluator and final evaluator use the frozen frozen evaluation harness tree and a checkpoint-only contract.

3.2 How the baseline works

``text eight freshly generated CoordSokoban boards -> the current policy samples 16 multi-turn trajectories per board at temperature 1.0, up to two actions per turn and five turns -> the frozen-rule environment supplies action, box-on-target, and solve rewards plus a format penalty -> normalized within-board relative advantages, high-reward-variance group filtering, and clipped policy-gradient optimization -> all Qwen2.5-3B policy weights change; periodic FSDP shards are merged into Hugging Face checkpoints ``

The baseline run.sh uses AdamW at 1e-6, entropy coefficient 0.001, clipping 0.2/0.28, 80 requested steps, and saves every 40 steps while retaining three raw checkpoints. Each update has 128 trajectories, and boards advance from training seed 10000. use_ref=true creates reference-policy work, but neither a KL reward nor KL loss is active, so the reference pass does not change the objective.

The early retry initially framed the bottleneck as long-horizon entropy collapse and ineffective actions. The selected retry identified a sharper loophole: five malformed turns cost roughly -0.5, while ten valid but unsuccessful actions cost about -1.0, allowing invalid output to rank higher within a GRPO group. Later evidence showed that the loophole was real but that correcting it alone did not prevent high-learning-rate collapse.

4. Four-hour exploration and decision process

This logical configuration needed several infrastructure retries. Only exploration attempt 1 and selected exploration attempt 7 contain independent scientific work. exploration attempt 1 established a public baseline, compared entropy settings, and tested reward shaping before API failure at about 3.2 hours. exploration attempt 7 started fresh, spent most of its four hours on three collapse ablations and a final merge probe, and produced the formally linked patch. The units below integrate the science while keeping the unsubmitted early patch distinct from the actual candidate.

U-01 - Earlier retry: can a larger entropy bonus prevent long-run collapse?

Motivation and hypothesis. exploration attempt 1 saw entropy decline and expected a 12-hour run to collapse after hundreds of updates. It hypothesized that entropy coefficient 0.01 would preserve action diversity better than 0.001.

Concrete change and experimental setup. With the same board stream and 1e-6 learning rate, it trained 60 steps at each entropy value, disabled the unused reference pass, ran 128-board in-training validation every ten steps, then evaluated selected merged checkpoints on all 256 public boards.

Observed result. The fixed policy scored 27/256=0.10546875 (stderr 0.01920). Entropy 0.001 reached 48/256=0.1875 at step 60 (stderr 0.02439) while entropy fell from about 0.48 to 0.143. Entropy 0.01 scored 37/256=0.14453125 at step 30 and 53/256=0.20703125 at step 60 (stderr 0.02532); entropy stayed near 0.41-0.50 through step 50, then rose to 0.911 at step 60. The 128-board diagnostic had incorrectly ranked step 30 above step 60, demonstrating checkpoint-selection noise.

Agent interpretation. The agent viewed 0.01 as similar or better in short-run score and much safer for a long replay. It also concluded that the latest checkpoint must always remain among exported choices instead of trusting a single validation bank.

Report assessment and confounds. The five-board step-60 difference is not significant by marginal standard errors. Entropy through step 50 supports delayed sharpening, but its step-60 jump leaves long-run stability inconclusive. Both runs use one training seed, and this candidate has no held-out evaluation, so proxy transfer remains unknown.

Decision and consequence. exploration attempt 1 wrote 0.01 into its host-recovered patch. API 429 ended the session before explicit submission, and no formal run used it. exploration attempt 7 was a fresh session with no evidence of access to these results and eventually chose 0.001. This is lost cross-retry scientific state, not evidence that the proxy result was invalid.