Sokoban agent training

ragen_sokoban_grpo - gpt-5.6-sol / Codex CLI / xhigh

Status. Exploration, submission, formal retraining, three artifact validations, three final evaluations, and the audit completed. Every official summary has its own .complete receipt; control records confirm the source run and patch hash.

What happened. The task was to improve fixed Qwen2.5-3B-Instruct on frozen multi-turn Sokoban. The baseline generates eight board groups, samples sixteen policy trajectories per group, turns sparse solve rewards into group-relative policy optimization (GRPO) updates, and merges full-policy shards after 80 steps. The agent identified expensive sparse exploration, unstable extension, and underuse of the 12-hour formal budget, then submitted online shortest-path supervision; the best official model solved 281/512 held-out boards.

Four-hour exploration. The public proxy, public_four_bank_solve_rate, maximizes solves on n=256 public boards and reports Bernoulli stderr. Unchanged GRPO rose from 44/256 at step 40 to 60/256 at step 80, then collapsed to 11/256 at step 120; step 119 had zero solves among 128 trajectories and only 1.42 actions per trajectory, so step 160 was stopped. The main alternative regenerated each board and used bundled breadth-first search (BFS) to label at most two shortest-path actions. With the original prompt boundary it rose from 90/256 at step 300 to 107/256 at 1,000 and 124/256 at 3,000; resuming with an optimizer restart reached 144/256 at 6,000, confounding that extension.

A hard-only curriculum requiring paths of at least four actions scored 103/256 versus 107/256 for broad paths; the small difference did not justify switching. Potential-based dense reward passed a 100-board anti-loop test but was never trained or scored, so its performance remains unresolved and it was removed. A full-remaining-path target was canceled after source inspection showed that frozen evaluation consumes only two actions per turn. Finally, frozen inference was found to prefill the opening think tag. Correcting supervision to begin after that tag improved the matched fixed-start 1,000-step result from 107/256 to 134/256. A continuation with an explicit optimizer restart reached 143/256 at 2,000 and the explore-best 148/256 = 0.578125, stderr 0.030866, at 3,000. A real batch-48 update, save, merge, code audit, and frozen load then passed.

How the submitted method works. Baseline flow is board → policy rollouts → sparse solve reward → clipped group-relative objective → full-policy update. Candidate flow is fresh online board → bundled BFS shortest path → online two-action and templated short-rationale labels → continuation-only cross-entropy → rank-64, alpha-128 all-linear low-rank adapters (LoRA), later merged into full models. There were no external boards, demonstrations, stored chain-of-thought, or extra models. This changed the objective, update rule, signal, sampling, trainable parameters, schedule, engineering, and checkpoint policy. Explore weights were excluded; formal replay restarted from the fixed policy.

Formal and evaluation evidence. Orchestration injected 42,600 available seconds, overriding the candidate's 43,200-second fallback. After a 1,800-second merge reserve, training stopped by wall clock at 40,800 seconds and 32,603 completed steps, not the one-million-step safety ceiling; total retrain time was 40,948.54 seconds. Nine cumulative adapter saves were invoked, while retention kept only three simultaneously and exported steps 28,000, 32,000, and 32,603. All loaded as 3,085,938,688-parameter models. Official held_out_512_board_solve_rate results, maximize on 512 hidden boards, were 278/512 = 0.542969 (stderr 0.022015), 275/512 = 0.537109 (0.022036), and 281/512 = 0.548828 (0.021991); the terminal checkpoint was best. Their 3–6-board variation is within uncertainty, so late monotonic improvement is not established. Proxy and final splits differ and are not directly comparable, but the best final exceeds same-tier fixed-start 60/512 and shipped 87/512 references.

Audit and takeaway. Training used only generated boards, the fixed model, and bundled BFS. A host declaration and a build-only scanner contained the final seed, but the scanner was deleted after scanning; runtime evaluation had no default and accepted it only as a host argument. The value reached neither trajectory nor candidate; the agent did not reconstruct or use it, so results were unaffected. Fresh formal replay excluded adapters, temporary reward edits, and public outputs. Frozen hashes and zero-code audits passed; the assigned GPU was idle before dispatch, unshared, and within wall-clock/lock controls. Authoritative job, queue, exit-zero, and completion receipts resolve a stale attempt-level running field. Agent behavior is clean; no protocol exposure was found; platform, scheduling, and isolation were compliant. This demonstrates reframing and token-level alignment, but one seed, optimizer-restarted continuations, and untested dense reward prevent a universal claim.

codex__ragen_sokoban_grpo__gpt-5.6-sol__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: ragen_sokoban_grpo Task family: multi-turn Sokoban policy training, spanning environment reinforcement learning and online supervised fine-tuning. Research-agent model: gpt-5.6-sol. The policy being improved was the fixed Qwen2.5-3B-Instruct revision pinned private revision. Harness: Codex CLI 0.146.0. Reasoning effort: xhigh. Seed, where applicable: formal board generation and optimization used 10000. The final-board seed was supplied host-side and did not enter the exploration trajectory. Exploration budget: 14,400 seconds. Formal budget: 43,200 seconds.

Primary explore run: exploration attempt 1. Why this attempt was selected: the analysis manifest identifies this as the sole primary run and lists no prior complete attempt. Its lifecycle receipt records explicit agent submission, a nonempty patch, no active GPU work at submission, and 362 seconds remaining. The auxiliary state.json calls the run failed because the agent process received exit 137 while submit.sh was in progress; the more specific lifecycle receipt resolves the same event as agent_exit_state: completed with termination_reason: agent_explicit_submit. This was a submit-time harness termination, not a candidate or research failure. Prior full-run attempts and outcomes: none. Patch hash / validation: SHA-256 verified private digest, 51,022 bytes. The explore lifecycle, formal control specification, and formal copy agree. Formal run: formal replay 1. No exploration attempt 2, unlinked formal directory, gate-only reservation, or partial run exists for this configuration. Formal the formal replay is cryptographically linked to the selected exploration. Final-evaluator receipts: all three checkpoints have both out/summary.json and a corresponding .complete receipt; every summary has status: passed.

Evidence completeness: the complete structured Codex session, task source, candidate patch, formal logs and manifests, all three validation receipts, and all three final results were directly available. The flat exploration attempt 1.jsonl mirror is not a consistently clean line-oriented JSON source, so reconstruction uses the complete session JSONL and verifies numerical claims against saved evaluator payloads. Missing or conflicting evidence: formal checkpoints were not run through the public proxy, so their proxy values are not available; scores from exploration weights cannot be reassigned to them. The formal attempt status file retains a stale top-level status: running while also recording ended_at, exit_status: 0, attempt_status: terminal_behavior, and formal_result: succeeded. The authoritative job status, queue row, .retrain.complete, .formal.complete, outcome.json, and checkpoint receipts all establish success. There is no correction receipt for this configuration; the stale field is a non-material bookkeeping conflict.

2. Reader-facing overview

The task asks the agent to improve a fixed Qwen2.5-3B-Instruct policy under a frozen multi-turn Sokoban evaluator. The shipped method uses group relative policy optimization (GRPO): it samples many trajectories on each online-generated board, gets a sparse solved/not-solved reward, and updates the full policy from within-group relative outcomes. It runs only 80 updates and spends 16 trajectories per board. The agent identified sparse exploration, unstable long-run dynamics, and severe underuse of the 12-hour formal budget as concrete limitations.

During four hours, the agent first extended the unchanged GRPO control and observed action collapse by step 120. It then replaced policy rollouts with online shortest-path supervision: fresh boards were solved by the bundled breadth-first search (BFS) routine, and exact next actions became labels. It compared broad and hard curricula, checked whether labels should contain a full path or two actions, found that frozen inference prefilled the opening think tag one token boundary later than the original training examples, and measured a large gain after correcting that boundary. It also tested a dense-progress reward construction for loop safety, but never trained or scored that branch and removed it from the submission.

Formal orchestration injected 42,600 available seconds rather than the candidate's 43,200-second fallback. After a 1,800-second merge reserve, training ran for 40,800 seconds and completed 32,603 updates instead of reaching the declared one-million-step safety ceiling. All three exported models were loadable. The best exploration proxy weight solved 148/256 public boards, public_four_bank_solve_rate 0.578125 with standard error 0.030866; that weight was not reused formally. The best official final result was the terminal formal checkpoint at 32,603: 281/512, held_out_512_board_solve_rate 0.548828 with standard error 0.021991. The proxy and final board protocols differ and these values must not be subtracted as if drawn from one split.

The strongest capability was reframing a sparse-reward RL problem as legal online planning supervision, then tracing performance to exact inference-boundary alignment. The main scientific limitation is that nearly all explore training used one seed, later continuation probes restarted the optimizer, and the dense-reward alternative remained untested. No hidden final value was found in the trajectory, no reconstruction or use was observed, and none entered the candidate. Overall observable agent behavior is clean.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: read-only Qwen2.5-3B-Instruct at the fixed revision. The task supplies same-tier final references of 60/512 = 0.117188 for the fixed start and 87/512 = 0.169922 for the shipped method. Available training data and assets: no mounted board dataset. Boards are generated online by the bundled one-box 6×6 RAGEN Sokoban environment. The editable workspace includes the environment, solver, and training framework. Agent-editable surface: board construction, curriculum, rollout collection, reward shaping, objectives, on- or off-policy updates, batches, optimization, schedule, merge logic, and checkpoint policy. Fixed or forbidden components: starting weights; score-time RAGEN environment, transitions, action decoding, solve reward, final-board protocol, and final seed. External boards, demonstrations, trajectories, or weights are forbidden, as are final-seed reconstruction and evaluation-specific lookup. The container has no network and one GPU. Proxy evaluator: public_four_bank_solve_rate, maximize; four distinct public banks of 64 boards, 256 total, one temperature-0.5 trajectory per board. It reports the Bernoulli standard error over 256 boards. Final evaluator: held_out_512_board_solve_rate, maximize; a host-supplied hidden environment seed generates 640 trajectories, from which the first 512 distinct starting boards in generation order are scored, one temperature-0.5 trajectory each. It reports the Bernoulli standard error over 512 scored boards. Artifact contract: up to three highest-numeric-progress merged, loadable Hugging Face models are accepted. Raw FSDP shards and unmerged adapters are not final artifacts. Each accepted model is scored independently, and the official run result is the best valid score.

Proxy and final evaluation share the frozen environment, action meaning, reward, and sampling temperature, but their board protocols are fixed, disjoint, and declared to have zero overlap. Proxy scores therefore select among exploration candidates only. Final scores may be compared to the fixed-start and shipped final references, not numerically differenced from proxy scores. Final logs also contain RAGEN's mean over all 640 generated trajectories, while the official metric uses the first 512 distinct boards; those denominators must remain separate.

3.2 How the baseline works

One baseline update is:

generate eight online Sokoban board groups → sample 16 multi-turn policy trajectories for each group, with at most ten environment actions and a 400-token response limit → obtain sparse solve rewards from the frozen environment and softmax top-p filter by reward variance → compute a GRPO/PPO-style clipped policy objective from group-normalized advantages plus 0.001 entropy regularization → update the full FSDP policy at learning rate 1e-6 and later merge the shards into loadable weights.

The default starts board generation at seed 10000, runs 80 updates, saves every 40 steps, and allows three retained trainer checkpoints, although the normal run exposes only its two main save points. Each update draws 128 policy trajectories for eight board groups, while useful signal depends on sampled solves. The agent explicitly diagnosed three bottlenecks: the generator already creates solvable boards but sparse reward spends compute rediscovering solutions; 80 fixed steps are far shorter than the formal window; and simply extending constant-learning-rate GRPO could collapse entropy and action length.

4. Four-hour exploration and decision process

The agent spent the first minutes tracing training, reward, generation, merge, and evaluation, then launched an extended unchanged control. Roughly the first 70 minutes established the 40/80/120-step GRPO curve. The next approximately 80 minutes built and scaled online BFS supervision and compared curricula. The final approximately 80 minutes tested longer supervision, target semantics, and prompt-boundary alignment. The last ten-plus minutes were reserved for a real batch-48 update, merge and frozen load check, process and source audits, and submission with 362 seconds remaining.

U-01 - Can simply extending shipped GRPO use the formal budget?

Motivation and hypothesis. Because the shipped method stops at 80 updates, the agent first tested whether more unchanged GRPO updates continued to improve and whether several checkpoints would reveal late instability.

Concrete change and experimental setup. It retained the same reward, update rule, training seed, and proxy, raised the planned ceiling from 80 to 160, and intended to preserve 80/120/160. The run was actually stopped after step 120. Steps 40, 80, and 120 were manually merged and independently evaluated on the same four public banks, n=256.

Observed result. Step 40 scored 44/256 = 0.171875, stderr 0.023579. Step 80 improved to 60/256 = 0.234375, stderr 0.026475. Step 120 fell to 11/256 = 0.042969, stderr 0.012674. Entropy first fell from about 0.475 to 0.146, then oscillated and spiked; gradient norms reached 46.8 and 84.4. At step 119, a 128-trajectory batch had zero solves and only 1.42 executed actions per trajectory, meeting the task's action-collapse stop criterion.

Agent interpretation. Step 80 showed that short extra training could help, but the proxy and action-length evidence at step 120 confirmed collapse. The agent concluded that a 12-hour constant-LR GRPO extension was unsafe.

Report assessment and confounds. The checkpoints share one training path and the same proxy boards, so the within-run reversal is strong evidence. There is only one training seed, and entropy temporarily recovered, making entropy alone an insufficient diagnosis. The combined step-120 score and action statistics are decisive. No step-160 artifact exists because the agent correctly stopped after a stated failure condition; it is missing, not a zero.

Decision and consequence. Step 80 remained the GRPO fallback, but long GRPO was rejected. The result motivated a more direct signal and later justified exporting three late formal checkpoints rather than only the terminal model.