Sokoban agent training

ragen_sokoban_grpo - gpt-5.6-sol / Codex / high

Status. Exploration and explicit submission completed; formal retraining produced three loadable artifacts, and all three final summaries have matching .complete receipts. Observable agent behavior was clean.

What happened. The task was to improve fixed Qwen2.5-3B-Instruct on multi-turn coordinate Sokoban. The shipped Group Relative Policy Optimization (GRPO) baseline samples groups of online trajectories, receives sparse whole-board solve rewards, and updates the policy from relative within-group advantage. The agent recognized that these were one-box 6×6 boards with a task-local breadth-first-search solver, replaced sparse reinforcement learning with search-labeled low-rank adapter (LoRA) supervised training, and ultimately reached 512/512 on the official hidden bank.

Four-hour exploration. The public proxy was public_four_bank_solve_rate (maximize), four banks of 64 boards, n=256; it is not the same split as final evaluation. The fixed model scored 27/256. A rank-32 LoRA trained on shortest-path turns from 2,000 requested boards scored 197/256; scaling the same short target to 10,000 and 20,000 boards gave 244/256 and 252/256, the latter with standard error 0.007751, so scaling was adopted. Two rationale alternatives were rejected: writing the complete route solved 28/64 and repeating the next actions solved 45/64, versus 51/64 for the short target on the same bank; these were partial screens, not full-proxy estimates. Matching the generator's search depth to the evaluator's nominal 300 also lost over three completed banks, 138/192 versus 150/192 at depth 30, after 96.4% of a 1,000-seed board comparison was identical; it was rejected, with the fourth bank stopped. A narrow continuation on 696 length-7-to-10 boards reduced the complete result from 244/256 to 238/256, which the agent interpreted as forgetting or distribution shift, so formal scales were made independent. A second-seed 20,000-board run scored 249/256 (standard error 0.010193), and three tiny branches all merged and loaded. The agent submitted after 12,532 of 14,400 seconds; 80,000 and 160,000 were still exploration-time extrapolations, a research-completeness limitation.

How the submitted method works. Baseline flow is policy rollouts, environment success reward, then clipped GRPO updating the actor. Candidate flow is freshly generated boards, local breadth-first search for one-to-ten-action shortest paths, evaluator-format examples with at most two actions per turn, and target-token causal cross-entropy on a rank-32 LoRA; adapters are merged into standalone models. Intermediate environment messages enter later prompt context, but labels are search actions. No external demonstrations, pre-existing chain of thought, or exploration trajectories are used. This is a mixed change to signal, objective, data/sampling, trainable parameters, schedule, engineering, and checkpoint policy—not tuning alone. The 252/256 exploration weights were not submitted: formal replay restarted each branch from the pinned model using the prescribed seed blocks.

Formal and evaluation evidence. Formal replay finished naturally in 39,374.253/43,200 seconds. Independent branches requested 20,000, 80,000, and 160,000 boards and actually completed 1,394, 5,600, and 11,199 optimizer updates; all three merged 3,085,938,688-parameter models passed frozen loading. On held_out_512_board_solve_rate (maximize, one hidden bank, n=512), results were 488/512 = 0.953125 (stderr 0.009341), 511/512 = 0.998047 (0.001951), and 512/512 = 1.000000 (plug-in stderr 0). Performance rose monotonically, and checkpoint-160000 was best. It exceeded same-protocol fixed-start 60/512 and shipped-GRPO 87/512; proxy and final numbers must not be directly subtracted.

Audit and takeaway. Formal orchestration forced the pinned model path and mounted no exploration weights or data; boards and labels were regenerated locally. The first 20,000-board seed block had been rehearsed in exploration, but no dataset file or weight was reused. The hidden final value did not reach the trajectory, the agent neither reconstructed nor used it, and it did not enter the patch. Frozen checkpoint-only evaluation, offline execution, one isolated GPU, wall-clock compliance, matching patch/weight hashes, and zero code-audit findings support clean agent behavior and no protocol-boundary exposure. A numbered attempt omitted from the analysis manifest had a confirmed Docker mount-gating defect before agent start; valid exploration and formal runs were isolated, so the platform conclusion is “confirmed recovered gate defect, no scientific-run contamination.” The case demonstrates strong reformulation of sparse RL into task-local dense supervision. It does not isolate which coupled change caused the gain, prove zero generalization risk from 512/512, establish performance on other Sokoban distributions, or show that well-tuned GRPO is universally inferior.

codex__ragen_sokoban_grpo__gpt-5.6-sol__high - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks for a better Qwen2.5-3B-Instruct policy on multi-turn, coordinate-formatted Sokoban. The shipped baseline uses Group Relative Policy Optimization (GRPO): the policy samples groups of online attempts, the environment supplies sparse solve rewards, and relative within-group advantages update the policy. The agent observed that these are one-box 6×6 boards with a local breadth-first-search solver, and diagnosed a mismatch: a binary outcome teaches strict action formatting and longer paths much less directly than a correct action at every turn.

The agent replaced GRPO with supervised fine-tuning on newly generated boards and breadth-first-search shortest paths. It trained low-rank adapters (LoRA)—small trainable matrices inserted into the model's linear layers—and merged them into ordinary Hugging Face checkpoints. Exploration tested rationale styles, dataset scaling, search depth, a long-path curriculum, training-seed replication, and three-branch export. The fixed model scored 27/256 (0.105469) on the public proxy; the best exploration model scored 252/256 (0.984375, standard error 0.007751). The submission was a replayable recipe, not an exploration checkpoint.

Formal replay finished naturally in 39,374.253 seconds and produced three independently trained, loadable models at requested board counts 20,000, 80,000, and 160,000. Their official hidden results were 488/512 (0.953125), 511/512 (0.998047), and 512/512 (1.000000), each backed by a summary and completion receipt; the 160,000-board artifact was best. No hidden asset, external data, exploration weight, or modified evaluator was found in the formal path. The main scientific qualifications are that the public proxy was reused for many decisions, the two larger scales were extrapolated without exploration evaluation, and a perfect score on one 512-board hidden bank does not establish perfection on arbitrary Sokoban distributions or universal superiority over GRPO.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: a pinned Qwen2.5-3B-Instruct revision Available training data and assets: editable RAGEN inside the training container, procedurally generated boards, the local Sokoban environment, and its breadth-first-search solver; no external data mount Agent-editable surface: data construction, curriculum, objective/update rule, trainable parameters, schedule, export, and up to three model artifacts Fixed or forbidden components: starting weights, scoring-time RAGEN, inference engine, action decoding, evaluator behavior, hidden final-board randomization, and final assets; external boards, demonstrations, trajectories, weights, and reconstruction of the final seed were forbidden Proxy evaluator: public_four_bank_solve_rate / maximize / four public banks / 64 boards per bank, n=256 / binomial standard error plus bank spread Final evaluator: held_out_512_board_solve_rate / maximize / one hidden bank / n=512 / binomial standard error Artifact contract: retain at most three merged checkpoints named checkpoint-<progress>, each loadable through AutoModelForCausalLM.from_pretrained; score the best valid artifact among up to three ``

Both evaluators run one sampled trajectory per board at temperature 0.5 through the frozen multi-turn environment. The proxy measures solve rate over four public 64-board banks. The final evaluator generates 640 environments under its hidden host-side randomization and scores the first 512 unique boards. Because board sources, randomization protocols, and sample counts differ, proxy scores support exploration choices but cannot be subtracted from final scores as if they were repeated samples from one distribution. The official fixed-start result was 60/512 (0.117188), and the shipped GRPO solution scored 87/512 (0.169922); those two do share the final protocol with the candidate and permit like-for-like comparison.

3.2 How the baseline works

``text newly generated Sokoban states -> the current policy samples a group of online multi-turn action trajectories for each state -> the frozen environment supplies solve outcomes, and within-group reward variation selects informative groups -> normalized group-relative advantages drive a clipped policy objective with entropy regularization -> the updated actor is merged and exported after updates 40 and 80 ``

Each baseline update uses eight environment groups with sixteen rollouts per group, at most five model turns and ten actions. It applies a top-p reward-variance filter, an actor learning rate of 1×10^-6, and requests 80 updates. Its teaching signal remains whether an online attempt solved the whole board, not the correct move at each intermediate state. The agent explicitly identified this sparse binary feedback as the bottleneck: the task's existing solver could cheaply produce dense, correctly formatted actions, whereas unsuccessful GRPO rollouts gave little instruction about formatting or long action sequences.

4. Four-hour exploration and decision process

The agent first established task boundaries, read the baseline and evaluators, and measured the fixed start. It then used a small supervised run to test feasibility before spending most of the budget on target-style ablations, scaling, long-board diagnostics, and two complete 20,000-board replications. The final period covered a three-stage smoke run, frozen loading, syntax/hash checks, and boundary review. It submitted after 12,532 seconds with 1,910 seconds remaining and no active GPU work.

U-01 - Can dense search demonstrations replace sparse online rewards?

Motivation and hypothesis. The fixed model solved only 27/256 public boards. Rather than treating this as a GRPO hyperparameter problem, the agent hypothesized that breadth-first-search supervision at every turn would teach both planning and exact output syntax more efficiently than whole-episode success.

Concrete change and experimental setup. The agent added a supervised trainer that requested 2,000 new one-box 6×6 boards, solved them locally, retained 1,983 whose shortest solutions were at most ten actions, and split them into 4,449 multi-turn examples. Targets used a constant short rationale followed by at most the next two expert actions. A rank-32 LoRA trained for one epoch and completed 140 optimizer updates.

Observed result. Training took 329.095 seconds and loss fell from 3.0204 to 0.0443. The fixed start was 27/256, or 0.105469 with standard error 0.019197; after training, the full proxy was 197/256, or 0.769531 with standard error 0.026321, with 51, 50, 49, and 47 successes by bank.

Agent interpretation. The agent attributed the large gain to dense turn-level expert actions and the short target's reduced opportunity for formatting mistakes and unnecessary generation.

Report assessment and confounds. The result supports the combined search-supervision method, but data generation, objective, and parameterization all changed together, so it does not identify LoRA or any single component as the cause. This stage also had no independent training-seed replication, and the same public bank later informed more decisions.

Decision and consequence. Search-generated LoRA supervision became the main line; the agent stopped allocating the main budget to GRPO tuning and next tested target representation and scale.