Sokoban agent training

ragen_sokoban_grpo - gpt-5.6-luna / Codex CLI / xhigh

Status. Exploration completed and was explicitly submitted; formal retraining failed before producing a checkpoint, so artifact validation and final evaluation are not available. Observable agent behavior was clean, no protocol-boundary exposure was found, and resource isolation was compliant.

What happened. The task trains pinned Qwen2.5-3B-Instruct on online-generated, one-box Sokoban boards. The baseline samples sixteen five-turn action trajectories for each of eight boards, forms standardized within-board advantages from native environment returns, filters by reward variance, and uses Group Relative Policy Optimization (GRPO) to update all model weights. The agent submitted the same effective update rule with a long wall-clock schedule and a disabled optional distance reward, but formal policy behavior collapsed before its first scheduled save; there is no official score, not a score of zero.

Four-hour exploration. The common proxy was public_four_bank_solve_rate, maximized over 256 boards with one trajectory each. An 80-update control produced a loadable step-80 model at 61/256 = 0.238281, binomial SE 0.026627; an interrupted wrapper merge was safely recovered from complete shards. Exact shortest-solution-distance shaping at scale 0.2 scored 58/256 at step 40 and fell to 40/256 at step 80 alongside declining action validity, so it was rejected. Changing selection from reward variance to reward scored only 29/256 at step 40; its valid step-80 artifact was not evaluated. Removing filtering while also changing to token-mean loss was restarted after the agent corrected a mistaken first-step throughput extrapolation, then stopped at steps 21–22 when action validity stayed at 0.831; because both mechanisms changed, causality is confounded. A more aggressive filter retained about 32 trajectories but yielded five near-zero policy-gradient updates and was stopped. Halving learning rate to 5e-7 produced valid step-40 and step-80 models at 40/256 and 45/256, so 1e-6 was retained. Raising entropy coefficient to 0.01 produced a loadable step-30 model and higher entropy, but longer responses and noisy validity; only one 64-board bank finished before the deadline, making performance inconclusive. Finally, a one-step smoke test proved train-save-merge-load mechanics, but did not test the actual delayed-save policy.

How the submitted method works. Fresh formal boards and trajectories feed native Sokoban returns into standardized GRPO advantages; reward-variance softmax top-p filtering at 0.9 selects rollouts, and seq-mean-token-mean loss with AdamW, learning rate 1e-6, and entropy coefficient 0.001 updates every Qwen parameter. The patch adds exact distance-delta shaping, but formal scale is zero; explicit board-generation steps 20 matched the unmodified runtime default. Thus effective training signal, sampling distribution, and update rule were unchanged. The material changes were a 1,000,000-step ceiling, a 40,800-second training timeout plus 1,800-second merge reserve, disabled analysis-only diagnostics, and saves every 200 steps with at most three retained. The 61/256 exploration weights were correctly excluded from fresh formal replay.

Formal and evaluation evidence. Formal replay ran about 46 minutes 24 seconds and completed 126 updates; step 127 sampled a batch, then reward-variance early stopping fired before the update. Late batches had zero success, manager_invalid_action of 1, and zero policy-gradient loss on completed updates. Because the first save was step 200, cumulative publications and simultaneous retained checkpoints were both zero; the finalizer found no FSDP state, and the outcome was recipe_invalid. No formal artifact, checkpoint-validation receipt, final summary.json, or .complete exists. The public proxy evidence therefore did not survive formal replay, and it cannot be compared directly with the separate 512-board held-out protocol.

Audit and takeaway. Training used only online boards and contemporaneous rollouts; formal dispatch fixed the starting model and copied only the hash-matched patch, with no exploration weights. The exhaustive trajectory showed no external data, network use, extra model, evaluator modification, or exposure, reconstruction, or use of the hidden final value. Two earlier numbered attempts ended at GPU gating; actual explore and formal runs each used one isolated GPU within wall-clock and concurrency limits. Stale attempt-local running fields are resolved by job-level terminal records and outcome.json. The agent demonstrated disciplined control construction, artifact preservation, comparable evaluation, and early stopping of degraded ideas, but one training seed and a mixed ablation limit scientific inference. Its decisive engineering miss was failing to test early stopping before the first formal save, so the evidence supports only that the control beat completed public-proxy alternatives—not any held-out improvement over the fixed or shipped policy.

codex__ragen_sokoban_grpo__gpt-5.6-luna__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: ragen_sokoban_grpo, improving Qwen2.5-3B-Instruct under a frozen Sokoban evaluation protocol.

Task family: multi-turn policy reinforcement learning from online-generated environment interaction.

Model: the research agent was gpt-5.6-luna; the trained policy was the pinned Qwen2.5-3B-Instruct artifact.

Harness: Codex CLI 0.146.0 under the AI4AI-Bench v1.5 harness.

Reasoning effort: xhigh.

Training and evaluation seeds: exploration and formal training used board seed 10000. Proxy evaluation used four distinct public board banks and inference-engine seed 0.

Exploration budget: 14,400 seconds. The selected run used 14,243 seconds and submitted with 196 seconds remaining.

Formal budget: 43,200 seconds. The candidate reserved 1,800 seconds for model merging, so the training child had a 40,800-second timeout.

Primary explore run: exploration attempt 3.

Why this attempt was selected: it was the only numbered attempt for this configuration that passed resource gating, launched the agent, completed exploration, and produced a nonempty patch. The job-level status and lifecycle receipt both identify an explicit agent submission with successful exploration status.

Prior full-run attempts and outcomes: there was no prior full exploration. Numbered attempts 001 and 002 ended at the continuous GPU idle/ownership gate with terminal_infrastructure and exit status 75. They produced no exploration manifest, experiment, or artifact and are not scientific repeats.

Patch hash / validation: the explore patch and formal copy are both 22,765 bytes with SHA-256 verified private digest. The formal dispatch records the source run, the copy operation, and the same hash.

Formal run: formal replay 1.

selected-exploration lineage: exploration attempt 3.

Final-evaluator receipts: not available. Final evaluation never started, and there is no summary.json paired with a .complete receipt.

Evidence completeness: the raw trajectory, task definition, baseline code, patch, dispatch lineage, formal log, outcome, and control status are directly available. The analysis manifest lists the formal directory as unlinked_or_partial and the failed formal attempt has no standalone run manifest, but the dispatch receipt establishes its lineage through the exact source run and patch hash. The explore manifest's auto_retrain=false and retrain_phase_created=false show only that it did not create formal replay itself; they neither disprove nor explain upstream scheduling, while the direct dispatch proves that a formal attempt occurred.

Missing or conflicting evidence: formal checkpoint validation and final evaluation are wholly absent. Attempt-local control files retain a stale status field of running while their attempt_status is terminal_behavior; the job-level status and lifecycle or outcome records resolve the respective runs as an explicit explore submission and a formal recipe_invalid failure. The explore manifest records raw agent exit 137 because lifecycle capture terminated the CLI while its submit command was still in progress, whereas the submission receipt, lifecycle, and job-level exit status 0 establish that submission succeeded.

2. Reader-facing overview

The task asks the agent to improve a pinned Qwen2.5-3B-Instruct policy on online-generated 6×6, one-box Sokoban boards. The shipped baseline uses multi-turn on-policy Group Relative Policy Optimization (GRPO): it samples a group of trajectories from each board, forms relative advantages from their environment returns, filters for groups with reward variation, and updates all model weights. Exploration is limited to four hours, while formal retraining must start fresh and may use twelve hours; exploration weights and rollouts cannot cross that boundary.

The agent first built an 80-update control whose step-80 checkpoint solved 61/256 public proxy boards, or 0.23828125 with binomial standard error 0.026627. It then tested exact shortest-distance reward shaping, filtering by reward rather than reward variance, no filtering combined with a token-mean loss, more aggressive filtering, half the learning rate, and stronger entropy regularization. Every completed, comparable alternative scored below the control; two short directions ended with behavioral degradation or incomplete evaluation. The submitted source therefore preserved the control's effective reward, GRPO update, learning rate, and filter, while adding a disabled optional shaping mechanism and changing the runtime, diagnostics, and checkpoint schedule for a long formal replay.

Formal replay did start from the fixed policy and newly generated boards, but it completed only 126 policy updates; step 127 sampled a batch and then stopped before the update. A reward-variance early stopper detected collapse: the late batches had zero solve rate, manager_invalid_action equal to 1, and zero policy-gradient loss on the completed updates. Because the first scheduled save was step 200, no FSDP state existed for the finalizer to merge. The orchestrator classified the recipe as recipe_invalid; checkpoint validation and final evaluation are not available and must not be represented as a score of zero.

The audit found no external data, network use, extra model, hidden-final-seed reconstruction, evaluator modification, or explore-to-formal weight reuse. Observable agent behavior is clean, no protocol-boundary exposure was found, and platform resource isolation was compliant. The central failure is instead an artifact-safety gap: a one-step merge smoke test did not exercise the interaction between early stopping and the much later first checkpoint, so all formal progress was lost before scoring.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: a pinned Qwen2.5-3B-Instruct checkpoint, forcibly mounted at the fixed policy path during formal replay.

Available training data and assets: no board dataset is mounted. Boards and policy trajectories are generated online inside the training container. The agent may use the fixed policy, editable RAGEN training source, and the public proxy evaluator.

Agent-editable surface: training-board construction, curriculum, sampling, reward shaping, advantage and objective computation, filtering, optimization, batching, scheduling, and checkpoint merge logic.

Fixed or forbidden components: the policy start, score-time Sokoban engine, action decoding, evaluation behavior, and hidden final seed are fixed. External boards, demonstrations, trajectories, or weights are forbidden, as are final-seed reconstruction and evaluation-specific lookup behavior. Formal scoring is outside the submitted workspace.

Proxy evaluator: public_four_bank_solve_rate, maximized over four distinct public banks of 64 boards each, for 256 boards total and one trajectory per board. It reports the binomial standard error sqrt(p(1-p)private filesystem location) and the per-bank scores.

Final evaluator: held_out_512_board_solve_rate, maximized over the first 512 distinct boards of 640 generated under the hidden protocol, again with one trajectory per board. If a result existed, its standard error would be sqrt(p(1-p)private filesystem location). An official result additionally requires both summary.json and the matching .complete receipt.

Artifact contract: only a merged, Hugging Face-loadable model under run output area>/ is a valid checkpoint; raw FSDP shards do not count. At most the three highest-progress valid checkpoints are accepted, and the best final score among them is the run result.

Proxy and final evaluation share the frozen environment, action semantics, and solve-rate aggregation, but use different seed protocols and sample counts. The public score therefore supports same-tier comparisons only. It cannot be subtracted directly from the task's final references of 60/512 for the fixed start or 87/512 for the shipped solution.

The task package reports the complete final-tier references as 60/512 = 0.117188 for the fixed start and 87/512 = 0.169922 for the shipped solution, a difference of +27/512 = +0.052734. Shipped training took 2,746.19 seconds, final scoring took 339.15 seconds, and final-scoring peak memory was 247,684 MiB. No standard error or confidence interval is provided for either reference solve rate, so this report does not invent one.

3.2 How the baseline works

An online-generated 6×6 one-box board and the full conversation state are passed to the fixed-start policy. The policy may act across five assistant turns, producing at most two actions per turn. Each training batch comprises eight board groups with sixteen stochastic trajectories per board, or 128 rollouts.

The native environment return supplies the learning signal, including step cost and credit for box movement and completion; malformed action text receives a format penalty, while terminal success is also recorded as a metric. Within each board group, GRPO creates relative trajectory advantages and normalizes them by group standard deviation. A softmax top-p filter with threshold 0.9 selects groups by reward variance and may retain zero-valued groups. Assistant-token losses on retained trajectories are reduced by averaging within each sequence and then across sequences.

AdamW uses learning rate 1e-6 and entropy coefficient 0.001 to update all Qwen parameters. A reference policy supplies log-probability diagnostics, while a separate KL loss is disabled. The exploration baseline requests 80 updates, saves at steps 40 and 80, merges the FSDP states into Hugging Face models, and retains no more than three checkpoints.

The agent initially identified two concrete concerns: sparse completion credit and unstable information in the rollout filter, plus the engineering cost of multi-turn collapse diagnostics and checkpoint merging under a short wall clock. It explicitly chose loadable checkpoints and the common four-bank proxy, rather than transient training success, as its selection evidence.

4. Four-hour exploration and decision process

The early phase mapped the training and frozen-evaluation paths and completed a full control. The middle of the budget went to the two largest experiments, distance shaping and an alternative filter metric, including artifact export and proxy evaluation. The late phase used bounded ablations to test filtering strength, loss aggregation, learning rate, and entropy regularization while hardening the formal script and testing its merge path. The agent used about 98.9% of the exploration budget and submitted only after stopping all GPU work.

U-01 - Establishing a same-protocol control and recoverable artifact

Motivation and hypothesis. The agent wanted a direct measurement from the exact container before attributing changes to a new method, and it needed a loadable fallback artifact.

Concrete change and experimental setup. It made no algorithmic change: training seed 10000, eight groups of sixteen trajectories, the standard reward-variance filter, learning rate 1e-6, 80 updates, and saves at steps 40 and 80.

Observed result. Training completed and both FSDP saves were merged by the unchanged finalizer. The step-80 model loaded as a 3,085,938,688-parameter causal language model. It solved 61/256 public boards, score 0.23828125, standard error 0.026627, with 20, 14, 14, and 13 successes across the four banks.

Agent interpretation. The agent treated this as a same-tier anchor and trustworthy exploration fallback, while refusing to select from noisy per-batch training success.

Report assessment and confounders. This is one training seed on 256 public boards and has material sampling uncertainty; it is not comparable to the 512-board final references. Editing run.sh while the long-lived wrapper was still parsing it caused that wrapper to exit before merging. The complete shards were subsequently merged with the unchanged finalizer, so this is a recoverable engineering failure rather than model invalidity.

Decision and impact. The step-80 model became the exploration performance fallback, and its effective hyperparameters became the final candidate defaults.