Sokoban agent training

ragen_sokoban_grpo - gpt-5.6-sol / Codex CLI 0.146.0 / medium

Status. Exploration completed in 13,182 of 14,400 seconds and submitted a nonempty patch; formal replay started but failed at zero updates, so artifact validation and final evaluation are not available. The boundary audit found clean observable agent behavior, no protocol exposure, and compliant platform/resource isolation.

What happened. The task asks for a better fixed Qwen2.5-3B-Instruct policy on frozen Sokoban. The shipped baseline generates boards and multi-turn trajectories online, converts within-group environment rewards into Group Relative Policy Optimization (GRPO) updates, and defaults to 80 updates. The agent instead submitted online exact-solver supervised fine-tuning: the container's breadth-first search (BFS) labels generated boards with shortest actions, and rank-128 low-rank adapters (LoRA) learn those labels. Its best public proxy was 222/256, but the formal timing bug prevented any official result.

Four-hour exploration. The public proxy is public_four_bank_solve_rate, maximized over four public banks of 64 boards each, with one trajectory per board and binomial standard error. It is not directly comparable to the held-out 512-board final tier.

The fixed start scored 27/256 (0.1055, stderr 0.0192). With 3,000 generated boards, rank-32 SFT rose from 133/256 at update 100 to 194/256 at 200, so the method was adopted. On 10,000 boards, one-bank results at updates 250, 500, and 750 were 51, 54, and 53/64; a rank-128 update-1,000 model regressed to 206/256, establishing a 500-update stopping point. Capacity then improved from rank 32 at 206/256 to rank 64 at 213/256 and rank 128 at 221/256. Rank 256 collapsed to 12/64 with mostly invalid actions; lowering its learning rate restored formatting but only reached 48/64, so it was rejected.

Fourfold action-token loss weighting fell to 52/64 and was rejected. Making the rationale state the planned actions reached 222/256 (stderr 0.0212), versus ordinary rationale at 221/256 (stderr 0.0215); the agent correctly treated the one-board edge as diversity, not proof. Seed variance was substantial: a second rank-32 seed scored 191/256, while another rank-128 seed collapsed to 34/64. Teacher-forced loss also ranked a 42/64 model above the known 55/64 model. The agent therefore adopted closed-loop solve rate on newly generated training-validation boards, while acknowledging that a 64-board test could not distinguish those two models. At learning rate 1e-4, the weak seed recovered to 210/256 and the known seed retained 219/256, supporting lower-rate restarts. Syntax, one-update smoke, merge, and representative model-load tests passed; an unrelated full evaluation failed once on Unix socket path length and succeeded after shortening the output path.

How the submitted method works. Baseline flow: generated board → sampled multi-turn policy actions → environment reward → filtered clipped GRPO loss → policy update and merged model. Candidate flow: generated 6×6 one-box board → BFS shortest path, split into two-action dialogue turns → action/rationale labels → assistant-token cross-entropy with AdamW over rank-128 LoRA → adapter merge. This changes the algorithm, training signal, data construction, trainable parameters, schedule, engineering, and checkpoint policy. Explore weights were not submitted. Formal replay was supposed to retrain two measured 500-update, 2e-4 rationale variants from the fixed start, then publish one 1e-4 restart selected on 512 generated validation boards.

Formal and evaluation evidence. The harness supplied a 43,200-second phase budget, forcing a 42,600-second training clock and 1,800-second reserve. The candidate overwrote its first arm's local clock with 1,800 seconds but inherited the same reserve. It generated 10,000 boards and loaded the model, then stopped before update one and raised “training ended before one update.” The authoritative correction classifies this as candidate behavior, not infrastructure. No formal checkpoint, validation receipt, final summary.json, or .complete receipt exists; the official result is unavailable, not zero, and the proxy conclusion did not survive formal replay.

Audit and takeaway. The candidate used only the fixed local model, in-container generator, and BFS; no external boards, demonstrations, weights, network input, hidden-final reconstruction, or evaluator lookup appears. Formal replay copied the hash-matched source patch, not 183 GB of explore outputs, and used one idle-gated, locked GPU. No task-package defect or early-submission breach was evidenced; the remaining 21 minutes are a completeness limit, not misconduct. The case demonstrates strong alternative-method design and counterexample-driven selection, but its decisive weakness is failure to test the exact formal environment variables; the evidence supports a large public-tier gain and serious seed sensitivity, not any held-out improvement.

codex__ragen_sokoban_grpo__gpt-5.6-sol__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

The explore supervisor killed the Codex process while the explicit submission command was in progress, so its one attempt records exit 137. In contrast, .explore.complete, submit.json, and lifecycle.json record an explicit agent submission, a nonempty patch, and no active work at submission. This report therefore treats exploration as completed while preserving the status conflict. The initial formal outcome was needs_review; the authoritative correction subsequently classified it as terminal_behavior, meaning candidate behavior rather than infrastructure caused the failure.

The analysis manifest places the formal directory under unlinked or partial formal evidence because no formal manifest was written. Nonetheless, the control job specification, dispatch receipt, and status record all name the selected-exploration lineage, source commit, and same patch hash; direct comparison confirms that the copied patch is identical. This establishes a specific partial lineage, but not a completed formal lineage with completion receipts.

2. Reader-facing overview

The task asks for a better policy than the fixed Qwen2.5-3B-Instruct start under a frozen Sokoban evaluator. The shipped baseline generates boards and trajectories online, lets the policy act over multiple turns, and uses Group Relative Policy Optimization (GRPO) to turn within-group environment rewards into policy updates. Its default schedule is only 80 updates. The task declaration reports held-out reference results of 60/512 for the fixed start and 87/512 for the shipped solution; those values use the held-out 512-board protocol and cannot be subtracted from exploration scores on public boards.

The agent measured the fixed start at 27/256 on the public four-bank proxy, then treated GRPO as a reference rather than a required objective. It used the in-container Sokoban generator and breadth-first search (BFS) solver to generate shortest-action labels online, and trained low-rank adapters (LoRA) by supervised fine-tuning (SFT). Exploration covered data and update scaling, LoRA capacity, learning rate, action-token weighting, rationale targets, training-seed variance, and formal restart selection. The strongest public results were 221/256 for ordinary rank-128 SFT and 222/256 when the training rationale stated the planned actions. Their one-board difference is far below the roughly 5.4-board standard-error scale and is not evidence that the latter is truly better.

The submitted suite planned two measured 500-update models plus a low-learning-rate restart selected on 512 newly generated training-validation boards. Formal replay did start from the fixed model and generated 10,000 training-board identities. However, the first arm set a local wall clock of 1,800 s while inheriting the formal harness's forced 1,800 s reserve. The training loop therefore stopped before its first update and raised an error. Actual completed updates were zero; no formal checkpoint, checkpoint validation, final summary.json, or .complete receipt exists. The official outcome is unavailable, not a score of zero.

The trajectory demonstrates strong alternative-method design, controlled ablation, and preservation of loadable fallbacks. Its decisive failure is that the final smoke and static checks did not reproduce the formal harness's forced environment variables, so none of the strong proxy evidence survived formal replay or held-out evaluation.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed Qwen2.5-3B-Instruct weights, read-only under fixed-asset mount in formal replay Available training data and assets: no mounted board dataset; boards may be generated online; in-container training source and solver utilities are available Agent-editable surface: training method, board construction, sampling, rewards, objectives, optimization, schedule, merge, and checkpoint logic under editable workspace Fixed or forbidden components: starting weights, frozen score-time Sokoban engine, action decoder, evaluation behavior, and held-out boards; no external boards, demonstrations, trajectories, weights, final-seed reconstruction, or evaluation lookup Proxy evaluator: public_four_bank_solve_rate, maximize; four distinct public banks of 64 boards, n=256, one trajectory per board, binomial stderr Final evaluator: held_out_512_board_solve_rate, maximize; generate 640 environments and score the first 512 distinct boards, one trajectory each, binomial stderr Artifact contract: complete merged and loadable Hugging Face model at run output area progress>; at most the three greatest valid progress values are accepted and the official run score is their best valid final score ``

Proxy and final evaluation share the frozen environment, reward, action parsing, temperature 0.5, and aggregation. Both measure single-trajectory solve rate, but they use different board-generation protocols and sample sizes, so comparisons are valid only within a tier. Explore proxy summaries do not need final-evaluation completion receipts. An official final result requires both its summary.json and corresponding .complete receipt; neither exists here.

3.2 How the baseline works

``text online-generated Sokoban board and current dialogue state -> Qwen samples multi-turn action sequences and executes them in the environment -> the frozen environment returns solve/effectiveness rewards; trajectories for a board group are compared -> GRPO forms group-relative advantages, top-p filters by reward variance, and applies a clipped policy loss with entropy regularization -> policy parameters change; FSDP trainer shards are saved and later merged into Hugging Face weights ``

The baseline defaults to eight environment groups with 16 online rollouts each, learning rate 1e-6, 80 requested updates, a save every 40 updates, and no more than three simultaneously retained trainer checkpoints. Its in-training proxy is disabled by default; finalization merges FSDP state because raw shards are not scoreable. The agent identified two concrete limitations. First, 80 fixed updates consumed only about 46 minutes in the shipped reference and did not naturally use a 12-hour formal budget. Second, the container already had a solver that could certify solvability and produce shortest actions, making lower-variance online supervised learning plausible. The agent did not retrain the shipped GRPO baseline during exploration; it measured only the fixed policy before testing this alternative.

4. Four-hour exploration and decision process

Exploration lasted 13,182 seconds. The opening minutes went to baseline inspection, correction of the fixed-model mount path, and a fixed-start measurement. Roughly the next two hours established exact-solver SFT and swept update progress and adapter capacity. The latter part tested target design and training-seed stability, then spent about the final hour on restart selection, lower learning rates, complete re-evaluation, artifact smoke tests, and the formal suite. Submission occurred with 1,259 seconds remaining.

U-01 - Can exact-solver supervised learning replace GRPO?

Motivation and hypothesis. The training container's generator could produce solvable boards, and its included BFS solver could return a shortest path. The agent hypothesized that this could produce evaluator-format supervision without external demonstrations, avoiding sparse-reward and on-policy variance.

Concrete change and experimental setup. It generated 6×6 one-box boards whose nonempty shortest path was at most ten actions, split each path into two-action turns, and retained state and reward feedback in the dialogue. The first valid probe used 3,000 boards, rank-32 LoRA, learning rate 2e-4, batch size 16, and 200 updates, with merged artifacts at updates 100 and 200. Two earlier fixed-start calls failed because they used the obsolete fixed-asset mount path; the agent corrected this to the harness-exported model path. The SFT prototype also initially lacked the verl Python path, which was fixed before training evidence was collected.

Observed result. The fixed start solved 27/256 public boards, or 0.105469 with stderr 0.019197 and bank counts 8, 7, 6, and 6. SFT at update 100 solved 133/256, or 0.519531 with stderr 0.031226. Update 200 solved 194/256, or 0.757812 with stderr 0.026775 and bank counts 48, 49, 49, and 48. Both training checkpoints were merged artifacts.

Agent interpretation. It called update 100 a strong fallback and interpreted the uniform update-200 gain as evidence that the method should be scaled.

Report assessment and confounds. The gain over the fixed start is much larger than the same-tier binomial uncertainty. It remained a single training seed and was not a direct comparison against a newly trained shipped GRPO run.

Decision and consequence. Exact-solver SFT became the main method. Updates 100 and 200 were preserved as exploration fallbacks, and later work focused on data, progress, and capacity rather than GRPO hyperparameters.