Preference reward modeling

ultrafeedback_bt_rm_rewardbench - claude-opus-5 / Claude Code 2.1.220 / low

Status. Exploration, submission, formal retraining, three artifact validations, and three final evaluations completed; every final summary.json has a matching .complete. Agent behavior was clean, with no protocol exposure or isolation defect found.

What happened. The task trains a scalar reward model from fixed Mistral-7B-Instruct-v0.2 and 8,192 UltraFeedback preference pairs. The baseline applies Bradley–Terry loss to chosen/rejected scores, freezes the backbone, and updates r=128 low-rank adapters (LoRA) plus a scalar head for 252 steps. It targeted length mismatch, random initialization, budget use, and seed volatility. It submitted zero-head, 2e-5, fresh-seed cycles with two endpoints and an exact LoRA average. Official best was 71.635607 ± 0.829178, below reference 74.568936.

Four-hour exploration. All proxy results used fixed 512-row rewardbench_proxy_512, maximize; its 1.8–2.0 descriptive errors are not seed uncertainty. A confounded short probe adopted zero initialization, microbatch 8, accumulation 8, and no activation checkpointing for throughput, not proven quality. The main run scored 68.81/72.45/72.60 at 120/240/360 steps; its 240/360 soup scored 73.21, so neither the third epoch nor soup clearly helped. A matched 240-step rate bracket gave 69.24 at 5e-6, 72.45 at 2e-5, and 70.15 at 3e-5; the low-rate run stopped at 257 after useful checkpoints, and 2e-5 was adopted.

Length-margin decorrelation was rejected: weights 2.0/0.5 scored 68.99/68.46 at 240 steps versus 72.45 without it. The mild penalty paradoxically had highest internal accuracy; its model remained evaluable despite a post-publication shell error from live-editing run.sh. A second seed scored 74.03 versus 72.60, while internal validation reversed their order; cross-seed soup scored 73.52. The agent diversified formal slots but retained this weak selector. Self-soup was nearly identical; heterogeneous four-model soup fell to 71.43. Tiny multicycle smokes, after a save-variable fix, proved publication and r=384 loading, not quality. Submission left 1,921 seconds; treating that as insufficient for a two-seed experiment is a completeness limitation, not an established violation.

How the submitted method works. The last 512 training pairs are internal validation; the first 7,680 train. Each independent cycle restarts fixed Mistral with a new seed and zero head, length-groups examples, and runs 360 Bradley–Terry steps plus 0.001*(r_chosen+r_rejected)^2 centering. Only LoRA and the head change, so the preference signal and core update rule remain. It publishes two validation-ranked endpoints and exactly averages four homogeneous LoRA deltas by rank concatenation. No generated label, external input, exploration weight, or chain-of-thought enters replay.

Formal and evaluation evidence. Orchestration overrode source fallbacks with a 42,600-second wall clock and 600-second end reserve. Formal replay completed 20 independent cycles and 7,200 aggregate steps in 39,763.5 seconds; each candidate itself received 360 steps. It stopped at a cycle boundary because 2,317 inner-window seconds remained versus 3,178 predicted for another cycle. Twenty endpoints plus three final slots were published cumulatively, but retention was three. All accepted artifacts had valid 1×4096 heads. On the full 2,985-row final, maximize: cycle-9 artifact-1001 scored 71.046429 ± 0.825883; cycle-20 artifact-1002 scored 71.635607 ± 0.829178 and was official best; the top-four soup artifact-1003 scored 71.617671 ± 0.809039. All passed zero-overlap checks. The internally top-ranked cycle was final-worst, and soup did not materially beat an endpoint. The 2.933-point deficit to the supplied reference is protocol-valid for this run, but absent baseline seed uncertainty and different hardware prevent a population-level method claim.

Audit and takeaway. Only the pinned base, pair file, and allowed proxy were used; 2,473 hidden rows were neither exposed nor reconstructed. Scorer source revealed only intended protocol logic. The patch has no weights, evaluation rows/labels, or per-row output; aggregate scores in NOTES.md are not read by training. Its explore/formal SHA-256 matches, fresh replay inherited no outputs, and one idle-gated GPU stayed within limits. An authorized commit waiver records an unchanged task tree; terminal receipts resolve stale running fields. The agent showed matched ablation and robust artifact engineering but knowingly retained a misranking selector. Evidence supports a valid 71.635607 run, not causal benefit, superior soup, or baseline improvement.

claude__ultrafeedback_bt_rm_rewardbench__claude-opus-5__low - Full English Analysis

1. Run identity, attempt selection, and lineage

Formal dispatch used an authorized source-commit waiver. The explore launch commit was 305d21c723dc…; replay used e2c7bd967cf6…, with a receipt limiting the difference to an orchestrator transport-retry hotfix and recording task_tree_unchanged=true. The frozen task-source hash, candidate-patch hash, and image digest match, so this was not a method or data change. There was no second formal attempt and no correction receipt for this configuration.

2. Reader-facing overview

The task is to train a scalar reward model from a fixed Mistral-7B-Instruct-v0.2 start and 8,192 decontaminated UltraFeedback preference pairs. The baseline freezes the backbone, trains low-rank adapters (LoRA, the exported incremental weights) and a new scalar head, and applies a Bradley–Terry pairwise logistic loss so the preferred answer scores above the rejected answer. Its concrete limitations were 4,096-token training despite 1,024-token evaluation, a fixed 252-step run that could underuse the 12-hour formal budget, a randomly initialized head, and reliance on one seed.

The agent aligned training to 1,024 tokens, zero-initialized the head, increased the microbatch, and investigated training depth, learning rate, length bias, seed sensitivity, and exact LoRA averaging. The submitted recipe retained the Bradley–Terry signal, enabled a small reward-centering term, used a 2e-5 learning rate, and repeatedly trained independent 360-step cycles from the fixed base. It planned to publish the two cycles ranked highest by an internal UltraFeedback holdout plus an exact average of the top four cycles.

Formal replay completed 20 independent cycles and 7,200 aggregate optimizer steps. All three artifacts loaded and received official final evaluations. The best was artifact-1002 at 71.635607 ± 0.829178 on all 2,985 RewardBench pairs, where higher is better. This is 2.933 points below the task's single-run shipped reference of 74.568936; the reported standard error is descriptive weighted-subset uncertainty, not seed uncertainty.

The trajectory demonstrates good matched ablation, negative-result preservation, and long-run artifact engineering. Its central scientific failure was selection: exploration had already shown that the internal holdout reversed the proxy ranking of two seeds, yet formal cycles were still ranked by that holdout. The internally best formal cycle became the worst of the three final artifacts. Observable agent behavior was clean, and no hidden final rows or labels were exposed; the valid official result simply did not improve the shipped reference.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: mistralai/Mistral-7B-Instruct-v0.2 at revision pinned private revision Available training data and assets: fixed 8,192-pair UltraFeedback projection; exploration additionally sees a fixed 512-pair RewardBench proxy Agent-editable surface: training objective, row selection/sampling, optimizer, schedule, trainable capacity, export, and checkpoint policy under editable workspace Fixed or forbidden components: base/data provenance, final RewardBench, scoring weights, and frozen evaluator; no external preference/evaluation rows, reward checkpoints, evaluator lookup, or train/evaluation overlap Proxy evaluator: rewardbench_proxy_512 / maximize / fixed stratified subset of final / n=512 / weighted-subset binomial descriptive stderr, not seed or paired uncertainty Final evaluator: rewardbench_v1_score / maximize / full RewardBench v1 filtered split / n=2,985 / the same kind of descriptive stderr; 2,473 rows are held out relative to the proxy Artifact contract: full model or compatible parameter-efficient delta on fixed Mistral, including a 1x4096 scalar head; at most the three greatest numeric progress checkpoints are accepted, and the best valid full-final score is official ``

Both evaluators use the same score units and aggregation: example-weighted subset accuracy within each of Chat, Chat Hard, Safety, and Reasoning, followed by an unweighted mean of the four sections. The proxy is a 512-row subset of the 2,985-row final, not an independent sample. Proxy runs are suitable for comparing exploration candidates under that proxy protocol, but proxy and full-final values are not direct training deltas. The final evaluator recomputes both the proxy rows and the remaining 2,473 rows for each formal artifact; their gap is a useful descriptive diagnostic, although its standard error is still not seed uncertainty.

Formal retraining mounts the fixed base and pair file but no RewardBench data. Scoring receives only the frozen artifact and mounts the full final asset offline. It also canonicalizes and checks all 8,192 source pairs for overlap against RewardBench.

3.2 How the baseline works

``text 8,192 two-answer conversations, one preferred and one rejected -> fixed Mistral chat-template tokenization, left-truncated to 4,096 tokens -> one sequence-classification model emits a scalar for each answer; the source preference supplies the chosen>rejected label -> minimize -log sigmoid(r_chosen-r_rejected) with paged_adamw_32bit, cosine decay, and gradient clipping -> freeze the Mistral backbone; update r=128 LoRA on attention/feed-forward projections plus the new scalar head; export a compatible delta ``

The baseline uses all 8,192 pairs, microbatch 1 and accumulation 64 for an effective 64-pair optimizer batch, activation checkpointing, learning rate 5e-6, four warmup steps, and 252 optimizer steps. Trainer checkpoints are saved at steps 126 and 252 with at most three retained, followed by a conventional export. Reward centering exists in the code but defaults to zero, so the main objective constrains only reward differences.

From early source inspection and probes, the agent identified the train/evaluation length mismatch, unstable initial rewards from the random head, and formal-budget underuse from a fixed endpoint. Later measurements added seed and section volatility. It did not run a full, unchanged baseline under the exploration proxy: probe t0 used the baseline trainer but already overrode length, learning rate, batch shape, and step count.

4. Four-hour exploration and decision process

The agent spent roughly the opening ten minutes on source, asset, and evaluator inspection, then used short runs to test initialization and batch shape. The middle of the session produced the main candidate, a low-learning-rate control, and a strong length-decorrelation run. The final hour completed a second seed, a mild length penalty, a high-learning-rate bracket, and end-to-end soup/publication checks. All proxy values below are rewardbench_proxy_512, maximize, n=512; their descriptive standard errors are about 1.79–2.02 and do not quantify seed variation.

U-01 - Aligning sequence length, stabilizing the head, and increasing throughput

Motivation and hypothesis. The agent expected 1,024-token training to remove wasted computation relative to the evaluator, zero head initialization to avoid arbitrary initial margins, and a larger microbatch without activation checkpointing to use the large GPU more efficiently.

Concrete change and experimental setup. Probe t0 retained the baseline trainer but overrode it to 1,024 tokens, 10 steps, 2e-5, microbatch 4, and accumulation 16. Probe t1 used the rewritten trainer, zero head, microbatch 8, accumulation 8, no activation checkpointing, and a 512-pair internal holdout for 12 steps. Both therefore used 64 pairs per optimizer update. Full-pool tokenization showed 558 chosen and 473 rejected sequences truncated at 1,024, with pre-truncation maxima of 4,240 and 4,469.

Observed result. t0 completed 10 steps in 156.32 trainer seconds and 186.13 wall seconds, peaked at 22,881,426,944 bytes, and reported training loss 2.0306. t1 completed 12 steps in 144.57 cycle seconds and 154.46 wall seconds, peaked at 163,057,149,952 bytes, began near the expected ln 2 pair loss, and ended at 0.7031 internal accuracy. Steady-state logs supported the agent's roughly twofold step-throughput claim; the eventual formal run settled near 5.49 seconds per step.

Agent interpretation. It treated the better-conditioned start and higher throughput as justification for zero initialization, evaluator-length training, and large-card batch shaping.

Report assessment and confounds. This was not a single-variable comparison: trainer implementation, initialization, validation overhead, batch shape, and activation checkpointing all changed. t0 was not the shipped default either. The throughput result is credible, but these probes do not establish a quality benefit. Max length, zero initialization, length grouping, and centering were never independently ablated on the final metric.

Decision and consequence. The final recipe adopted 1,024 tokens, zero head, microbatch 8, accumulation 8, and no activation checkpointing on large cards. It added lower-memory shapes and up to two OOM retries as fallbacks; neither fallback was triggered in the actual explore or formal runs.