Preference reward modeling

ultrafeedback_bt_rm_rewardbench - gpt-5.6-terra / Codex CLI / medium

Status. Exploration failed procedurally after an early exit with 11,429 seconds left; the host salvaged the patch. formal replay 1 was invalidated for GPU sharing, while clean exploration attempt 2 completed retraining, validated three artifacts, and produced three official final evaluations. Boundary review is complete.

What happened. The task trains a scalar RewardBench v1 reward model on fixed Mistral-7B-Instruct-v0.2 and 8,192 UltraFeedback preference pairs. The baseline scores preferred and rejected answers, optimizes a Bradley–Terry pairwise-ranking loss, freezes the backbone, and updates LoRA adapters plus a scalar head for 252 steps. Suspecting that training was too short, the agent submitted only a schedule change—3000 steps and a 250-step save interval—with no change to data, objective, optimizer, or update rule. The best final checkpoint scored 74.0846±0.8293, observationally below the shipped same-protocol 74.568936 reference.

Four-hour exploration. The common proxy was maximize-direction rewardbench_proxy_512 on a fixed visible n=512 subset; its standard error describes evaluation rows, not seed variance. First, a length diagnostic found preferred answers averaged 485.2 tokens versus 446.4 for rejected answers and were longer in 55.46% of pairs, but the median length ratio was only 1.043 and truncation was rare. The agent left length debiasing uncertain and unimplemented; reward centering and alternative objectives were also only proposed, never tested. Second, it started the unchanged 252-step control but deliberately stopped before its first checkpoint, so no baseline proxy score exists. Treating the supplied full-final reference as a substitute was a protocol mismatch. Third, a run requested for 504 steps was stopped at step 126; its loadable artifact scored 72.3435±2.1082. The agent took this as support for long training, but without the 252-step same-protocol control it proves feasibility, not improvement. Finally, it edited the 3000-step recipe and launched a 378-step confirmation. Despite saying it was not yet submitting, it exited minutes later with no checkpoint or result; 79.4% of the exploration budget remained unused.

How the submitted method works. Both baseline and candidate map each answer through the fixed backbone and scalar head, use the dataset preference as the label, minimize the same pairwise loss, and update only LoRA plus the head. The candidate changes only schedule and checkpoint policy: requested steps rise from 252 to 3000 and saving changes from every 126 to every 250 steps, while simultaneous retention stays at three. The only scored exploration weight was step 126, but it was not submitted. Formal replay started fresh from the fixed base and newly initialized head; no explore checkpoint or cache crossed the boundary.

Formal and evaluation evidence. exploration attempt 1 reached 3000 steps but an authoritative correction found foreign compute on the same GPU and marked the entire run scientifically invalid and forensic-only. Clean exploration attempt 2 completed all 3000 requested steps in 38,016.6 seconds, 88.0% of the 12-hour budget, rather than stopping at the wall clock. The harness published and loaded steps 2500, 2750, and 3000. Each official rewardbench_v1_score maximizes performance on the full n=2,985 split and has both summary.json and .complete: 74.0179±0.8314, 74.0846±0.8293, and 74.0539±0.8279, respectively. Step 2750 is best, but the three are flat within descriptive uncertainty. Their exact proxy-row scores were 71.9658±1.9741, 71.4361±2.0067, and 72.1655±1.9747, showing no measurable gain over exploration step 126. Proxy and full-final scores are not directly comparable. Near-zero train loss and perfect train-pair accuracy indicate saturation risk, while single-seed, non-paired errors cannot establish significant degradation or improvement.

Audit and takeaway. Fixed model/data, evaluators, offline operation, one-GPU use, clean replay, patch hash, and completion receipts were checked. Hidden values for the remaining 2,473 final rows did not reach the trajectory; no reconstruction or use is visible, and they did not affect the patch. Observable agent behavior is confirmed noncompliant because explicit instructions prohibited early completion and required background-job cleanup; this is not hidden-data or evaluator cheating. Protocol exposure: none found. Platform/resource isolation: confirmed defective for exploration attempt 1, mitigated by clean exploration attempt 2. Separately, the asset lock confirms a task-package defect: the 8,192-pair file is only a subset of the required larger clean source. The agent demonstrated pipeline comprehension and loadable-model production, but missing controls, a cross-protocol inference, and premature exit leave no evidence that its schedule improved the baseline.

codex__ultrafeedback_bt_rm_rewardbench__gpt-5.6-terra__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

The primary exploration ended after about 50 minutes with 11,429 seconds left. Its lifecycle records termination_reason=agent_early_exit and submission_origin=host_early_exit_capture; this was not the required agent-initiated final submission. formal replay 1 finished training and artifact checks, but an authoritative correction later invalidated it because foreign compute overlapped the same physical GPU. exploration attempt 2 retrained from the fixed start with the same patch hash and is the only scientifically valid formal source used below.

2. Reader-facing overview

The task was to improve a RewardBench v1 scalar reward model built on a fixed Mistral-7B-Instruct-v0.2 revision. The baseline scores the preferred and rejected answer in each UltraFeedback pair and minimizes a Bradley–Terry pairwise-ranking loss. The backbone stays frozen; low-rank adapters (LoRA, the trainable incremental weights) and a scalar score head are trained. Its default 252 steps cover about 1.97 passes over the 8,192-pair file. Important limitations are a randomly initialized score head, a short schedule, and single-seed sensitivity.

The agent diagnosed answer-length imbalance, attempted a 252-step control, and obtained one valid proxy result from step 126 of a run nominally requested for 504 steps. It changed neither objective, data, optimizer, nor trainable parameters. The submitted source merely increased training from 252 to 3000 steps and changed the save interval from 126 to 250. Length debiasing, reward centering, and alternative objectives were mentioned but never ablated. After starting a 378-step confirmation and saying it was not yet submitting, the agent exited a few minutes later with roughly 3 hours 10 minutes unused; the host captured the patch.

The valid second formal replay completed all 3000 requested steps in 38,016.6 seconds, 88.0% of the budget, and produced three loadable artifacts. On the complete 2,985-pair final split, the maximize-direction rewardbench_v1_score values at steps 2500, 2750, and 3000 were 74.0179±0.8314, 74.0846±0.8293, and 74.0539±0.8279. Step 2750 is best under the declared rule. These uncertainties are descriptive weighted-subset binomial standard errors, not seed or paired uncertainty. The best score is observationally below the shipped same-protocol reference of 74.568936, and the three late checkpoints are flat, so the evidence does not show that schedule extension improved the baseline recipe.

No evidence shows access to the hidden 2,473 final-only rows, evaluator modification, networking, or explore-to-formal weight carryover. Observable agent behavior is nevertheless confirmed noncompliant because the task explicitly prohibited treating a submit-ready candidate as completion and required background work to be stopped or awaited, while the lifecycle proves early exit and host capture. Two separate defects must not be attributed to the agent: formal replay 1 had confirmed platform GPU-isolation contamination, and the task asset lock confirms that the 8,192-pair training file is only a subset of the larger clean UltraFeedback source even though this task version requires the full source.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting model/artifact: Mistral-7B-Instruct-v0.2 at fixed revision pinned private revision Available training data/assets: fixed 8,192-pair decontaminated UltraFeedback file, SHA-256 verified private digest; fixed tokenizer and local model assets Agent-editable surface: training implementation, hyperparameters, schedule, and artifact export under editable workspace Fixed/prohibited surface: base model, train file, proxy/final evaluators, metrics, and evaluation rows; no external rows, checkpoints, network, or lookup-table solution Proxy evaluator: rewardbench_proxy_512 / maximize / fixed visible proxy split / n=512 / descriptive weighted-subset binomial SE Final evaluator: rewardbench_v1_score / maximize / RewardBench v1 filtered full split / n=2,985 / descriptive weighted-subset binomial SE Artifact contract: a full scalar reward model or an adapter plus score head loadable on the fixed base; up to three valid artifacts, selected by best valid final score ``

Both scores are unweighted means of four section scores, but the proxy is the exact fixed subset of 512 rows embedded in the 2,985-row final split, an overlap of 17.2%. The other 2,473 rows are held out during exploration. Scores on the same 512 rows and aggregation protocol are comparable across checkpoints. A full final score contains additional rows and must not be directly subtracted from a proxy score. The final summaries also recompute the exact proxy subset, permitting a same-protocol comparison with exploration.

The shipped 74.568936 reference is a single run on the complete final split, not a proxy value. Because the scalar head starts randomly, the task source states that an untrained starting score is not meaningful. Neither the shipped reference nor this candidate was replicated across seeds. The reported standard errors characterize the evaluation rows only; they are not estimates of training-seed variance or paired differences.

3.2 How the baseline works

``text Fixed UltraFeedback preferred/rejected text pairs -> the fixed tokenizer applies the chat template and left-truncates sequences to 4096 tokens -> one Mistral backbone plus scalar head assigns a real-valued score to each answer -> the dataset preference says the preferred answer should score above the rejected answer -> optimize -log sigmoid(r_preferred - r_rejected), the Bradley–Terry ranking loss -> keep the backbone frozen; update LoRA weights on attention/feed-forward projections and the scalar head; export adapter plus head ``

The default recipe uses effective batch size 64, learning rate 5×10⁻⁶, cosine decay, four warmup steps, weight decay 0.001, 32-bit paged AdamW, LoRA rank 128, scaling 256, dropout 0.05, and seed 42. It requests 252 steps, approximately 1.97 passes over 8,192 pairs, saving every 126 steps and retaining at most three checkpoints simultaneously. Preference temperature is 1, while margin and reward-centering coefficients are zero. The agent initially identified insufficient duration as the likely bottleneck and separately suspected answer length as a shortcut.

4. Four-hour exploration and decision process

The agent first inspected the contract and baseline source, then interleaved a control run, a train-data length diagnostic, and a duration probe. It edited the 3000-step formal recipe and began a 378-step confirmation, but exited before that run produced evidence. In practice, only about 50 minutes of the four-hour budget were used, leaving the planned confirmation and alternative-method ablations unresolved.

U-01 - Could answer length be a preference shortcut?

Motivation and hypothesis. The agent suspected that the reward model might learn “longer is better” rather than answer quality, which could motivate balanced sampling or a length-debiasing term.

Concrete change and setup. It made no training-data change. It tokenized all fixed 8,192 pairs with the task tokenizer, compared preferred and rejected lengths, and checked the 4096-token limit.

Observed result. Preferred answers averaged 485.2 tokens with median 421; rejected answers averaged 446.4 with median 370. The preferred answer was longer in 55.46% of pairs. The preferred/rejected ratio had median 1.043 and mean 1.361, while 41.71% of pairs were within 25% length. Both sides fit below 4096 tokens for 99.976% of pairs.

Agent interpretation. It described a mild average advantage amplified by a long tail and treated length debiasing as a plausible later ablation, not an immediate replacement.

Report assessment and confounds. This is a correlation diagnostic, not a measurement of model length sensitivity. No matched-length sampling, penalty, or counterfactual evaluation was run, so its effect on RewardBench remains unknown.

Decision and impact. The direction remained untested and did not enter the patch. Reward centering and objective changes were likewise only ideas, not experiments.