Language-model preference optimization

dpo_preference_alignment - gpt-5.6-sol / Codex / high

Status. Exploration attempt 2 completed and submitted a nonempty source patch; attempt 1 failed only at the platform storage gate and produced no scientific run. Formal retraining completed 10,000 steps, all three retained artifacts loaded successfully, and all have official final summaries plus completion receipts. Observable agent behavior was clean; the platform had a confirmed but non-contaminating pre-run defect.

What happened. The task starts from a fixed Mistral-7B plus Zephyr supervised-fine-tuning adapter and allows only 61,135 UltraFeedback preference pairs. The baseline uses Direct Preference Optimization (DPO), which raises a chosen response's likelihood advantage over a rejected response relative to a frozen reference, while updating only low-rank adapters (LoRA). The agent found late proxy regression and response-length imbalance, then submitted length-desensitized DPO. Its best formal artifact scored 189/413, above the declared fixed-start reference of 164/413 but below the packaged solution's 210/413.

Four-hour exploration. The common proxy was strict IFEval accuracy on 128 visible items, maximized with descriptive binomial standard error. First, data profiling found 7,387 score-tied pairs, longer chosen responses, and 1,496 prompts dropped because they filled the 1,024-token limit. The agent considered filtering ties, but positive-margin data had stronger length association, so filtering remained disabled and untested. Second, standard DPO completed 772 steps: the midpoint scored 51/128, while the endpoint regressed to 46/128 despite lower loss and negative drift in both chosen and rejected relative rewards. Editing the still-running script caused an exit-127 publication failure; intact weights were manually published and validated, and the endpoint was rejected. Third, LD-DPO gave shared-length tokens weight 1 and unmatched longer-response tails weight 0.5. Its midpoint was 49/128, but its endpoint reached 55/128 (SE 0.043755) and reduced generation-cap hits from the standard endpoint's 11 to 5, so it was adopted, although mean output was only slightly shorter. Fourth, adding a 0.1 chosen-response supervised loss to standard DPO stopped at 381/772 steps under the remaining wall clock and scored 52/128; it was rejected without establishing how a completed run would behave. Finally, the untouched start also scored 55/128, with 19 paired gains and 19 losses versus LD-DPO. This counterexample made improvement over the start inconclusive, but the agent retained LD-DPO based on its trained-standard controls, cap-hit diagnostic, and executability.

How the submitted method works. Baseline and candidate use fixed chosen/rejected labels, current and frozen-reference log probabilities, a beta=0.01 sigmoid DPO loss, and the same rank-128 LoRA parameters. The candidate changes the objective by downweighting only unmatched long-tail tokens to alpha=0.5; it generates no reward, label, synthetic data, or chain of thought. It also changes the formal schedule to 10,000 steps at 5e-7, saves every 3,000 steps, and retains three model-only artifacts. Exploration weights do not cross the boundary: formal applies the hash-matched source patch and trains fresh. Because objective and schedule change together, the formal gain cannot be attributed to LD-DPO alone.

Formal and evaluation evidence. Formal retraining used 40,744 of 43,200 seconds and stopped normally after all requested steps, not at the wall limit. Steps 6,000, 9,000, and 10,000 were loadable. On official ifeval_strict_accuracy_hidden413 (maximize, 413 items), they scored 186/413 = 0.450363 (SE 0.024482), 188/413 = 0.455206 (SE 0.024504), and 189/413 = 0.457627 (SE 0.024515); step 10,000 was best. Full-set accuracy rose modestly, while the visible subset peaked earlier at 61/128 on step 6,000, showing that proxy-only selection would differ. The best result gains 25 items over the fixed-start reference but loses 21 to the packaged solution; neither reference has repeated-seed uncertainty.

Audit and takeaway. Training used the fixed data and start, the patch contains no weights or evaluator changes, no network or extra model was used, and no exploration artifact crossed into formal. The hidden 285 final items did not reach the trajectory, were not reconstructed or used, and did not affect the candidate; the visible-128 overlap is declared protocol design. Selected runs used one GPU with no external active GPU process and remained within wall-clock controls. Thus agent behavior is clean, protocol/evaluation-boundary exposure is none found, and platform compliance has confirmed storage-gating and post-submit exit-127 bookkeeping defects without selected-run contamination. The case demonstrates useful diagnosis-to-implementation and artifact-validation skill, but one seed, a small proxy, an incomplete alternative, and confounded objective/schedule changes prevent claiming that LD-DPO itself beats standard DPO.

codex__dpo_preference_alignment__gpt-5.6-sol__high - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks the agent to improve instruction-following accuracy from a fixed Mistral-7B plus Zephyr supervised-fine-tuning (SFT) adapter start, using only UltraFeedback preference pairs. The baseline applies Direct Preference Optimization (DPO): it contrasts the chosen and rejected response log probabilities under the trainable policy against a frozen reference and trains only low-rank adapters (LoRA) on a quantized backbone. Its observed weakness was not failure to optimize. Rather, proxy accuracy fell late in training, both chosen and rejected relative rewards drifted negative, and the data showed response-length imbalance and truncation.

Within four hours, the agent audited ties and truncation, measured the standard-DPO learning curve, tested length-desensitized DPO (LD-DPO), tested a chosen-response supervised anchor, and evaluated the untouched policy start. The strongest exploration artifact downweighted by 0.5 the unmatched tail tokens of the longer response. At 772 steps it scored 55/128 on the visible proxy, versus 51/128 for standard DPO at step 386 and 46/128 at step 772. The mixed supervised objective stopped at step 381 and scored 52/128, so it was not selected. Crucially, the untouched start also scored 55/128: the proxy did not establish a net accuracy gain over the start. The agent nevertheless submitted LD-DPO because it had fewer generation-cap hits and the task declared a lower full-set reference for the start.

Formal retraining started fresh, completed all 10,000 requested steps, and simultaneously retained loadable adapters at steps 6,000, 9,000, and 10,000. Their official 413-example scores were 186/413, 188/413, and 189/413. The best, step 10,000, scored 0.457627 with descriptive standard error 0.024515. It was 25 items above the declared fixed-start reference of 164/413, but 21 below the packaged solution's 210/413. The strongest defensible conclusion is therefore that this combined recipe improved the fixed start in one formal run, not that it beat the task baseline or that LD-DPO alone caused the gain: the formal recipe also changed learning rate and training duration.

Observable agent behavior was clean. No hidden value from the 285 held-out final items reached the trajectory, was reconstructed or used, or entered the patch; the overlap of the visible 128 items with the final 413 is explicitly part of the protocol. The platform did have one confirmed pre-run storage-gating failure and some submission/status bookkeeping noise, but the selected exploration, formal replay, single-GPU isolation, artifact validation, and final-evaluation lineage show no resulting scientific contamination.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting model/artifact: a fixed Mistral-7B-v0.1 backbone merged with a Zephyr SFT adapter; the same start serves as the frozen training reference. Available training data/assets: 61,135 fixed UltraFeedback train preference pairs, a visible 128-item IFEval proxy, and read-only task/harness source. Agent-editable surface: workspace training entry point and implementation, including objective, schedule, data filtering, and artifact export. Fixed or forbidden: start weights, UltraFeedback assets, proxy/final evaluators, and the hidden 285 final items; no network, external rows, external weights, or exploration weights crossing into formal retraining. Proxy evaluator: ifeval_strict_accuracy_public128 / maximize / visible 128-item split / descriptive binomial standard error; greedy generation capped at 1,280 new tokens. Final evaluator: ifeval_strict_accuracy_hidden413 / maximize / 413 items comprising the same visible 128 plus 285 held-out items / descriptive binomial standard error; the same greedy generation rule. Artifact contract: apply only the source patch and retrain fresh from the fixed start; publish checkpoint-<progress>; accept at most three simultaneously retained artifacts; each must be a loadable parameter-efficient fine-tuning (PEFT) adapter or merged model; official selection is best_valid_of_up_to_3. ``

Both evaluators measure strict satisfaction of IFEval instructions one generated answer at a time. Results on the shared 128 items can be matched item by item because the protocol is the same. The 128-item aggregate and the full 413-item aggregate are different splits and must not be subtracted as though they were identically distributed evaluations. The reported standard errors describe finite-sample binomial uncertainty only; there are no repeated training seeds from which to estimate run-to-run variance.

3.2 How the baseline works

``text An UltraFeedback prompt with one chosen and one rejected response -> the current policy and frozen reference compute token-level log probabilities for both responses -> the dataset's chosen/rejected relation supplies the preference label -> a sigmoid DPO loss with beta=0.01 increases the chosen response's policy advantage over the rejected response -> only rank-128 LoRA weights on seven linear-layer classes of the 4-bit backbone are updated and exported ``

The baseline shuffles all 61,135 pairs with data seed 42, uses maximum sequence length 1,024, microbatch 4 and accumulation 4 for global batch 16, and trains for 772 steps. It uses learning rate 5e-6, 10% warmup, cosine decay, paged 32-bit AdamW, and seed 42. It saves around steps 386 and 772 with a retention limit of three. The reference is the same fixed start with its adapter disabled, rather than a second trainable model. The agent first suspected that 7,387 score-tied pairs, chosen-response length bias, and truncation could create a shortcut; the measured standard-DPO curve then revealed a more immediate problem, as proxy accuracy regressed from the midpoint to the endpoint.

4. Four-hour exploration and decision process

The agent spent roughly the first forty minutes auditing the task, baseline source, and data. It then used about fifty-five minutes for a standard-DPO run and about half an hour for two proxy evaluations. A comparable training window and two evaluations tested LD-DPO, followed by a shorter mixed-objective run and one evaluation. The final minutes covered the untouched-start comparator, formal-recipe construction, patch preflight, and submission. Repeated polling during GPU work is consolidated below.

U-01 - Are tied preferences, response length, and truncation the primary data bottleneck?

Motivation and hypothesis. Chosen responses were often longer. The agent suspected that standard DPO could exploit length rather than instruction quality, and initially considered removing pairs whose scalar scores differed by less than 0.5 because score ties might be noisy.

Concrete change and experimental setup. It directly profiled scores and character lengths in the fixed UltraFeedback pool and added optional minimum-margin filtering plus preprocessing receipts to the training implementation. The submitted default remained 0.0, so the formal candidate still selected all 61,135 pairs. Tokenization diagnostics used the training maximum length of 1,024.

Observed result. Of 61,135 pairs, 7,387 were score ties and the other 53,748 had margin at least 0.5. Chosen responses averaged about 1,295 characters versus 1,121 for rejected responses, and the chosen answer was longer in 33,657 pairs. Preprocessing later showed that 1,496 prompts alone filled the length limit and were dropped by the trainer. Among the remaining 59,639 pairs, 3,959 chosen and 3,380 rejected responses were truncated; mean completion lengths were about 322.5 and 279.8 tokens.

Agent interpretation. The agent first treated ties as low-information samples. It then noticed that positive-margin pairs had an even stronger length association, so deleting ties alone would not address the likely mechanism. It redirected effort toward changing the length contribution to the objective.

Report assessment and confounders. The audit establishes length imbalance and truncation, but no tie-filtered model was trained. The effect of filtering is therefore unresolved. “Full 61,135-pair pool” describes selection before tokenization; only 59,639 pairs survived the trainer's prompt-length preprocessing.

Decision and impact. Margin filtering was not adopted and remains disabled instrumentation. The length finding directly motivated U-03.