Language-model preference optimization

dpo_preference_alignment - gpt-5.6-terra / Codex CLI 0.146.0 / medium

Status. Exploration ended early after two short trainings; normal submission and the candidate's proxy evaluation were not completed. Host recovery captured a patch, after which formal retraining, validation of three artifacts, three official final evaluations, and the boundary audit all completed.

What happened. The task was to improve IFEval instruction following from a fixed Zephyr/Mistral 7B policy using only pinned UltraFeedback preferences. The shipped baseline compares preferred and rejected response probabilities under the trainable policy and a frozen reference, applies sigmoid Direct Preference Optimization (DPO), and updates only rank-128 Low-Rank Adaptation (LoRA) deltas. The agent submitted margin filtering plus a longer schedule; its best final score was 185/413 = 0.44794, above the fixed start's 164/413 but below the shipped solution's 210/413.

Four-hour exploration. The common proxy was ifeval_strict_accuracy_public128, maximize on a visible 128-row subset; its descriptive binomial error does not make it directly comparable with 413-row final references. First, a full-data audit found 12.0831% equal-score pairs and no reversed margins; margin at least one retained 42,482/61,135 pairs. In a 2,000-example length audit, 9.25% of preferred and 8.75% of rejected responses exceeded 1,024 tokens. The agent adopted margin filtering but did not test thresholds, longer context, or the observed supervised-fine-tuning data. Second, a matched 200-step all-pairs control completed and loaded, scoring 52/128 = 0.40625 (SE 0.04341). The agent called this close to the fixed start and retained it as fallback, but no fixed-start public128 receipt exists, so that cross-protocol interpretation was unsupported. Third, margin-one training completed 200 steps with lower loss, but changed data makes loss non-comparable; its proxy run produced no receipt, while a duplicate was correctly rejected by the output lock. Its performance remained inconclusive, yet it entered the patch, so the only proxy-scored fallback was not the submitted method. Finally, the agent raised the ceiling from 772 to 1,000,000 steps and widened saving from 386 to 2,000 steps to let wall time govern. It did not test that long schedule, inspect the final patch, or submit normally; it exited during evaluation with 11,135 seconds remaining.

How the submitted method works. Baseline flow is pinned preferred/rejected pairs -> policy and frozen-reference log probabilities -> dataset preference label -> beta=0.01 sigmoid DPO -> LoRA updates. The candidate first admits only pairs whose provided chosen-minus-rejected score is at least one, then uses the identical loss, optimizer, frozen reference, and LoRA targets. It creates no labels, synthetic data, external-model output, or reasoning traces. The change is data/sampling, schedule, and checkpoint policy—not a new update rule. Formal replay used only the hash-matched source patch and restarted from the fixed model, not exploration weights.

Formal and evaluation evidence. Replay used 41,463.24 of 43,200 seconds, stopped normally by its wall-clock callback, and completed 10,167/1,000,000 requested steps; all three adapters validated valid. Official ifeval_strict_accuracy_hidden413, maximize on sealed n=413, was 183/413 = 0.44310 at step 8,000 (SE 0.02444), 184/413 = 0.44552 at 10,000 (SE 0.02446), and a best 185/413 = 0.44794 at 10,167 (SE 0.02447). Each summary has .complete. The two-row spread is not a reliable trend. At the endpoint, the final pass's visible subset was 63/128 while held-out was 122/285, a gap 0.06412 with SE 0.05303: a risk signal, not confirmed overfitting. The candidate had no exploration proxy conclusion to “survive.” Critically, retaining 10% warmup with a million requested steps created 100,000 warmup steps, so all formal training remained in warmup and ended at learning rate 5.083e-7 rather than the 5e-6 peak.

Audit and takeaway. Fixed data/model, frozen evaluation, no-network execution, one-GPU isolation, wall time, and patch lineage were intact; no held-out prompt, answer, or per-row outcome reached the exploration trajectory, was reconstructed or used, or entered the candidate. A nested proxy metric-name error is a task-package reporting defect, not exposure. Observable agent behavior is confirmed noncompliance because explicit continuation/closure rules and the early-exit receipt establish a breach; protocol/evaluation-boundary exposure was not found, and platform/scheduling/resource isolation was compliant. The run demonstrates fast data auditing and replayable engineering, but not that margin filtering helps: missing matched proxy evidence and the warmup coupling prevent isolating filtering, duration, learning rate, or random variation.

codex__dpo_preference_alignment__gpt-5.6-terra__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

The selection and lineage are independently supported by the raw explore manifest, explore control status, and formal control status. The explore manifest's auto_retrain=false only means that manifest did not create a formal phase. The analysis manifest links a formal run, so it would be wrong to infer that formal replay was not scheduled.

2. Reader-facing overview

The task asks for better IFEval strict instruction following from a fixed Zephyr/Mistral 7B instruction model, using only a pinned UltraFeedback preference asset. The shipped method applies Direct Preference Optimization (DPO): for a preferred and rejected response to the same prompt, it trains the policy to increase the preferred response's advantage relative to a frozen reference. Only Low-Rank Adaptation (LoRA) deltas are trained and exported. The shipped recipe uses all 61,135 pairs for 772 optimizer steps and has a same-final-protocol reference of 210/413, versus 164/413 for the fixed start.

The agent audited margins and lengths, finding that 12.08% of pairs had equal scores and that roughly 9% of sampled responses exceeded the 1,024-token training limit. It completed a 200-step all-pairs control whose ifeval_strict_accuracy_public128 proxy score was 52/128 = 0.40625 with descriptive binomial standard error 0.04341. It then submitted source that retained only the 42,482 pairs with score margin at least 1, raised the formal step ceiling from 772 to 1,000,000, and changed saving from every 386 to every 2,000 steps. The filtered 200-step training completed, but its matched proxy evaluation did not. The agent did not submit normally: the host captured the workspace after the agent exited with 11,135 seconds remaining.

Formal replay successfully ran for 41,463.24 seconds and completed 10,167 of the requested 1,000,000 steps. All three published adapters loaded. Official ifeval_strict_accuracy_hidden413 results, maximize on 413 sealed rows, were 183, 184, and 185 correct; the best was 185/413 = 0.44794 at step 10,167, with descriptive standard error 0.02447. That is 21 more correct than the fixed start but 25 fewer than the shipped solution. A consequential schedule coupling was not recognized during exploration: retaining a 10% warmup while declaring 1,000,000 steps made warmup 100,000 steps, so the entire formal run remained in warmup. The endpoint learning rate was about 5.083e-7 rather than the intended 5e-6 peak. This is the report author's diagnosis from formal records, not an agent claim.

The fixed data/model, hidden assets, evaluator, network boundary, one-GPU isolation, and explore-to-formal patch lineage show no contamination. No held-out prompt, answer, or per-row outcome was found in the exploration trajectory, reconstructed by the agent, or introduced into the patch. However, the task explicitly required continued meaningful exploration when budget remained, completion or explicit stopping of background work, final patch inspection, and normal submit.sh use. The lifecycle proves early exit during an unfinished evaluation with about 3.1 hours left. Observable agent behavior is therefore confirmed execution-protocol noncompliance, not data or evaluator cheating; the formal artifacts and scores retain valid lineage.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting model or artifact: a fixed merged Mistral-7B-v0.1 plus zephyr-7b-sft-qlora 7B policy start. Available training data and assets: the pinned HuggingFaceH4/ultrafeedback_binarized revision, including 61,135 train_prefs pairs; the exploration container also sees a 128-row IFEval proxy. What the agent may modify: workspace source, including data subsets or weights, preference objectives, reference treatment, trainable capacity, batching, truncation, optimizer, schedule, attention implementation, and checkpoint policy. What is fixed or forbidden: the policy start and UltraFeedback assets are read-only; no external rows or weights, hidden-prompt training, evaluator-specific lookup, network, or additional GPU; formal scoring is frozen outside the submitted workspace. Proxy evaluator: ifeval_strict_accuracy_public128 / maximize / visible 128-row subset / n=128 / descriptive binomial standard error, not seed or paired uncertainty. Final evaluator: ifeval_strict_accuracy_hidden413 / maximize / sealed 413-row final set, comprising the 128 proxy rows and 285 rows hidden during exploration / n=413 / descriptive binomial standard error. Artifact contract: at most the three valid checkpoints with greatest numeric progress are accepted; each must be a complete PEFT adapter or merged causal LM loadable with the fixed base; the official run result is the best valid final score. ``

Both evaluators use the model chat template, greedy generation, up to 1,280 new tokens, and prompt-level strict instruction-following accuracy. The proxy nevertheless covers only 128 rows, while final evaluation adds 285 held-out rows. The task explicitly requires within-protocol comparison. The exploration score 52/128 therefore cannot be directly subtracted from the fixed-start or shipped 413-row references. Final summaries' proxy/held-out decomposition comes from a single final-evaluation pass and can diagnose split behavior, but it still does not measure seed variability.

The contract is direct in the task instruction, task configuration, proxy evaluator, and final evaluator.

3.2 How the baseline works

One baseline training pass has this data flow:

``text prompt, preferred response, and rejected response from pinned UltraFeedback -> current policy and frozen reference each compute conditional log probabilities for both responses -> the dataset's chosen/rejected relation supplies the preference label -> sigmoid DPO compares each response's policy-over-reference advantage and pushes the preferred advantage higher -> only LoRA deltas attached to the fixed 4-bit-quantized backbone are changed and exported ``

The reference is the same fixed policy start with the adapter disabled. The policy trains rank-128, alpha-128, dropout-0.05 LoRA modules on q/k/v/o attention projections and gate/up/down feed-forward projections. The baseline uses sigmoid DPO with beta=0.01, maximum length 1,024, global pair batch 16 (device batch 4 and four-step accumulation), learning rate 5e-6, cosine decay, 10% warmup, seed 42, all 61,135 pairs, and 772 optimizer steps. It saves every 386 steps with a simultaneous trainer retention of three and exports the endpoint adapter; its internal 128-pair diagnostic is not used to select IFEval checkpoints. The shipped reference also records 3,352.75 seconds training time, 2,090.04 seconds final-scoring time, and 19,720 MiB peak training memory.

The agent initially identified two concrete bottlenecks: equal-score pairs may lack a reliable preference direction, and the 772-step endpoint is much shorter than what a 12-hour replay can execute. It also measured approximately 9% response truncation at 1,024 tokens and noticed parallel supervised-fine-tuning data, but did not turn either observation into an experiment. The task's shipped 210/413 reference establishes that the overall baseline is useful, not the causal effect of any individual setting.

4. Four-hour exploration and decision process

Exploration actually lasted about 54 minutes. The first few minutes covered the baseline, data schema, and evaluation boundary. The agent then trained and evaluated a 200-step all-pairs control, trained a matched 200-step filtered model, and started the latter's proxy evaluation. Much of the elapsed time was repetitive polling of background training and scoring. The filtering and long-formal-schedule source changes were already written before the control proxy result arrived. The agent exited before the filtered score, threshold ablations, length changes, supervised alternatives, seed replication, or final patch review.

U-01 - Are preference margin and truncation actionable data bottlenecks?

Motivation and hypothesis. The agent reasoned that pairs assigned equal reward scores but forced into chosen/rejected roles could inject arbitrary DPO directions, while larger score margins might be cleaner. It also checked whether 1,024-token truncation was severe and whether parallel responses supported supervised fine-tuning.

Concrete change and experimental setup. This unit first audited the fixed asset rather than training. Margin counts covered all 61,135 pairs; length statistics used 2,000 examples after deterministic shuffling. The agent selected score_chosen - score_rejected >= 1.0 as the sole tested filtering threshold and kept maximum length 1,024 for a matched comparison.

Observed result. Positive margin accounted for 87.9169% of pairs, equal margin 12.0831%, and negative margin 0%. Thresholds of at least 0.25, 0.5, 1, 1.5, 2, 3, and 4 retained 53,748, 53,748, 42,482, 30,482, 23,493, 14,896, and 10,152 pairs. In the length sample, 9.25% of preferred and 8.75% of rejected responses exceeded 1,024 tokens, with an upper tail beyond 1,600. Supervised and generated-response fields existed, but no training result used them.

Agent interpretation. It called equal/low-margin removal a principled denoising direction and kept length fixed to make the first comparison interpretable.

Report assessment and confounders. The descriptive counts are sound, but margin 1 was not selected by a proxy-backed threshold sweep, and a low margin does not necessarily imply a bad label. The length audit covered only 2,000 examples; no longer-context run measured downstream effect. Merely inspecting supervised data does not constitute an explored method.

Decision and effect. Margin filtering entered the candidate. Maximum length, label source, and DPO objective remained unchanged. Threshold alternatives, truncation mitigation, and supervised fine-tuning remained unresolved.