Language-model preference optimization

dpo_preference_alignment - gpt-5.6-luna / Codex CLI 0.146.0 / xhigh

Status. Four-hour exploration, submission, twelve-hour formal replay, validation of three retained artifacts, final evaluation, and the boundary audit all completed. Every final result has both summary.json and .complete; the step-6,000 artifact is official best.

What happened. The task starts from a fixed Zephyr/Mistral 7B instruction model and 61,135 pinned UltraFeedback preference pairs. The baseline uses Direct Preference Optimization (DPO): relative to the adapter-disabled frozen start, it raises the probability margin of a chosen answer over a rejected answer while updating only a rank-128 low-rank adapter (LoRA). The agent found that its 772 requested steps underused the formal budget and that 7,160 pairs exceeded the 1,024-token limit. It submitted DPO plus weight-0.2 token-level supervision on chosen answers, longer wall-clock training, and staged artifact publication.

Four-hour exploration. The maximize-direction public proxy was ifeval_strict_accuracy_public128, n=128; its binomial standard errors are descriptive, not seed or paired uncertainty. Data profiling found 53,748 pairs with a strictly higher-rated chosen response and 3,171 pairs truncated on both sides. The agent retained all data because filtering would discard substantial signal, but no filtered control was trained, so that decision remains uncertain. Pure DPO completed 772 steps and scored 47/128 (0.3672, SE 0.0426). Its launcher then failed during publication with exit 127; the agent verified intact weights, published them manually, and repaired copy invocation. Adding chosen-response cross-entropy at weight 0.1 raised the endpoint to 53/128; its step-386 artifact scored 51/128. Weight 0.2 reached 57/128 (0.4453, SE 0.0439) after 300 steps and was adopted, although progress and warmup were not perfectly matched and no seed was repeated. As a countertest, pure DPO with beta=0.03 scored 10/32 on only the first 32 prompts, versus 11/32 for the 0.1 mix; the agent rejected it, but this partial result cannot be compared directly with full-128 scores. Alternative losses, capacity changes, and curricula remained untested. For formal readiness it added a 20,000-step ceiling, 200-step warmup, 3,000-step saves, three-slot retention, and wall-clock stopping. A static check accidentally started training and the first patch preview lacked a module path; the agent stopped the run and then replayed the patch successfully. No accidental weight was submitted.

How the submitted method works. The current LoRA policy and frozen reference score each chosen/rejected pair. Pair labels supply the unchanged sigmoid-DPO signal at beta=0.01; chosen completion tokens additionally supply cross-entropy targets, and the summed loss uses weights 1.0 and 0.2. Only LoRA changes. This is an objective and training-signal change plus schedule, checkpoint, and engineering changes; fixed data, reference, optimizer, learning rate, global batch 16, and seed 42 remain. The highest-proxy objective was submitted, and formal replay transferred source only and restarted from fixed weights.

Formal and evaluation evidence. Orchestrator overrides—not source fallbacks—set 42,600 seconds with a 1,200-second reserve. Training requested 20,000 steps and stopped cleanly at step 9,849 after 41,402 seconds, using 95.98% of the formal budget. Saves at 3,000, 6,000, and 9,000 plus the endpoint made four cumulative publications; three were retained, so step 3,000's final score is not available. All retained PEFT artifacts load. On maximize-direction ifeval_strict_accuracy_hidden413, full split n=413, step 6,000 scored 233/413 (0.5642, descriptive SE 0.0244; 25 length clips), step 9,000 scored 227/413 (0.5496, SE 0.0245; 29 clips), and step 9,849 scored 225/413 (0.5448, SE 0.0245; 40 clips). Best formal exceeds the fixed start by 69 correct prompts and the shipped solution by 23 under the same protocol. The mixed recipe succeeded, but later training degraded; without a formal pure-DPO control, the gain cannot be assigned solely to auxiliary supervision.

Audit and takeaway. Training used only fixed policy_start and UltraFeedback; no task-package defect was found. No hidden prompt value reached the trajectory, no reconstruction or use is visible, the evaluator was unchanged, no network/external model appeared, source-only lineage and patch hash match, and each phase used one idle-gated GPU within its wall clock. Observable agent behavior is clean; no protocol/evaluation-boundary exposure was found. Platform scheduling and GPU isolation comply, but a confirmed metadata defect remains: an attempt-level top status says running while its own result, job status, outcome, and completion markers say succeeded. Receipts resolve it. The case demonstrates controlled ablation and reliable replay, but one seed, imperfectly matched probes, no formal pure-DPO control, and loss of step 3,000 prevent a causal or cross-seed claim.

codex__dpo_preference_alignment__gpt-5.6-luna__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task is to improve instruction following from a fixed merged Zephyr/Mistral 7B supervised-fine-tuned policy, using only a pinned UltraFeedback preference dataset. The shipped baseline performs pairwise Direct Preference Optimization (DPO): it makes a trainable low-rank adapter prefer a chosen answer over a rejected one relative to the frozen starting policy. Its 772-step endpoint underuses the twelve-hour formal budget, and 7,160 training pairs exceed its 1,024-token sequence allowance.

The agent established a pure-DPO proxy baseline, profiled truncation and rating consistency, tested chosen-response supervised losses at weights 0.1 and 0.2, and ran a higher-DPO-beta countertest. The decisive exploration result was 57/128 (0.4453) on the public proxy after 300 steps with a 0.2 chosen-response cross-entropy term, versus 47/128 (0.3672) for the 772-step pure-DPO baseline and 51/128 (0.3984) for the 0.1 mix at a nearby step. It submitted that mixed objective together with wall-clock stopping, longer training, and periodic artifact publication.

Formal replay started fresh, requested 20,000 steps, and stopped cleanly at step 9,849 under the orchestrator's effective 41,400-second training threshold. Four progress points were published cumulatively; the three retained artifacts at steps 6,000, 9,000, and 9,849 all validated. Each has an official final summary.json and .complete receipt. On maximize-direction ifeval_strict_accuracy_hidden413, step 6,000 was best at 233/413 (0.5642, descriptive binomial standard error 0.0244), followed by 227/413 and 225/413.

The trajectory demonstrates competent controlled comparison, negative testing, recovery of an artifact-publication failure, and conversion of a short probe into a replayable formal recipe. Its main scientific limitation is causal: there is one seed, no formal long-run pure-DPO control, and the weight-0.2 probe is not perfectly progress-matched to weight 0.1. The evidence therefore supports the complete submitted recipe, not an isolated effect size for the auxiliary loss. Observable agent conduct is clean and no hidden value exposure or use was found; the only confirmed platform defect is the stale status field, which does not invalidate the direct receipts.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting model or artifact: fixed merged Zephyr/Mistral 7B SFT policy_start. Available training data and assets: all 61,135 preference pairs in the pinned UltraFeedback training split; the 128-row public IFEval proxy is additionally available during exploration. Agent-mutable content: training source, objective, hyperparameters, schedule, and artifact-publication logic under solution; only a source patch may cross into formal replay. Fixed or prohibited content: starting weights, data revisions, final evaluator, and metric. Retraining may not use IFEval, network access, external data, or external models; exploration weights may not cross into formal replay. Proxy evaluator: ifeval_strict_accuracy_public128 / maximize / public split / n=128 / descriptive binomial standard error when reported. Final evaluator: ifeval_strict_accuracy_hidden413 / maximize / 413 rows comprising the public 128 plus a hidden held-out 285 / n=413 / descriptive binomial standard error. Artifact contract: a PEFT/LoRA adapter loadable with the fixed base, or a merged Hugging Face causal language model. At most three valid artifacts are accepted; the highest valid final score is official best. ``

Both evaluators apply the chat template, greedy generation, and strict prompt-level instruction checks. They measure whether a response satisfies every verifiable requirement for a prompt. The 128-row public split is smaller than the 413-row final set, so an exploration proxy score must not be subtracted from a full-final score as though both sampled the same population. The reported standard errors are descriptive binomial quantities from one evaluation; they do not capture seed variance and are not paired confidence intervals between candidates.

The task source supplies two references under the same 413-row final protocol: the fixed start scores 164/413 (0.3971), and the shipped solution scores 210/413 (0.5085). Direct comparisons with formal final results are valid for these references.

3.2 How the baseline works

``text [Fixed start, prompt, chosen answer, and rejected answer] -> [Current policy and frozen reference compute token log probabilities for both answers] -> [UltraFeedback chosen/rejected labels supply the pairwise preference signal] -> [A sigmoid DPO loss with beta=0.01 increases the policy-vs-reference preference margin] -> [Only the LoRA adapter is updated and exported; the base and reference remain fixed] ``

The baseline loads the start in four-bit NF4 and trains rank-128, alpha-128, dropout-0.05 LoRA modules on the attention and feed-forward projections. Disabling the adapter yields the frozen reference policy. It shuffles all 61,135 pairs with seed 42, uses a maximum sequence length of 1,024, and forms a global batch of 16 pairs from microbatch 4 and gradient accumulation 4. Optimization is paged 32-bit AdamW at 5e-6, cosine decay, warmup ratio 0.1, zero weight decay, and gradient clipping at 1. The baseline requests 772 steps, saves every 386 steps, and retains at most three adapters.

The agent identified two concrete bottlenecks. A 772-step endpoint makes weak use of the formal budget. Also, 7,160 pairs exceed the sequence limit and 3,171 truncate both responses, yet filtering them would discard substantial data. This motivated a training signal that could stabilize the chosen answer without removing examples.

4. Four-hour exploration and decision process

The agent first inspected the code and data, spent roughly the next hour establishing pure DPO and repairing its post-training publication failure, then tested auxiliary supervision and a higher-beta alternative. The latter part compared supervision weights and used the remaining time for wall-clock logic, checkpoint policy, patch replay, and artifact loading. The manifest's phase clock reports 13,991 seconds, while the attempt's remaining-budget counter reports 447 seconds at explicit submission; the source of the roughly 38-second accounting difference is inconclusive, but both show compliance with the 14,400-second limit.

U-01 - Should truncation and preference noise be handled by filtering?

Motivation and hypothesis. The agent suspected that noisy rating relations and 1,024-token truncation weakened DPO, and asked whether filtering should precede an objective change.

Concrete change and experimental setup. It profiled rating order and post-template sequence length over all 61,135 fixed training pairs. It did not mutate the data or run a filtered-training control.

Observed result. The chosen answer had a strictly higher rating in 53,748 pairs. A total of 7,160 pairs exceeded the sequence budget, including 3,171 for which both responses were truncated.

Agent interpretation. Aggressive filtering would remove too much signal, while truncation-only handling would not directly anchor chosen-response generation. The agent therefore prioritized an auxiliary supervised term. It inspected other loss and capacity possibilities but did not turn them into evaluable runs.

Report assessment and confounders. These are data diagnostics, not efficacy evidence. With no filtered control, the record cannot establish that filtering would be inferior; rating order is also not a direct measure of label correctness.

Decision and impact. Filtering was rejected for this budget. The full dataset was retained, and chosen-response supervision became the primary direction.