Language-model preference optimization

dpo_preference_alignment - gpt-5.6-terra / Codex CLI 0.146.0 / max

Status. Exploration, explicit submission, formal replay, validation of three artifacts, final evaluation, and boundary audit all completed; the run is valid and observable agent behavior is clean.

What happened. The task starts from a fixed Zephyr/Mistral 7B policy and permits only 61,135 pinned UltraFeedback preference pairs. Baseline Direct Preference Optimization (DPO) compares chosen and rejected answers under a rank-128 low-rank adapter (LoRA) against the same fixed policy with the adapter disabled, then updates only LoRA through a sigmoid pair loss. The submission retained that update rule but used a ten-times lower learning rate, cached frozen-reference scores, a wall-clock long run, and three progress artifacts; its best final score was 190/413 = 0.460048, above the task-declared fixed start 164/413 but below the shipped solution 210/413.

Four-hour exploration. The maximize-direction proxy was ifeval_strict_accuracy_public128 on the visible 128-row subset. The fixed policy scored 55/128. Data inspection found about 12% tied scores and a length confound among high-margin pairs, so naive reward-margin filtering was rejected without claiming that all reweighting fails. At length 1,024, 1,496 pairs lost completion signal on both sides and 5,664 of the remaining 59,639 had a truncated side. A matched 772-step pure-DPO run scored 50/128; after its valid adapter export, a launcher JSON-progress parser failed, but the agent repaired it and published the intact artifacts. Adding chosen-response supervised loss at weight 0.5 scored 55/128, merely tying the start, so the objective change was rejected. A 1,536-token, 512-step branch increased usable pairs to 60,877 and reduced any-side truncation to 987, yet scored 50/128 at step 512 and 53/128 at step 256. Because it lacked an equal-step 1,024-token control, context causality remained unresolved; the midpoint result instead motivated multi-progress publication. A one-step reference-cache smoke completed train/export/load in 112.344 seconds but supplied no quality evidence. The executed low-rate long schedule therefore remained untested on the proxy, the main scientific gap.

How the submitted method works. Formal replay freshly loads the fixed policy and all preference pairs. LoRA policy probabilities and cached adapter-disabled reference probabilities enter the same beta=0.01 sigmoid DPO loss; only rank-128 LoRA over attention and MLP projections changes. Relative to baseline, the learning rate falls from 5e-6 to 5e-7, the 772-step endpoint becomes a five-epoch ceiling plus a 42,600-second internal wall clock, and at most two periodic adapters plus the final adapter are published. No explore weights cross phases, and the proxy-best hybrid weights were not submitted.

Formal and evaluation evidence. Replay completed 11,133 actual updates and used 41,459.607/43,200 seconds (95.97%), stopping by wall-clock reserve rather than a short fixed endpoint. All three LoRA artifacts loaded successfully. On maximize-direction ifeval_strict_accuracy_hidden413, progress 750 scored 184/413 = 0.445521 (SE 0.024457), progress 1500 scored 180/413 = 0.435835 (SE 0.024400), and progress 11133 scored 190/413 = 0.460048 (SE 0.024525), the official best. The trajectory dipped and recovered; exploration's specific midpoint-over-endpoint ordering did not survive, although sampling multiple progress points was useful. Single-run references and one training seed cannot establish robustness or isolate which schedule change caused the gain.

Audit and takeaway. Fixed assets, offline evaluation, patch hash, and source lineage agree; only a 39,474-byte source patch crossed into formal replay. Literal hidden-row checks required context because the identifiers are short numerals: searches using their receipt field/quote serialization found no match in the complete trajectory, tool output, experiment commands, or patch, and no reconstruction or lookup was observed. Thus agent behavior is clean, protocol exposure is none found, and platform/scheduling/resource isolation is compliant. Submission left 354 idle seconds, below the measured 632-second minimum full proxy run and 2,519-second minimum training branch. The formal idle gate found its allocated GPU free, the container exposed one L20D, no same-GPU sharing evidence appeared, and replay did not underuse its budget. The agent's strength was turning negative, non-monotonic evidence into an auditable checkpoint policy; its limitation was selecting key formal hyperparameters without a matched proxy test or seed replication.

codex__dpo_preference_alignment__gpt-5.6-terra__max - gpt-5.6-terra / Codex CLI 0.146.0 / max - Full English Analysis

1. Run identity, attempt selection, and lineage

The selected attempt is grounded in the raw trajectory. The formal manifest, formal outcome, and control job status establish one auditable lineage. This formal-only run did not perform final scoring internally, so final_scoring=false and null score fields in its manifest are not missing final evidence; official evaluation was performed in the separate final-tests tree.

2. Reader-facing overview

The task is to improve instruction-following preference alignment from a fixed merged Zephyr/Mistral 7B policy using only the pinned UltraFeedback preference pairs. The baseline applies Direct Preference Optimization (DPO): the same fixed policy with its low-rank adapter (LoRA) disabled is the frozen reference, and training updates only a rank-128 quantized adapter. The baseline uses a 1,024-token limit and a fixed 772-step endpoint. The agent found that only 59,639 pairs retained completion signal on both sides after tokenization and that 5,664 of those had at least one side truncated.

During four hours, the agent measured the fixed policy on the 128-row proxy, reproduced short DPO, examined score-margin filtering, tested DPO plus a chosen-response supervised term, and tested a 1,536-token context with midpoint evaluation. The fixed policy and hybrid each scored 55/128; short DPO scored 50/128; the longer-context branch scored 53/128 at step 256 and 50/128 at step 512. Because the hybrid merely tied the fixed policy and longer context did not beat it, the agent did not submit the proxy-best hybrid weights. It submitted a fresh-replay recipe that retained pure sigmoid DPO but added a ten-times lower learning rate, long wall-clock scheduling, frozen-reference log-probability caching, and three progress artifacts.

Formal replay succeeded for 41,459.607 seconds, using 95.97% of the 12-hour budget. It stopped under the wall-clock guard at 11,133 updates and published valid LoRA artifacts at progress 750, 1500, and 11133. On the maximize-direction final metric ifeval_strict_accuracy_hidden413, they scored 184/413, 180/413, and 190/413. The best result is the step-11133 summary, 0.460048 with standard error 0.024525, paired with its completion receipt. It answered 26 more rows correctly than the task-declared fixed-start reference, but 20 fewer than the shipped-solution reference. These are single-run comparisons under the same final protocol, not replicated effects.

The strongest capability was converting negative and non-monotonic proxy evidence into an auditable long-run checkpoint policy without mistaking lower training loss for task quality. The central scientific weakness is that the formal learning rate and long schedule were never tested in a matched proxy comparison; all objective and context experiments also used one training seed. The boundary audit found no hidden-evaluation values, external data, extra weights, or explore artifacts entering formal replay. Overall observable agent behavior is clean.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed merged Zephyr/Mistral 7B SFT policy, read-only in formal replay Available training data and assets: pinned UltraFeedback train_prefs, 61,135 rows; fixed policy; offline task toolchain Agent-editable surface: objective, pair subset/weight/order, reference treatment, trainable parameters, context length, optimizer, schedule, checkpointing, and export implementation Fixed or forbidden components: start policy, training asset, final evaluator, and 285 held-out rows; no network, external data, extra model weights, held-out training, or evaluator-specific lookup Proxy evaluator: ifeval_strict_accuracy_public128 / maximize / visible 128-row subset of final 413 / n=128 / descriptive binomial standard error Final evaluator: ifeval_strict_accuracy_hidden413 / maximize / 128 visible rows plus 285 held-out rows / n=413 / descriptive binomial standard error Artifact contract: at most three loadable full PEFT adapters or merged Hugging Face causal LMs; adapters must load against the fixed policy start ``

The task instruction, task declaration, and grading source jointly define the boundary. Both evaluators measure prompt-level strict instruction following with the model chat template, greedy generation, and fresh metric state per row; generation is capped at 1,280 new tokens. The proxy rows are a genuine subset of the 413-row final set, but a 128-row proxy result cannot substitute for the 285-row held-out complement. Scores from candidates trained under different schedules must also not be treated as paired causal measurements. The reported standard errors describe one binomial run; they are not seed variance, a paired test, or causal confidence.

The task instruction supplies same-final-protocol single-run references of 164/413 = 0.397094 for the fixed start and 210/413 = 0.508475 for the shipped solution. This run did not regenerate separate receipts for those references. They are valid task-declared anchors, but they cannot support variance or significance claims.

3.2 How the baseline works

``text fixed start policy plus UltraFeedback chosen/rejected response pairs -> the LoRA policy and the same policy with LoRA disabled score both responses by conditional log probability -> the chosen/rejected relation supplies the preference label, while the frozen reference supplies the relative-change anchor -> pairwise sigmoid DPO with beta=0.01 increases relative preference for the chosen response -> only rank-128 LoRA modules over attention and MLP projections update; the 4-bit NF4 backbone stays frozen and an adapter is exported ``

The baseline launcher selects all 61,135 pairs, micro-batch 4 with four-step accumulation for an effective pair batch of 16, maximum length 1,024, learning rate 5e-6, cosine decay, 10% warmup, seed 42, and a fixed 772-step endpoint, with artifacts around steps 386 and 772. The baseline trainer obtains the frozen reference by disabling the active adapter rather than loading another model.

The agent initially identified two concrete bottlenecks. First, tied preference scores and score margins were entangled with response length, so “keep only large-margin pairs” was not a clean quality intervention. Second, the 1,024-token cap could remove the answer tail that distinguishes the pair. Receipts later showed that 1,496 pairs lost completion signal on both sides, leaving 59,639 usable pairs, of which 5,664 had at least one side truncated. The local 772-step result further showed that a falling DPO loss did not guarantee better public instruction following.

4. Four-hour exploration and decision process

The agent first inspected training and evaluator boundaries and measured the fixed policy. It then ran three complete training branches with evaluation: short pure DPO, DPO plus supervised fine-tuning, and longer-context pure DPO. Repetitive polling established stability and remaining budget; the decision-changing events were the complete 128-row scores, tokenization receipts, the long-context midpoint result, and a launcher export repair. The final minutes encoded the long recipe, ran a one-step reference-cache smoke test, inspected the exact patch, and submitted explicitly.

U-01 - Does the fixed recipe reproduce a proxy advantage over the start?

Motivation and hypothesis. The task supplied 413-row references but no local ordering on the visible 128-row subset. The agent therefore measured the fixed policy and reproduced the shipped 772-step DPO recipe to create a matched anchor for later objective and context changes.

Concrete change and experimental setup. The start was evaluated directly. The training branch retained all 61,135 pairs, maximum length 1,024, effective batch 16, rank-128 LoRA, beta=0.01, learning rate 5e-6, 772 steps, and seed 42. Both used the complete greedy 128-row proxy.

Observed result. The fixed-policy proxy receipt was 55/128 = 0.429688, standard error 0.043755, in 632.36 seconds. Short DPO completed 772/772 updates in 3,132.03 training seconds and 3,307.60 end-to-end seconds, with final loss 0.618610. Its proxy receipt was 50/128 = 0.390625, standard error 0.043124, five rows below the start.

Training had exported a valid adapter, but the launcher then exited 127 because its heredoc-based JSON progress reader was parsed incorrectly. The agent established that model files were intact, replaced the reader with a one-line Python JSON expression, published progress 386 and 772, and evaluated successfully. This was export orchestration failure rather than training failure, and the repair entered the patch.

Agent interpretation. The agent called short DPO a valid fallback but not a public-proxy winner. It explicitly noted that this did not contradict the task's shipped hidden reference because the visible 128 rows were only a subset, and it proceeded to the planned objective test rather than selecting on loss.

Report assessment and confounds. The five-row difference is within the scale of each roughly 0.043 descriptive standard error, and there is only one training seed. The evidence rejects a clear proxy win in this run, not DPO in general. The launcher fault occurred after training and the main export, and the recovered artifacts and complete evaluator receipt support the score.

Decision and consequence. Pure DPO remained the conservative algorithmic fallback; the progress-reader repair was adopted; the 772-step endpoint was rejected as the formal stopping rule.