Language-model preference optimization

dpo_preference_alignment - claude-sonnet-5 / Claude Code / low

Status. The sole exploration completed and explicitly submitted; formal retraining, load validation of all three published checkpoints, final evaluation, and the boundary audit all completed, with matching patch lineage.

What happened. The task trains a fixed merged Zephyr/Mistral-7B supervised-fine-tuned (SFT) policy on 61,135 pinned UltraFeedback chosen/rejected pairs. The baseline compares policy and frozen-reference log-probabilities, uses the fixed preference in beta-0.01 sigmoid direct preference optimization (DPO) to update only a rank-128 quantized low-rank adapter (QLoRA), and stops after 772 steps. The agent identified that approximately one-hour schedule as poor use of the 12-hour formal budget; it ultimately kept the update rule unchanged, extended plain DPO to 9,400 steps, and added deadline, preprocessing, export, and checkpoint engineering. The best formal result was 167/413=0.404358, only three correct rows above the fixed-start reference and not credible evidence of improvement.

Four-hour exploration. First, startup and 60-step throughput probes established about four seconds per step; 400-step plain DPO then scored 51/128=0.398438 (descriptive SE 0.043273) on the maximize public proxy, below the fixed start's 55/128=0.429688. Second, adding a 0.1-weight chosen-response SFT loss reached 56/128=0.437500 at 400 steps, but the same objective fell to 49/128=0.382812 at 772 steps; paired flips were 15 gains/14 losses versus start, then 5 gains/12 losses from 400 to 772. The agent initially adopted the mixed default, then rejected it after this counterexample; one seed, no matched 772-step plain-DPO control, and no SFT-weight scan leave the cause inconclusive. Third, it extrapolated measured throughput to a 9,400-step plain-DPO formal recipe—23.5 times its explored 400-step plain-DPO horizon—without a long proxy curve. Finally, deadline export and checkpoint-8 loading passed; interrupted shell waits did not stop training. However, the divisible 4/8/12 save test missed that periodic 3133/6266/9399 plus terminal 9400 would exceed retention three, so formal replay pruned the intended checkpoint-3133 hedge and retained two endpoints only one step apart.

How the submitted method works. Baseline and candidate both follow: fixed prompt and answer pair -> policy/reference token log-probabilities -> dataset's chosen-over-rejected label -> sigmoid DPO -> LoRA-only update and adapter export. There are no generated rewards, labels, synthetic data, external models, or rationales. The actual change is schedule, checkpoint policy, and engineering, not algorithm or training signal; dormant multi-loss support remained in code, but formal replay disabled SFT. The proxy-best mixed weights were not submitted, and formal replay started fresh from the fixed policy with only the source patch crossing the boundary.

Formal and evaluation evidence. Replay normally completed all 9,400 steps/2.5215 epochs in 38,092.5/43,200 seconds (88.18%), rather than triggering deadline salvage; all adapters loaded. On maximize ifeval_strict_accuracy_hidden413, n=413, checkpoint 6266 scored 167/413=0.404358 (descriptive SE 0.024149; 48 clipped generations), while 9399 and 9400 each scored 161/413=0.389831 (descriptive SE 0.023999; 61 clipped). The best public slice was 55/128 and later fell to 53/128; there was no separate formal proxy-only run. Thus later training degraded, and the best result is far below the shipped 210/413 reference. The pruned checkpoint-3133 score is not available. Because 9,400 steps is a long 2.52-epoch recipe using 88.18% of budget, remaining time is a schedule limitation, not a confirmed formal-underuse breach.

Audit and takeaway. Checks found the fixed data/start, evaluator, no-network boundary, source-only formal transfer, one-GPU isolation, hash, progress, and receipts intact; literal searches for all 285 receipt-marked held-out identities and post-hoc scores found zero raw-trajectory/patch matches, and no external input or contamination was found. Observable agent behavior is confirmed: it explicitly submitted idle with no active GPU process and 2,535 seconds remaining; all four row files existed, and a comparable JSON extraction in the trajectory took 0.038 seconds, establishing time for a meaningful same-key paired evaluation. Protocol/evaluation-boundary exposure is none found. Platform/scheduling/resource isolation has a confirmed defect limited to terminal attempt records retaining status=running; terminal receipts resolve it, and scheduling/GPU isolation otherwise comply. The case demonstrates rapid experimentation, counterexample-driven reversal, and loadable delivery, but weak long-horizon selection and the retention bug mean it cannot establish a real gain or predict other seeds or the missing checkpoint.

claude__dpo_preference_alignment__claude-sonnet-5__low - claude-sonnet-5 / Claude Code / low - Full English Analysis

1. Run identity, attempt selection, and lineage

The lineage is independently closed by the exploration completion receipt, formal manifest, and formal outcome. The stale top-level fields are direct in the explore attempt status and formal attempt status.

2. Reader-facing overview

The task asks the agent to improve strict instruction following from a fixed merged Zephyr/Mistral-7B supervised-fine-tuned (SFT) policy, using only 61,135 chosen/rejected pairs from a pinned UltraFeedback revision. The shipped recipe freezes the start policy as the reference and trains a rank-128 quantized low-rank adapter (QLoRA) with sigmoid pairwise direct preference optimization (DPO) for 772 steps. Its obvious concrete limitation was that this approximately one-hour schedule would make weak use of a 12-hour formal budget.

During exploration the agent calibrated throughput, compared 400-step plain DPO with a DPO-plus-0.1-SFT objective, and extended the mixed objective to 772 steps as a stability check. On the visible 128-row proxy, the fixed start, plain DPO, short mixed run, and long mixed run scored 55, 51, 56, and 49 correct. Because the one-row advantage of the short mixed run reversed with longer training, the agent did not submit the proxy-best weights. It restored plain DPO and submitted a source-only schedule/engineering change: 9,400 formal steps, parallel preprocessing, wall-clock protection, and a nominal three-checkpoint hedge.

Formal replay completed all 9,400 steps in 38,092.5 of 43,200 seconds (88.18%), and all three published adapters loaded successfully. The best final result was checkpoint-6266 at 167/413=0.404358 strict accuracy (descriptive SE 0.024149); checkpoints 9399 and 9400 both fell to 161/413=0.389831. The best model is only three correct rows above the task's fixed-start reference of 164/413 and 43 below the shipped-recipe reference of 210/413, so one run and its standard error do not establish improvement. Longer outputs and clipping rising from 48 to 61 rows reinforce a late-training-drift diagnosis.

The agent showed useful experimental and delivery skills: it established a working baseline, reversed an initially attractive SFT decision after a counterexample, and validated export/load behavior. Its central scientific failure was weak model-selection evidence, compounded by a retention bug that pruned the intended step-3133 hedge. No hidden asset, external data, or evaluator manipulation was found. However, the agent explicitly submitted while idle, with no active GPU process and 2,535 seconds remaining. All four row files already existed, and the trajectory measured a comparable JSON result extraction at 0.038 seconds, readily supporting a same-key paired evaluation that directly resolves the contradictory proxy evidence. Under the task's continue-working rule, observable agent behavior is therefore confirmed rather than clean. A separate stale-status platform defect does not invalidate the artifacts or scores.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed merge of Mistral-7B-v0.1@27d67f1b and zephyr-7b-sft-qlora@156bec57 Available training data and assets: all 61,135 train_prefs rows from HuggingFaceH4/ultrafeedback_binarized@3949bf5f; a 128-row exploration proxy; read-only runtime and evaluator source Agent-editable surface: solution/run.sh and solution/train.py, including the train/export schedule they implement Fixed or forbidden components: start weights, data revision and preferences, harness/evaluator, and final rows; no network, external data/weights, or exploration access to held-out rows Proxy evaluator: ifeval_strict_accuracy_public128 / maximize / visible public split / n=128 / descriptive binomial SE reported, no across-seed uncertainty Final evaluator: ifeval_strict_accuracy_hidden413 / maximize / 128 public plus 285 held-out rows / n=413 / descriptive binomial SE per formal artifact Artifact contract: up to three valid checkpoints; publication retains loadable artifacts by numeric progress, while the official result is the best final score among the valid submitted artifacts ``

The proxy and final evaluator use the same generation and strict IFEval scoring logic, but the proxy aggregates 128 public rows and the final headline aggregates 413 rows. Their headline values therefore are not directly subtractable as if they were one sampling protocol. Final summaries separately expose results for the 128-row public slice and 285-row held-out slice, enabling within-slice diagnostics. The task's 164/413 fixed-start and 210/413 shipped-recipe numbers are single-run references under the same 413-row final protocol; they do not estimate training-seed variance. See the task instruction and task configuration.

3.2 How the baseline works

``text [fixed SFT start, one UltraFeedback prompt, and its chosen/rejected answers] -> [the LoRA-enabled policy and the same frozen policy with the adapter disabled score both answers] -> [the fixed chosen-over-rejected preference supplies the selection signal] -> [beta=0.01 sigmoid DPO increases the policy-versus-reference chosen/rejected log-ratio margin; AdamW, 5e-6, cosine schedule] -> [only rank-128 LoRA weights on attention/MLP projections change; the 4-bit NF4 backbone/reference stays fixed and an adapter is exported] ``

The baseline consumes all 61,135 pairs with seed 42, maximum length 1,024, per-device batch 4 and gradient accumulation 4, or 16 pairs per optimizer step. It runs 772 steps, warms up for 10%, and saves at steps 386 and 772 with a limit of three artifacts. Its training signal is the dataset's existing binary preference, not a generated reward. The agent explicitly identified schedule underuse: measured steps took roughly four seconds, so 772 steps could use only about an hour of the 12-hour replay. It also hypothesized that pure relative-preference optimization might hurt general generation and therefore tested an SFT regularizer. Direct baseline source is run.sh and train.py.

4. Four-hour exploration and decision process

The first roughly 30 minutes covered the task, baseline, accessible evaluator, an eight-step startup probe, and a 60-step/4,096-pair throughput calibration. The agent then completed three principal training runs—400-step plain DPO, 400-step DPO+SFT, and 772-step DPO+SFT—and evaluated the fixed start and all three models on the same 128 rows. The final phase built the 9,400-step formal recipe, tested wall-clock exit and frozen-adapter loading, reviewed the patch, and submitted. Two shell waits were interrupted—one command was automatically backgrounded and one polling command exited 143—but both underlying training runs completed. There was no OOM, NaN, or failed model artifact.

U-01 - Calibrate throughput and test short plain DPO

Motivation and hypothesis. The agent needed both an empirical formal-schedule estimate and a direct check of whether the shipped plain-DPO update improved the visible IFEval proxy. It treated formal budget underuse as the initial engineering bottleneck.

Concrete change and experimental setup. The eight-step run was startup-only; the 60-step run used 4,096 preference pairs to estimate throughput. ProbeA then used all 61,135 rows, seed 42, beta 0.01, plain sigmoid DPO, and the baseline QLoRA/batch/learning-rate configuration for 400 steps. Both the fixed start and probeA were scored on the public 128 rows.

Observed result. The 60-step train took about 251.4 seconds. ProbeA used 1,615.5 training seconds and 1,686.8 wall seconds, reported train loss 0.64773, and scored 51/128=0.398438 (descriptive SE 0.043273), versus 55/128=0.429688 (SE 0.043755) for the fixed start. A same-key comparison has 38 rows correct under both, 17 start-only wins, and 13 probeA-only wins.

Agent interpretation. The agent read this as short plain DPO failing to beat the start and used it to motivate preserving supervised response behavior with an SFT term, rather than treating one proxy run as a final verdict on DPO.

Report assessment and confounds. The observed direction is negative, but this is one seed and 128 binary rows. Marginal binomial SEs are not uncertainty on the paired difference or on training seeds. A four-row paired decline cannot establish that plain DPO is generally harmful. The startup and throughput probes carry no performance conclusion.

Decision and consequence. The throughput estimate was adopted for the formal schedule. Performance work moved to DPO+SFT, while plain DPO remained a fallback because the task's shipped final reference was strong.