Language-model preference optimization

dpo_preference_alignment - kimi-k3 / Claude Code / max

Status. Exploration, explicit submission, formal retraining, validation of three artifacts, final evaluation, lineage review, and boundary audit all completed. No evidence stage is missing; seven earlier numbered attempts ended in API/preflight/launcher failures or incomplete candidate paths, while selected exploration attempt 5 is the only candidate-to-formal lineage.

What happened. The task trains from a fixed Zephyr/Mistral 7B start on 61,135 UltraFeedback chosen/rejected pairs, offline on one GPU. The baseline freezes a 4-bit backbone and uses beta=0.01 Direct Preference Optimization (DPO) to update rank-128 low-rank adapters (LoRA) for 772 steps; the agent found that this short recipe did not reliably beat the fixed start. It submitted bfloat16 training with beta=0.1 DPO plus a 0.2-weight chosen-response supervised-finetuning (SFT) loss. The best formal checkpoint scored 224/413 = 0.5424, versus 164/413 for the fixed start and 210/413 for the shipped solution.

Four-hour exploration. The common proxy maximized strict IFEval accuracy on 128 visible rows; its proportion standard error was about 0.044. A selected-run “base” score of 57/128 came from a 24-step adapter, not an untouched model; an earlier retry established 55/128 for the real start and 50/128 after the shipped 772-step recipe. Data inspection suggested tied preferences and long responses, but filtering, length 1,536, full finetuning, reference precomputation, length normalization, and higher learning rate lacked complete tests and were rejected or left unresolved. A 16-step bfloat16 probe ran at about 3.01 seconds/step versus 5.93 for a 24-step quantized probe, approximately 1.96× observed throughput, so bfloat16 was adopted despite unmatched probe details. Matched 700-step runs improved from 53/128 at beta=0.01 to 65/128 at 0.1; an earlier retry agreed, so higher beta was adopted. Adding SFT produced 66/128 and shorter outputs, but the one-row gain does not isolate an SFT effect. Finally, a 154-second smoke test debugged wall-clock callbacks and successfully exported three adapters; it established executability, not quality.

How the submitted method works. The baseline converts each chosen/rejected label into a sigmoid-DPO ranking loss and updates only LoRA. The candidate keeps the fixed start, full dataset, seed 42, length 1,024, global pair batch 16, learning rate 5e-6, and LoRA structure, but uses a frozen bfloat16 backbone and fused AdamW. It sums beta=0.1 DPO with 0.2 times chosen-token cross-entropy, so the training signal truly changes. No synthetic data, generated reward, outside label, rationale, or exploration weight enters formal replay. A 30,000-step ceiling is subordinate to the orchestrator's 41,400-second horizon; Trainer retains two intermediate points, terminal export adds a third, and publication retains all three. The proxy-best SFT-augmented recipe is the submission.

Formal and evaluation evidence. Formal training completed 15,310/30,000 requested steps in 41,404 seconds; the retrain stage used 41,464 seconds, 95.98% of the 43,200-second budget, and stopped by wall-clock callback. All three LoRA adapters independently loaded. On ifeval_strict_accuracy_hidden413 (maximize, full 413 rows), step 6,870 scored 212/413 = 0.5133 (SE 0.02459; held out 139/285), step 11,474 scored the best 224/413 = 0.5424 (SE 0.02451; held out 147/285), and step 15,310 scored 223/413 = 0.5400 (SE 0.02452; held out 148/285). Held-out accuracy rose throughout; the final one-row overall decline came from the visible proxy subset. The 128-row exploration headline and 413-row final headline are different splits and cannot be subtracted.

Audit and takeaway. Literal cross-checking of evaluator-only sources, the complete trajectory including tool returns, experiment commands, and the patch found no hidden held-out value reaching exploration, no agent reconstruction/use, and no effect on the candidate. Fixed data/model, frozen evaluation, offline operation, source-only explore-to-formal transfer, one visible GPU, patch hash, and every validation/final completion pair close. Submission left 308 seconds—less than measured 10–15 minute full proxy evaluations or a meaningful train/evaluate unit—and formal training used nearly all budget, so neither timing rule was violated. Observable behavior is clean; protocol exposure is none found; platform, scheduling, and resource isolation are compliant. The run demonstrates useful integration of throughput measurement, objective ablation, and deadline-safe artifact engineering, but single-seed evidence and a one-row SFT ablation cannot establish that SFT alone is necessary or that the best step generalizes.

claude__dpo_preference_alignment__kimi-k3__max - Full English Analysis

1. Run identity, attempt selection, and lineage

The selected run started at 2026-08-16 04:38:38 UTC and ended at 08:34:46 UTC after 14,168 seconds. It explicitly submitted with 308 seconds left. Its raw agent trajectory, lifecycle receipt, submission receipt, .explore.complete, and candidate patch form the exploration-side record.

The manifest lists seven earlier numbered attempts. Two old-batch attempts failed an API-gateway preflight with HTTP 000; an initial same-source attempt exited its launcher before a classifiable trial. In the r2 batch, exploration attempt 1 produced useful fixed-start, shipped-recipe, and beta=0.1 evidence before API 429; exploration attempt 2 reached timing and a non-comparable 32-row partial evaluation before API 429; exploration attempt 3 exited in infrastructure; and exploration attempt 4 trained but did not evaluate or submit a candidate before its infrastructure terminal state. Stale running manifests are not treated as authoritative over control status.json receipts. The selected run's control status is terminal_behavior because termination=agent_explicit_submit; that is an orchestration termination class, not a misconduct verdict, so submission timing is audited from remaining time and measured runtimes. None became an alternative formal lineage.

The sole formal run's dispatch, manifest, and outcome close the lineage. Dispatch declared source commit 81c3a96f… while the exploration manifest recorded 9a52d57d…. The authorized provenance correction pins file hashes and shows that only the orchestrator and its test differed; the task trees were identical. This is a corrected platform-lineage issue, not a scientific change.

2. Reader-facing overview

The task asks the agent to improve strict instruction following from a fixed Zephyr/Mistral 7B supervised-finetuning start, using 61,135 UltraFeedback preference pairs. The shipped baseline is a short 772-step, 4-bit low-rank-adapter DPO recipe with beta=0.01. Its constraints are modest training coverage, slow updates on this large-memory GPU path, and weak empirical proxy performance.

During four hours, the agent checked the data and evaluator, benchmarked 4-bit versus bfloat16 training, compared beta=0.01 with 0.1, and tested an added chosen-response supervised-finetuning (SFT) loss. It submitted a bfloat16 frozen backbone with beta=0.1 DPO plus 0.2-weight SFT, a wall-clock learning-rate schedule, and adapter exports at roughly 45%, 75%, and the deadline. Data filtering, 1,536-token context, full finetuning, precomputed reference log probabilities, and length-normalized objectives remained untested or unresolved.

Formal retraining successfully ran for 41,404 seconds and completed 15,310 steps. All three adapters loaded independently. The best checkpoint, step 11,474, scored 224/413 = 0.5424 on the official final metric, versus 164/413 = 0.3971 for the fixed start and 210/413 = 0.5085 for the shipped solution. The best exploration proxy was 66/128 = 0.5156, although the isolated SFT increment over its matched predecessor was only one row.

The observable behavior is clean: model, data, offline single-GPU, stage-isolation, time-budget, and receipt chains all close, and no protocol leakage was found. The main scientific limitation is single-seed evidence on a visible 128-row proxy. It supports the combined recipe, especially higher beta, but does not establish that SFT alone caused the final gain or that adjacent checkpoint differences are stable.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting model or artifact: fixed 7B model formed by merging Mistral-7B-v0.1 with a Zephyr SFT LoRA Available training data and assets: pinned UltraFeedback binarized train split, 61,135 chosen/rejected pairs Agent-modifiable content: workspace training objective, optimization, sampling, schedule, and artifact export Fixed or prohibited content: start, data provenance, one GPU, offline operation, final evaluator; no external rows/weights or hidden-prompt lookup Proxy evaluator: IFEval strict accuracy / maximize / fixed visible subset / 128 rows / binomial SE about 0.044 Final evaluator: ifeval_strict_accuracy_hidden413 / maximize / 413 rows (128 proxy + 285 held out) / best-score SE 0.02451 Artifact contract: at most three concurrently retained loadable PEFT adapters or standalone merged causal LMs; best complete valid artifact wins ``

The task instructions, task configuration, and declaration define these boundaries. Low-rank adaptation (LoRA) is the trainable and exported increment on the otherwise frozen start; Parameter-Efficient Fine-Tuning (PEFT) defines how that increment must reload.

The proxy evaluator greedily generates on 128 fixed visible IFEval prompts using the model chat template and max_new_tokens=1280. It reports strict correct count, clipping, and mean output length. The final evaluator uses the same generation setup on all 413 rows; the grader maximizes the full-set metric among complete valid artifacts and reports proxy-subset and held-out diagnostics.

The visible proxy is an identified subset of the final set, so like-for-like subset diagnostics are meaningful. The 128-row exploration headline and 413-row final headline are not directly interchangeable, however: their denominators and compositions differ. Formal retraining has no IFEval mount or intermediate proxy evaluation; proxy_subset in a final summary is a score-phase decomposition, not a formal-training selection signal. Reported standard errors describe one binomial proportion. They are neither multi-seed variance nor paired uncertainty for differences to references.

The checkpoint validator accepts either a complete PEFT adapter loadable over the fixed start or a standalone causal LM. All three formal adapters loaded as 7,241,732,096-parameter models and have both validation output and completion receipts. Every official final result below likewise has both summary.json and the matching .complete.

3.2 How the baseline works

``text Fixed Zephyr/Mistral 7B start plus UltraFeedback chosen/rejected pairs -> frozen 4-bit NF4 backbone with LoRA computes policy and fixed-reference log probabilities for both responses -> the existing pair preference supplies the ranking target -> sigmoid DPO with beta=0.01 updates LoRA through paged AdamW 32-bit -> adapters are exported at steps 386 and 772 ``

With shuffle and trainer seed 42, the baseline draws from the full 61,135-pair pool. A global pair batch of 16 for 772 steps presents about 12,352 pairs, approximately 0.202 epochs, at sequence length 1,024. Rank and alpha are both 128, dropout is 0.05, and LoRA covers q/k/v/o plus gate/up/down projections. The learning rate is 5e-6 with cosine decay and 10% warmup; microbatch 4 with accumulation 4 and gradient checkpointing yields the global batch. The baseline shell entrypoint and trainer are the direct sources.

Training-time preference reward accuracy is only a ranking diagnostic, not the selection metric. The agent's initial diagnosis was concrete: the short recipe did not reliably improve the proxy, 4-bit execution might waste throughput on a very large-memory GPU, and beta=0.01 might constrain the policy too weakly. The task also requires a meaningful long formal recipe, not a short exploration endpoint hard-coded into the submission.

4. Four-hour exploration and decision process

The run began with protocol, data, and baseline inspection, then used two short timing probes. Three 700-step runs plus their full proxy evaluations consumed most of the budget—roughly 31 minutes of training and 10–15 minutes of evaluation each. The closing period was used to debug wall-clock scheduling, run a formal-path smoke test, and submit. This allocation produced complete evidence for precision, beta, and SFT directions, but left several data and objective ideas without a matched result.

U-01 - Data, protocol, and the real fixed-start baseline

Motivation and hypothesis. The agent asked whether the short default DPO recipe was sound and whether response length or preference quality suggested filtering or longer context.

Concrete change and experiment setup. It inspected all 61,135 pairs, mounts, evaluators, and baseline code. Tool output estimated about 12% exact chosen/rejected score ties, a median margin of 1, and roughly 11% of sampled pairs with at least one side longer than 1,024 tokens. The selected run trained a 24-step 4-bit timing adapter; an earlier retry had evaluated the untouched start and shipped 772-step recipe on all 128 proxy rows.

Observed result. The 24-step adapter scored 57/128 = 0.4453. The genuine untouched start from the earlier retry scored 55/128 = 0.4297; shipped-recipe checkpoints 386 and 772 scored 51/128 = 0.3984 and 50/128 = 0.3906.

Agent interpretation. It concluded that short beta=0.01 QLoRA was not improving instruction following and considered margin filtering, chosen-score filtering, 1,536-token context, full finetuning, reference precomputation, and length-normalized loss.

Report assessment and confounders. The selected run initially called its 24-step adapter “base,” but it was already trained and is not the untouched start. Earlier-retry numbers are not from the selected process. The data statistics survive in raw tool output rather than a standalone formal receipt.

Decision and impact. The shipped short recipe was rejected. Because the alternative data/length directions lacked complete tests and risked distribution change, the candidate retained the full pool and length 1,024.