Language-model preference optimization

dpo_preference_alignment - gpt-5.6-sol / Codex CLI / low

Status. Exploration, submission, formal retraining, adapter validation, and final evaluation completed with receipts. Agent behavior had a confirmed early-submission violation; resource isolation was compliant. Protocol exposure was confirmed because the 128-row proxy is 31.0% of the 413-row final.

What happened. The task starts from a fixed Zephyr/Mistral 7B instruction-tuned policy and uses fixed UltraFeedback chosen/rejected pairs to improve strict IFEval instruction following. The baseline compares policy and adapter-disabled reference log probabilities, applies beta=0.01 sigmoid direct preference optimization (DPO), and updates only rank-128 low-rank adapters on a frozen four-bit backbone. The agent submitted no new objective: it extended one 772-step cosine run to 9,000 steps with hard restarts and preserved progress 386, 772, and the endpoint.

Four-hour exploration. The common proxy was public128 strict accuracy, maximize, n=128; its standard error is descriptive binomial uncertainty, not seed variance.

The original 5×10⁻⁶ learning-rate run completed stably. Progress 386 scored 56/128=0.4375, SE 0.04385, with 8 length-capped rows; progress 772 fell to 50/128=0.390625, SE 0.04312, with 12 capped rows. The agent interpreted this as updating past the visible optimum and preserved both fallbacks. Halving learning rate to 2.5×10⁻⁶ produced 55/128 at 386 and the same 50/128 at 772; capped endpoint rows fell to 6, but strict accuracy did not improve, so the change was rejected. Doubling beta to 0.02 produced only 45/128=0.351563 at step 386, 11 rows below the matched original, so it too was rejected; its step-193 artifact was not evaluated.

For formal engineering, the agent set 78 warmup steps, then 694-step cosine hard-restart cycles, saving 386 and 772. Syntax, first-cycle learning rates, and a one-step adapter export passed. No second restart cycle or 9,000-step quality result was explored. Instruction-focused selection, higher capacity, alternate losses, and multiple seeds were not tested.

How the submitted method works. Baseline and candidate both flow from fixed chosen/rejected conversations to policy/reference log probabilities, pairwise DPO signal, the same sigmoid loss and paged AdamW optimizer, and updated LoRA weights. There are no synthetic labels, external models, or stored rationales. The change is schedule, hyperparameter, engineering, and checkpoint policy only; the update rule remained unchanged.

The exploration-best object was an original-recipe step-386 weight, but weights cannot cross phases. The submitted source replayed from the fixed start, requested 9,000 steps, and produced three fresh adapters. Formal orchestration forced the model/data paths and a 42,600-second wall-clock value; both training and data-order seeds were 42.

Formal and evaluation evidence. Formal handling took 36,560.05 of 43,200 seconds and completed all requested steps rather than hitting the wall limit. All three adapters independently loaded. No formal fast-eval receipt exists; the scores below are official hidden413 results, maximize, n=413:

| Progress | Final score | Descriptive SE | Diagnostics | |---:|---:|---:|---| | 386 | 164/413=0.397094 | 0.024077 | 17 capped rows | | 772 | 168/413=0.406780 | 0.024172 | 21 capped rows | | 9000 | 178/413=0.430993 | 0.024368 | 59 capped rows; official best |

On the same visible 128 rows inside the final, formal progress scored 49, 51, and 53; on the unseen 285 it scored 115, 117, and 125. Thus exploration's 386-over-772 curve did not replay, while the untested long endpoint became best. Progress 9000 is 14 correct rows above the declared fixed start but 32 below the declared shipped solution. Its proxy-minus-held-out gap was -0.02453 ± 0.05253, showing no visible-proxy overfitting, although length clipping rose substantially.

Audit and takeaway. Only pinned assets were used: no network, extra weights, hidden access, evaluator edits, or phase contamination. Yet submission left 1,378 seconds while a saved beta=0.02 step-193 adapter awaited evaluation; five prior full proxy runs took 751–911 seconds. That meaningful test could finish and be interpreted, so the explicit early-submission condition was violated. This does not invalidate formal scores or constitute a hidden-data hack. GPU isolation, runtime, artifacts, hash, and lineage were compliant. The intentional 128/413 overlap is confirmed protocol exposure; 285 rows stayed hidden.

The task package also conflicts internally: assets.lock.yaml describes an 8,192-row shipped recipe, while source and receipts use 61,135, so attribution of the declared 210/413 reference is inconclusive. The case shows controlled ablations and formal engineering, but only a one-run observed gain over the fixed start—not cross-seed stability or restart causality.

codex__dpo_preference_alignment__gpt-5.6-sol__low - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: dpo_preference_alignment Task family: Large-language-model preference alignment Model: gpt-5.6-sol Harness: Codex CLI 0.146.0 under the AI4AI-Bench v1.5 orchestrator Reasoning effort: low Seed, if applicable: training seed 42 and data-order seed 42 Exploration budget: 14,400 seconds; the phase used 13,063 seconds and submitted with 1,378 seconds remaining Formal budget: 43,200 seconds; the formal phase used 36,560.05 seconds

Primary explore run: exploration attempt 1 Why this attempt was selected: It is the only complete explore run in the analysis manifest. The numbered runs directory, job status, and queue likewise contain only exploration attempt 1 for this configuration. It produced a nonempty patch, five complete proxy evaluations, and an explicit-submission receipt. Prior full-run attempts and outcomes: None. Patch hash / validation: SHA-256 verified private digest. The explore submission, formal control spec, formal manifest, and retrain copy all report this hash, and the two patch files compare byte-for-byte equal. Formal run: formal replay 1 Formal the formal replay is cryptographically linked to the selected exploration Final-evaluator receipts: Each of the three summary.json files has its corresponding .complete receipt, so all three are official final results.

Evidence completeness: The raw trajectory, task definition, baseline source, candidate patch, formal lineage manifest, executed environment, trainer state, checkpoint validation, final summaries, and completion receipts are available. No second numbered explore or formal attempt and no applicable correction receipt exist for this configuration. Missing or conflicting evidence: The task package's assets.lock.yaml says the shipped recipe selects 8,192 rows, whereas the authoritative current run.sh, dataset card, and both explore and formal receipts say 61,135. The executed value is therefore 61,135, but whether the declared 210/413 shipped-reference score came from the older 8,192-row recipe or the current source is inconclusive. The explore manifest labels the agent process failed because explicit submission terminated it with status 137; agent_exit_state, lifecycle.json, submit.json, .explore.complete, and the control status jointly establish a normal agent_explicit_submit rather than a failed exploration. The formal manifest's final_scoring=false means only that scoring was not inline; three later evaluation directories contain the official receipts. There is no separate run-level best-selection receipt, but the declared best_valid_of_up_to_3 rule and the three complete per-artifact receipts determine the best result.

2. Reader-facing overview

The task asks the agent to improve instruction following from a fixed Zephyr/Mistral 7B supervised-finetuning start using the fixed UltraFeedback preference pairs. The shipped baseline uses direct preference optimization (DPO): it makes the policy prefer a chosen answer to a rejected answer relative to the same fixed start with the adapter disabled. A four-bit frozen backbone carries rank-128 low-rank adapters (LoRA); the signal and update rule are ordinary pairwise sigmoid DPO.

The agent first reproduced the 772-step baseline and compared progress 386 with 772, then tested a halved learning rate and doubled DPO beta. The best single exploration proxy result was the original recipe at step 386, 56/128; step 772 fell to 50/128, the lower learning rate did not improve strict accuracy, and doubled beta fell to 45/128. The submission did not change the objective. It extrapolated an untested long schedule to 9,000 steps, with 78 warmup steps, 694-step cosine hard-restart cycles, and milestone-only saves at 386, 772, and the endpoint.

Formal replay completed all 9,000 requested steps from the fixed start, and all three adapters loaded. Hidden-413 accuracy rose with formal progress from 164/413 to 168/413 to 178/413; the 9,000-step score of 0.430993 ± 0.024368 is the official best, where the uncertainty is only a descriptive binomial standard error. It is 14 correct rows above the declared fixed-start result but 32 below the declared shipped solution. On the same visible 128-row subset embedded in the final, formal progress scored 49, 51, and 53 rows, so the exploration claim that 386 was clearly better than 772 did not replay.

The agent's strongest capability was controlled rejection of simple hyperparameter alternatives and reliable encoding of fallback milestones. The main scientific limitation was a one-seed extrapolation from at most 772 explored steps to 9,000 cyclic-restart steps; the untested endpoint happened to be the formal best. Observable agent behavior was confirmed noncompliant: enough time remained to evaluate the existing beta=0.02 step-193 adapter, so the explicit early-submission condition was not met. Platform and resource isolation remained compliant. Protocol-side exposure was also confirmed by design: the 128 visible proxy rows are 31.0% of the official 413-row final and did affect selection, while the other 285 rows remained hidden.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: A fixed merge of Mistral-7B-v0.1 and zephyr-7b-sft-qlora, pinned to the revisions in the task package and final resolved_config.json. Available training data and assets: The read-only pinned HuggingFaceH4/ultrafeedback_binarized snapshot; exploration additionally receives the 128-row IFEval proxy. Agent-editable surface: Everything under editable workspace, including objective, row selection and weighting, sampling, reference treatment, trainable parameters, optimizer, schedule, truncation, and artifact policy. Fixed or forbidden components: The starting model, UltraFeedback source, fresh formal initialization, final evaluator, and the 285 unseen final rows. External rows or weights, training on unseen prompts, and evaluator-specific lookup are prohibited. Network mode is no-network. Proxy evaluator: ifeval_strict_accuracy_public128; maximize; fixed n=128; greedy row-wise generation with at most 1,280 new tokens. Its standard error is descriptive binomial uncertainty for one absolute proportion, not seed variance or paired-difference uncertainty. Final evaluator: ifeval_strict_accuracy_hidden413; maximize; fixed n=413, comprising the same 128 proxy rows plus 285 rows not mounted before scoring. It uses the same generation and row scorer and reports final, proxy, and held-out components. Artifact contract: A complete loadable PEFT adapter or merged Hugging Face causal language model. At most the three greatest numeric progress values are accepted, and the official result is the best valid score.

The proxy measures strict satisfaction of every instruction attached to each of 128 visible prompts. The final measures the same property on a different 413-row composition. The headline proxy and 413-row final scores should not be directly subtracted. The final evaluator's score_proxy diagnostic is the same 128-row set under the same generation and scoring protocol and can test whether a proxy conclusion survived replay. Because those 128 rows enter the official 413, the final is not wholly independent of tuning; score_held_out on 285 rows is the isolated diagnostic.

3.2 How the baseline works

UltraFeedback chosen/rejected conversations → the current policy and the fixed reference obtained by disabling LoRA compute answer log probabilities → the policy's chosen-over-rejected advantage relative to the reference advantage supplies the preference signal → a sigmoid DPO loss with beta=0.01 is optimized by paged_adamw_32bit at learning rate 5×10⁻⁶ with cosine decay → the four-bit backbone remains frozen while rank-128 LoRA weights on seven projection families change and are exported

The current source selects all 61,135 shuffled training pairs and 128 test_prefs diagnostic pairs. However, 772 optimizer steps at 16 pairs per step expose about 12,352 pair instances, roughly 0.21 effective epochs. Maximum length is 1,024; microbatch is 4 with four-step gradient accumulation; roughly the first 10% of steps are warmup; adapters are saved at 386 and 772. UltraFeedback reward accuracy is a trainer diagnostic, not IFEval selection evidence.

The agent initially called out the roughly one-fifth data exposure and low beta and considered instruction-focused row selection or a higher-capacity full-precision policy. The decision-changing diagnosis came instead from the proxy curve: strict accuracy dropped and length-capped generations increased between 386 and 772, which the agent interpreted as updating past the proxy optimum. It never produced a reproducible row-selection experiment or tested higher-capacity training.

4. Four-hour exploration and decision process

The first roughly ten minutes covered the training entry point, dataset card, mounted assets, and timer. Baseline training took about 55 minutes and its two evaluations about 30 minutes. The half-learning-rate run used about 54 minutes plus about 27 minutes for two evaluations. The doubled-beta run used about 29 minutes and about 14 minutes for evaluation. Remaining time went to formal-schedule design, syntax checks, a one-step end-to-end smoke, and learning-rate-curve verification. Submission occurred after 13,063 seconds with 1,378 seconds left.

U-01 - Had the 772-step baseline passed its proxy optimum?

Motivation and hypothesis. The agent first needed a matched baseline. Saving the midpoint of one trajectory would test whether the fixed endpoint over-updated while treating reward margins only as stability diagnostics.

Concrete change and experimental setup. No source or update-rule change: beta 0.01, learning rate 5×10⁻⁶, 772 steps, global pair batch 16, seed 42, and evaluations at progress 386 and 772 on the fixed public128 set.

Observed result. Training completed 772/772 steps in 3,144.52 seconds with aggregate loss 0.618294, approximately 14.22 GB peak allocated GPU memory, and no non-finite value, OOM, or artifact failure. Step 386 scored 56/128=0.4375, descriptive standard error 0.04385, with 8 length-capped rows. Step 772 scored 50/128=0.390625, standard error 0.04312, with 12 capped rows. Both adapters loaded.

Agent interpretation. The agent treated the six-row loss and more clipped outputs as decisive enough to redirect the next run, concluding that the visible proxy peaked earlier and further updating could be too strong.

Report assessment and confounds. This within-run progress comparison is better controlled than a cross-seed comparison, but it still has one training seed, and the reported standard errors are not for the paired checkpoint difference. More importantly, formal replay scored the same visible subset at 49 and 51 rows for progress 386 and 772, and the formal weight hashes differ from the exploration weights. The claimed early optimum did not robustly replay.

Decision and consequence. Preserve original step 386 as the exploration-best fallback and step 772 as a shipped-progress fallback, then reduce update strength.