Diffusion aesthetic alignment

ai4ai/ddpo-sd15-aesthetic - gpt-5.6-luna / Codex CLI / xhigh

Status. Exploration, explicit submission, formal retraining, validation of three artifacts, official final evaluation, and boundary audit all completed; source-run and patch-hash lineage is closed.

What happened. The task optimizes animal images from fixed Stable Diffusion v1.5. Baseline Denoising Diffusion Policy Optimization (DDPO) scores online-generated images with a frozen aesthetic model, applies a clipped Proximal Policy Optimization loss, and updates only Low-Rank Adaptation (LoRA) weights in U-Net attention; the agent identified training all 50 denoising steps as the throughput bottleneck. It submitted the same active update with a longer 121-epoch schedule and 40-epoch saving, ultimately scoring 5.738819 ± 0.025576.

Four-hour exploration. On one public 64-image proxy stream, where higher mean aesthetic score is better, the fixed model scored 5.441924 ± 0.054287 and the four-epoch, 50-step baseline scored 5.498931 ± 0.042570 in 1,037 seconds. Cutting training to 20 steps reduced runtime to 410 seconds but scored 5.487423 ± 0.053382; a longer 20-step run also failed to win, so the agent rejected it. Disabling prompt-conditioned reward statistics scored 5.481266, while rank-normalized advantages scored 5.479892 against the matched 5.498931 baseline; global and rank normalization were rejected, although long-horizon rank behavior remains uncertain. Two inner passes scored 5.440750; learning rates of 1×10^-4 and 5×10^-4 both scored about 5.45; 64 rather than 32 trajectories per epoch scored 5.440622; and widening the PPO clip from 10^-4 to 0.2 scored 5.468462 with lower diversity. All were rejected despite single-seed uncertainty. Longer 50-step training produced a saved weight at 5.536280 ± 0.047184, leading the agent to select duration rather than an update-rule change. The requested eight-epoch long runs executed eight epochs, but their last saved weights represent seven completed epochs because saving used zero-indexed epoch numbers. Finally, the agent found and fixed an outer save-interval default that would have overridden 40 with 1; a one-step smoke run published a loadable checkpoint.

How the submitted method works. Each formal epoch samples 32 animal prompts, generates 50-step diffusion trajectories, obtains frozen aesthetic rewards, converts them to prompt-history z-score advantages, and uses AdamW at 3×10^-4 for one clipped-PPO inner pass over LoRA parameters. The executed change was schedule, checkpoint-policy, and configuration engineering—not a new signal or update rule: 13 default epochs became 121 and every-epoch saving became every 40. Optional rank and inner-update controls stayed on the baseline path. No explore weight was submitted; formal replay restarted from the fixed model.

Formal and evaluation evidence. Formal replay completed all 121 configured epochs in 29,686 seconds, stopping at the epoch cap after 68.7% of the 12-hour budget. Three LoRAs were retained, validated, and officially evaluated with both summaries and completion receipts. At 41 completed epochs, artifact 1 scored 5.610320 ± 0.025617; at 81, artifact 2 scored 5.613724 ± 0.023948 with a diversity warning; at 121, artifact 3 scored 5.738819 ± 0.025576 with a diversity warning. All use the hidden 256-image final metric, higher is better. Artifact 3 was best and exceeded the shipped-solution reference 5.526373 by 0.212446. The proxy direction—retain 50 steps and train longer—survived directionally, but proxy and final streams are not directly comparable. Declining pairwise CLIP distance indicates increasing mode-concentration risk.

Audit and takeaway. Formal replay used only the fixed model, embedded prompt distribution, and online images; no external data, network input, explore weights, evaluator modification, GPU sharing, or lineage corruption was found. Agent-readable evaluator source did expose a value intended to remain hidden, and the trajectory read that source, but no later command, proxy configuration, or patch used the value: observable agent behavior is clean, protocol exposure is confirmed, and platform/scheduling/resource isolation is compliant. The case demonstrates broad ablation and checkpoint-path debugging, but one training seed, a small exploration gain, unused formal time, and diversity degradation prevent claims that epoch count alone caused the gain or that it will reproduce across seeds.

codex__ddpo_sd15_aesthetic__gpt-5.6-luna__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

The explore manifest fields auto_retrain=false and retrain_phase_created=false say only that the explore manifest did not itself create a formal phase. The independent formal job specification, control status, formal manifest, and result receipts connect the same source run and patch hash to the formal run. There is no higher-numbered explore or formal attempt for this configuration and no correction receipt that supersedes this run.

The explore manifest records agent_state=failed and raw exit status 137 because the lifecycle controller ended the still-running Codex process while its explicit submission command was being finalized. The same record's agent_exit_state=completed and termination_reason=agent_explicit_submit, together with the submission receipt and .explore.complete, establish a successful submission. At the formal level, a low-level attempt record retains a stale status=running field, while attempt_status=terminal_behavior, job-level status, outcome.json, .formal.complete, and exit code 0 all establish successful completion.

There is also a small task-package conflict in the fixed-start score. The instruction reports 5.397311, whereas assets.lock.yaml records 5.395526885986328 for the current scoring reference. The exact improvement over the fixed start is therefore inconclusive, although the roughly 0.0018 disagreement is immaterial to the conclusion that the best candidate is well above either value. The shipped-solution reference, 5.526373, is unconflicted.

2. Reader-facing overview

The task asks the agent to improve the mean aesthetic score of images generated from a fixed Stable Diffusion v1.5 start. The baseline is Denoising Diffusion Policy Optimization (DDPO): it samples animal prompts, generates diffusion trajectories, scores the resulting images with a frozen aesthetic model, and applies a clipped Proximal Policy Optimization (PPO) loss to Low-Rank Adaptation (LoRA) weights in the U-Net attention layers. Its concrete cost bottleneck was training across all 50 denoising steps of every trajectory.

During exploration, the agent established a four-epoch baseline and tested 20-step trajectories, prompt-conditioned versus batch and rank reward normalization, extra inner updates, two learning-rate alternatives, 64 trajectories per epoch, a wider PPO clip, longer 20-step training, and longer 50-step training. Most alternatives reduced the public 64-image proxy score. The only clear upward point estimate retained the baseline 50-step update and trained longer: its last saved weight scored 5.536280 ± 0.047184 versus 5.498931 ± 0.042570 for the four-epoch baseline. The submitted active method therefore did not change the reward or PPO update. It extended the formal default from 13 to 121 epochs and changed saving from every epoch to every 40 epochs; optional controls added by the patch remained inactive in formal replay.

Formal replay restarted from the fixed model, completed all 121 configured epochs, and produced three validated LoRA artifacts. Their official 256-image final scores were 5.610320, 5.613724, and 5.738819. The last was best, with standard error 0.025576, and exceeded the shipped-solution single-run reference by 0.212446. The later two artifacts nevertheless triggered advisory diversity warnings, and the fixed epoch cap used only 68.7% of the 12-hour formal budget.

The strongest capability was a broad, mostly matched ablation sequence coupled with decision-changing checkpoint-path debugging and a closed formal lineage. The main scientific limitations are the absence of a second training seed, an exploration gain smaller than its evaluation uncertainty scale, and unused formal budget. Separately, a value intended to be hidden by the final protocol was embedded in agent-readable evaluator source. The trajectory read that source, but no subsequent experiment, command, or patch shows reconstruction or use of the value. Protocol leakage and observable agent behavior must therefore be classified separately.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

~~~text Starting artifact / model: runwayml/stable-diffusion-v1-5 at revision pinned private revision Available training data and assets: the fixed model, fixed CLIP-L/14, the frozen aesthetic MLP, and the 45-item simple_animals prompt distribution embedded in source; training images are generated online by the current policy Agent-editable surface: training source under editable workspace, including prompts, reward construction, sampling, optimization, trainable parameters, schedule, and checkpoint policy Fixed or forbidden components: formal starting weights and scoring-side model/source; the final reference stream is intended to be mounted only during scoring; external data, external weights, network input, and lookup against the final stream are forbidden Proxy evaluator: mean_aesthetic_score_public64 / maximize / independent public generation stream / n=64 / sample standard error Final evaluator: mean_aesthetic_score_final256 / maximize / independent hidden generation stream / n=256 / sample standard error Artifact contract: a complete loadable Diffusers LoRA under a numeric progress directory; at most three valid checkpoints compete and the highest final score wins ~~~

Both evaluators use fixed 20-step Denoising Diffusion Implicit Model (DDIM) sampling, classifier-free guidance 5, and the same frozen aesthetic scorer. They also report CLIP text-image alignment and mean pairwise CLIP distance. These are advisory drift and diversity diagnostics, not validity gates and not components of the primary score. Proxy and final scores come from different random generation streams and different sample counts, so they must not be subtracted as if they were a shared test set. Their reported standard errors quantify image-level sampling uncertainty within one evaluation, not variability across independent training runs.

3.2 How the baseline works

~~~text sample text from the 45-animal prompt distribution, starting from fixed Stable Diffusion v1.5 -> generate 32 image trajectories of 50 denoising steps and record old-policy transition log probabilities -> score each image with frozen CLIP-L/14 plus a five-layer aesthetic MLP, then standardize rewards into advantages using prompt history or batch statistics -> apply a clipped PPO probability-ratio loss and one inner AdamW optimization pass -> update and export only LoRA parameters in the U-Net attention processors ~~~

With training seed 43, each baseline epoch has four sampling batches of eight, or 32 trajectories. Training uses batch size four and gradient accumulation four, for an effective batch of 16 and two optimizer updates per epoch. The learning rate is 3×10^-4; the PPO ratio clip is 10^-4; advantages are clipped to ±5. A per-prompt tracker with capacity 32 uses prompt-specific mean and standard deviation after at least 16 observations and otherwise falls back to a global batch z-score.

The baseline formal default is 13 epochs, saves after every epoch, and retains three checkpoints; the exploration proxy profile forces four epochs. The agent initially identified the 50 trained denoising steps as the main bottleneck: one training minibatch took roughly 22 seconds, limiting the number of updates possible in the formal budget. It also explicitly questioned whether sparse per-prompt histories and only one inner pass wasted samples. Those hypotheses were tested rather than inferred by this report.

4. Four-hour exploration and decision process

The agent spent the opening period reading the training loop, reward path, evaluators, and checkpoint publisher, then about 17 minutes established a four-epoch 50-step proxy baseline. The middle of the run focused on the faster 20-step recipe and ablations of normalization, update intensity, learning rate, and sample count. A longer 50-step run occupied much of the later period while the agent prepared the formal schedule. The last portion checked save-variable precedence, ran a one-step formal-entry smoke test, completed PPO-clip and rank-normalization tests, and preflighted the patch and artifacts. Total explore time was about 3 hours 28 minutes, leaving about 32 minutes.

U-01 - Can the published update establish a reproducible proxy baseline, and where is its cost concentrated?

Motivation and hypothesis. Before changing the method, the agent needed to verify that the published update improved the fixed model on the local public stream and to measure the cost of a 50-step trajectory.

Concrete change and experimental setup. Source was unchanged. The agent ran the four-epoch proxy profile with 32 50-step trajectories per epoch, learning rate 3×10^-4, one inner pass, and prompt-conditioned statistics. It separately evaluated the unadapted fixed model on the same 64-image public stream.

Observed result. The fixed model scored 5.441924 ± 0.054287, with CLIP alignment 0.235890 and pairwise distance 0.340275. Four-epoch LoRA training took 1,037.09 seconds and scored 5.498931 ± 0.042570, with alignment 0.236080 and distance 0.338769. The three retained checkpoints and final export loaded successfully. Timing showed that training through 50 steps dominated cost.

Agent interpretation. The agent regarded this as a primary-metric improvement without diagnostic collapse and identified fewer training sampling steps as the most promising throughput direction.

Report assessment and confounds. The shared public stream matches image-sampling conditions across evaluated models, but only one training seed was used. The 0.0570 difference is on the scale of the two reported evaluation errors and does not substitute for retraining. The four-epoch proxy recipe is also not the shipped 13-epoch formal recipe.

Decision and consequence. This run became the reference for matched four-epoch ablations and a credible fallback.