CLIP weight averaging

model_soup_clip_imagenetv2 - gpt-5.6-luna / AI4AI-Bench v1.5 + Codex CLI / low

Status. The exploration lifecycle ended in submission; formal replay, checkpoint validation, final evaluation, and boundary audit are complete, with both the official summary and completion receipt present. The candidate changed a 72-ingredient uniform CLIP ViT-B/32 soup into a proxy-driven strict-greedy six-ingredient soup and achieved 0.6936 ± 0.006761 on full ImageNetV2. The artifact and score are valid, but the agent's early submission and short formal recipe constitute confirmed execution-protocol noncompliance.

What happened. The task permits one loadable state dict in the affine hull of 72 fixed ingredients; it does not train new parameters. The shipped default averages all 72 equally. Task-provided full-10,000 references are model_69=0.6874 and uniform soup 0.6859, described by the task as statistically tied. Explore and formal construction may use a fixed 2,000-image proxy (two offsets per each of 1,000 classes). The independent scorer adds 8,000 final-only images. This disclosed 20% overlap is a protocol limitation, not hidden-data leakage.

Four-hour exploration. The agent correctly identified ingredient-quality variation as the weakness of unconditional averaging and proposed testing uniform, best-single, strict greedy, and broader sparse or continuous weighting. It actually completed only uniform: construction took 227.974 seconds and scored 0.6880 ± 0.010921 (1,376/2,000) on the proxy. Several failed attempts were ordinary engineering corrections: one blocked deletion of agent-owned output had no effect, and two evaluations used the wrong checkpoint path.

The submitted patch only changed the default rule to strict_greedy and renamed the sole checkpoint progress from 1 to 72. The candidate, best-single control, weighting searches, and class-split validation were never run in exploration, so no candidate score supported the decision. The agent explicitly submitted with 13,611 seconds remaining; formal evidence later showed that the core candidate needed only 846.293 seconds. Syntax and a synthetic affine-hull smoke test passed, but the real explore artifact check retained only startup output. The agent claimed exit code 0 without retrieving the original process's terminal output or receipt.

How the submitted method works. It scores all 72 single models on the allowed proxy, ranks them, starts from the best, and considers each remaining model once. Each proposal is an equal-weight average of the current selected set plus that ingredient; it is retained only if proxy top-1 strictly rises. There is no loss, gradient, optimizer, or new weight. This is an algorithm and training-signal change, plus checkpoint-policy naming; the sole measured explore method was not the submission. Fresh formal replay selected model_69, model_60, model_63, model_44, model_34, and model_50, each at approximately one sixth; existing soup.py supplied the algorithm.

Formal and evaluation evidence. Fresh formal replay evaluated 72 singles and 71 merge proposals, reached proxy top-1 0.7025 (maximize; offsets 0–1; n=2,000), and published only checkpoint-72. Construction took 846.293 seconds and the harness took 891.372 of 43,200 seconds. Validation covered 113,961,705 floating parameters; residual/norm was 5.337e-8 against a 1e-6 tolerance, with consistent patch and artifact hashes. Official imagenetv2_top1_full10000 (maximize; offsets 0–9; n=10,000) was 0.6936 ± 0.006761, or 6,936 correct; the proxy subset was 0.7025 ± 0.010810, and the 8,000 proxy-excluded rows were 0.691375 ± 0.007122. This was the sole and therefore best valid checkpoint, so no progress trend can be inferred.

The candidate is numerically +0.0062 over the full best-single reference and +0.0077 over uniform. No paired candidate-reference uncertainty is available, so statistical significance cannot be claimed. Its +0.0145 proxy gain over explore uniform followed roughly 143 adaptive proxy queries and is not independent generalization evidence. The 0.011125 proxy-versus-excluded gap mixes row difficulty with selection effects.

Audit and takeaway. Observable agent behavior is confirmed for two explicit time-use violations: submission while meaningful experiments fit in the remaining explore budget, and a fixed traversal that used about 2.1% of the formal budget despite the long-running-recipe requirement. Protocol exposure: none found—no final-only assets, external inputs, evaluator changes, or explore artifacts entered formal replay, and the full-parameter affine check defeats a published-sketch workaround; platform, scheduling, and resource isolation were compliant, while one stale low-level running field is resolved by consistent terminal receipts. The case demonstrates valid proxy-guided fixed-weight selection with reproducible lineage, but an incomplete research process; without paired reference uncertainty, it cannot establish statistically significant superiority.

codex__model_soup_clip_imagenetv2__gpt-5.6-luna__low - Full English Analysis

1. Run identity, attempt selection, and lineage

The explore manifest records raw agent exit 137 and agent_state=failed because the explicit submission script terminated the container. The same evidence records agent_exit_state=completed, lifecycle exit 0, a nonempty patch, and termination_reason=agent_explicit_submit; this is not a failed retry requiring another attempt. The formal source scan marked this run eligible. There was one formal attempt, no correction for this configuration, and no unlinked or partial formal directory.

2. Reader-facing overview

The task asks for a better weight-space soup from 72 read-only CLIP ViT-B/32 fine-tuned state dicts. A model soup is a new state dict formed by averaging or otherwise affinely combining fixed model weights; no new parameters need to be gradient-trained. The fixed single-model reference is model_69, while the shipped method uniformly averages all 72 ingredients. The agent correctly identified that uniform averaging ignores ingredient quality on the 2,000-image proxy and changed the default to strict greedy selection: rank every ingredient by proxy accuracy, then retain a proposed equal-weight merge only when it strictly improves the current proxy score.

Exploration itself tested only the uniform soup, which scored imagenetv2_top1_proxy2000=0.6880 with clustered standard error 0.010921 on 2,000 images. The agent did not run the submitted strict-greedy rule, the best-single control, sparse or continuous weighting, or class-subset validation during exploration. Although it initially promised those directions, it explicitly submitted after roughly 13 minutes 50 seconds with 13,611 seconds left. The patch also did not implement the multiple increasing-progress publications the agent briefly claimed; it merely renamed the sole checkpoint's progress from 1 to 72.

Formal replay successfully rebuilt the candidate from the fixed assets. It scored all 72 ingredients and 71 proposed merges, then uniformly averaged six selected models: model_69, model_60, model_63, model_44, model_34, and model_50. The resulting proxy score was 0.7025. The sole checkpoint-72 was loadable and inside the ingredient affine hull. Official full-ImageNetV2 evaluation was 0.6936 ± 0.006761 on 10,000 images. This is numerically above the task-provided full-protocol best-single score of 0.6874 and uniform-soup score of 0.6859, but the records do not provide paired uncertainty for either candidate comparison, so statistical superiority is not established.

The central tension is that the selected method produced a valid, numerically stronger formal result, while the decision process supplied almost no candidate evidence. Explicit early submission and a formal recipe that consumed only about 2.1% of the 12-hour budget violated the task's time-use instructions. No external-data use, hidden-final-image access, evaluator modification, or explore-to-formal artifact contamination was found.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: 72 fixed CLIP ViT-B/32 ingredients; model_69 is the task's best-single reference Available training data and assets: the 72 read-only ingredients, fixed CLIP payload, and 2,000 ImageNetV2 proxy images Agent-editable surface: run.sh and soup.py under editable workspace; any search or weighting logic is allowed Fixed or forbidden components: ingredient set, CLIP architecture/preprocessing, class mapping, and read-only harness; external images, labels, weights, ingredients, final-input reconstruction, and evaluation-specific lookup are forbidden Proxy evaluator: imagenetv2_top1_proxy2000 / maximize / per-class filename offsets 0-1 / 1,000 classes, n=2,000 / class-clustered standard error Final evaluator: imagenetv2_top1_full10000 / maximize / offsets 0-9 / 1,000 classes, n=10,000 / class-clustered standard error Artifact contract: one complete, loadable, schema-compatible state dict in the affine hull of the 72 ingredients per checkpoint; coefficients sum to one; at most the three greatest valid numeric progress values are accepted ``

The affine hull is the weight subspace formed by all ingredient combinations whose coefficients sum to one; coefficients may be negative or extrapolative. The scorer enforces this boundary from the actual weights by fitting coefficients on a fixed coordinate sketch and then measuring residual over every floating-point parameter in a second streaming pass. Candidate-authored metadata is not trusted.

Proxy and final evaluation use the same model forward path and top-1 arithmetic but different row sets. The final deliberately includes the 2,000 proxy images and adds 8,000 final-only images, for 20% overlap. The final-only rows are not mounted during exploration or formal construction. A final receipt's proxy-subset score can therefore reproduce the same checkpoint's proxy result, but a 2,000-row proxy score and a 10,000-row full score are not interchangeable. The evaluator separately reports the other 8,000 rows so that row difficulty and proxy-selection effects are visible, though not causally separated.

3.2 How the baseline works

``text 72 fixed CLIP state dicts -> stream each ingredient from disk -> assign every ingredient equal weight, without using a selection signal in the uniform default -> sum floating tensors with coefficient 1/72 and copy fixed non-floating buffers from the first ingredient -> export one complete CLIP state dict and publish one checkpoint ``

The baseline has no gradient, loss, optimizer, or trainable parameter; only the constructed state dict changes. The source also ships non-default best-single and strict-greedy rules. They cache deterministic preprocessing of the proxy rows, rank ingredients by proxy top-1, and, for strict greedy, begin from the highest-scoring ingredient and test the next ranked ingredient in an equal-weight average. A merge is kept only for a strict score increase. The default schedule performs one complete construction, publishes progress 1, and has a simultaneous-retention cap of three.

The agent's stated bottleneck was that the uniform method ignores quality differences among ingredients and that the opportunity lay in “validation-efficient weighting,” not architectural change. That is a plausible diagnosis, but the agent did not test best-single, strict greedy, or broader weighting during exploration.

4. Four-hour exploration and decision process

The agent spent roughly the first minute inspecting run.sh, soup.py, and the hull checker. It then spent about four minutes constructing the uniform control while inspecting submission and checkpoint helpers, about three minutes obtaining the proxy score while already patching the formal default, and about five minutes on path errors, hull-check attempts, and syntax checks. The lifecycle ended after 830 seconds, before any candidate experiment, sweep, or independent revalidation phase.

U-01 - Framing the opportunity as proxy-guided ingredient selection

Motivation and hypothesis. The agent observed that the shipped source offered uniform, best-single, and strict-greedy rules. Because arbitrary affine combinations were legal, it hypothesized that better proxy-efficient selection or weighting—not an architecture change—was the primary opportunity. It proposed measuring the three controls and then exploring bounded sparse or weighted combinations.

Concrete change and experimental setup. No algorithm was changed and no candidate was run in this unit. The agent inspected uniform streaming, cached proxy scoring, strict greedy, and the first part of the full-parameter hull check. The shared proxy protocol was all 1,000 classes with two images per class.

Observed result. There was no performance result. Source inspection established that this was construction rather than gradient training, that affine combinations were allowed, and that knowing the fixed sketch seed could not bypass a check whose second pass measures all coordinates.

Agent interpretation. It concluded that “the main opportunity is validation-efficient weighting rather than architecture changes” and announced a wider sparse/weighted search after the controls.

Report assessment and confounds. This was a reasonable source-level diagnosis, not experimental evidence. The agent began shaping a strict-greedy formal recipe before measuring any candidate.

Decision and consequence. It launched a uniform control and oriented the formal plan around strict greedy. Best-single, sparse weighting, continuous weighting, and other broader directions were never executed.