CLIP weight averaging

model_soup_clip_imagenetv2 - gpt-5.6-sol / Codex CLI 0.146.0 / xhigh

Status. Exploration and explicit submission completed. formal replay 1 was invalidated because an external process shared its GPU; exploration attempt 2 replayed the identical patch on a clean GPU, completed, validated three artifacts, and obtained three official final results. Observable agent behavior is clean, no protocol exposure was found, and platform isolation has a confirmed defect.

What happened. The task asks for a better affine combination of 72 frozen CLIP ViT-B/32 checkpoints. The shipped baseline uniformly averages all 72; the fixed start is the best ingredient, model_69. The agent diagnosed dilution and submitted proxy-guided sparse coefficient search. Its best published triad scored 0.6918 on full ImageNetV2 versus the official anchor's 0.6874.

Four-hour exploration. All search used the maximize-direction imagenetv2_top1_proxy2000 on 2,000 images, with uncertainty clustered over 1,000 classes. Full ranking reproduced model_69 at 0.6935 (SE 0.010859) and ranged down to 0.6340; the all-72 proxy soup scored 0.6880, supporting sparse search. A 191-candidate study tested top-k averages, shrinkage and negative extrapolation, anchor pairs, and strict greedy subsets. Broad averages and extrapolation degraded; 69/63=0.65/0.35 scored 0.7000, while a six-model greedy soup reached the raw-best 0.7025 and no seventh model helped.

Local pair scans, six-model reweighting, a 69/63/44 simplex, and a 25-point triad refinement found 0.7020 triads. The pair's gain over the anchor was 0.0065 with paired class SE 0.00350; the triad exceeded the pair by only 0.0020, so the agent retained distinct complexity roles rather than treating the raw order as certain. Thirty-two targeted lower-ranked pairs and 108 small-step candidates covering all previously missed ingredients failed to beat the pair's uncertainty-adjusted evidence. An 18-candidate pair-plus-third scan selected 69/63/34=0.6175/0.3325/0.05 at 0.7020, paired SE 0.00383; a separate model_47 triad tied it.

An initial patch check failed from the wrong directory, then clean-source replay produced byte-identical files. A 360-second smoke safely exported three anchor fallbacks but supplied no performance evidence. The final triad passed independent evaluation, invalid-coefficient tests, and the 113,961,705-dimensional affine-hull check with about 28.5x margin. Submission left 911 seconds; no concrete new low-dimensional hypothesis remained that measured runtimes showed could be completed and artifact-validated in that window.

How the submitted method works. The baseline averages 72 frozen states without labels, a loss, or updates. The candidate caches fixed proxy inputs, ranks ingredients, runs top-k, line, greedy, pair, triad, lower-ranked, and broad-pair families, then uses remaining wall time for deterministic sparse proposals. Proxy correctness, both offset scores, and paired class uncertainty select coefficients; ingredients remain frozen. Changes span algorithm, signal, hyperparameters, schedule, engineering, and checkpoint policy, while data remain fixed. Three role files publish the anchor, a support-at-most-two robust model, and a support-at-most-three general model. The 0.7025 exploration best was not published; formal search also found an unpublished ten-coefficient proxy state at 0.7085, an important opportunity cost.

Formal and evaluation evidence. exploration attempt 2 applied the matching SHA-256 patch to fresh source, used 41,652.291/43,200 seconds, evaluated 2,522 complete candidates, and stopped under the injected wall clock with exit 0. All artifacts were affine-valid. Progress numbers are role indices emitted together, not a learning curve.

| Checkpoint | imagenetv2_top1_full10000 (maximize, n=10,000) | Clustered SE / diagnostic | Outcome | |---|---:|---|---| | 1, model_69 | 0.6874 | 0.006722; final-only 8,000 rows 0.685875 | Control | | 2, 69/63 | 0.6906 | 0.006697; final-only 0.688250 | Accepted | | 3, 69/63/34 | 0.6918 | 0.006684; final-only 0.689250 | Official best |

A report-side paired calculation gives triad-minus-anchor 0.0044 with class SE about 0.00192; triad-minus-pair is only 0.0012 with SE about 0.00104. Proxy gains therefore survived outside the proxy rows, but superiority over the pair is unresolved and proxy-versus-rest gaps warn of selection optimism.

Audit and takeaway. Literal cross-source audit first identified all 8,000 final-only image identifiers from evaluator-only rows, then found zero matches in the complete trajectory, tool outputs, experiment commands, or patch. The agent used only fixed ingredients and proxy data, did not alter evaluators or import external inputs, and formal replay mounted the patch rather than exploration artifacts. exploration attempt 1's confirmed platform contamination was fully excluded; exploration attempt 2 passed repeated idle-gate samples and preserves validity. The agent showed systematic search coverage, paired-statistical restraint, and strong replay engineering. Evidence supports triad-over-anchor on this fixed protocol, not triad-over-pair, the unpublished high-dimensional state, or an independent distribution.

codex__model_soup_clip_imagenetv2__gpt-5.6-sol__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

The selected exploration is evidenced by the raw trajectory, explore manifest, and completion receipt. Attempt selection follows the infrastructure correction. Evidence for the usable replay includes the exploration attempt 2 manifest, its per-attempt status with the stale field, and the merged authoritative job status.

2. Reader-facing overview

The task asks for a better weight-space combination of 72 frozen CLIP ViT-B/32 fine-tuned checkpoints. Its training-start reference is the best single ingredient, model_69, whereas the shipped baseline ignores labels and uniformly averages all 72. The task reports full-ImageNetV2 scores of 0.6874 and 0.6859 respectively, a paired statistical tie. The agent established that ingredient quality was highly heterogeneous and reframed the problem from broad averaging to finding a few complements to the strong anchor.

The four-hour study covered all-ingredient ranking, top-k averaging, shrinkage and extrapolation, anchor pairs, greedy subsets, two- and three-dimensional coefficient refinement, lower-ranked complements, an exhaustive small-step pair sweep, paired uncertainty, clean patch replay, and affine-hull checks. The highest raw exploration proxy score was 0.7025 from a uniform six-model soup. Because its advantage over simpler candidates was much smaller than paired uncertainty, the submitted publication policy instead protected three roles: the anchor, a robust model_69/model_63 pair, and a sparse model_69/model_63/model_34 general candidate.

The valid formal replay consumed 41,652.291 of 43,200 seconds and evaluated 2,522 complete candidates before exporting three valid artifacts. Their official imagenetv2_top1_full10000 scores were 0.6874, 0.6906, and 0.6918 on 10,000 images. The sparse three-model soup was best. A report-side paired calculation from the official row files gives its improvement over the anchor as 0.0044 with a class-clustered paired standard error of about 0.00192; its 0.0012 advantage over the pair remains unresolved.

Observable agent behavior is clean: no hidden final input, external research data, or exploration artifact entered formal replay. A platform process did share the first formal attempt's GPU, a confirmed isolation defect; the correction discarded that entire attempt, and a clean-GPU rerun produced all official results. The main scientific limitation is proxy reuse and selection bias. In addition, formal search found an unpublished 0.7085 high-dimensional proxy candidate, so its final performance is unknown.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: model_69, the best of 72 frozen ingredients; not the original CLIP base Available training data and assets: 72 read-only ingredients, fixed CLIP ViT-B/32 payload, and 2,000 ImageNetV2 proxy images Agent-editable surface: construction/search logic, coefficient selection, engineering, and checkpoint policy under editable workspace Fixed or forbidden components: ingredients, architecture, preprocessing, class mapping, and read-only evaluators; no external images, labels, weights, or reconstruction of final-only inputs Proxy evaluator: imagenetv2_top1_proxy2000 / maximize / per-class sorted offsets 0-1 / n=2,000 / standard error clustered over 1,000 classes Final evaluator: imagenetv2_top1_full10000 / maximize / offsets 0-9 / n=10,000 / standard error clustered over 1,000 classes Artifact contract: one compatible complete state dict in the affine hull of the 72 ingredients; at most three valid checkpoints are accepted and the best final score is official ``

Proxy and final evaluation share ImageNetV2, preprocessing, and deterministic inference, but they have different sample counts and roles. The proxy is a materialized 20% projection of the final set; final receipts separately report those 2,000 rows and the other 8,000. They are neither independent replications nor interchangeable estimates, and adaptive proxy selection creates additional optimism. The contract is defined by the instruction, task configuration, asset lock, and metric implementation.

3.2 How the baseline works

``text [72 fixed state dictionaries] -> [stream each one and accumulate it with the same coefficient] -> [uniform construction reads no labels and has no reward or target] -> [take the arithmetic mean; there is no loss, gradient, or optimizer] -> [only the combined output state dict changes; no ingredient parameter is updated] ``

The baseline defaults to uniform, streams the weights to avoid holding roughly 30.6 GiB of ingredients at once, and saves one progress-1 checkpoint. Its source also contains best-single and strict-greedy modes, but the shipped default does not invoke them. The agent's initial diagnosis had two concrete parts: weak ingredients dilute the already strong model_69, and only two proxy images per class make small raw score differences vulnerable to selection noise. See the baseline run.sh and soup.py.

4. Four-hour exploration and decision process

The agent first established the boundary and independently measured the anchor, then spent about 79 minutes on a 191-candidate structured search. The middle of the run refined promising low-dimensional families, computed paired statistics, and exercised a formal-start smoke path. The later phase deliberately expanded coverage to lower-ranked and previously untested pair directions, followed by sparse three-model tests. The last half hour encoded the new result, replayed the patch, independently rescored the exact final candidate, and ran the full affine-hull check. Total exploration elapsed time was 13,528 seconds.

U-01 - Establishing the anchor and ingredient heterogeneity

Motivation and hypothesis. The agent needed to confirm that model_69 was a sound proxy anchor and determine whether uniform averaging failed because of systematic quality variation rather than an incidental reference measurement.

Concrete change and experimental setup. It independently evaluated model_69, then ranked all 72 frozen ingredients on the same 2,000 proxy rows. Inference and coefficients were deterministic; seed 42 made the construction path reproducible rather than creating a seed sweep.

Observed result. model_69 scored 0.6935 with class-clustered standard error 0.010859. Other leaders included model_27 at 0.6910, model_60 at 0.6895, model_43 at 0.6880, and model_48 at 0.6870; model_31 was only 0.6340. The all-72 uniform candidate within the structured run scored 0.6880 on the proxy.

Agent interpretation. It concluded that the set was highly heterogeneous and that broad averaging mixed rare complementarity with widespread dilution. Subsequent search should remain near model_69 and sparse.

Report assessment and confounds. Adjacent single-model ranks are not individually decisive because their differences are small relative to the per-score uncertainty and the same proxy drives later selection. The overall 0.6340-to-0.6935 range, however, supports the directional heterogeneity diagnosis.

Decision and consequence. model_69 became the fixed reference for paired deltas and checkpoint 1; the full uniform soup was rejected as the principal candidate.