CLIP weight averaging

model_soup_clip_imagenetv2 - gpt-5.6-sol / Codex CLI / medium

Status. Exploration, submission, formal replay, three artifact validations, three final evaluations, and the boundary audit completed; every official final has both summary.json and .complete.

What happened. The task asks for a better affine-hull model from 72 frozen CLIP ViT-B/32 state dictionaries. The baseline performs no gradient training: it averages all ingredients equally and publishes one model. Its 0.6859 full-ImageNetV2 score is statistically tied with the 0.6874 best-single start, model_69. Diagnosing dilution, the agent submitted a five-ingredient signed soup plus deadline-bounded affine search. Two artifacts tie at the best official 0.6950.

Four-hour exploration. Search used the fixed 2,000-image proxy top-1 metric, maximized. Ranking all ingredients put model_69 first at 0.6935. Equal averages of the top 1–24 peaked at only 0.6945 and worsened at larger k, so they were rejected. Seventy-five pair interpolations found a smooth model_44 complement at 0.7015. An interrupted triple sweep was corrected for duplicate trials and rerun; adding model_63 reached 0.7020, while tested alternatives were lower.

A signed-screen helper had a syntax error. Once fixed, tested positive moves failed while several negative moves helped; after documenting a partial-screen stop, the agent completed 144 negative candidates over all ingredients. Moving away from model_13 reached 0.7060, but model_2 collapsed to 0.6500, refuting a universal negative-weight rule. Three local sweeps totaling 225 candidates reached 0.7085 and bracketed greater-extrapolation decline. Splitting negative mass across models 13 and 17 then yielded 0.7090 and a robust 0.7095 (proxy SE 0.01068). The agent treated nominal paired significance as selection-biased because roughly 900 candidates reused the proxy.

The static solution then became a long formal recipe. Smoke tests exposed duplicate-branch bookkeeping, a patch-context failure, and rounding risk; distinct export branches, finite guards, and one-pass reconstruction fixed them. Short replay proved loadability, not performance. Submission left about 50 minutes unused; because no specific omitted study is proven completable and interpretable, this is a completeness limitation rather than confirmed noncompliance.

How the submitted method works. Baseline flow is ingredients, no selection signal, equal tensor average, one artifact. The candidate uses permitted proxy labels for signed coordinate proposals, accepting lower cross-entropy while accuracy remains at least 0.700, then exports three artifacts. Seed 42 controls order; coefficient magnitude, L1 norm, and finite values are constrained. No ingredient or neural parameter is trained. The changes are coefficient-search objective/rule, wall-clock engineering, and checkpoint policy. The 0.7095 explore-best is the robust branch; fresh formal replay uses only the identical-hash patch and fixed assets.

Formal and evaluation evidence. The orchestrator's 42,600-second deadline overrides the source fallback within a 43,200-second budget. Formal construction exited normally after 41,687 seconds: 4,679 proposals, 57 cycles, and 102 accepted moves. Progress 1/2/3 are publication labels, not training steps; all three retained artifacts validated in-hull.

On imagenetv2_top1_full10000 (maximize, n=10,000), progress 1 scores 0.6950 with clustered SE 0.00667 and proxy 0.7095; progress 2 scores 0.6950 with SE 0.00670 and proxy 0.7090; progress 3, the minimum-cross-entropy solution, scores 0.6946 with SE 0.00676 and proxy 0.7005. The first two tie for best. The proxy is 20% of the final set, not an independent holdout: on the other 8,000 rows the three scores are 0.691375, 0.6915, and 0.693125. Proxy advantage therefore shrinks, but subset difficulty and selection bias are confounded. The 0.6950 point estimate is +0.0076 over the fixed start and +0.0091 over the uniform baseline; no paired final test supports a significance claim.

Audit and takeaway. Direct evidence shows only fixed ingredients and proxy data; final-only values were neither exposed, reconstructed, used, nor incorporated. Explore and formal each used one assigned GPU without external sharing; no external input, explore-artifact reuse, wall overrun, or unauthorized concurrency was found. Job terminal state and completion receipts resolve stale attempt-level running fields. Agent behavior is clean, boundary exposure none found, and platform/resource isolation compliant. Disciplined ablation and replay engineering improved the point estimate, but repeated proxy selection precludes statistical-significance or split-independent-generalization claims.

codex__model_soup_clip_imagenetv2__gpt-5.6-sol__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks the agent to build a better CLIP ViT-B/32 weight-space soup from 72 frozen fine-tuned state dictionaries. The shipped baseline does not train the network: it streams all 72 ingredients with equal coefficients and publishes their parameter-wise average. That uniform soup scores 0.6859 on full ImageNetV2, statistically tied with the 0.6874 best-single starting ingredient, model_69. The agent diagnosed indiscriminate averaging as dilution of the strongest ingredient and of a few useful complementary directions.

Exploration ranked every ingredient, rejected broad equal-weight top-k soups, found positive pair and triple complements, screened negative extrapolation directions, refined the best local surface, and tested whether splitting negative mass over two ingredients was more stable. The submitted method preserves a five-ingredient extrapolative solution as a robust branch, then uses proxy accuracy and cross-entropy in a wall-clock-bounded affine coordinate search from a conservative three-ingredient start. It exports robust, alternate-accuracy, and minimum-loss state dictionaries.

Formal replay completed successfully. The candidate ran for 41,642 s inside a 41,687 s formal construction phase, evaluated 4,679 coefficient proposals, completed 57 cycles, and accepted 102 moves. All three artifacts are loadable and in the affine hull. Their official full-ImageNetV2 scores are 0.6950, 0.6950, and 0.6946, with class-clustered standard errors near 0.0067; the first two tie for best. The best point estimate is 0.0076 above the fixed start and 0.0091 above the shipped uniform soup, but the available evidence does not establish a statistically significant paired improvement.

The trajectory demonstrates strong narrowing from broad ablation to a replayable formal recipe. Its main scientific limitation is repeated selection on the same 2,000 proxy images. The proxy-leading solutions score less well on the final-only 8,000 rows, which is a selection-risk diagnostic but does not by itself distinguish overfitting from fixed-subset difficulty. No hidden-asset access, external-data use, explore-to-formal contamination, or unauthorized resource use was found.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: 72 frozen fine-tuned CLIP ViT-B/32 state dictionaries; model_69 is the fixed training-start reference Available training data and assets: the 72 read-only ingredients, fixed OpenAI CLIP ViT-B/32 payload and preprocessing, and the labeled 2,000-image ImageNetV2 proxy Agent-editable surface: selection, coefficient search, construction, engineering, and checkpoint-publication code under editable workspace Fixed or forbidden components: architecture, preprocessing, class map, read-only ingredients, and frozen evaluator; external images, labels, weights, ingredients, and reconstruction of final-only inputs are forbidden Proxy evaluator: imagenetv2_top1_proxy2000 / maximize / matched-frequency offsets 0-1, two images per class / n=2,000 / class-clustered stderr only for frozen fast_eval receipts Final evaluator: imagenetv2_top1_full10000 / maximize / matched-frequency offsets 0-9, ten images per class / n=10,000 / class-clustered stderr Artifact contract: one complete compatible state dictionary in the affine hull of the 72 ingredients; coefficients must sum to one, but negative or extrapolative coefficients are permitted ``

The proxy is fixed top-1 accuracy on 2,000 images. The final evaluator covers 10,000 images from the same benchmark: the proxy rows are a 20% subset and the other 8,000 rows are absent during explore and formal construction. Final receipts report the two parts separately. Proxy and full scores differ in sample count, selection role, and composition, so their difference is not an independent, identically distributed generalization test. Likewise, a proxy-versus-other-rows gap mixes repeated selection with any intrinsic difference between the fixed subsets.

3.2 How the baseline works

``text [72 fixed state dictionaries] -> [assign coefficient 1/72 to every ingredient] -> [use no labels, rewards, or search signal] -> [stream ingredients and average every tensor] -> [publish one complete CLIP state dictionary as checkpoint-1] ``

There are no trainable parameters, loss, or gradient updates in the default baseline. The only changed object is the exported floating-point state dictionary. The reference construction took 248.13 s, full scoring took 1,198.02 s, and peak GPU memory was 2,222 MiB. Optional best-single and strict-greedy modes exist in the source, but the shipped default constructs the uniform soup once and retains one checkpoint. The task's full-evaluation references are 0.6859 for the uniform soup and 0.6874 for model_69; their -0.0015 difference is only about 0.56 paired standard errors and must be treated as a statistical tie. The agent's explicit initial diagnosis was that averaging all ingredients dilutes strong ones, so it made targeted complementarity around model_69 the primary question.

4. Four-hour exploration and decision process

The agent first inspected the task boundary, builder, and evaluator and created a reusable proxy-scoring cache. It then spent most of the run narrowing from all single ingredients to top-k, pair, triple, signed, and local coefficient searches. The later portion revalidated the best state dictionary, checked affine-hull compatibility, and converted a static solution into a deadline-aware formal recipe. Exploration lasted about 3 h 10 min; the last phase emphasized source replay, distinct-checkpoint behavior, finite-value safeguards, and submission auditing.

U-01 - Does single-ingredient ranking expose the uniform-soup bottleneck?

Motivation and hypothesis. The agent first needed to determine whether ingredient quality varied enough to explain the uniform average's ceiling and to choose an anchor for targeted merging.

Concrete change and experimental setup. It evaluated all 72 individual state dictionaries on the fixed 2,000-row proxy and recorded aggregate accuracy plus the two 1,000-row offset accuracies. Fixed images and labels make this a deterministic comparison, not a training-seed sweep.

Observed result. model_69 ranked first at 0.6935, with offset scores 0.681 and 0.706. The next ingredients were model_27 at 0.6910, model_60 at 0.6895, model_43 at 0.6880, and model_48 at 0.6870. Frozen re-evaluation of model_69 reported class-clustered stderr 0.01086 at n=2,000. After image caching, the complete rank pass took about 520 s.

Agent interpretation. The agent took the quality gradient as support for its dilution diagnosis. It also saw model_69's offset imbalance as room for a complementary ingredient to repair offset-0 errors.

Report assessment and confounds. The ranking is reproducible, but all later choices use the same proxy. Internal ranking entries do not have independent uncertainty receipts, and proxy rank need not equal full-10,000 rank.

Decision and consequence. model_69 became the anchor, and its ranking ordered subsequent top-k and pair candidates. Uniform averaging over all ingredients was rejected as the main approach.