CLIP weight averaging

model_soup_clip_imagenetv2 - claude-opus-5 / Claude Code / xhigh

Status. Exploration and explicit submission completed. formal replay 1 was authoritatively classified as an infrastructure failure after shared-storage Errno 70; the identical-hash retry succeeded, all three artifacts validated, and all three official final evaluations have both summary.json and .complete. Observable agent behavior is nevertheless confirmed noncompliant because of premature submission.

What happened. The task combines 72 frozen CLIP ViT-B/32 state dictionaries into a sum-one affine soup. The baseline streams their uniform average without reading labels; its task-package full score is 0.6859, statistically tied with best ingredient model_69 at 0.6874. The agent diagnosed that maximizing the visible 2,000-image proxy was easy but selecting a rule that transfers beyond repeatedly queried rows was hard. It submitted a long-running complementary-half rule selector, whose best official full score was 0.6959.

Four-hour exploration. Proxy top-1 is maximized on 2,000 images, two per class. First, a 30.6 GiB resident GPU matrix reduced a soup score from roughly 250 seconds to 0.39 seconds; three internal scores matched official fast evaluation, so this engineering change was adopted. Individual model_69, best top-six averaging, and full-proxy greedy scored 0.6935, 0.6945, and 0.7025. Greedy fit on one fixed offset held 0.708 on the other, which the agent initially called transfer, but reversing the offsets held only 0.678; greedy became insurance rather than the final rule.

Direct cross-entropy optimization lowered loss but lost held accuracy late, changed directionally, and nearly collapsed at learning rates 0.5 and 2.0, so it was rejected. Calibrated soft greedy also reversed by fold; exhaustive top-eight subsets overfit their fit half. Shrinkage, mild extrapolation, greedy multiplicity, and bagging produced scattered signals: exhaustive/soft selection was rejected, while extrapolation and resampling variants remained candidates. The agent then compared 29 rules—greedy variants, stability, consensus, bagging, top-k, blends, and softmax weights—on random complementary halves. At seed 42 with 10 directed splits, 50% consensus held 0.6947 while greedy fit/held was 0.7050/0.6897; at seed 7 with six splits, stable greedy led at 0.6933. Changing leaders motivated formal repetition rather than a fixed exploration weight.

Ten-split evidence also put best-single below uniform, so the agent removed its forced “floor” slot and selected highest held mean, best unused-family lower bound, and a third family. Tests exposed zero-gain handling, candidate-exception isolation, and a missing import; all were fixed. Obsolete probes were stopped and one slow probe remained partial, while final two-seed exports and affine checks passed.

How the submitted method works. Baseline flow is frozen states → label-free uniform average → one state dictionary. Candidate flow is the same states plus true proxy labels → fast candidate correctness vectors → repeated fit/held rule comparisons → all-row refit of three mechanism families → float64 reconstruction and one float32 cast. It has no optimizer, trainable parameter, generated label, external data, or reused exploration artifact. The change is search algorithm, sampling, schedule, engineering, and checkpoint policy, not model training.

Formal and evaluation evidence. Orchestration overrode source fallbacks with a 42,600-second maximum and 900-second reserve. Seed 42 completed 258 directed splits (129 complementary pairs), 95,876 forwards, and 38,015/43,200 formal seconds before the 92% phase share stopped it. Four checkpoints were cumulatively published—insurance plus three finals—but retention kept only the latter three. All are affine-valid. On imagenetv2_top1_full10000 (maximize, n=10,000), stable-greedy 37956 scored 0.6877±0.006602 clustered SE, top-five 37957 scored 0.6895±0.006712, and greedy-extrapolation 37958 scored 0.6959±0.006645. Their other-8,000 scores were 0.685375, 0.688750, and 0.693875. These adjacent progresses are different methods, not a learning curve. Held ordering matched final ordering, but proxy-row advantages reveal selection bias. No paired difference error supports a significance claim against the references.

Audit and takeaway. Transcript and patch use only fixed proxy data, ingredients, and CLIP; no network, external model, hidden-image reconstruction, evaluator replacement, or explore-to-formal reuse was found. Reading accessible evaluator source revealed only the disclosed offset rule, so protocol exposure and task-package defects are absent. Successful replay used one isolated permitted GPU; platform status is a confirmed storage defect on exploration attempt 1, resolved by exact-hash retry. Agent behavior is confirmed because submission left 2,714 seconds while a demonstrated complementary-pair experiment required under 600 seconds, contrary to an explicit condition. This limits exploration completeness but not formal lineage or scoring. The case shows strong counterexample-driven redesign; it does not establish statistical significance or generalization beyond this protocol.

claude__model_soup_clip_imagenetv2__claude-opus-5__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

The explore manifest's auto_retrain=false and retrain_phase_created=false mean only that this explore directory did not create its own formal phase. The analysis manifest, numbered directories, queue, statuses, and correction establish two later upstream formal dispatches, so those explore fields are not used to infer non-scheduling.

2. Reader-facing overview

This task does not train a new CLIP model. It asks for an affine combination of 72 frozen ViT-B/32 state dictionaries. The shipped baseline streams the ingredients into their uniform average and publishes one complete state dictionary. Its task-package final score is 0.6859, statistically tied with the best individual ingredient, model_69, at 0.6874. The agent identified the central limitation correctly: maximizing accuracy on the visible 2,000-image proxy is easy, but repeated proxy queries can select noise that does not transfer to images excluded from selection.

The agent first made soup evaluation fast enough for statistical experiments by placing the 72 flattened states in a 30.6 GiB GPU matrix, reducing a proxy score to about 0.39 seconds. It then tested individual ranking, top-k averages, greedy selection, cross-entropy coefficient optimization, calibrated soft selection, exhaustive subsets, shrinkage and extrapolation, stability resampling, consensus, and bagging. A promising one-way held-out greedy result failed when the two fixed offsets were reversed. That counterexample changed the submission into a long-running procedure that compares 29 construction rules over repeated complementary half-splits, refits winners on all proxy rows, and exports three distinct mechanism families.

formal replay 1 failed at 11,680.9 seconds with a shared-filesystem Errno 70; control-plane review classified it as infrastructure and retried the identical patch. exploration attempt 2 completed 258 directed held-out splits—129 complementary pairs—95,876 proxy forwards, and three valid artifacts. Official imagenetv2_top1_full10000 scores, where higher is better, were 0.6877, 0.6895, and 0.6959 for progress 37956, 37957, and 37958. Each result has a completion receipt, and greedy extrapolation artifact 37958 is the official best.

The strongest capability shown is evidence-driven self-correction: the agent withdrew an initially confident transfer claim after a reverse-fold counterexample and encoded the uncertainty into the formal recipe. The main scientific limitation is that all rule selection still reuses the same 2,000 proxy images, so split-level standard errors omit correlated reuse and model-selection uncertainty. There is also confirmed execution-protocol noncompliance: the agent explicitly submitted with 2,714 seconds left even though at least one more experiment of the demonstrated type could finish and be interpreted. That limits research completeness but does not break the patch lineage, formal reconstruction, or the official final scores.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: 72 fixed fine-tuned CLIP ViT-B/32 state dictionaries; the training-start reference is the best ingredient, model_69 Available training data and assets: the 72 read-only ingredients, fixed CLIP payload, and labeled 2,000-image ImageNetV2 proxy Agent-editable surface: source under editable workspace for selecting, weighting, searching, constructing, and publishing soups Fixed or forbidden components: ingredients, architecture, preprocessing, class mapping, and final evaluator; no external images, labels, weights, or ingredients, no reconstruction of final-only inputs or lookup logic, and no network Proxy evaluator: imagenetv2_top1_proxy2000 / maximize / 1,000 classes, offsets 0-1 / n=2,000 / class-clustered stderr in official fast-eval receipts Final evaluator: imagenetv2_top1_full10000 / maximize / 1,000 classes, offsets 0-9 / n=10,000 / class-clustered stderr plus proxy-row and other-8,000 diagnostics Artifact contract: one compatible, loadable full state dictionary in the affine hull of the 72 ingredients; coefficients may be negative but must sum to one; at most three valid retained checkpoints, with the best official final score counting ``

The proxy is a disclosed 20% subset of the final set, not an independent sample: it contains two fixed offsets per class, while the other 8,000 images are physically absent from exploration and formal construction. Proxy and full scores therefore differ in sample count, row set, and use in selection and must not be treated as independent, directly comparable measurements. The evaluator's final_minus_proxy_rows and overfitting fields are the protocol-defined diagnostics. Class-clustered standard errors are the relevant uncertainty estimates; naive binomial errors in the summaries are diagnostic only.

3.2 How the baseline works

``text [72 frozen CLIP state dictionaries] -> [stream each state and add every floating tensor with weight 1/72] -> [the default uniform rule reads no proxy labels and has no reward or loss] -> [normalize coefficients to sum to one and form one complete averaged state] -> [change only the exported combination; do not train or alter any ingredient] ``

Baseline run.sh defaults to uniform and publishes one checkpoint at progress 1. Reference alternatives in the source can choose the best individual model on the proxy or greedily accept ingredients that strictly raise proxy accuracy, but the shipped default uses neither. Streaming keeps host memory near one state rather than all 72, but a new candidate still requires reconstructing and forwarding a large model; the task reports 248.13 seconds for the uniform construction. The agent's stated diagnosis was that only two visible images per class make proxy maxima easy to obtain and hard to distinguish from selection noise.

4. Four-hour exploration and decision process

The agent began by inspecting the task boundary, baseline, forward path, artifact gate, and final evaluator. It then spent the early part of the run building the resident-matrix evaluator and establishing simple baselines; the middle tested discrete, gradient, and regularized alternatives; the later part converted contradictory folds into a repeated complementary-split rule selector and revalidated export, retention, and affine-hull behavior. Probes whose source became obsolete or whose execution slowed were explicitly stopped rather than treated as completed evidence. Exploration used 11,744 of 14,400 seconds, or 81.6% of the budget.

U-01 - Can soup scoring be accelerated enough to study selection variance?

Motivation and hypothesis. Baseline reconstruction made the number of statistically meaningful comparisons too small. The agent hypothesized that keeping all ingredients as one flattened GPU matrix and caching decoded proxy images would turn each soup into a coefficient-vector matrix product.

Concrete change and experimental setup. It implemented Lab and Ranker, loading a 72 by 113,961,705 matrix on the GPU, exposing parameter views to the unchanged model forward path, and memoizing correctness vectors for repeated integer-count structures. Data, preprocessing, labels, and the 2,000-row accuracy metric were unchanged.

Observed result. Matrix initialization used about 30.6 GiB and peak telemetry was 35.7 GiB. Uniform proxy accuracy was 0.6880, and a new soup could be scored in about 0.39 seconds instead of the roughly 250-second construction scale. Official fast evaluation of three temporary exports returned 0.6935, 0.6995, and 0.6970, exactly matching the internal scores.

Agent interpretation. The agent treated the speedup as what made repeated “honest” held-out comparisons possible, not merely as a runtime improvement.

Report assessment and confounds. The fast-evaluator agreement supports metric equivalence. The implementation traded speed for high device memory, but it used one permitted L20D with no observed external process on the selected GPU, so there is no evidence of resource overreach.

Decision and consequence. The resident evaluator became final souplab.py and enabled nearly 100,000 formal proxy forwards.