CLIP weight averaging

model_soup_clip_imagenetv2 - claude-sonnet-5 / AI4AI-Bench v1.5 / medium

Status. The exploration run completed and submitted; formal replay succeeded, three retained artifacts validated, and three final summaries have matching .complete receipts. Boundary review is complete: observable agent behavior was clean, no protocol exposure was found, and platform/resource isolation was compliant.

What happened. The task supplied 72 fine-tuned CLIP ViT-B/32 checkpoints and asked for a higher-accuracy weight-space model soup, meaning one model formed by linearly combining their parameters. The baseline uniformly averaged all state dictionaries without labels, loss, optimizer, or gradients; its full-set reference was 0.6859, versus 0.6874 for the best ingredient, model_69. The agent submitted repeated cross-fitted shrinkage and coefficient averaging, ultimately reaching an official best imagenetv2_top1_full10000 point estimate of 0.6927.

Four-hour exploration. Proxy results maximized accuracy on 2,000 images, two per class. After preprocessing timeouts, GPU-caching images, weights, and predictions cut candidate evaluation to about 0.4 seconds. Uniform scored 0.688 with class-clustered stderr 0.010921, while 72 single models ranged roughly 0.634–0.6935. Full-proxy strict greedy selected six models and scored 0.7025, but the agent treated this as selection-biased. Across 16 bidirectional half-folds, greedy averaged 0.69325 versus 0.68800 for uniform and 0.69350 for model_69; counterexamples reached -0.010 versus model_69. Ten smaller 20% holdouts were noisier and also failed to beat model_69, so pure greedy was rejected. A model_69/greedy blend chose its coefficient on the greedy-fitting half, a circular experiment that underperformed both. Across 12 corrected train/tune/test partitions, the same two-component blend averaged +0.00114 over model_69 with SD 0.00500 and negative cases. The agent used its lower variation to motivate bagging and added uniform shrinkage in submission; evidence remains weak because all partitions reused one proxy pool and submitted code uses two cross-tuned halves and split-selected best models. A short probe fell back to uniform; a 1,200-second probe completed 16 one-direction rounds, scored 0.6945 ± 0.010888, and produced valid affine-hull checkpoints. One malformed validator invocation was corrected, and a fresh-workspace smoke proved clean reconstruction.

How the submitted method works. Baseline flow was fixed ingredients → equal coefficients → direct parameter average → one state dictionary. Candidate flow was cached ingredient predictions → random A/B split → select A's best single and fit greedy on A → on B select a 0.1-grid convex blend of the A-selected single, A-trained greedy soup, and uniform → reverse halves → average all winning 72-dimensional coefficient vectors → export. Proxy labels were the only selection signal; there was no generated data, external input, loss, or gradient update. This changed signal, search, sampling, schedule, engineering, and checkpoint policy. The in-sample-best six-model soup was not submitted, and formal replay rebuilt solely from fixed assets.

Formal and evaluation evidence. Orchestrator overrides, not source fallbacks, set a 43,200-second outer budget, 42,600-second internal wall time, and 900-second reserve. With seed 42, replay completed 758 one-direction rounds and exited normally after 41,699.798 seconds. It cumulatively published 42 periodic plus one final checkpoint but retained only progress 40551, 41543, and 41655. All were loadable and inside the ingredient affine hull. Their maximized full-set scores at n=10,000 were respectively 0.6927 ± 0.006735, 0.6927 ± 0.006731, and 0.6926 ± 0.006730, where uncertainty is class-clustered stderr; corresponding proxy-row scores were 0.6930, 0.6935, and 0.6935. Proxy improvement over uniform survived, but none exceeded model_69's 0.6935. The first two tie for best, while the last lost one correct image, so progress was not monotonic but showed no collapse. The best point estimate exceeds both references, but no paired-difference uncertainty supports a claim of statistical significance.

Audit and takeaway. The trajectory used only the fixed proxy, ingredients, and CLIP assets; no external data/model, evaluator modification, or explore artifact entered formal replay. Source exposed split mechanics but no hidden final image content, prediction, or score; the agent neither reconstructed nor used hidden assets, and none entered the patch. One isolated GPU was used within wall-clock limits, patch hashes and the authorized orchestrator-only commit waiver preserve lineage, and stale status: running fields are outweighed by consistent terminal manifests and receipts. Submission left about 2,006 seconds; whether another meaningful experiment could finish is inconclusive, a research-completeness limitation rather than confirmed misconduct. The case demonstrates strong throughput engineering and self-critical validation, but cannot establish that repeated proxy reuse generalizes independently or that 0.6927 is significantly better than the fixed references.

claude__model_soup_clip_imagenetv2__claude-sonnet-5__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

The task was to construct a better weight-space model soup from 72 fixed fine-tuned CLIP ViT-B/32 ingredient checkpoints. A model soup is one model produced by linearly combining the parameters of compatible models. Its task family was ai4ai/model-soup-clip-imagenetv2; this was offline construction and search, not gradient training. The evaluated model was claude-sonnet-5 under the AI4AI-Bench v1.5 harness at medium reasoning effort. Candidate search used seed 42. The exploration budget was 14,400 seconds and the formal-replay budget was 43,200 seconds.

The primary explore run was exploration attempt 1. Its manifest, numbered sibling directories, and control records show no earlier full rerun and no same-configuration attempt that stopped at gating, reservation, or queueing. It ran from 2026-08-13 10:07:12 UTC to 13:34:50 UTC, explicitly submitted after 12,457 seconds, and exited with status 0.

The candidate patch changed only run.sh and soup.py. Its SHA-256 was verified private digest. The only formal run was formal replay 1, whose selected-exploration lineage identifies that explore run; the formal copy of the patch is byte-identical. Exploration launched from task-source commit 305d21c723dc…, whereas formal replay used e2c7bd967…. The orchestrator-hotfix-e2c7bd96.json receipt explicitly authorizes an orchestrator-only commit waiver and states that the task tree was unchanged, so the source transition does not break lineage.

Formal replay ran only once. The job manifest lists no unlinked or partial formal directory, and numbered-directory and control-record review found no second formal attempt, gate-only attempt, or correction run. It retained checkpoints 40551, 41543, and 41655. Each has a terminal checkpoint-validation attempt with exit status 0, a valid validation result, and a completion marker. Each corresponding final-test directory also contains both summary.json and .complete, so all three final results are official.

The direct evidence chain is complete from raw trajectory and task source through patch, formal manifest, retrain result, artifact validation, and final-evaluator completion. There is one field-level conflict: attempt-level control/status.json files for both explore and formal retain status: running, while their attempt_status fields, job-level status, manifests, process exit records, and result receipts all establish terminal or successful outcomes. This is best explained as a stale generic status field; multiple authoritative terminal records resolve the run outcome. The explore manifest's auto_retrain=false and retrain_phase_created=false mean only that this explore manifest did not create a formal phase. Separate upstream dispatch and formal receipts directly establish that formal replay occurred.

2. Reader-facing overview

The task constrained the agent to 72 supplied CLIP checkpoints, supplied CLIP code, and a 2,000-image ImageNetV2 proxy. The shipped baseline uniformly averaged all 72 state dictionaries without gradients; its reference score on the 10,000-image final protocol was 0.6859, while the best fixed ingredient, model_69, scored 0.6874. Because the proxy has only two images per class, the central scientific problem was not merely finding a high proxy score but limiting selection overfitting.

The agent first cached all weights and proxy predictions on the GPU to make model-soup evaluation sub-second. It then compared individual ingredients, an in-sample strict-greedy soup, random holdout validation, a rejected circular blend-selection experiment, and a corrected three-partition blend test. The submitted method repeatedly selects a best single model and fits a greedy soup on one random half, tunes a convex blend of those two models and the uniform soup on the opposite half, swaps the halves, and averages every winning 72-dimensional coefficient vector.

Formal replay completed 758 one-direction search rounds in 41,699.798 seconds. It cumulatively published 43 checkpoints but, under retention, left only the three latest valid artifacts. Their official imagenetv2_top1_full10000 scores were 0.6927, 0.6927, and 0.6926, with class-clustered standard errors near 0.00673. The best point estimate was 0.0053 above the fixed best ingredient and 0.0068 above the shipped uniform soup. No paired-difference uncertainty against those references is available, so the evidence establishes higher official point estimates, not statistically significant superiority.

The trajectory demonstrates strong evaluation-throughput engineering, active diagnosis of selection bias, and a credible transition from exploratory code to a long-running recoverable recipe. Its main scientific limitation is that all split experiments reused the same 2,000 proxy images, and the submitted two-half, three-component procedure is not identical to the three-partition, two-component experiment that provided its most favorable evidence. The boundary audit found no agent misconduct, hidden-final-image exposure, or explore-to-formal contamination.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

The starting material was 72 fine-tuned CLIP ViT-B/32 checkpoints with one fixed architecture; the task identifies model_69 as the best-ingredient starting reference. Available construction assets were those checkpoints, the fixed CLIP implementation and cache, and the ImageNetV2 proxy. The agent could edit the solution's construction algorithm, search schedule, engineering, and checkpoint policy. It could not change the architecture, preprocessing, class mapping, evaluator, or fixed mounts, and it could not use external images, labels, weights, ingredients, or final-only content.

The proxy metric was imagenetv2_top1_proxy2000, maximized on 1,000 classes with two sorted images per class, offsets 0 and 1, for n=2,000. Reported uncertainty is a standard error clustered by class. The final metric was imagenetv2_top1_full10000, also maximized, with ten images per class at offsets 0 through 9, for n=10,000 and the same uncertainty convention. Proxy rows are 20% of the final set, but the other 8,000 images are unavailable during exploration and formal construction. Thus the two scores overlap but use different splits; they are neither independent replications nor directly interchangeable measures.

The artifact had to be one CLIP-compatible state dictionary in the affine hull of the 72 ingredients. Formal code had to publish model.pt under checkpoint-<progress>. At most three valid checkpoints could be accepted; when more were published, the three greatest progress values were retained, and the official policy was best_valid_of_up_to_3.

3.2 How the baseline works

The baseline construction flow was:

72 fixed CLIP state dictionaries as input → stream each dictionary and add it with coefficient 1/72 → use no label, reward, or selection signal in the default construction; proxy labels are only for evaluation → apply no loss, optimizer, or gradient, only direct parameter averaging → export one new CLIP-compatible state dictionary.

There were no trainable parameters or gradient-update schedule. The component checkpoints stayed fixed; only the exported model weights changed. The default published one checkpoint at progress 1. The task package reports 248.13 seconds to construct the uniform soup, 1,198.02 seconds for full scoring, and 2,222 MiB peak GPU memory. Its full score was 0.0015 below model_69, but only about 0.56 paired standard errors apart, so the task describes them as effectively tied. The baseline source also contained best_single and strict_greedy alternatives. Strict greedy ranks ingredients by proxy accuracy, starts from the best, tries equal averages with later ingredients, and accepts only strict proxy improvements, but the shipped default remained uniform averaging.

The agent identified two concrete bottlenecks. First, CPU preprocessing of 2,000 images caused initial end-to-end scripts to outlive a two-minute interactive wait window. Second, selecting repeatedly on a small proxy could overfit its binary outcomes. The agent addressed the engineering bottleneck by background preprocessing and GPU-resident caches, then addressed the scientific one with held-out proxy splits.

4. Four-hour exploration and decision process

The early part of exploration established asset shapes, GPU capacity, baseline behavior, and the evaluation protocol, then converted slow repeated evaluation into a one-time cache. The middle concentrated on individual-model scanning, greedy construction, and multiple holdout schemes. The final part tested blend selection, implemented a deadline-aware long-running recipe, and revalidated it with short, long, and fresh-workspace probes. The run used 12,457 of 14,400 seconds, or about 86.5%, and submitted with approximately 2,006 seconds remaining.

U-01 - Establishing a reproducible proxy baseline and fast soup evaluation

Motivation and hypothesis. The agent needed trustworthy uniform, single-model, and combination measurements. Re-reading checkpoints and preprocessing images for every candidate would prevent enough robustness tests within four hours. It hypothesized that keeping the proxy tensors and all 72 models on the GPU would reduce candidate evaluation from minutes to below one second.

Concrete change and experimental setup. Inspection showed float32 ingredients of roughly 0.456 GB each and an estimated 32.8 GB stacked footprint. Two initial end-to-end invocations returned at the 120-second tool wait limit while preprocessing continued in the background. The agent then cached the two supplied images per class, stacked the weights, cached individual predictions, and evaluated linear combinations with tensor operations. All scientific comparisons in exploration used the fixed n=2,000 proxy.

Observed result. The uniform soup scored 0.688 with class-clustered stderr 0.010921. Its offset-0 and offset-1 halves scored 0.679 and 0.697, revealing substantial half-to-half difficulty variation. The 72 individual ingredients ranged approximately from 0.634 to 0.6935; model_69 was best at 0.6935, followed by such strong alternatives as model_27 at 0.691 and model_60 at 0.6895. A CPU matrix merge of all weights took about 17.3 seconds, too slow for hundreds of combinations per round. With GPU caching, a merge took about 0.17 seconds, the 2,000-image forward pass about 0.54 seconds, and steady candidate evaluation about 0.38–0.43 seconds.

Agent interpretation. The agent concluded that the fast path made large cross-validation sweeps feasible. It also treated ingredient heterogeneity and the 0.018 offset-half difference as reasons not to trust a single all-proxy selection.

Report assessment and confounds. The two timeouts were interactive waiting and preprocessing failures, not failed scientific trials; later results and the official fast evaluator confirmed the uniform score. Each fixed half contained only 1,000 images, so the half difference mixes row difficulty with model behavior and is not by itself a selection-variance estimate.

Decision and consequence. High-frequency CPU bagging was rejected in favor of the GPU-resident path. If that path was unavailable, the later candidate would fall back to lower-cost strict greedy rather than attempt the same search on CPU. Uniform remained the no-search fallback and model_69 the best-single anchor.