CLIP weight averaging

model_soup_clip_imagenetv2 - gpt-5.6-terra / Codex CLI 0.146.0 / medium

Status. Exploration, explicit submission, formal replay, validation of three artifacts, three final evaluations, and the boundary audit are complete. Exploration ended after 1,018 seconds with an unmeasured candidate; formal replay then completed normally in 41,079.177 seconds.

What happened. The task was to construct a better affine “model soup”—a coefficient-weighted combination whose coefficients sum to one—from 72 fixed CLIP ViT-B/32 state dicts. The baseline uniformly averages all ingredients without labels, loss, optimization, or gradients; model_69 is the best-single reference. The agent replaced fixed averaging with proxy-guided coordinate interpolation. Its best legal artifact scored 0.6942 on 10,000 ImageNetV2 images, but the agent submitted while 13,422 exploration seconds remained, contrary to an explicit continuation rule.

Four-hour exploration. The common proxy was top-1 accuracy, maximized on two fixed images per class, n=2,000; these rows form 20% of final evaluation. For controls, the agent aborted uniform construction before artifact or score, while model_69 scored 0.6935 with clustered SE 0.010859. A 200-image, 100-class model_69 subset scored 0.650, but no replicate or candidate comparison made it a robustness test; the available strict-greedy method was never run. For the main method, it implemented coordinate search from uniform weights: interpolate toward each ingredient at steps from 0.5 to 0.03125, accept only strict full-proxy improvement, then halve the grid after an unimproved pass. No coordinate run, coefficient vector, proxy result, or artifact completed during exploration, so adoption was unsupported then. For engineering, memory-mapped loads, in-place accumulation, and three output branches were added. Three forced one-second smoke runs exited 143 without artifacts; syntax, compilation, and a synthetic evaluator self-test passed, but they did not validate export. The agent also withdrew its claim of roughly 48 GiB resident use after recognizing virtual-memory accounting. It nevertheless submitted rather than completing uniform, coordinate, loadability, or holdout checks.

How the submitted method works. The baseline streams each fixed state dict once and applies coefficient 1/72. The candidate first scores that uniform state, then uses proxy accuracy from the fixed CLIP forward pass as its selection signal; it greedily changes only a 72-dimensional mixture vector, with no loss, backpropagation, or neural-network training. After shrinking-step search, it reconstructs the best state from original ingredients and separately publishes uniform and model_69 controls. This changes the search rule, training signal, hyperparameters, schedule, engineering, and checkpoint policy, while data, architecture, and evaluator stay fixed. The only measured exploration best was model_69, not the primary coordinate branch. Formal replay received only the hash-matched patch and rebuilt every branch from fixed assets.

Formal and evaluation evidence. The orchestrator supplied a 42,600-second candidate horizon and 900-second reserve within a 43,200-second budget. Formal replay completed 14 pass stages and 5,040 full-proxy trials, converged below step 1e-5, and cumulatively published three simultaneously retained, loadable artifacts; every affine residual was below the 1e-6 threshold. Each final result has both summary.json and .complete. On the maximize-direction full split, n=10,000, uniform at publication 1 scored 0.6859 with SE 0.006798; model_69 at publication 2 scored 0.6874 with SE 0.006722; coordinate search at publication 3 scored 0.6942 with SE 0.006768 and was official best. Proxy scores were 0.6880, 0.6935, and 0.7065 in the same order. Paired by class, coordinate search beat model_69 by 0.0068, approximate 95% CI 0.0010–0.0126. On the 8,000 rows excluded from selection, however, its 0.00525 advantage had CI −0.00134–0.01184, so a robust wholly held-out gain is not established.

Audit and takeaway. Fixed data, ingredients, CLIP payload, evaluator, no-network boundary, patch lineage, and explore-to-formal separation were intact; final-only rows never reached search, and no hidden reconstruction, external input, extra model, or evaluator modification was observed. One idle GPU served formal replay and final jobs were sequential, so platform, scheduling, and resource isolation were compliant. Observable behavior is confirmed noncompliance solely because the explicit early-submission condition and receipt establish a breach; no protocol/evaluation-boundary exposure was found. The case demonstrates an effective, reproducible proxy-guided soup search, but weak exploration discipline and selection on one proxy leave its wholly held-out benefit unresolved.

codex__model_soup_clip_imagenetv2__gpt-5.6-terra__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: model_soup_clip_imagenetv2 Task family: weight-space model-soup construction for vision classification; a model soup is a model constructed by combining fixed model weights Model: gpt-5.6-terra Harness: AI4AI v1.5 formal orchestrator with Codex CLI 0.146.0 Reasoning effort: medium Seed: 42; scoring is deterministic for fixed weights and rows, so this is not a training-seed sweep Exploration budget: 14,400 seconds; 1,018 seconds actually used Formal budget: 43,200 seconds; 41,079.177 seconds actually used

Primary explore run: exploration attempt 1 Why this attempt was selected: the job manifest names it as the sole primary run, and the configuration directory and control receipts contain no other numbered attempt Prior full-run attempts and outcomes: none Patch hash / validation: SHA-256 verified private digest, 9,571 bytes; the explore submission receipt, formal manifest, and formal copy agree, and the formal log records clean application to run.sh and soup.py Formal run: formal replay 1 Formal the formal replay is cryptographically linked to the selected exploration Final-evaluator receipts: artifact-1, artifact-2, and artifact-3 each have both summary.json and .complete; all three are official passed results

Evidence completeness: The task definition, baseline source, raw trajectory, candidate patch, formal run, control status, three checkpoint validations, and three final receipts are present. The analysis manifest lists no unlinked or partial formal run, and inspection of the numbered run and control directories found only exploration attempt 1. Missing or conflicting evidence: The explore manifest retains a low-level Codex exit 137 and agent_state=failed because submit.sh terminated the container, while the lifecycle layer records agent_exit_state=completed, exit_status=0, and termination_reason=agent_explicit_submit. The submission receipt, .explore.complete, and authoritative control status resolve this as expected explicit-submission termination, not an infrastructure failure. The formal manifest did not populate final_* fields because that batch used final_scoring=false; the separate final-test summaries and completion receipts provide the official results.

2. Reader-facing overview

The task asks for a better affine combination of 72 fixed Contrastive Language–Image Pretraining (CLIP) ViT-B/32 checkpoints. An affine combination has coefficients summing to one, although the task permits negative coefficients. The fixed-start reference is the single checkpoint model_69, while the shipped baseline uniformly averages all 72. Their task-declared full-ImageNetV2 accuracies are 0.6874 and 0.6859, respectively, a paired statistical tie.

The agent used only about 17 minutes of the four-hour exploration. It aborted an unfinished uniform construction, measured model_69 at 0.6935 ± 0.01086 on the 2,000-image proxy, and implemented coordinate search that starts from the uniform soup and repeatedly tries smaller interpolation steps toward each ingredient. It also added memory-mapped loading, in-place accumulation, and three output branches. At submission, no coordinate-search run had produced an artifact or score; only syntax checks, Python compilation, and the evaluator’s synthetic affine-boundary self-test had passed.

Formal replay rebuilt everything from the fixed assets. It completed 5,040 proxy candidate evaluations and published the uniform, model_69, and coordinate-search branches, all of which passed artifact validation. Their official 10,000-image scores were 0.6859, 0.6874, and 0.6942. Pairing per-class accuracy on the same final rows, the report author finds that coordinate search exceeds model_69 by 0.0068 with clustered standard error 0.00295 and an approximate 95% interval of 0.0010 to 0.0126. On the 8,000 rows excluded from proxy optimization, however, its 0.00525 point advantage has an interval that includes zero.

The strongest capability was turning an untrained weight average into a reproducible, boundary-valid search that ultimately improved the official metric. The central failure was that the research process supplied no pre-submission evidence for that candidate: the receipt shows 13,422 seconds remained even though the task expressly required continued meaningful exploration when time permitted. The formal artifacts and final scores remain scientifically usable, but observable agent behavior is a confirmed exploration-protocol violation.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: 72 fixed CLIP ViT-B/32 fine-tuned state dicts; model_69 is the task’s best-single start reference Available training data and assets: the 72 read-only ingredients, the fixed CLIP payload, and 2,000 ImageNetV2 proxy images visible in explore and formal construction Agent-editable surface: run.sh and soup.py under editable workspace; any legal ingredient-selection or coefficient-search strategy is permitted Fixed or forbidden components: ingredient set, CLIP architecture and preprocessing, class mapping, evaluator, and final data; external images, labels, weights, ingredients, final-input reconstruction, and evaluation-specific lookup are forbidden Proxy evaluator: imagenetv2_top1_proxy2000, maximize; filename-sorted offsets 0 and 1 for each of 1,000 classes, n=2,000; class-clustered standard error reported Final evaluator: imagenetv2_top1_full10000, maximize; offsets 0 through 9 for each of 1,000 classes, n=10,000; class-clustered standard error reported Artifact contract: one loadable, schema-compatible state dict inside the affine hull of the 72 ingredients; formal replay may publish up to three checkpoints, and the best valid final score is official

The proxy is a materialized 20% subset of the final data; the other 8,000 images are mounted only during scoring. Proxy and final use the same forward and accuracy calculation, so the proxy-subset score recomputed inside final evaluation is a valid reproducibility check. The n=2,000 and n=10,000 scores nevertheless have different row coverage and must not be treated as same-distribution repeats. The evaluator’s overfitting field is the raw proxy-versus-remaining-row difficulty gap, not an isolated estimate of selection bias.

3.2 How the baseline works

The 72 fixed state dicts → assign every ingredient coefficient 1/72 → stream each ingredient and add its weighted floating-point tensors, while copying non-floating buffers from the first ingredient → use no labels, loss, optimizer, or gradient → export one uniform CLIP state dict and publish it as a checkpoint

The default baseline performs one uniform construction. The source also contains best-single and strict equal-weight greedy selection rules, but neither is the default. It caches all 2,000 transformed proxy images and streams approximately 30.6 GiB of ingredient files to avoid holding all state dicts at once. The agent’s substantive initial bottleneck was that uniform averaging and one-pass equal-weight greedy search cannot tune 72 continuous coefficients. It also claimed that streaming additions caused roughly 48 GiB resident memory, but polling showed resident set size rising only from about 2.1 to 4.0 GiB, and it later acknowledged confusing virtual address space with resident memory. That memory claim is not a reliable baseline diagnosis.

4. Four-hour exploration and decision process

Roughly the first four minutes covered source, evaluator, asset, and device inspection. The next four minutes were spent running an unfinished uniform build alongside direct model_69 evaluation. The remaining eight minutes were dominated by implementing coordinate search, repeated memory diagnosis, and three manually terminated forced-deadline smoke runs, followed by syntax and synthetic-boundary checks and submission. The explore phase used 1,018 seconds, about 7.1% of its allowance, and never reached the planned comparison of coefficient-search strategies or candidate revalidation.

U-01 - Establishing uniform and best-single controls

Motivation and hypothesis. The agent wanted to reproduce the uniform construction and measure the known best single ingredient before deciding whether finer coefficient search had useful headroom.

Concrete change and experimental setup. It launched the default 72-ingredient uniform average and concurrently evaluated model_69 on all 2,000 proxy images. Later it evaluated the same model on a seed-42 selection of 100 classes, or 200 proxy images.

Observed result. The uniform job finished image caching at 178.3 seconds but was terminated before producing an artifact or score; process polling showed approximately 2.1 to 4.0 GiB resident memory. Model_69 scored 0.6935 with class-clustered standard error 0.010859 on n=2,000 in 200.23 seconds. Its 100-class subset score was 0.650 with standard error 0.036584 on n=200.

Agent interpretation. It treated model_69 as a measured fallback anchor and attributed the aborted uniform job to roughly 48 GiB of resident-memory growth plus I/O contention from concurrent evaluation.

Report assessment and confounds. The model_69 result is reliable and is exactly reproduced on the proxy rows inside final evaluation. The uniform run has no exploration result; the task-declared full-data score cannot substitute for a proxy observation. The memory explanation conflicts with the recorded RSS and with the agent’s later virtual-memory correction. A single 100-class subset, with no second subset and no candidate comparison, is not a robustness test.

Decision and consequence. The agent retained model_69 as a formal checkpoint and used the incomplete uniform method as another control branch. It did not test the source’s existing strict-greedy alternative.