CLIP weight averaging

model_soup_clip_imagenetv2 - gpt-5.6-sol / Codex CLI / high

Status. The sole explore run submitted successfully; the same-hash patch completed fresh formal replay, all three artifacts passed frozen validation, and all three final summaries have matching .complete receipts. Boundary audit is complete.

What happened. The task requires an affine weight-space soup from 72 frozen CLIP ViT-B/32 state dictionaries, without gradient training. The shipped all-72 equal average scores 0.6859 on the full set, statistically tied with the fixed model_69 start at 0.6874. A deterministic coefficient search produced a three-ingredient soup scoring 0.6972 on 10,000 final images.

Four-hour exploration. The maximize proxy metric is deterministic top-1 on 2,000 images. First, scoring all endpoints put model_69 first at 0.6935; the agent built a resident exact-averaging/partition tool and corrected an omitted model_36/37 ranking transcription before search. Sparse top-k averaging plateaued at 0.6945, while anchor shrinkage reached 0.6970, so it adopted an anchor-dominant fallback. Pair tests then showed that second-ranked model_27 was unhelpful and 50% model_44 reached 0.7015 but hurt one offset; the agent therefore introduced a worst-of-four offset/class-parity filter and retained a safer 35% model_44 pair at 0.7010. An exhaustive 240-trial screen of 60 remaining partners found robust model_42 and a surprising low-ranked model_59; combining the latter with model_44 yielded a broad all-partition-positive three-model region up to 0.7040, while a 0.7030 model_69/model_44/model_10 mixture became the safety artifact. A 403-second formal smoke replay published three loadable checkpoints. A generic validator call failed and accidentally concurrent hull checks were stopped; the surviving check was valid at about 3.3×10^-8 relative residual, so this was debugging, not an invalid result. Finally, all 25 local 69/44/59 mixtures beat the anchor on every partition; weights 0.15/0.45/0.40 scored 0.7075 and an independent reload reproduced it. Thirteen zero/negative-anchor extrapolations did not improve it, so extrapolation was rejected. Submission used 12,146 of 14,400 seconds; the remaining 2,303 seconds are a research-completeness limitation, not an established violation.

How the submitted method works. Baseline flow is fixed states to label-free uniform tensor averaging to one model. Candidate flow is fixed states and true proxy labels to cached candidate predictions, pooled accuracy plus a four-partition gate, deterministic coefficient enumeration, then exact affine averaging. There is no loss, optimizer, or trainable parameter; the averaging update is unchanged. The changes are search algorithm, hyperparameters, schedule, engineering, and checkpoint policy. The exploration leader was encoded as a fixed formal seed, while all explore models and caches were excluded from fresh replay.

Formal and evaluation evidence. Harness overrides supplied a 42,600-second brake and 900-second reserve, producing a 41,700-second search limit. Beyond endpoint reranking, formal construction recorded 2,187 mixture trials in 41,674.508 seconds and stopped safely during pair-simplex search before the planned random-convex branch. It cumulatively published and simultaneously retained three valid artifacts:

| Progress | Proxy, maximize, n=2,000 | Final, maximize, n=10,000 | Class-clustered SE / off-proxy score | Status | |---|---:|---:|---|---| | 1, model_69 | 0.6935 | 0.6874 | 0.00672 / 0.685875 | start | | 2, balanced soup | 0.7030 | 0.6947 | 0.00666 / 0.692625 | valid fallback | | 3, exploration leader | 0.7075 | 0.6972 | 0.00676 / 0.694625 | official best |

Full and untuned 8,000-row scores preserve the proxy direction. From official row files, artifact 3 beats the start by +0.0098 with paired class-clustered SE 0.00288 and approximate 95% interval [0.00415, 0.01545]; its +0.0025 over artifact 2 has an interval crossing zero. The raw proxy/off-proxy gap grows, but evaluator source says row difficulty and selection effects are confounded.

Audit and takeaway. Only fixed ingredients, CLIP payload, and proxy images were used. Final-only image content and candidate-specific predictions were not exposed, reconstructed, queried, or used. No external input, evaluator modification, explore-artifact reuse, GPU sharing, or wall-clock breach was found. Observable behavior is clean, protocol exposure is none found, and scheduling/resource isolation was compliant. The platform nevertheless has a confirmed bookkeeping defect: attempt files retain running and explore exit layers conflict, while job-level termination, formal succeeded, matching hashes, and terminal receipts resolve the run. The case demonstrates disciplined complementarity search and formal engineering, but not global coefficient optimality or statistical superiority of artifact 3 over artifact 2.

codex__model_soup_clip_imagenetv2__gpt-5.6-sol__high - Full English Analysis

1. Run identity, attempt selection, and lineage

A same-configuration directory and control-receipt scan found only this explore run and this formal run, with no applicable correction receipt. The explore fields auto_retrain=false and retrain_phase_created=false show only that the explore manifest did not create formal replay; the upstream scheduler later created the linked formal run.

2. Reader-facing overview

The task asks for a better weight-space soup from 72 frozen CLIP ViT-B/32 state dictionaries. The delivered artifact must be an affine combination of those ingredients; there is no gradient training. The fixed start is model_69, whose published full-set accuracy is 0.6874. The shipped baseline instead uniformly averages all 72 and scores 0.6859; its published paired uncertainty makes that a statistical tie with the start, not a demonstrated regression. The agent diagnosed two linked problems: weak or non-complementary ingredients dilute the best model, while repeated coefficient selection on only 2,000 proxy images can overfit the selection set.

Exploration re-ranked every ingredient, then tested sparse top-k mixtures, anchor shrinkage, individual directions, four fixed proxy partitions, an exhaustive screen of lower-ranked directions, complementary three-model soups, a local plateau, and permitted negative-weight extrapolation. The submitted method is not a new gradient update. It is a deterministic, long-wall-clock proxy coefficient search that immediately preserves the start, separately preserves a prevalidated balanced soup, and selects the highest proxy-accuracy candidate whose worst of four partitions does not fall by more than 0.001.

Formal construction completed and published exactly three simultaneously retained checkpoints, all accepted by the frozen affine-hull validator. Their formal proxy scores were 0.6935, 0.7030, and 0.7075; their official 10,000-row final scores were 0.6874, 0.6947, and 0.6972. The best third artifact uses weights 0.15/0.45/0.40 on model_69/model_44/model_59. Recomputing a paired, class-clustered difference from the official row files gives +0.0098 over the start with SE 0.00288 and an approximate 95% interval [0.00415, 0.01545]. On the 8,000 rows not used for coefficient tuning, it gains +0.00875 with paired SE 0.00324.

Observable agent behavior is clean: no final-only image content, external data, or extra model entered the trajectory or patch, and no protocol leakage was found. The main scientific limitation is that 2,187 formal candidates were still selected on the same 2,000 proxy rows. The raw proxy-versus-off-proxy gap grows, and artifact 3's final gain over artifact 2 is inconclusive under paired uncertainty. The platform has a confirmed status-bookkeeping defect, but independent completion receipts, hash lineage, and GPU records support the scientific result.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: model_69, the best of the 72 ingredients on the published full evaluation; score 0.6874 Available training data and assets: 72 read-only state dictionaries, fixed CLIP ViT-B/32 payload, and 2,000 ImageNetV2 proxy images Agent-editable surface: construction, search, engineering, and checkpoint policy under editable workspace Fixed or forbidden components: ingredient set, CLIP architecture/preprocessing, class mapping, and frozen evaluator; no external images, labels, weights, or final-only inputs Proxy evaluator: imagenetv2_top1_proxy2000 / maximize / filename-sorted per-class offsets 0–1 / n=2,000 / class-clustered SE available in final receipts Final evaluator: imagenetv2_top1_full10000 / maximize / offsets 0–9 / n=10,000 / class-clustered SE, plus proxy-row and remaining-8,000-row diagnostics Artifact contract: one complete compatible state dictionary in the affine hull of all 72 ingredients; formal output is checkpoint-<progress>private filesystem location and at most the three greatest progress values are accepted ``

Both evaluators compute deterministic top-1 accuracy. The 2,000 proxy rows are also 20% of the 10,000 final rows, so final evaluation can reproduce the proxy portion for the same checkpoint. The full score also contains 8,000 offset-2–9 images absent from exploration and formal construction, however, so proxy and full scores are not the same split and must not be subtracted as if they were repeated measurements. The evaluator's overfitting diagnostic is proxy score minus off-proxy score; its own source says this raw gap combines row-set difficulty with any selection effect and cannot be attributed wholly to overfitting.

3.2 How the baseline works

``text 72 frozen, shape-compatible CLIP state dictionaries -> stream each tensor; assign every ingredient the same coefficient by default -> uniform construction needs no labels; optional best_single/strict_greedy modes use accuracy on the 2,000 proxy rows -> normalized weighted averaging for floating tensors; copy non-floating tensors from the first ingredient -> export one complete model.pt; no ingredient is trained and there is no loss or optimizer ``

The shipped run.sh defaults to uniform, makes one all-72 equal-weight soup, publishes it at progress 1, and has a simultaneous retention limit of three. soup.py also offers nondefault best_single, which evaluates all endpoints, and strict_greedy, which adds a ranked ingredient only when the newly uniform set strictly improves proxy accuracy. The task's full references are 0.6874 for model_69 and 0.6859 for the uniform soup. Their -0.0015 difference is published as about 0.56 paired standard errors and therefore a tie. Uniform construction took 248.13 seconds, full scoring took 1,198.02 seconds, and peak GPU memory was 2,222 MiB.

The agent explicitly identified that two examples per class make proxy coefficient selection the bottleneck, not stochastic training noise. It also reasoned that endpoint accuracy does not measure whether the weight-space direction from model_69 is complementary.

4. Four-hour exploration and decision process

The agent spent roughly the first 16 minutes obtaining an exact 72-ingredient ranking, then kept preprocessed images and every state dictionary resident so most coefficient trials cost about 9–20 seconds. The middle of the run was dominated by ingredient screens and complementary three-model grids, with offset and class-parity diagnostics added as the selection criterion matured. It then built and debugged a bounded formal replay, refined the local optimum, independently reloaded the leader, tested negative-weight extrapolation, and audited the final source. Submission occurred after 12,146 seconds with 2,303 seconds remaining.

U-01 - Establish the ingredient baseline and correct the measurement tool

Motivation and hypothesis. The agent needed to confirm that model_69 was the proxy anchor, determine whether the all-72 failure reflected dilution rather than an implementation error, and avoid reloading 30.6 GiB of states and preprocessing images for every candidate.

Concrete change and experimental setup. It first used the baseline's best_single mode to score all 72 ingredients on the fixed 2,000 rows with batch size 1,024. It then added a persistent exact-soup driver that cached image tensors and states, reconstructed each coefficient vector exactly, and reported pooled accuracy, two offsets, and later even/odd class partitions. An initial manual ranking transcription omitted model_36 and model_37; the agent corrected it against the complete output before consequential searching.

Observed result. model_69 ranked first at 0.6935, followed by model_27 at 0.6910, model_60 at 0.6895, and model_43 at 0.6880; model_31 was worst at 0.6340. The ranking run took 966.05 seconds. The later official receipt reports class-clustered SE 0.01086 for model_69 on those same 2,000 rows.

Agent interpretation. It retained model_69 as an unconditional fallback and treated the ranking only as a list of directions to test, not as proof that high-ranked endpoints would mix well.

Report assessment and confounds. The scores are deterministic for fixed rows; a seed repeat is not an independent replicate. The SE describes class heterogeneity and does not correct selection across 72 endpoints. The transcription error was repaired before it could define the formal method.

Decision and consequence. Adopt model_69 as anchor and checkpoint 1, express later gains relative to its partitions, and retain the persistent averaging function in the submitted source because formal search imports it.