CLIP weight averaging

model_soup_clip_imagenetv2 - gpt-5.6-sol / Codex CLI / low

Status. Exploration, submission, formal replay, three artifact validations, and three official final evaluations completed. Formal artifacts are valid; observable agent behavior is confirmed noncompliant because of an idle early submission.

What happened. The task asks for one affine combination of 72 frozen CLIP ViT-B/32 checkpoints that improves ImageNetV2. The baseline uniformly averages every ingredient, with no gradients or optimizer; declared full-set references are 0.6874 for best-single model_69 and 0.6859 for the uniform soup. The agent submitted three low-dimensional fallbacks plus long coefficient search, and final checkpoint 2 achieved the best official point estimate, 0.6941.

Four-hour exploration. All search scores used top-1 accuracy (higher is better) on the public offsets 0–1 proxy, n=2,000. Ranking all 72 ingredients found model 69 at 0.6935. A 69/44 half-mixture produced a broad 0.7000–0.7015 plateau; adding 10% model 63 reached 0.7020, while negative extrapolation and top-k averaging were rejected. Cross-entropy supported the pair/triangle, but adding strong models 60/27/43/48 reduced top-1 to roughly 0.697–0.6995; model-60 mixtures lowered loss further without exceeding 0.699, so loss became only an acceptance guard.

An exhaustive 142-pair scan found standalone-weak model 59 (0.6625) complementary: 5% added to 69/44 scored 0.7030, with offsets 0.701/0.705 and nearby weights similar. Model 34 did not improve the established soup. The first follow-up failed after a helper rename caused ImportError; it was fixed and rerun. A 144-query ±3% direction scan peaked at 0.7045 for negative model 53, but the agent rejected this as multiple-comparison overfit. Its claim that negative directions systematically worsened loss is contradicted by negative models 53 and 43, so that mechanism is inconclusive.

Class bootstrap intervals crossed zero: checkpoint 3 versus the pair gained 3/2,000 images with about [-0.0015,0.0045], while checkpoint 2 gained one. Bridge mixtures did not beat endpoints. Clean replay, syntax, hashes, and checkpoints 1/3 affine validation passed; a local git-clone ownership failure was bypassed with an archive-built clean tree. Checkpoint 2 received proxy evaluation but no explore-stage affine validation.

How the submitted method works. It immediately publishes 69/44, 69/44/63, and 69/44/59 soups, caches all ingredient logits, ranks singles, searches a 71×9 anchor-pair grid, then loops over global sparse and local affine perturbations. Proxy top-1 supplies selection; cross-entropy and offset gap guard replacement. Slot 3 changes only for robust gain ≥0.004 with nearly nondegrading splits/loss. No labels, external data, or CLIP parameters are generated/trained: the change is coefficient search, hyperparameters, schedule, engineering, and checkpoint policy, not a new gradient update rule. The 0.7045 peak was not submitted; the neighborhood-supported 0.7030 positive soup was.

Formal and evaluation evidence. Formal overrides were 42,600 seconds with 900 seconds reserve, seed 42, batch 512. Replay made 6,646 queries in 41,726.60 seconds (96.59% of 12 hours) and stopped normally; no search result crossed the 0.706 replacement threshold. All startup artifacts passed affine validation.

| Checkpoint | Official full top-1 ↑, offsets 0–9, n=10,000 | Class-clustered SE | Result | |---|---:|---:|---| | 1: 69/44 | 0.6912 (complete) | 0.00672 | valid | | 2: 69/44/63 | 0.6941 (complete) | 0.00671 | best | | 3: 69/44/59 | 0.6926 (complete) | 0.00671 | proxy-best, not final-best |

Checkpoint 2 is +0.0067/+0.0082 in point estimate versus declared references, but paired delta uncertainty is unavailable, so significance is not established.

Audit and takeaway. Fixed data/models, final-only 8,000 rows, evaluator, external inputs, fresh formal boundary, single-GPU isolation, hashes, and receipts were clean; no protocol exposure was found, and platform/resource controls were compliant. However, submission was explicit and idle with 5,044 seconds remaining. The task required continued useful work while feasible and final source/artifact verification before the last action, yet checkpoint 2—one of three final artifacts—was unvalidated; formal validation took only 269.63 seconds. Observable behavior is therefore confirmed, independently of compliant formal-budget use. The case demonstrates effective complement search and reproducible engineering, but small-proxy reuse and rank reversal limit the conclusion to a higher official point estimate, not statistical superiority or benefit from adaptive search.

codex__model_soup_clip_imagenetv2__gpt-5.6-sol__low - gpt-5.6-sol / Codex CLI / low Full English Analysis

1. Run identity, attempt selection, and lineage

Explore manifest, formal manifest, and outcome establish lineage directly.

2. Reader-facing overview

The task supplies 72 frozen CLIP ViT-B/32 checkpoints and asks for one better ImageNetV2 model formed with arbitrary affine coefficients. The shipped baseline uniformly averages all ingredients, ignoring quality and error complementarity. Declared full-set references are 0.6874 for the best single ingredient, model_69, and 0.6859 for the uniform soup.

On the public 2,000-image proxy, the agent studied ingredient ranking, two/three-model interpolation, cross-entropy diagnostics, exhaustive complement scans, signed directions, and class stability. It submitted three fixed fallbacks—69/44 at 0.5/0.5; 69/44/63 at 0.6/0.3/0.1; and 69/44/59 at 0.45/0.50/0.05—followed by wall-clock-filling global/local coefficient search with a strict checkpoint-3 replacement rule.

Formal replay completed 6,646 proxy queries and retained the initial artifacts. All three passed affine-hull validation. Checkpoint 2 won the official 10,000-image evaluation at 0.6941 (class-clustered SE 0.00671); proxy-best checkpoint 3 scored 0.6926. These are higher point estimates than both declared references, but no paired uncertainty for candidate-minus-reference is available, so statistical superiority is not established.

The strongest capability was reducing a large coefficient space to reproducible, valid fallbacks and a durable formal recipe. The task required continued meaningful work while feasible and final source/artifact verification before the last action, yet submission was explicit and idle with 5,044 seconds left while checkpoint 2—one of the three final artifacts—lacked affine validation. Measured validations took only about 270–502 seconds, making this confirmed execution-protocol noncompliance. Formal validation later passed, so official artifacts and scores remain scientifically usable.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: 72 read-only CLIP ViT-B/32 ingredients; model_69 is the fixed best-single full-set reference Available training data and assets: ingredients, fixed CLIP payload, ImageNetV2 offsets 0–1 proxy Agent-editable surface: selection, affine coefficients, construction/search code, schedule, checkpoint policy Fixed or forbidden components: ingredients, architecture, proxy images, final-only other 8,000 images, evaluators; no network or off-hull fine-tuning Proxy evaluator: imagenetv2_top1_proxy2000 / top-1 higher / offsets 0–1 / n=2,000 / clustered SE available for retained artifacts Final evaluator: imagenetv2_top1_full10000 / top-1 higher / offsets 0–9 / n=10,000 / clustered SE plus 2,000/8,000 diagnostics Artifact contract: one compatible state_dict in the ingredients' affine hull; at most three valid formal checkpoints ``

The proxy is 20% of the final rows but uses only two offsets, so it is a selection protocol, not an independent identically distributed replication. Proxy-versus-other-8,000 gaps also confound selection with offset difficulty. Execution is deterministic; uncertainty is sampling/class uncertainty, not seed variance.

3.2 How the baseline works

``text 72 frozen state dictionaries -> stream each with the same positive coefficient -> no label, reward, or feedback in default mode -> parameter-wise uniform mean, with no gradients or optimizer -> one compatible state dictionary and checkpoint ``

Source also provides best-single and proxy-accuracy strict-greedy modes. The agent explicitly diagnosed forced inclusion of weak ingredients and failure to use the allowed affine space, while recognizing repeated selection on a small proxy as an overfitting risk.

4. Four-hour exploration and decision process

The run started at 21:18 UTC. Roughly 13 minutes established the ingredient ranking; the middle studied low-dimensional mixtures, losses, complements, signed directions, and stability, including one import fix; the end replayed the recipe and audited patch/artifacts. It explicitly ended at 23:55 after about 9,356 of 14,400 seconds.

U-01 - Ingredient quality and the uniform bottleneck

Motivation and hypothesis. Weak ingredients could dilute the uniform mean; ranking supplies an anchor without proving complementarity.

Concrete change and experimental setup. All 72 ingredients were scored on the fixed 2,000 rows with seed 42 and batch 512.

Observed result. Ranking took 775.49 seconds. Models 69, 27, 60, 43, and 48 scored 0.6935, 0.6910, 0.6895, 0.6880, and 0.6870; models 34 and 59 scored only 0.6700 and 0.6625.

Agent interpretation. Use 69 as anchor, but do not equate standalone rank with complementarity.

Report assessment and confounds. Adjacent differences are much smaller than roughly percentage-point sampling uncertainty.

Decision and consequence. Start with strong low-dimensional mixtures, then scan every ingredient.