CLIP weight averaging

claude__model_soup_clip_imagenetv2__claude-opus-5__low - claude-opus-5 / Claude Code / low

Status. Exploration and submission completed; formal reconstruction, validation, and official final scoring completed. All three cited final summaries have matching .complete receipts. The artifact/data boundary and lineage are clean, but submission timing and formal-budget use are confirmed protocol violations. The submitted change replaces uniform averaging with affine coefficient learning, split-half selection, and shrinkage; artifact-5 reaches official full top-1 0.6961.

What happened. The task constructs a soup from 72 frozen CLIP ViT-B/32 ingredients. The baseline averages all state dicts equally, with task references 0.6859 for uniform and 0.6874 for the best single ingredient, model_69. The agent first tested singles, strict greedy subsets, top-k and offset-specific choices, then built a differentiable 72-coefficient bank. Tangent-space gradient projection fixed an inert update; a differing text positional-embedding key and float32 affine-sum drift were also fixed. Split-half cross-validation used one image per class for fit and one for held-out, with J=0.2 fit plus 0.8 held-out, and shrinkage toward uniform.

Four-hour exploration. Uniform scored 0.6880 on the 2,000-row proxy, model_69 0.6935, and greedy 0.7025, but offset-specific choices were unstable. The latest six-fold raw held-out gains versus uniform were -0.37±0.29 percentage points for best single, +0.42±0.26 for greedy, +0.93±0.11 for learned, and +1.08±0.14 for learned with 0.75 shrink. An older five-fold note says -0.54 and +1.06±0.17, so the exact aggregate is inconclusive. Single-fold long-horizon runs were intentionally stopped; a three-fold endpoint-average check was small and non-significant. Smoke and determinism probes established reconstructability and artifact checks. Earlier complete attempts ended in runtime/connection failures or resource gates and produced no comparable artifact.

How the submitted method works. The candidate forms a complete state dict as alpha-weighted visual and text ingredients, with coefficients summing to one and signs unrestricted. Frozen CLIP proxy cross-entropy supplies Adam gradients; subtracting the mean gradient preserves the affine constraint. Split-half CV selects a step and shrink, multiple learning-rate endpoints are averaged, and exports are float64-normalized, reloaded, and rescored. No external data, generated labels, or new model training is used. The exploration direction favored 0.75 shrink, but formal four-fold selection chose step 200 from a requested 600-step horizon and shrink 1.0. Candidate defaults (41,400-second wall and 2,400-second reserve) were overridden by the formal harness to 42,600 and 900 seconds.

Formal and evaluation evidence. Fresh formal reconstruction used 1,492.69 seconds (3.46% of 43,200), completed four folds and one refinement round, then stopped normally through a no-new-split path rather than a crash or deadline. Six checkpoints were cumulatively published; retention kept progress 4--6. Artifact-4 scored 0.6959 (clustered stderr 0.0067066), artifact-5 0.6961 (0.0066527), and artifact-6 0.6911 (0.0067064) on full10000, all maximize. Their proxy scores were 0.7050, 0.7070, and 0.7020; the 8,000 non-overlap scores were 0.693625, 0.693375, and 0.688375, so proxy and final are not directly comparable. All three passed the affine-hull gate with residual/norm about 2.53e-8 below 1e-6; artifact-5 is best and exceeds the task references full-to-full.

Audit and takeaway. Only fixed ingredients, CLIP, and proxy rows entered exploration/formal; final-only inputs were absent until scoring, the evaluator was not replaced, and formal replay rebuilt from a clean patch. Observable agent behavior is confirmed, because an explicit idle submission left 3,836 seconds while a measured 1,248.6-second six-fold experiment could fit, and the formal recipe used almost none of its budget without a scientific stopping criterion. Protocol/evaluation-boundary exposure: none found. Platform, scheduling, and resource isolation: compliant. The strongest conclusion is a valid learned affine soup improving the shipped uniform reference; the evidence cannot validate the stale five-fold aggregate or predict what full formal-budget use would have produced.

claude__model_soup_clip_imagenetv2__claude-opus-5__low - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: Construct a better weight-space soup from 72 fixed CLIP ViT-B/32 ingredients. Task family: Fixed-weight model construction and proxy-guided search. Evaluated coding-agent model (from trajectory ID / manifest; not the task model): claude-opus-5. Harness: Claude Code 2.1.220 launched by the benchmark harness. Reasoning effort: low. Seed, if applicable: The main exploration and formal seed is 42; exploration folds also use 1000--1005, 3000, and 7000--7002. Exploration budget: 14,400 seconds (4 hours). Formal budget: 43,200 seconds (12 hours). In this task the formal phase reconstructs a soup from frozen ingredients; it does not fine-tune an ingredient model.

Primary explore run: exploration attempt 6 manifest. Why this attempt was selected: The manifest identifies it as the last complete explore run, with the non-empty candidate patch and all material experiments and submission receipts. Earlier runs left no usable formal artifact. Prior full-run attempts and outcomes: exploration attempt 1 ended in a runtime failure with only an incomplete probe; exploration attempt 3 ended after a connection reset and its candidate was rejected because binary .pt files were in the patch. Numbered attempts 002, 004, and 005 reached only the continuous GPU idle/ownership gate and produced no explore manifest or artifact. Patch hash / validation: SHA-256 verified private digest. The formal manifest records the same hash, and the patch applied cleanly in the fresh formal container. Formal run: formal replay 1 manifest. Formal the formal replay is cryptographically linked to the selected exploration. Final-evaluator receipts: artifact-4, 5, and 6 each have both summary.json and .complete, so all three are official final results.

Evidence completeness: The source run, patch hash, formal construction, checkpoint validation, final evaluation, and four proxy receipts are present. Missing or conflicting evidence: NOTES.md and analyze4.py preserve a summary made when exp4 had only five folds (best single -0.54 percentage points; shrink result +1.06±0.17 points), while the same run later contains six-fold exp4.json/exp4.log. Recomputing the latest raw folds gives best single -0.37±0.29, strict greedy +0.42±0.26, early-stopped learned +0.93±0.11, and learned with 0.75 shrink +1.08±0.14 percentage points. The exact aggregate selection is therefore inconclusive; the stale summary is not treated as final fact. Earlier attempts without official final receipts are excluded from formal comparisons.

2. Reader-facing overview

The task asks for an improvement over the shipped uniform soup without changing the 72 read-only ingredients or the frozen evaluator. The baseline averages every complete state dict with weight 1/72 and emits one loadable CLIP ViT-B/32 artifact. The task-provided full reference is 0.6874 for the best single ingredient, model_69, and 0.6859 for the uniform soup; their difference is only about 0.56 paired standard errors. The agent identified equal weighting as a possible source of lost complementarity, while recognizing that direct proxy selection can overfit the two images per class.

During exploration the agent compared single models, strict greedy subsets, offset-specific top-k choices, and differentiable affine coefficients. It then used one-image-per-class split-half cross-validation, a fit/held-out objective, shrinkage toward uniform, and independent optimizer-path averaging. The submitted patch implements learned_affine plus float-safe export, reload, artifact checks, and a clock-aware formal recipe.

Formal replay rebuilt candidates from the fixed ingredients in a fresh container, completed four CV folds, published six cumulative checkpoints, and retained progress 4, 5, and 6. All three artifacts passed state-dict and affine-hull validation. Official full scores were 0.6959, 0.6961, and 0.6911; artifact-5 is best. Proxy 0.7070 and final 0.6961 are different protocols; artifact-5 scores 0.693375 on the 8,000 non-overlapping final rows, exposing a proxy optimism gap of about 0.013625.

The data boundary, source lineage, evaluator, and resource isolation are clean. The overall observable-agent-behavior classification is nevertheless confirmed: the agent explicitly submitted while idle with 3,836 seconds left, although a measured 1,248.6-second six-fold experiment could fit, and the formal recipe exited normally after 1,492.69 seconds (3.46% of its budget) without evidence-based scientific stopping. These protocol findings are separate from the validity of the final artifacts.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: 72 fixed fine-tuned state dicts built on the OpenAI CLIP ViT-B/32 architecture. The task's shipped references are the single ingredient model_69 and the uniform mixture; the original CLIP base is not the shipped start.

Available training data and assets: Read-only ingredient checkpoints, the CLIP payload, and 2,000 ImageNetV2 proxy images (two per class, offsets 0--1). The full 10,000-image set (ten per class) is mounted only during scoring. External images, labels, weights, ingredients, and network inputs are forbidden.

Agent-editable surface: The explore workspace source and any search/construction logic. Formal replay applies only candidate.patch to a fresh workspace.

Fixed or forbidden components: The ingredient set, frozen CLIP architecture and preprocessing, class mapping, proxy/final evaluators, no-network boundary, and final-only asset mount.

Proxy evaluator: imagenetv2_top1_proxy2000, maximize, 2,000 rows and 1,000 classes. Exploration receipts show class-clustered standard errors of about 0.01064--0.01092. It is a search signal, not the final score.

Final evaluator: imagenetv2_top1_full10000, maximize, 10,000 rows. Artifact-4/5/6 class-clustered standard errors are 0.0067066, 0.0066527, and 0.0067064. Each summary has a matching .complete receipt, making each result official.

Artifact contract: One compatible state dict in the affine hull of the 72 ingredients, with coefficients summing to one. Negative or extrapolative coefficients are allowed. Final scoring reports the 2,000-row overlap and the remaining 8,000 rows separately. Proxy and final scores have different protocols and sample counts and must not be treated as directly comparable.

3.2 How the baseline works

72 read-only ingredient state dicts -> fixed coefficient 1/72 for every ingredient and a per-key average (non-floating keys are copied) -> frozen CLIP produces top-1 accuracy on the proxy images -> no gradient update or label generation; the averaged state dict is written as a checkpoint -> the harness publishes checkpoint-1 and later checks loadability and affine-hull membership.

The baseline has no trainable parameters; “training” is one model-construction pass. The agent explicitly diagnosed a possible loss of complementarity from uniform weights and proxy overfitting when only two images per class are available.

4. Four-hour exploration and decision process

The agent spent the early period establishing uniform, single-model, and evaluator controls, then focused on proxy overfitting, differentiable coefficient learning, split-half validation, and formal-pipeline checks. The main run lasted 10,616 seconds and ended with 3,836 seconds remaining.

U-01 - Are a single model, greedy subset, or top-k enough?

Motivation and hypothesis. Determine whether the task is solved by choosing one strong ingredient or whether complementary ingredients help; also test whether the two proxy offsets induce incompatible rankings.

Concrete change and experimental setup. exp1 scored all 72 singles, uniform, and strict_greedy on 2,000 proxy rows and separately ranked offset-0 and offset-1. exp2 scanned top-k subsets under those rankings and measured fit versus held-out halves.

Observed result. Uniform scored 0.6880 and the best single, model_69, 0.6935. The all-row strict greedy subset scored 0.7025. Offset-0 greedy had 0.697/0.708 on offsets 0/1; offset-1 selection had 0.678/0.714, showing unstable offset-specific choices. A representative offset-0 top-k=6 score was 0.6965, while all 72 returned to 0.688; fitted candidates often lost on held-out rows.

Agent interpretation. The agent viewed greedy as evidence of complementarity but regarded offset-specific and direct top-k selection as chasing individual proxy rows.

Report assessment and confounds. All measurements use the same 2,000-row proxy and single evaluations have roughly 0.011 standard error. Offset difficulty differs, so absolute cross-offset comparisons are limited. The unit supports a regularized continuous search, not a demonstrated greedy generalization gain.

Decision and consequence. Keep greedy as an ablation and fallback; reject single-model, top-k, and offset-specific rules for submission; move to continuous affine coefficients.