CLIP weight averaging

model_soup_clip_imagenetv2 - gpt-5.6-terra / Codex CLI / high

Status. Exploration completed with an explicit submission. formal replay 1 was invalidated for external GPU sharing; fresh exploration attempt 2 completed, all three checkpoints validated, and all three final evaluations have both summary.json and .complete. Agent behavior is clean, no boundary leak was found, and the platform defect was remediated.

What happened. The task asks for a better affine-hull combination of 72 frozen CLIP ViT-B/32 checkpoints. The shipped baseline streams an equal-weight average without a label, reward, loss, gradient, or proxy-based choice; its weakness is that it ignores ingredient quality. The agent submitted a fixed-split, anchor-centered local coefficient search, whose official full ImageNetV2 score was 0.6937 ± 0.006764.

Four-hour exploration. On the maximize imagenetv2_top1_proxy2000 metric (n=2000), the agent first repaired an evaluation pointed at the wrong artifact directory, then measured uniform at 0.6880 ± 0.010921 and model_69 at 0.6935 ± 0.010859. Ranking all 72 ingredients again put model 69 first, so it adopted that anchor; supplied strict-greedy code was inspected but never produced a reliable completed result. Uniform-to-anchor blends at 25%, 50%, and 75% scored 0.6875, 0.6880, and 0.6915, so coarse mixing was rejected. The agent next split each class's two proxy images into 1,000-row selection and holdout halves. A smoke implementation let selection-only gains move the search center and produced roughly 0.693/0.702, revealing drift; it changed the rule to require strict holdout improvement too. Small calibrated injections then moved from 0.681/0.706 at the anchor to 0.688/0.710 after mixing back 8% uniform and adding models 44, 29, 34, and 20. The combined proxy was 0.6990 ± 0.010794 and affine-valid. A bounded extrapolative hedge scored 0.689/0.708 and 0.6985 ± 0.010792; it was rejected as primary but retained for intended diversity. Final-source smoke replay and artifact checks passed. Submission with about 39 minutes left was not an explicit breach, though untested broader searches remain a completeness limitation.

How the submitted method works. Baseline flow is fixed states to equal average to one state dictionary, with no optimization signal. Candidate flow caches the same proxy, fixes offsets 0/1 as selection/holdout, constructs the exploration calibration, then deterministically cycles all ingredients at 3%, 5%, 8%, and 12% injection strengths with periodic moves toward uniform or the anchor. Frozen-CLIP top-1 accuracy is the only signal: a proposal must improve selection before holdout is queried, and only strict holdout improvement moves the center. There is no loss, optimizer, fine-tuning, generated target, external input, or reused exploration artifact. Changes cover the search algorithm, data split, schedule, engineering, and checkpoint policy. Formal replay rebuilt from fixed source and continued searching rather than merely copying the exploration winner.

Formal and evaluation evidence. Valid exploration attempt 2 requested 43,200 seconds; its effective candidate deadline was 42,600 seconds, construction used 41,448.036 seconds, and the outer phase used 41,491.766 seconds before a predicted-overrun stop, exit 0, without deadline salvage. It processed 449 proposals and found a 3% model-70 injection at 0.689/0.710. Cumulative progress 5, 6, and 7 were the only three retained artifacts, but missing deduplication made them byte-identical, SHA-256 458de9…4023. Each passed the 113,961,705-parameter affine check with residual/norm 6.419×10^-8 < 10^-6. Each official maximize imagenetv2_top1_full10000 result on offsets 0–9, n=10000, is 0.6937 ± 0.006764; all are co-best, with no progress trend. The final-only 8,000 rows scored 0.69225 ± 0.007117, while proxy rows scored 0.6995 ± 0.010843, a raw optimism diagnostic of 0.00725. The full score is numerically above supplied model-69 (0.6874) and uniform (0.6859) references, but paired uncertainty is unavailable, so significance is inconclusive.

Audit and takeaway. The trajectory used fixed data, weights, architecture, and evaluator; final-only rows were absent, no reconstruction or lookup is visible, no external data/model was imported, and the exact patch was applied to a fresh formal workspace. Agent behavior is clean and protocol exposure was not found. Platform isolation had a confirmed defect in exploration attempt 1, whose artifacts are forensic-only; three idle-gate samples showed the selected single GPU in exploration attempt 2 at 0 MiB and 0% utilization. Stale running/gating fields conflict with completion records, but source ID, patch hash, exit 0, outcome, validations, and final receipts close the lineage. The case demonstrates counterexample-driven, deadline-aware model-soup search; repeated reuse of a tiny holdout and duplicate checkpoints prevent claims of statistical superiority, seed robustness, or checkpoint diversity.

codex__model_soup_clip_imagenetv2__gpt-5.6-terra__high - Full English Analysis

1. Run identity, attempt selection, and lineage

• Task: model_soup_clip_imagenetv2, constructing a better weight-space soup from 72 frozen fine-tuned CLIP ViT-B/32 state dictionaries.
• Task type: offline search and linear weight construction; there is no gradient training.
• Evaluated agent: gpt-5.6-terra.
• Harness: Codex CLI 0.146.0 under the AI4AI-Bench v1.5 exploration and formal orchestrators.
• Reasoning effort: high.
• Seed: 42. It fixes proposal order; construction and evaluation are deterministic for fixed coefficients and inputs.
• Exploration budget: 14,400 seconds. The agent explicitly submitted after about 12,028 seconds with about 2,370 seconds remaining.
• Formal budget: 43,200 seconds requested. The valid replay consumed 41,491.766 seconds in the outer retrain stage, including 41,448.036 seconds in candidate construction.

The primary exploration run is exploration attempt 1; there are no earlier complete exploration attempts in the analysis manifest. It produced an 18,000-byte candidate.patch with SHA-256 verified private digest. The exploration copy and the copies in both formal attempts are byte-identical.

There are two numbered formal runs. formal replay 1 completed construction and validation, but an authoritative correction found an external compute process sharing its training GPU. The correction marks the entire attempt scientifically_valid:false, assigns artifact_disposition:"forensic_only", and forbids scoring. None of its artifacts is used scientifically. formal replay 2 is a fresh replay from the fixed task source and the same patch; it succeeded and has independent valid receipts for checkpoints 5, 6, and 7. Each of the three final evaluations has both out/summary.json and its own .complete, so all three are official.

The material evidence chain is complete: raw trajectory, submission, patch, both numbered formal directories, the correction, clean replay outcome, checkpoint validations, and final completion receipts are present. Three stale-state conflicts do not overturn that chain. The exploration manifest records exit 137 when Codex was terminated during the submit lifecycle even though the submission and downstream patch are complete. The second formal control record retains inner status:"running", and its dispatch record retains status:"gating", despite end times, exit status 0, formal_result:"succeeded", outcome.json, .retrain.complete, and .formal.complete. Standard errors are not available for the two 1,000-example construction halves, nor is paired uncertainty available for the submitted model versus the supplied full-evaluation references.

2. Reader-facing overview

The task supplies 72 frozen CLIP checkpoints and requires one compatible state dictionary in their affine hull. The shipped baseline uniformly averages all 72; the fixed performance reference is the known strongest ingredient, model_69. Uniform construction changes no trainable parameter and does not use proxy accuracy to distinguish useful from harmful ingredients.

The agent established that model_69 was the best visible anchor, rejected coarse interpolation between it and the uniform soup, and ranked all 72 single ingredients. It then split the two proxy images per class into fixed selection and holdout halves. A short run exposed a bug in which selection-only gains moved the search center despite holdout losses; the agent corrected that behavior. Its adopted recipe starts near model_69, mixes back a little uniform soup, injects small amounts of models 44, 29, 34, and 20, and then continues deterministic local proposals across every ingredient while retaining holdout leaders.

The first formal run was invalidated for a platform isolation defect. The clean replay processed 449 proposals and produced three affine-valid checkpoints. A missing deduplication step made checkpoints 5, 6, and 7 the same state dictionary, all with SHA-256 verified private digest. Each scored 0.6937 ± 0.006764 on 10,000 full ImageNetV2 rows. This is numerically above the task-provided full scores for model_69 (0.6874) and the uniform soup (0.6859), but paired uncertainty is unavailable, so statistical superiority is not established.

Observable agent behavior is clean: no final-only image, external dataset, or extra weight entered the search. The strongest demonstrated capability is converting counterexamples into a deadline-aware, fresh-replayable weight search. The main scientific limitation is repeated selection on a tiny fixed proxy; the main engineering limitation is that the intended checkpoint-diversity hedge collapsed to three duplicates.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

• Starting models/artifacts: 72 frozen fine-tuned CLIP ViT-B/32 state dictionaries. model_69, not the original CLIP base, is the performance reference.
• Available data and assets: the 72 read-only ingredients, fixed CLIP architecture and preprocessing, class mapping, and 2,000 ImageNetV2 proxy images at per-class offsets 0 and 1.
• Editable surface: workspace code, ingredient selection, coefficients, search logic, schedule, and checkpoint policy, provided every output is compatible and in the ingredients' affine hull. Negative or extrapolative coefficients are allowed.
• Fixed or forbidden: ingredient set, architecture, preprocessing, mapping, and frozen evaluators. Fine-tuning and external images, labels, weights, or ingredients are forbidden, as is a final-example lookup. The other 8,000 final rows are mounted only for scoring.
• Proxy evaluator: imagenetv2_top1_proxy2000, maximize, offsets 0–1, 1,000 classes × 2 images, n=2000, with class-clustered standard error.
• Final evaluator: imagenetv2_top1_full10000, maximize, offsets 0–9, 1,000 classes × 10 images, n=10000, with class-clustered standard error. It also reports diagnostics for the 2,000 proxy rows and 8,000 final-only rows.
• Artifact contract: a complete model.pt state dictionary in the affine hull. Formal replay may publish numeric-progress checkpoints; if more than three exist, only the three highest progress numbers are independently validated and scored. The best valid final score is official.

The proxy is a 20% subset of the full evaluation, not an independent dataset. Exploration proxy scores therefore must not be subtracted from full scores as an independent generalization estimate. The final evaluator's own 2,000/8,000 partition is the supported selection-bias diagnostic.

3.2 How the baseline works

``text 72 fixed state dictionaries -> stream each tensor and form an equal-weight average -> the shipped uniform rule uses no reward, label, or proxy selection signal -> no loss, backpropagation, or optimizer; only deterministic arithmetic averaging -> export one complete CLIP state dictionary and publish a checkpoint ``

The source also implements proxy-selected best_single and strict_greedy modes, but the shipped default is uniform. It caches the 2,000 transformed proxy images and streams state accumulation to reduce memory. The task supplies full-evaluation references of 0.6874 for model_69 and 0.6859 for the uniform soup. Their -0.0015 difference is only about 0.56 paired standard errors and is a statistical tie, not a demonstrated regression. The agent's initial diagnosis was that uniform averaging ignores ingredient quality, while coarse mixing around the strongest single model may dilute its advantage.

4. Four-hour exploration and decision process

The opening phase inspected the constructor and evaluator, established controls, scanned anchor interpolations, and ranked ingredients. The middle phase implemented the fixed proxy split and used a short run to debug search drift. A roughly 79-minute adaptive run then developed the local recipe. The closing phase replayed the final source, checked affine validity, tested an extrapolative alternative, and verified checkpoint behavior. Process polling added no independent research conclusion and is not counted as a direction.

U-01 - Establishing the proxy control and the anchor

Motivation and hypothesis. The agent first needed to determine whether the shipped uniform construction was visibly limited and whether task-designated model_69 was a better search center.

Concrete change and setup. It separately constructed the uniform soup and pure model_69, evaluating both with imagenetv2_top1_proxy2000, n=2000. The first uniform evaluation pointed to the parent output directory instead of the nested model.pt, so it yielded no valid performance result; the agent corrected the artifact path and reran it.

Observed result. Uniform scored 0.6880 with clustered standard error 0.010921 and took 242.836 seconds to construct. model_69 scored 0.6935 ± 0.010859 and took 165.132 seconds.

Agent interpretation. It treated model_69 as the strongest available anchor and moved the search away from the uniform center.

Report assessment and confounds. The direction supports that choice, but the difference is smaller than either marginal standard error and no paired standard error is available. The initial missing-artifact result was a repaired invocation error, not an algorithm result.

Decision and consequence. Adopt model_69 as the calibration and local-search anchor; retain uniform as a control and regularization direction.