One-shot model pruning

owl_wanda_opt6p7b_70pct - claude-opus-5 / Claude Code / max

Status. Exploration, submission, formal retraining, checkpoint loading, and final scoring completed. Formal retraining used 38,745.271 seconds; checkpoints 2–4 were valid and every final summary has a matching .complete. Numbered runs and control/queue/correction records show no extra resource-only attempt; exploration auto_retrain=false describes that manifest only, while the independent formal manifest confirms replay. The agent submitted explicitly while idle with 2,929 seconds left, so timing is confirmed noncompliance; other data, evaluator, and resource boundaries are clean. The recipe's best gate-valid test perplexity is 13.041221616727807.

What happened. The task builds a loadable globally 70% unstructured-sparse OPT-6.7B model from fixed C4. The baseline uses Wanda importance plus OWL allocation, then zeros weights without updating survivors; validation proxy was 57.8382 on 122 blocks. The agent identified missing reconstruction and submitted a four-file recipe adding reconstruction, masked refinement, and dense-teacher KD. Formal replay rebuilt from the dense start.

Four-hour exploration. All proxy values are WikiText2 validation perplexity (lower is better, 122 blocks). e1 replaced zeroing with SparseGPT/OBS masking plus CG reconstruction and reached 22.4131 in about 584 seconds, so reconstruction was adopted. Cross-moment reconstruction then failed (e2 6,808.86; e3 with 100 Adam steps 51.7521), while self-target CG (e4) reached 22.3626; the agent blamed dense/sparse trajectory mismatch and fixed cross=0,target=self. Layer-refinement/calibration runs e5–e7 reached 20.5691, 19.7005, and 19.6064; v2 with 512 calibration samples reached 19.2417 and loaded, supporting wider calibration but not single-factor causality. KD runs e8/e9 (1e-5/4e-5, 552 steps) reached 15.5911/15.3386; refined-start e10 (3e-5, accumulation 8, 200 steps) reached 14.8465 and was adopted. The e8 held-out guard conflict is inconclusive, though its proxy receipt is complete. v1/v2 tested staging; v1 published valid models but ended with an unbound variable after evaluation, while later checks and formal replay succeeded. Measured idle time still fit another e10-sized experiment, confirming early-submission noncompliance.

How the submitted method works. OWL allocates layer sparsity; SparseGPT/OBS freezes a mask and damped CG reconstructs each layer with the dense layer on sparse inputs as target. Masked layer-wise Adam updates surviving weights, biases, and LayerNorm. A fixed dense teacher supplies KL targets on C4 blocks; AdamW, accumulation, checkpointing, cosine decay, and mask reapplication perform KD. No external data, rewards, labels, or pre-existing reasoning are used. This is a mixed algorithm/training-signal/schedule/engineering/checkpoint change. KD_STEPS=1,000,000 is only an upper bound; formal receipts show 10,292 actual steps from a fresh dense start.

Formal and evaluation evidence. Runtime receipts record 512 calibration sequences, CG 80, refinement ceiling 3,000, KD learning rate 3e-5, internal wall limit 41,700 seconds, and one GPU; the run exited normally after 38,745.271 seconds. Four progress publications accumulated, while retention kept three (2–4). Final test perplexities (minimize, 140 disjoint blocks) were 16.5098154224176 (artifact 2, sd 0.25733), 13.276840550074192 (artifact 3, sd 0.26704), and 13.041221616727807 (artifact 4, sd 0.26673). All sparsities were about 0.700001, all loaded, and artifact 4 was selected. Formal replay has no independent proxy receipt, so protocols are not directly compared; the improving direction survived. No seed sweep or standard error supports robustness or one-parameter causality.

Audit and takeaway. Pinned OPT-6.7B, C4, the evaluator, no-network rule, one-GPU limit, and patch hash were checked. Searches of evaluator-only code, the complete trajectory/tool results, patch, and commands found no test text, hidden score, external weight, or network input; visible metrics and public references are protocol metadata. No task-package defect affecting science was found. Observable agent behavior is confirmed only for early submission; protocol exposure is none found; platform, scheduling, and resource isolation are compliant. Formal underuse is clean because the recipe is clock-driven, not a short fixed endpoint. The case shows strong structured ablation and reproducible sparse construction, but early submission and one seed limit the conclusion.

claude__owl_wanda_opt6p7b_70pct__claude-opus-5__max - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: owl_wanda_opt6p7b_70pct (ai4ai/owl-wanda-opt6p7b-70pct) Task family: construction and quality improvement of a globally 70% unstructured-sparse model from a fixed dense language model Evaluated coding-agent model (from trajectory ID / manifest; not the task model): claude-opus-5 Harness: Claude Code 2.1.220 / AI4AI runtime Reasoning effort: max Seed, if applicable: 0 in both exploration and formal configuration; no seed replication Exploration budget: 14,400 seconds (4 hours); 11,533 seconds elapsed, with 2,929 seconds remaining at submission Formal budget: 43,200 seconds (12 hours)

Primary explore run: explore manifest and raw trajectory JSONL Why this attempt was selected: job_manifest.json lists one primary exploration run and no prior_attempt_dirs. The run has a normal .explore.complete, a complete lifecycle record, and 14 proxy-evaluation receipts; there is no same-configuration full rerun to choose instead. Prior full-run attempts and outcomes: none. This is one exploration attempt, not a successful retry after a failed full run. Same-configuration audit: I also checked the numbered run directories, both control/attempts and control/jobs status records, and the formal corrections directory; no extra attempt stopped at resource gating, queueing, or reservation without producing a manifest. Exploration auto_retrain=false and retrain_phase_created=false describe that exploration manifest only and do not explain upstream scheduling; the independent job_manifest and formal manifest confirm the formal run. Patch hash / validation: both are 62,920 bytes with SHA-256 verified private digest. The formal log records all four files applied cleanly; the exploration tail records shell syntax, Python compilation, and idle-GPU checks. Formal run: formal manifest and formal outcome Formal selected-exploration lineage: exploration attempt 1; the formal manifest's source path matches the primary explore path exactly. Final-evaluator receipts: artifacts 2, 3, and 4 each have both summary.json and a matching .complete, and all three summaries are passed; artifact 4 is the official best by the minimize-direction test metric.

Evidence completeness: the exploration, submission, formal retrain, checkpoint validation, and final-evaluation layers all have direct file evidence. The manifest reports no prior attempt and no unlinked or partial formal run. Missing or conflicting evidence: the formal control status.json still ends with status="running", although it also reports formal_result="succeeded" and exit_status=0; .retrain.complete, .formal.complete, outcome.json, and all validation/final receipts agree that the run completed, so the lifecycle field itself is inconclusive and the terminal/result receipts are used. Exploration e8's build log reports held-out C4 loss 2.62965 while kd_metrics.json leaves kd_best_val at the initial 2.86559; that guard field is inconclusive, but e8's independent proxy score is usable. The formal manifest's termination_reason=agent_explicit_submit is a lineage label and does not by itself say when formal training stopped; the formal retrain result records normal exit.

2. Reader-facing overview

The task asks for a loadable, globally 70% unstructured-sparse model built from a fixed OPT-6.7B dense model and fixed C4 calibration data. The baseline uses activation-aware Wanda importance and OWL layer-wise sparsity allocation to zero weights, but never reconstructs the removed behavior or updates the surviving weights. Its exploration proxy score was 57.8382 on 122 validation blocks.

During the four-hour window, the agent tested whether SparseGPT/OBS masking plus reconstruction was the main missing ingredient, compared self and dense-trajectory cross reconstruction targets, varied conjugate-gradient (CG) and layer refinement, changed calibration width, and added global knowledge distillation (KD). The cross target produced a decisive 6,808.86 failure (and only 51.7521 after refinement), while the best exploration result, e10, combined a refined 512-sequence starting point with 200 actual KD steps and reached validation perplexity 14.8465. The submitted object was the four-file recipe patch, not e10's weights, so formal replay rebuilt the model from the dense start.

Formal replay completed in 38,745.271 seconds (89.688% of the 12-hour wall-clock budget), published progress 1 through 4, and retained 2, 3, and 4 simultaneously. All three retained models loaded and passed the sparsity gate. Official WikiText2 test perplexities were 16.5098, 13.2768, and 13.0412, with artifact 4 best. Proxy and final numbers use different splits and block counts, so they are not direct differences, but the direction that continued distillation improved quality was preserved.

The strongest capability is structured ablation followed by a reproducible, clock-aware long-run recipe. The main failure is protocol-related: the agent explicitly submitted while idle with 2,929 seconds left even though an e10-sized construction took about 709 seconds, despite the instruction to continue useful work. Literal searches found no test text, external weights, or score lookup in the visible trajectory or patch. The final score is a valid fixed-protocol result, but one seed and no formal proxy sweep cannot establish cross-seed robustness or isolate one hyperparameter's causal effect.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: facebook/opt-6.7b@pinned private revision, mounted in the run container as fixed-asset mount.

Available training data and assets: a pinned C4 English shard (allenai/c4@1588ec...) at fixed-asset mount; exploration additionally exposes WikiText2 validation, while formal construction has no WikiText2 mount and final scoring mounts the test split. External data, weights, and network content are forbidden.

Agent-editable surface: source, scripts, and construction choices under editable workspace, including masks, layer allocation, calibration count, reconstruction target, optimization, and checkpoint scheduling. Fixed or forbidden components: the dense start, C4 shard, frozen final evaluator, global sparsity window [0.699,0.701], no-network policy, and one-GPU resource boundary.

Proxy evaluator: wikitext2_validation_perplexity, minimize; WikiText2 validation, 122 complete 2,048-token blocks. block_nll_per_token_sd is between-block dispersion, not a standard error or confidence interval; exploration uses seed 0.

Final evaluator: wikitext2_test_perplexity, minimize; disjoint WikiText2 test, 140 complete 2,048-token blocks. The summaries report the same block spread and no sampling standard error; overfitting_delta=null does not prove absence of overfitting.

Artifact contract: emit complete loadable models under run output area>; at most three valid checkpoints are accepted and the official result is the best valid final score. The dense zero-sparsity reference is a quality reference, not a gate-valid submission. Shipped sparse references have test perplexities 53.9975, 52.6180, and 53.4615 for seeds 0–2 (mean 53.3590); these are protocol references, not reruns of this candidate.

Both evaluators compute perplexity from the same basic arithmetic, but their split and block count differ. Thus 14.8465 on validation and 13.0412 on test must be reported separately rather than subtracted.

3.2 How the baseline works

Fixed dense OPT-6.7B plus 128 random C4 windows (sequence length 2,048, seed 0) → collect mean-square input activation scalers for each decoder linear layer and compute Wanda importance |W| × sqrt(scaler) → let OWL allocate nonuniform layer sparsities from outlier ratios, then remove the lowest-scoring weights row by row → export one masked sparse model and check global sparsity → perform no loss optimization, learning-rate update, or surviving-weight training.

The agent identified “zeroing without reconstruction or fine-tuning” as the concrete bottleneck, supported by the baseline measurement rather than inferred only from the eventual improvement.

4. Four-hour exploration and decision process

The run first read the task and evaluator, established the baseline, and created the output structure. It then ran e1–e7 reconstruction/refinement ablations, e8–e10 distillation experiments, and v1/v2 end-to-end entry checks. The final roughly three minutes were spent on syntax, compilation, idle-GPU, patch, and snapshot checks before submission. All proxy scores below use WikiText2 validation, 122 blocks, minimize direction; sd means block dispersion.

U-01 - Is plain zeroing the bottleneck?

Motivation and hypothesis. The baseline zeros weights by importance but does not compensate for the lost layer output. The agent hypothesized that an OBS/SparseGPT mask followed by local reconstruction would recover quality.

Concrete change and experimental setup. e1 retained OWL layer allocation and 128 C4 calibration windows, replaced simple zeroing with SparseGPT/OBS masking plus CG reconstruction, and did not add gradient refinement. Every model was checked against the global sparsity gate.

Observed result. The baseline scored 57.8382 (sd=0.28225, sparsity 0.699909); e1 scored 22.4131 (sd=0.19829, sparsity 0.700001) after about 584 seconds and exited successfully.

Agent interpretation. The agent attributed the roughly 35-point improvement to reconstructing layer-output information lost by zeroing and treated e1 as a credible fallback.

Report assessment and confounds. This is strong same-protocol, same-seed evidence, but there is no second calibration seed to separate method effect from seed variance.

Decision and consequence. Adopt reconstruction as the base for later directions and reject pure Wanda zeroing.