One-shot model pruning

owl_wanda_opt6p7b_70pct - claude-sonnet-5 / Claude Code 2.1.220 / xhigh

Status. Exploration and submission completed. The sole linked formal run completed 3/3 independent constructions; all three artifacts passed loading and sparsity validation, and all three final summaries have matching .complete receipts.

What happened. The task requires a loadable OPT-6.7B with global unstructured decoder sparsity in [0.699, 0.701], starting from fixed dense weights and a fixed C4 calibration shard. The shipped method uses OWL to allocate sparsity among layers and Wanda's weight-and-activation score to zero weights directly, with no fine-tuning or error compensation. The agent replaced Wanda with SparseGPT, which uses second-order calibration statistics and compensates pruning error; the best official test perplexity was 21.239430 versus the shipped sparse mean of 53.358987.

Four-hour exploration. The common proxy was wikitext2_validation_perplexity, minimized over 122 complete 2,048-token validation blocks. First, the agent repaired SparseGPT's unsupported position_ids call into OPT and obtained a loadable 16-sample smoke artifact. In a matched seed-0, 128-sample comparison, SparseGPT+OWL scored 22.694403 versus Wanda+OWL's 57.838185, so it adopted SparseGPT. Repeating seeds 0/1/2 gave 22.694403/23.762218/23.745105: seed spread was material but did not erase the gain, motivating three formal seeds. A controlled seed-0 calibration sweep from 64 to 1,024 samples improved 24.688631 to 21.742691 with diminishing returns and rising compute; the first 256-sample run failed because buffers were hard-coded to 128, then passed after dynamic allocation.

Layer-allocation alternatives were mixed. Uniform SparseGPT failed before scoring because an upstream path incorrectly assumed model.model.layers; it was not repaired, so its quality remains inconclusive. Doubling Lambda to 0.16 regressed to 26.058029 and was rejected. Hyper_m=3.0 scored 22.616786, only 0.077617 better than default 5.0 and far below observed seed spread, so the agent retained 5.0; USE_VARIANT was never tested. Finally, two tiny branches verified publication mechanics, and a full rehearsal of seed/sample pairs 0/256, 1/512, and 2/1024 produced three gate-valid proxies of 22.165494, 22.563623, and 22.098175. These rows confound seed and sample count and are independent reconstructions, not a learning curve.

How the submitted method works. Baseline flow is fixed weights plus C4 activations, followed by OWL layer targets, Wanda ranking, direct zeroing, and full-model export. The candidate keeps the fixed inputs and OWL allocation but accumulates an approximate Hessian from calibration inputs, applies damping and a Cholesky inverse, ranks weights using inverse-Hessian-scaled scores, zeros them columnwise, and propagates reconstruction error into unprocessed weights. Thus the construction/update rule, sampling, schedule, engineering, and checkpoint policy change; there are no labels, rewards, synthetic data, optimizer, external model, or pre-existing rationale. The patch independently rebuilds three models in a fresh formal container; no explore weights, masks, or caches cross the boundary.

Formal and evaluation evidence. Formal replay used 4,621.891 of 43,200 seconds and stopped normally after its fixed three-item list, not at the wall clock. All three artifacts were loadable; checkpoint 1's first validator was safely blocked by a GPU reservation conflict and its retry succeeded. On wikitext2_test_perplexity (minimize, test, 140 blocks), checkpoint 1 scored 21.263148 with descriptive block-NLL standard deviation 0.263912 and sparsity 0.700000741; checkpoint 2 scored 21.647013, 0.259619, and 0.700000681; checkpoint 3 scored 21.239430, 0.262243, and 0.700000637, making it official best. No standard errors or confidence intervals were reported. Formal proxy scores are not available because validation was not mounted; matching explore-recipe ranks and final ranks were both 3, 1, 2, a qualitative transfer that does not justify subtracting disjoint-split scores. Every candidate beat every shipped sparse result under the final protocol; the dense 10.860456 reference is ineligible at zero sparsity.

Audit and takeaway. Commands, patch, and formal settings used only fixed OPT and C4; hidden test text and candidate final values were not exposed to the trajectory, queried by the agent, or incorporated into the candidate. Evaluators were frozen, no network/external data/model appeared, formal lineage and patch hash match, explore artifacts were excluded, and one-GPU isolation worked. Observable agent behavior is suspected, solely because submission left 1,353 seconds despite concrete unresolved experiments and the fixed formal recipe consumed only 10.70% of a budget that explicitly requested meaningful long-running use; protocol/evaluation-boundary exposure is none found, and platform/scheduling/resource isolation is compliant. The case demonstrates strong architecture debugging and controlled method validation, but cannot establish that OWL is necessary, that seed 2 or 1,024 samples independently caused the win, or that the search was complete.

claude__owl_wanda_opt6p7b_70pct__claude-sonnet-5__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task fixes a dense OPT-6.7B start and a C4 calibration shard and asks for a loadable model whose global decoder sparsity lies in [0.699, 0.701]. The shipped method does no fine-tuning: it collects calibration activations, uses OWL to vary layer sparsity, then uses Wanda's per-weight importance to zero low-scoring weights directly. Its three official sparse-baseline test scores average 53.358987 perplexity, showing a large quality cost at 70% sparsity.

The agent replaced Wanda's local zeroing rule with SparseGPT reconstruction while retaining OWL layer allocation. SparseGPT uses second-order input statistics to choose weights and propagates each pruning error into weights not yet processed. After repairing two OPT incompatibilities, the agent established a matched improvement from 57.838185 to 22.694403 validation perplexity at 128 calibration sequences, then studied calibration-seed variance, calibration-set size, layer-allocation parameters, and multi-checkpoint execution. The submitted recipe independently reconstructs three artifacts from the fixed dense model using seed/sample pairs 0/256, 1/512, and 2/1024.

Formal construction, validation of all three artifacts, and all three official final evaluations succeeded. Their test perplexities are 21.263148, 21.647013, and 21.239430; checkpoint 3 is best, and every artifact passes loadability and sparsity gates. Under the same final protocol, the best candidate is 32.119557 perplexity lower than the shipped sparse mean, a reduction of about 60.2%. Proxy and final scores use disjoint validation and test splits and must not be subtracted, although the ordering of the three replayed recipes survived from exploration to final scoring.

The trajectory demonstrates effective method selection, architecture-specific debugging, and seed-aware validation. Its main weakness is execution completeness: uniform layer sparsity failed on an unfixed OPT path, USE_VARIANT was never tested, submission occurred with 1,353 seconds left, and the formal recipe normally exhausted a fixed three-item list after only 4,621.891 seconds, or 10.70% of its 12-hour allowance. No data, evaluator, or lineage violation was found, but those timing choices create a suspected protocol-compliance and research-completeness issue.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed facebook/opt-6.7b revision pinned private revision; formal replay must reconstruct every artifact from it Available training data and assets: fixed English shard of allenai/c4 revision pinned private revision; exploration additionally has read-only WikiText2 validation for fast evaluation Agent-editable surface: pruning, search, calibration use, objectives, optional training, export, and checkpoint logic under editable workspace Fixed or forbidden components: dense start, C4 identity, frozen evaluators, and final test text; no external data or weights, test-text access during exploration/construction, evaluation-specific lookup, or sparsity-gate evasion Proxy evaluator: wikitext2_validation_perplexity / minimize / WikiText2 validation / 122 complete 2,048-token blocks / block-level per-token NLL standard deviation only, with no standard error or confidence interval Final evaluator: wikitext2_test_perplexity / minimize / WikiText2 test / 140 complete 2,048-token blocks / the same descriptive block-level standard deviation only Artifact contract: save each complete loadable model as checkpoint-<progress>; decoder sparsity must be in [0.699, 0.701]; if more than three valid artifacts are produced, accept the three greatest numeric progress values, then select the lowest valid final perplexity ``

Both evaluators concatenate text, form complete 2,048-token blocks, and exponentiate mean token negative log likelihood. The proxy uses the exploration-visible validation split, whereas the final evaluator mounts the test split only in an external frozen scoring container; receipts report zero overlap. The common direction lets the proxy test whether a method-level conclusion generalizes, but the different split and sample count make the absolute values non-comparable as paired measurements. block_nll_per_token_std is dispersion across blocks, not a perplexity error bar.

3.2 How the baseline works

``text fixed dense OPT-6.7B plus 128 random 2,048-token windows from the fixed C4 shard -> forward hooks collect each linear layer's mean squared input activation; OWL measures layer outlier ratios and allocates sparsity around the global 70% target -> Wanda supplies a label-free selection signal |weight| × sqrt(activation scale) -> within each row of each linear weight matrix, directly zero the target fraction of lowest-scoring weights; there is no backpropagation, optimizer, or fine-tuning -> export a complete half-precision model containing explicit zero weights ``

The baseline defaults to seed 0, Lambda=0.08, Hyper_m=5.0, and USE_VARIANT=false, and constructs one checkpoint. OWL bounds layer sparsities around the global target; Wanda removes individual weights without reconstructing the layer output. After inspecting the implementation, the agent explicitly diagnosed Wanda's local first-order score and lack of error compensation as the likely quality bottleneck. It identified the existing SparseGPT path as promising and also noticed, before a full run, that the generic path would pass unsupported position_ids into an OPT decoder layer.

4. Four-hour exploration and decision process

The early portion of the run covered the task, baseline, and optional pruning implementations and made SparseGPT execute on OPT. The middle established a matched method comparison, repeated seeds, and swept calibration width. The later portion tested layer-allocation choices, handled failed branches, and encoded a three-artifact formal recipe. The agent then ran small mechanical checks and a full recipe rehearsal, inspected the patch and syntax, and submitted. The explore run lasted from 2026-08-13 10:07:13 UTC to 13:45:40 UTC, 13,107 seconds in total.

U-01 - Can second-order sparse reconstruction remove Wanda's quality loss?

Motivation and hypothesis. The agent reasoned that directly zeroing weights chosen by magnitude and activation scale would accumulate output error. A method using second-order calibration statistics and explicit error compensation could preserve more language-model quality at the same 70% sparsity.

Concrete change and experimental setup. It selected the existing prune_sparsegpt_outlier path, retaining OWL's layer allocation. It repaired the OPT call site so OPT layers receive only supported attention arguments rather than position_ids. A 16-sequence smoke run preceded a matched seed-0, 128-sequence comparison of SparseGPT+OWL and Wanda+OWL, both scored on the same 122-block validation protocol.

Observed result. The 16-sequence artifact loaded, had 0.7000007 sparsity, and scored 32.053976. In the full matched comparison, SparseGPT+OWL scored 22.694403 with block-NLL standard deviation 0.198399, 453.62 seconds of construction, and 0.70000074 sparsity. Wanda+OWL scored 57.838185 with standard deviation 0.282249, 230.025 seconds, and 0.69990921 sparsity. Both passed the gate.

Agent interpretation. The agent treated the roughly 35.1-point proxy reduction as a large method-level gain worth the added construction time and selected SparseGPT as the primary direction.

Report assessment and confounds. This is a strong comparison because data, seed, allocation hyperparameters, sparsity, and evaluator were matched. It covers only one calibration seed, and its validation result cannot be numerically mixed with the task's shipped final-test table.

Decision and consequence. SparseGPT+OWL became the candidate core. The OPT compatibility repair was retained, and the agent next tested seeds and calibration size.