One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-luna / AI4AI-Bench v1.5 / xhigh

Status. Exploration and explicit submission completed; the sole linked formal replay, three artifact validations, and three official final evaluations all completed. Boundary review found one confirmed early-submission protocol violation, but no data, hidden-test, network, lineage, or GPU-isolation breach.

What happened. The task requires a loadable OPT-6.7B whose decoder linear weights are about 70% sparse. Shipped Wanda ranks weights by magnitude times activation scale, while OWL assigns different sparsities to layers; it scored 57.8382 validation perplexity. The agent replaced this with OPT-correct SparseGPT, which uses calibration activations to approximate second-order pruning error and compensates retained weights, then added mask-preserving next-token fine-tuning. The best official test perplexity was 16.1624, versus 53.3590 for the shipped sparse baseline mean under the same final protocol.

Four-hour exploration. The common proxy was lower-is-better WikiText-2 validation perplexity over 122 fixed 2,048-token blocks. Uniform Wanda worsened to 181.7436, so OWL was retained for Wanda. Uniform SparseGPT improved to 21.3580; adding OWL at two allocation strengths worsened it to 22.5182 and 21.5114, so the agent adopted uniform SparseGPT and rejected that combination. Seeds 0, 1, and 2 scored 21.3580, 22.2979, and 22.2875, supporting a robust large gain but showing calibration-sample variance. After repairing a hard-coded 128-row activation buffer, increasing calibration from 128 through 256 and 512 to 1,024 sequences improved monotonically from 21.3580 to 20.6880. Because the last doubling gained only 0.0418 while raising pruning time to about 1,082 seconds and memory near 50 GiB, the agent selected 512.

Mask-preserving causal-language-model fine-tuning first failed because finetune_steps was not forwarded; a float16 retry produced NaN on its second update. Bfloat16 fixed stability. At 512 calibration sequences, 8, 32, 128, and 256 updates improved to 19.1489, 18.4038, 17.5627, and 17.1114. The last point also increased fine-tuning data from 128 to 256 sequences, so steps and data coverage are confounded. All points were fresh reconstructions, not one continuous curve. Formal-driver debugging fixed a zero-wall-clock interpretation and verified patch application, loading, hashes, sparsity, atomic publication, and three-artifact retention. The submitted 512-update setting and fine-tuned seeds 1–2 were not scored during exploration.

How the submitted method works. The baseline flow is C4 activations to Wanda ranking to OWL allocation to one-shot zeroing. The candidate uses 512 C4 sequences for SparseGPT's curvature estimate, zeros 70% uniformly with error compensation, then uses 256 C4 sequences for 512 bfloat16 AdamW updates at learning rate 1e-5. Next-token cross-entropy is the only training target; gradients and post-step re-zeroing preserve the sparse mask, while other dense parameters may update. There are no external rewards, synthetic labels, or saved rationales. This changes the algorithm, training signal, trainable parameters, schedule, engineering, and checkpoint policy. Exploration weights did not cross the boundary; formal replay rebuilt each candidate from the fixed dense model.

Formal and evaluation evidence. Formal replay used 41,517.382 of 43,200 seconds and stopped after cumulative publication 51 because about 1,124 seconds remained, below its 1,800-second start reserve. Publications were independent fresh candidates; only three existed simultaneously. Checkpoints 49, 50, and 51 used seeds 0, 1, and 2, all completed 512 updates, loaded successfully, and passed sparsity at approximately 0.700001. Official lower-is-better WikiText-2 test perplexities over 140 blocks were 16.2461, 16.1624, and 16.4791; checkpoint 50 was best. Their block-NLL standard deviations were 0.253726, 0.256059, and 0.252065, diagnostic spreads rather than error bars. Formal artifacts lacked proxy scores, and validation used a disjoint split, 122 blocks, and 256 updates, so proxy and final values cannot be directly differenced or used to prove that 512 beats 256.

Audit and takeaway. The fixed C4 source, OPT revision, withheld test text, evaluator, no-network rule, clean explore-to-formal rebuild, one-GPU isolation, wall clock, patch hash, and all three summary.json/.complete pairs checked out. Test content was not exposed, reconstructed, or used; no task-package defect or protocol exposure was found, and platform/scheduling/resource isolation was compliant. Observable agent behavior is nevertheless confirmed: the task explicitly forbade early submission while a meaningful experiment could finish, yet the agent submitted with 5,346 seconds left and an idle GPU; comparable formal candidates later took 780–857 seconds. This limits research completeness without invalidating the artifacts. The evidence establishes a large SparseGPT-plus-masked-fine-tuning gain, but not that 512 updates or 512 calibration sequences are optimal, nor that small seed differences are statistically significant.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-luna__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks the agent to start from a fixed dense facebook/opt-6.7b revision and a fixed English C4 calibration shard, construct a loadable model with about 70% unstructured sparsity in decoder linear weights, and minimize WikiText-2 perplexity. The shipped baseline is Wanda, or activation-aware weight pruning: it ranks weights using weight magnitude times input-activation scale. OWL then varies sparsity across layers using their outlier rates. The procedure has no gradient-training stage. The agent reproduced a validation perplexity of 57.8382, found that activation capture was hard-coded to 128 samples, and found that the repository's pre-existing SparseGPT path was not safe for OPT without adapter work.

The principal method change was OPT-correct SparseGPT, a second-order pruning method that approximates the curvature of weight-reconstruction error from calibration activations and compensates unpruned weights as each block is zeroed. Uniform per-layer sparsity reduced validation perplexity to 21.3580, while reintroducing OWL allocation made it worse. The agent then tested calibration seeds, expanded calibration from 128 to 1,024 sequences, and added causal-language-model fine-tuning that permanently masks pruned positions. After fixing one missing-argument failure and float16 NaNs, bfloat16 fine-tuning with 512 calibration sequences, 256 fine-tuning sequences, and 256 steps produced the exploration best of 17.1114. The submitted source extrapolated the formal setting to 512 steps, which had not itself been scored during exploration, and formal replay rebuilt models from the fixed start rather than carrying over exploration weights.

Formal replay succeeded for 41,517.4 seconds and cumulatively published 51 freshly constructed candidates. Retention kept only three at any one time, ending with progress 49–51. All three passed standard model loading and tensor-sparsity validation. Their official WikiText-2 test perplexities were 16.2461, 16.1624, and 16.4791; seed 1 at progress 50 was best. These results are dramatically better than the shipped sparse baseline's 53.3590 mean under the same final protocol. They must not be numerically differenced against exploration validation scores because the splits and sample counts differ.

The run demonstrates strong algorithm replacement, ablation, numerical debugging, and formalization. No data, model, hidden-test, network, lineage, or GPU-isolation violation was found. There is nevertheless a confirmed execution-protocol violation: the task expressly required continued work when a meaningful experiment could still be completed and interpreted, but the agent explicitly submitted with 5,346 seconds left, an idle assigned GPU, and its intended 512-step configuration still untested. Comparable formal candidates later took about 780–857 seconds each. This finding concerns early submission; it does not invalidate the lineage, artifacts, or official scores that were subsequently produced.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: dense facebook/opt-6.7b at fixed revision pinned private revision Available training data and assets: fixed C4 English shard 0; WikiText-2 validation proxy during exploration; task-provided source, baseline, and local model assets Agent-editable surface: candidate implementation under editable workspace, including OWL code, schedule, export, and checkpoint logic Fixed or forbidden components: starting weights and prescribed data source, evaluators, final test data, no-network boundary, one-GPU and wall-clock limits; no test-set use during exploration or formal replay Proxy evaluator: wikitext2_validation_perplexity / lower is better / WikiText-2 validation / 122 blocks of 2,048 tokens / no statistical error bar Final evaluator: wikitext2_test_perplexity / lower is better / WikiText-2 test / 140 blocks of 2,048 tokens / no statistical error bar Artifact contract: checkpoint-<numeric progress>; at most the three greatest-progress checkpoints retained and accepted; decoder-linear tensor sparsity in [0.699, 0.701]; loadable through the standard model loader ``

Both evaluators exponentiate autoregressive negative log likelihood over fixed-length text blocks into perplexity. The validation proxy drives exploration decisions, while the disjoint test split is used only for official evaluation. Validation has 122 blocks and test has 140, with 0% overlap, so they support qualitative transfer claims but are not repeated measurements from one distribution. The reported block_nll_per_token_sd is the spread of per-token negative log likelihood across the fixed blocks. It is a diagnostic standard deviation, not a standard error of the mean or a confidence interval.

3.2 How the baseline works

``text fixed dense OPT-6.7B plus 128 length-2,048 C4 calibration sequences -> collect inputs to each decoder linear layer and score every weight as |W| times the square root of its input channel's mean squared activation -> no external reward or label; activation scale supplies the importance-selection signal -> OWL uses each layer's above-threshold outlier rate to assign nonuniform layer sparsities around the global 70% target, then each row zeros its lowest Wanda scores -> decoder linear weights become about 70% sparse and the otherwise unchanged full model is exported as a checkpoint ``

The baseline uses seed 0, 128 C4 sequences, lambda=0.08, Hyper_m=5, and no optimizer, training loss, or fine-tuning. It constructs one model and publishes checkpoint-1. The agent identified two concrete initial bottlenecks. First, the activation buffer had 128 rows regardless of the requested calibration count, so wider calibration could not actually work. Second, the repository's upstream SparseGPT route contained LLaMA-specific paths and could not be treated as a correct OPT alternative until its module dispatch was repaired.

4. Four-hour exploration and decision process

Exploration began at 16:50:58 UTC. The agent first established the shipped baseline and a uniform-Wanda control, then spent most of the run implementing and comparing OPT SparseGPT, layer allocation, calibration seeds, and calibration width. The latter portion added sparse fine-tuning, repaired numerical and argument-plumbing failures, swept update counts, and hardened the formal driver. Syntax, patch application, loadability, hashes, sparsity, and retention were rechecked before submission. Exploration ended at 19:22:32 UTC after 9,094 seconds, leaving the remaining budget unused.

U-01 - Is Wanda limited by its score or rescued by OWL layer allocation?

Motivation and hypothesis. The agent needed a measured shipped baseline and a uniform-sparsity control to isolate whether OWL's cross-layer allocation mattered.

Concrete change and experimental setup. Both candidates used seed 0, 128 C4 calibration sequences, and the same Wanda row-wise ranking. One retained OWL at lambda=0.08; the other imposed approximately 70% sparsity on every layer. Both used the common 122-block WikiText-2 validation proxy, with lower perplexity better.

Observed result. The shipped method scored 57.838185 at tensor sparsity 0.699909, block-NLL standard deviation 0.282249, and about 159 seconds of pruning. Uniform Wanda deteriorated to 181.743562 at sparsity 0.699951, standard deviation 0.385130, and about 64 seconds. An initial evaluation pointed at the wrong artifact subdirectory and failed; correcting the path to the actual export produced the valid baseline result.

Agent interpretation. The agent treated uniform Wanda as a strong negative control and concluded that Wanda's importance estimate depended heavily on OWL's nonuniform layer allocation.

Report assessment and confounds. The gap is large but comes from one calibration seed. It establishes that OWL helps this Wanda implementation, not that OWL will help every pruning rule.

Decision and consequence. Uniform Wanda was rejected, the shipped method became the reference, and the agent sought a stronger within-layer update rule.