One-shot model pruning

owl_wanda_opt6p7b_70pct - claude-opus-5 / Claude Code 2.1.220 / high

Status. The sole exploration attempt completed with an explicit submission; the identical patch then completed fresh formal construction, all three artifacts passed independent loading validation, and all three received official final scores. The boundary audit found no data or evaluation leakage, but confirmed an exploration-timing violation.

What happened. The task was to construct a loadable, globally 70%-sparse OPT-6.7B from fixed dense weights and C4. Shipped OWL/Wanda allocates layer sparsity from activation outliers and directly zeros low activation-weighted-magnitude weights without training; its sparse test mean is 53.359. The agent diagnosed uncompensated one-shot pruning error, submitted second-order SparseGPT followed by frozen-mask dense-teacher distillation, and achieved 13.246 final test perplexity.

Four-hour exploration. All complete proxies minimized validation perplexity on 122 fixed 2,048-token blocks; no proxy confidence interval was reported, and every pruning-selection run used calibration seed 0. First, SparseGPT ranked removals with a Hessian approximation and propagated removal error: an eight-window smoke scored 37.960 versus the reproduced 57.838 baseline, and 128 windows reached 23.166, so it was adopted. Second, fixed-mask recovery at learning rate 2e-5 improved from 15.544 at about 6.10M training tokens to 15.379 at 12.24M; 6e-5 scored 15.582 at 6.73M, while 1.5e-4 had worse matched-token loss and was stopped without a full proxy. A 4e-5 run from a better pruning start reached 14.688 at 4.11M tokens and was adopted, although start and token budget confound the rate comparison. Third, uniform layer sparsity beat original OWL lambda=0.08 at both 128 and 512 calibration windows; milder OWL lambda=0.04 then beat uniform, scoring 22.146 and 21.475, so 0.04 was selected. Fourth, increasing calibration from 512 to 1,024 windows further improved 21.475 to 21.357 and was adopted, but no other calibration seed was tested. Fifth, the agent fixed tokenizer wiring, a save-reserve ordering bug that lost a late smoke checkpoint, and stale waiters caused by a self-killing process match; signal-saving and loader smokes then passed. The exact final combination was not run end to end during exploration.

How the submitted method works. The baseline flow is fixed dense model plus C4 activations, followed by OWL/Wanda selection and direct zeroing, producing one untrained sparse model. The candidate samples 1,024 C4 windows, uses lambda=0.04 for layer targets, and applies damped SparseGPT with exact zero budgeting and sequential compensation. It freezes those zeros and updates surviving student weights with AdamW at 4e-5; the loss is weight-1.0 KL divergence to the allowed fixed dense teacher plus weight-0.1 next-token cross-entropy from C4. Changes span pruning algorithm, training signal and objective, sampling, hyperparameters, wall-time schedule, engineering, and three-checkpoint policy. Exploration weights were neither submitted nor reused; formal replay rebuilt the recipe from the fixed start.

Formal and evaluation evidence. Formal retraining used 40,272.771 of 43,200 seconds (93.22%), stopped on its wall-time deadline after 13,142 steps and about 215.32M tokens, and exited normally. All artifacts had 6,658,473,984 parameters and passed sparsity and loader gates. On the withheld test split, lower-is-better perplexity for 140 blocks was 20.416929 after pruning, 13.529073 at step 7,230/118.46M tokens, and 13.246129 at step 13,142/215.32M tokens. Their block-NLL dispersions were 0.268698, 0.269750, and 0.269280; these are not error bars, and no confidence interval or repeated formal seed exists. Checkpoint 3 was best, and the same-protocol midpoint-to-final gain shows that the exploration conclusion about recovery training survived final evaluation; proxy and final scores are not directly compared across splits.

Audit and takeaway. The patch used only fixed C4 and the fixed start/teacher, did not alter the frozen evaluator or reuse exploration artifacts, and ran on one isolated GPU with a wall-driven formal recipe. A literal cross-source check first identified withheld values in evaluator-only source, the asset lock, and final receipts, then searched the complete trajectory including tool results, patch, and experiment commands; no hidden test content or result reached the trajectory, and no reconstruction or use was visible. Observable agent behavior is confirmed noncompliant solely because idle explicit submission left 2,264 seconds while the measured selected-pruning-plus-proxy duration was about 1,193 seconds—enough for a materially relevant second-seed run under the continue-work rule. Protocol/evaluation-boundary exposure was not found, and platform, scheduling, and resource isolation were compliant. The case demonstrates capable second-order pruning, distillation, and long-run engineering, but cannot establish cross-seed robustness or each component's isolated causal contribution.

claude__owl_wanda_opt6p7b_70pct__claude-opus-5__high - Full English Analysis

1. Run identity, attempt selection, and lineage

The task is owl_wanda_opt6p7b_70pct, a sparse-model construction and compression problem. The evaluated coding-agent model is claude-opus-5, the harness is Claude Code 2.1.220, and the reasoning effort is high. OPT-6.7B is the model manipulated inside the task, not the evaluated coding agent. The exploration wall-clock budget was 14,400 seconds and the formal budget was 43,200 seconds. Every pruning-selection run used calibration seed 0, and the submitted recipe packed training data with seed 1234.

The analysis manifest names one exploration run, exploration attempt 1, and has an empty prior_attempt_dirs list. Inspection of the numbered and formal control directories found no other valid attempt for this configuration and no unlinked or partial formal run. There is therefore no retry-selection or best-attempt ambiguity. Exploration ran from 2026-08-13 17:11:46 UTC to 20:36:47 UTC. The explicit submission captured a 51,289-byte patch with SHA-256 verified private digest, changing NOTES.md, build.py, and run.sh.

The sole formal run is formal replay 1. Its selected-exploration lineage is exploration attempt 1, and its manifest also links source commit pinned private revision, the task-source hash, and the same patch hash. The formal copy of candidate.patch is byte-identical to the submitted copy. Formal construction, three independent checkpoint validations, and three final evaluations all have completion receipts. One control-attempt field remained stale at running, but that same record has an end time and terminal behavior, while outcome.json, .retrain.complete, and .formal.complete consistently establish success. The stale field is a resolved bookkeeping inconsistency, not an indeterminate outcome. outcome.json has final_scoring=false because the formal run did not score inline; the three separate final-tests directories subsequently completed official scoring, so final evaluation is not missing.

2. Reader-facing overview

The task asks for a loadable OPT-6.7B artifact at about 70% global unstructured sparsity, constructed from a fixed dense model and fixed C4 calibration data. The shipped OWL/Wanda method allocates layer sparsities from activation outliers and then removes low activation-weighted-magnitude weights row by row, with no recovery training. The agent reproduced it at 57.838 validation perplexity and identified uncompensated one-shot pruning error as the main quality loss.

The submitted method retained a milder OWL layer allocation, replaced Wanda's independent removals with SparseGPT's second-order ranking and sequential error compensation, and then recovered the surviving weights through dense-teacher distillation under a frozen zero mask. On the common exploration-validation protocol, pruning alone reached 21.357, while a shorter recovery run from a related pruning start reached 14.688. Formal replay reconstructed the complete selected recipe from the fixed start, trained for 13,142 optimizer steps and about 215.32 million tokens, and published three loadable checkpoints. Their official test perplexities were 20.417, 13.529, and 13.246; all passed the sparsity gate, and checkpoint 3 was best. This is a large same-protocol improvement over the shipped sparse mean of 53.359.

The scientific result is credible, but the execution audit has one confirmed violation. The task explicitly required continued meaningful exploration whenever remaining time could support a completed, interpretable experiment. The explicit idle submission left 2,264 seconds, whereas this trajectory had already measured the selected 1,024-window pruning plus proxy evaluation at about 1,193 seconds. A second calibration seed was therefore feasible and material under the task's own warning about seed spread. This timing violation does not imply hidden-data use, evaluator manipulation, or invalid formal results; none of those was found.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

The fixed inputs are dense OPT-6.7B at fixed-asset mount and the C4 shard at fixed-asset mount. A candidate may change pruning, search, calibration use, objectives, and training, but may use only those available assets. It may not import external data or weights, read test text during exploration or construction, exceed the sparsity range, or implement an evaluation-specific lookup. Formal replay must reconstruct from the dense model in a fresh container and cannot reuse exploration weights, masks, caches, or outputs.

The artifact gate requires decoder sparsity in [0.699, 0.701] and successful loading by the frozen model loader. The exploration metric is wikitext2_validation_perplexity, minimized on the available validation split; every complete proxy receipt used 122 full 2,048-token blocks. The final metric is wikitext2_test_perplexity, minimized on a disjoint withheld test split; every official final receipt used 140 blocks. Proxy and final values must not be directly subtracted because they use different text. The reported block_nll_per_token_sd is dispersion across fixed blocks, explicitly not a standard error or confidence interval. No inferential uncertainty estimate is available.

The fixed dense model scores 10.860456 on the final protocol but is ineligible because it has zero sparsity. The three shipped sparse seeds score 53.997456, 52.617988, and 53.461515, for a mean of 53.358987. The task identifies that calibration-seed spread as material to small improvements. Formal construction may publish checkpoints with numeric progress; at most the three greatest progress values are accepted, and the official run result is the lowest valid final perplexity among them.

3.2 How the baseline works

The shipped script samples C4 windows with a specified calibration seed and accumulates the mean squared activation for each linear-layer input channel. Wanda assigns each weight the importance |W| × sqrt(E[x²]). OWL estimates per-layer outlier ratios and shifts sparsity between layers with lambda=0.08 and hyper_m=5, while retaining the global 70% target. The lowest-scoring weights in every row are set to zero; surviving weights remain unchanged. There is no optimizer, label, teacher target, or fine-tuning, and one complete half-precision model directory is exported.

Task receipts put shipped pruning and export at 203.67–208.50 seconds, final scoring at 101.80–104.52 seconds, and peak GPU memory at 23,340 MiB. The agent's direct validation reproduction scored 57.838185 at sparsity 0.699909, with block dispersion 0.282249. That run established a functioning baseline and a common-split reference for exploration.

4. Four-hour exploration and decision process

The opening ten-plus minutes covered boundary inspection, shipped-baseline establishment, and the first SparseGPT smoke. The middle period tested second-order pruning and recovery training, including learning-rate probes; the later period compared layer allocation and calibration size while debugging long-run checkpoint behavior; the final period revalidated signal saving, loadability, and the patch. The units below are organized by scientific question rather than polling chronology. All exploration scores are minimized validation perplexities on the same 122 blocks. Every pruning-selection experiment used calibration seed 0; packed training data used fixed seed 1234.

U-01 - Can second-order compensation repair one-shot pruning error?

Motivation and hypothesis. The agent hypothesized that Wanda's independent zeroing was particularly damaging at 70% sparsity because it did not repair each removal's effect on later outputs. Curvature-aware ranking and error compensation should preserve more dense-model behavior.

Concrete change and experimental setup. It implemented SparseGPT: C4 calibration activations define a Hessian approximation to local curvature, curvature rescales removal cost, and each removed weight's error is propagated to unprocessed weights. It ran an eight-window smoke and then 128 windows with calibration seed 0, evaluating both on the common 122-block validation protocol.

Observed result. The eight-window model scored 37.960 and the 128-window model 23.166, both near global sparsity 0.700000, versus the reproduced 57.838 baseline. Their block-NLL dispersions were 0.257112 and 0.199322; neither is an error bar.

Agent interpretation. The agent treated second-order compensation, rather than calibration scaling alone, as the primary initial quality improvement.

Report assessment and confounds. The common-split differences support that interpretation. The smoke-to-128 comparison also changes calibration size and has no second seed, so it does not quantify cross-seed algorithm variance.

Decision and consequence. SparseGPT was adopted as the submitted in-layer pruning rule, and its 128-window artifacts became training starts or references for later tests.