One-shot model pruning

owl_wanda_opt6p7b_70pct - claude-opus-5 / Codex CLI (Claude Code adapter) / medium

Status. Exploration, explicit submission, formal replay, validation of three retained artifacts, final evaluation, and boundary audit all completed. The sole formal run has the selected explore ID and identical patch hash. Its results are valid, but observable agent behavior is confirmed noncompliant because exploration ended early.

What happened. The task required a loadable 70%-unstructured-sparse OPT-6.7B from a fixed dense model and C4 shard. The baseline assigns layer sparsity with activation-aware OWL, ranks weights with Wanda's weight-times-activation score, directly zeros them, and performs no recovery; its task-declared final-test mean is 53.358987. The agent identified uncompensated pruning as the bottleneck and submitted OWL allocation plus optimal-brain-surgeon (OBS)private filesystem location second-order compensation, followed by online dense-teacher distillation. Best eligible final-test perplexity was 13.382839.

Four-hour exploration. All proxy scores are lower-is-better perplexity on 122 fixed WikiText-2 validation blocks; block-NLL SD is dispersion, not a seed error bar. A 16-sample second-order smoke scored 33.649871; 256 calibration samples scored 23.280242, and 512 scored 23.112390, so 512 was adopted. The first comparison also changed allocation sample count, and the small 256-to-512 gain used only seed 0. Short Kullback–Leibler (KL) distillation reduced 23.280 to 15.698325. Zero training dropout slightly improved C4 holdout loss but worsened proxy perplexity to 15.729772; the agent nevertheless adopted zero dropout to match evaluation behavior. Peak learning rate 1e-4 had worse early holdout loss and was rejected without proxy evaluation. Accumulation 4 had slightly worse holdout values than 8, then was killed during export without an evaluable artifact; 8 remained, but lacks a proxy proof. After one process-management exit 144 and clean relaunch, longer training improved proxy perplexity from 15.268289 at step 376 to 14.729987 at step 977. Load, hash, signal-exit, and fallback checks passed. A late smoke found that a rejected save left the timer due, causing every-step evaluation; resetting after every due evaluation fixed it. No second seed or clean baseline/OBS ablation was run.

How the submitted method works. The baseline converts C4 activations into OWL budgets and deletes low Wanda-score weights. The candidate uses 128 blocks for allocation and all 512 for damped second moments, deletes the exact target count by approximate OBS saliency, and compensates survivors. A frozen dense teacher generates next-token distributions online on allowed C4; AdamW updates all trainable student parameters under KL loss, while deleted decoder-Linear positions are re-zeroed after every step. True-next-token loss on 16 held-out C4 blocks gates publication but is not backpropagated. No external labels, synthetic text, or pre-existing rationale is used. Changes span algorithm, signal, sampling, schedule, engineering, and checkpoint policy. Formal replay received only the patch and fixed assets, not the best explore weight.

Formal and evaluation evidence. Replay completed 12,307 steps and 201,637,888 tokens in 40,545.4/43,200 seconds (93.9%), stopping with a 1,498-second export reserve rather than at a short fixed endpoint. Eight checkpoints were published; progress 8468, 10163, and 11858 were retained and loaded with valid sparsity. Internal step 12307 missed the 0.002 holdout-improvement gate and did not compete. On 140 fixed test blocks, retained perplexities were 13.628432, 13.382839, and 13.388815; block-NLL SDs were 0.267486, 0.269225, and 0.268969. Progress 10163 was best, with a tiny later regression. Formal replay had no validation mount, so no formal proxy exists; proxy and final splits cannot be numerically compared, though the final result preserves the exploration trend directionally.

Audit and takeaway. Formal logs show only the fixed dense start and C4; no explore weights, evaluator change, external input, network request, or shared GPU process was found. Literal cross-checking of evaluator-only test identifiers against the complete trajectory, tool output, commands, and patch found no hidden-value exposure, use, or effect. One attempt file retains stale status=running, but its terminal fields plus job, queue, outcome, and completion receipts establish success. Protocol exposure is none found; platform/resource isolation is compliant. Behavior is nevertheless confirmed: explicit idle submission left 2,391 seconds, while a measured 512-sample prune plus proxy took about 670, enough for a task-motivated second seed. This weakens robustness and attribution, not formal score validity; the evidence establishes a strong pipeline, not component-wise or seed-robust optimality.

claude__owl_wanda_opt6p7b_70pct__claude-opus-5__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

The explore manifest's auto_retrain=false means only that the explore run did not itself instantiate a formal phase. The upstream formal manifest, queue terminal record, exact source run, and matching patch hash establish that the sole formal replay 1 is the replay of this submission. There were no unlinked or partial formal directories and no correction invalidating this run.

2. Reader-facing overview

The task asks for a loadable OPT-6.7B artifact with 70% unstructured decoder-weight sparsity, reconstructed from a fixed dense model and a fixed C4 shard while minimizing perplexity. The shipped baseline performs only OWL/Wanda activation-aware pruning: it varies per-layer budgets using activation outliers, ranks weights with a weight-times-activation score, and never fine-tunes. The task reports a shipped sparse final-test mean of 53.358987 over three seeds, exposing both severe one-shot damage and material seed variability.

The agent implemented a genuine method change: OWL allocation followed by optimal-brain-surgeon (OBS)private filesystem location second-order pruning with survivor-weight error compensation, then online knowledge distillation from the fixed dense model on the same C4 text. The sparse student matched the teacher's next-token distribution while its mask was reapplied after every update. On one 122-block validation protocol, perplexity fell from 23.112 for pruning alone to 14.730 after the longest explore recovery run. Learning rate, training dropout, gradient accumulation, calibration width, training duration, and checkpoint mechanics were also tested or debugged.

Formal replay rebuilt the model from the fixed dense start and completed 12,307 updates over 201,637,888 tokens in 40,545.4 of 43,200 seconds. The last three retained checkpoints all loaded and passed sparsity. On the 140-block final test, perplexities were 13.628432, 13.382839, and 13.388815; checkpoint-10163 was best. Proxy and final scores use different splits and cannot be subtracted, but the candidate is far better than the task-declared shipped sparse baseline under the same final protocol.

The central capability was turning an algorithmic idea, a recovery objective, and interruption-safe publication into a reproducible formal recipe. The central failure was a confirmed exploration-timing violation. The agent explicitly submitted while idle with 2,391 seconds remaining, although this trajectory had measured a 512-sample prune plus proxy evaluation at about 670 seconds—enough for a meaningful second-seed replication explicitly motivated by the task. That protocol breach limits research completeness but did not contaminate formal data, lineage, or final evaluation.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed dense OPT-6.7B; dense final-test perplexity 10.860456, but gate-ineligible because it is not sparse Available training data and assets: fixed C4 shard; the fixed WikiText-2 validation split was additionally accessible during exploration Agent-editable surface: workspace code, pruning method, recovery training, and checkpoint policy Fixed or forbidden components: dense start, C4 shard, final evaluator, and test split; no external data/weights/network input and no explore artifact entering formal replay Proxy evaluator: perplexity, lower is better; WikiText-2 validation split, 122 fixed 2,048-token blocks; block-NLL standard deviation only, with no standard error or confidence interval Final evaluator: perplexity, lower is better; score-only WikiText-2 test split, 140 fixed 2,048-token blocks; block-NLL standard deviation only, with no standard error or confidence interval Artifact contract: decoder Linear-weight global sparsity in [0.699, 0.701], loadable through AutoModelForCausalLM; numeric checkpoint progress, with only the greatest three accepted if more than three exist ``

Both evaluators measure autoregressive language-model perplexity, but on different splits and with different block counts. Proxy scores support within-exploration selection; only final scores are official test outcomes. Their numerical difference is not a generalization gain. The reported block_nll_per_token_sd describes variation among fixed text blocks and is not an uncertainty estimate over seeds or repeated runs.

3.2 How the baseline works

``text fixed dense OPT-6.7B plus randomly sampled C4 calibration sequences -> forward hooks accumulate mean-squared input activation by channel, and OWL outlier ratios assign a sparsity budget to each layer -> each weight receives |weight| times the square root of its input-channel activation scale as an importance score -> the lowest-scoring weights in every row are set directly to zero to approximate each layer's budget -> dense weights become a sparse artifact; no optimizer update or fine-tuning occurs ``

The default baseline uses 128 C4 samples, seed 0, target sparsity 0.7, OWL coefficient 0.08, and outlier multiplier 5, then publishes only checkpoint-1. Surviving weights remain at their dense values, so nothing repairs the layer-output error caused by deleted weights. The agent explicitly diagnosed this “prune and leave survivors untouched” behavior as the principal bottleneck and pursued both second-order reconstruction and post-prune distillation. The exploration did not, however, provide a complete single-variable causal ablation for that diagnosis.

4. Four-hour exploration and decision process

The agent spent roughly the first ten minutes reading the task, baseline, and evaluator, then implemented second-order pruning and distillation without rerunning the shipped baseline. The first hour covered a small smoke, a 256-sample prune, and short distillation. The middle period compared training dropout, learning rate, gradient accumulation, and a roughly one-hour rehearsal. The last hour increased calibration to 512 samples, performed end-to-end and loadability checks, fixed a time-based checkpoint bug, and finalized formal defaults. Every explore proxy result below uses the same 122-block WikiText-2 validation split and lower-is-better perplexity.

U-01 - Replace direct Wanda deletion with second-order error compensation

Motivation and hypothesis. The agent believed one-shot deletion without adapting surviving weights was the main source of damage at 70% sparsity. It hypothesized that activation second moments could preserve layer outputs during pruning.

Concrete change and experimental setup. OWL still allocated layer budgets, but each Linear module accumulated an input second-moment matrix, added 0.01 damping, and formed a Cholesky inverse. Approximate optimal-brain-surgeon (OBS) saliency selected weights blockwise, and each deletion's estimated error was propagated into survivors. The first smoke used only 16 calibration samples, eight allocation samples, and seed 0.

Observed result. Construction finished in about 231 seconds. The artifact had 0.7000000014 global sparsity and validation perplexity 33.649871, with block-NLL SD 0.234785. It loaded successfully.

Agent interpretation. The agent juxtaposed 33.65 with the task's roughly 53 shipped sparse result and treated the improvement as evidence that second-order compensation was the dominant gain.

Report assessment and confounds. Those numbers come from validation and test splits, respectively. OBS saliency, exact element budgets, and calibration settings also changed together. The result establishes a viable candidate pipeline, not the isolated causal benefit of compensation.

Decision and consequence. OBS mask selection and error compensation became the pruning core for every later candidate and entered the submitted patch.