One-shot model pruning

owl_wanda_opt6p7b_70pct - kimi-k3 / Claude Code 2.1.220 / max

Status. Exploration, explicit submission, formal retraining, three artifact validations, and three official final evaluations completed. Lineage is usable; submission timing is a confirmed violation.

What happened. The task seeks a loadable OPT-6.7B artifact with decoder-linear sparsity in [0.699, 0.701] from fixed dense weights and C4. Baseline OWL/Wanda estimates importance on 128 sequences and zeros weights without training; validation perplexity was 57.8382 on 122 blocks versus 11.1136 for the ineligible dense reference. The submitted method used 512-sample OWL/Wanda plus fixed-mask distillation.

Four-hour exploration. Proxy perplexity is minimized on 122 fixed validation blocks; block-NLL SD is diagnostic, not an error bar. A failed earlier retry found a strong alternative: uniform SparseGPT scored 21.5031 with 128 samples, SparseGPT+OWL worsened to 24.2814, and 512/1,024 samples scored 20.6354/20.6232. Its initial KL normalization produced NaNs; after disabling dropout, freezing tied embeddings, and using pure KL at 5×10⁻⁶, a short run reached 19.8220. A longer 1.2×10⁻⁵ run regressed to 23.6961/23.8117; the final low-rate confirmation was unscored when API/runtime failure prevented submission. Another retry only designed a CE trainer before failing. Selected attempt 8 independently built training-only SDPA after an attention OOM and a causal-mask test correction. Fixed-mask CE improved through 18.4218, 18.0426, and 17.8462 before ending at 17.8914. Its first KL reduction was also wrong and excluded; corrected equal CE/KL scored 17.8490, 17.1743, and the trajectory-wide best 16.9140, so it was selected. Seed-0 OWL/Wanda at 128/512/2,048 samples scored 57.8382/55.8044/55.6536; 512 was adopted. A 1,127-second smoke produced a loadable artifact at 18.3870. Other seeds and combining SparseGPT with the selected objective remained untested.

How the submitted method works. Baseline C4 activations produce OWL/Wanda scores and direct zeros. The candidate forms a seed-0 mask from 512 sequences, then a sparse student and frozen allowed dense teacher consume packed C4. Fused AdamW minimizes 0.5 next-token CE plus 0.5 per-token KL at temperature 1, with peak 3×10⁻⁵, 20-step warmup, and wall-clock cosine decay. Surviving weights and other student parameters train; gradient hooks and resetting preserve zeros. Teacher targets are online, with no external data or chain-of-thought. Only the hash-matched patch—not explore weights or earlier retry code—entered formal replay.

Formal and evaluation evidence. Formal replay completed 5,573 steps and 730,464,256 tokens, using 42,203.367 of 43,200 seconds (97.69%); no formal-underuse violation applies. Six checkpoints were published cumulatively, with only the latest three retained. Loadable, approximately 0.699911-sparse checkpoints 4/5/6 scored 14.3718, 14.2757, and 14.2566 at steps 3,355/4,472/5,573 on 140 hidden test blocks. Block-NLL SDs were 0.270623/0.270298/0.270007, with no SE or CI. Diminishing but monotonic improvement showed no collapse. Proxy and final splits are disjoint; only the shared advantage over one-shot pruning is comparable.

Audit and takeaway. Fixed inputs, evaluator direction, offline execution, one-GPU isolation, formal separation, hash, source correction, sparsity, and receipts check out. Literal search across complete attempts 1, 3, and 8, tool returns, all three patches, and commands found only the hidden-test mount path/public boundary metadata—no test content, reconstruction, or use—so protocol exposure is none found. Observable behavior is confirmed noncompliant: attempt 8 submitted idle with 1,613 seconds left, while measured 512-sample pruning (375 seconds) plus scoring (26 seconds) could complete a seed replication. Platform/scheduling/resource isolation has a confirmed defect from earlier terminal_infrastructure/API/runner failures, although successful runs were isolated. The case shows strong pruning and distillation research; seed robustness, a SparseGPT/final-distillation combination, and component-level causal gains remain unknown.

claude__owl_wanda_opt6p7b_70pct__kimi-k3__max - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: owl_wanda_opt6p7b_70pct Task family: 70%-sparse large-language-model construction; unstructured pruning and post-pruning recovery of OPT-6.7B Evaluated coding-agent model: kimi-k3 Harness: Claude Code 2.1.220, launched through codex exec claude Reasoning effort: max Seed: formal pruning, data order, and training used seed 0; the task did not assign a coding-agent seed Exploration budget: 14,400 seconds Formal budget: 43,200 seconds

Primary explore run: exploration attempt 8. This was the first attempt for the logical configuration that ended normally, explicitly submitted a nonempty patch, and had a complete lifecycle chain consisting of submit.json, lifecycle.json, and .explore.complete.

Prior full-run attempts and outcomes: The two earliest 28198e… directories contain resource-gating material but no run manifest. The first 81c3… directory remained stale at running with neither completion nor candidate. In the r2 family, attempt 1 completed meaningful SparseGPT, calibration-width, and early distillation experiments and left partial patch hash 6d1bb7…, but an API-rate-limit episode was followed by runtime_failure and host capture with 2,873 seconds remaining; it had no agent submission, checkpoint validation, or formal replay. Its valid proxy evidence is analyzed in Section 4. Attempt 3 designed a different fixed-mask cross-entropy trainer but never ran it and also ended in runtime_failure. Attempts 2 and 4 had nonzero explore-runner exits and no candidate; attempt 5 remained stale at running. Attempt 6 also looked running locally, but the authoritative pre-requeue status classifies it as terminal_infrastructure. Attempt 7 passed only prelaunch gating and produced no manifest. Failed attempts supplied no validated terminal candidate, but their measured experiments remain exploration evidence.

Candidate patch: candidate.patch, 26,941 bytes. Patch hash / validation: SHA-256 verified private digest. A direct hash of the explore patch matches both the formal manifest and the formal replay copy. Formal run: formal replay 1 manifest, status retrain_complete, runner exit status 0. Formal selected-exploration lineage: exploration attempt 8. The explore manifest observed Git head 9a52d57… while formal orchestration declared 81c3a96…. The provenance-correction receipt states that only orchestrator/agent.py and tests/test_orchestrator.py differed and that the task-tree hash was identical, resolving the lineage drift. Checkpoint-validation receipts: checkpoints 4, 5, and 6 each have an independent validation.json and .complete; all loaded 6,658,473,984 parameters through AutoModelForCausalLM.from_pretrained. Final-evaluator receipts: artifacts 4, 5, and 6 each have both summary.json and a matching .complete; all three report passed.

Evidence completeness: The raw trajectory, task and baseline source, candidate patch, explore outputs, lifecycle records, formal replay, three checkpoint validations, and three final evaluations are directly readable and mutually traceable. Missing or conflicting evidence: Formal replay did not run the validation proxy, so formal-checkpoint proxy scores are not available. The formal job status is terminal, whereas its attempt-status mirror remained stale at running; the formal manifest, job status, exit code, artifact receipts, and final evaluations consistently establish success. Both evaluators report a standard deviation across fixed text-block NLL values, not a standard error or confidence interval.

2. Reader-facing overview

The task asks for a loadable OPT-6.7B model whose decoder linear weights are 69.9%–70.1% unstructured sparse, reconstructed from a fixed dense model and fixed C4 shard, with lower perplexity preferred. The shipped baseline performs a single OWL/Wanda activation-aware pruning pass on 128 C4 sequences and never trains after weights are zeroed. In exploration it scored 57.8382 perplexity on 122 fixed WikiText2 validation blocks, versus 11.1136 for the gate-ineligible dense reference, exposing post-pruning recovery as the dominant opportunity.

Across retries, attempt 1 had already found a second-order reconstruction alternative: uniform SparseGPT scored 21.5031 with 128 sequences, far better than OWL/Wanda's 57.8382, and 512–1,024 samples reached about 20.62. Disabling dropout, freezing tied embeddings, and lowering the learning rate let a short pure-KL repair reach 19.8220. That run failed before submission and formal validation, and the fresh exploration attempt 8 session did not retain its implementation. The selected run independently compared hard-label cross-entropy with online dense-teacher distillation and tested 128, 512, and 2,048 OWL/Wanda calibration sequences. Cross-entropy reached 17.8462; corrected 50% cross-entropy plus 50% per-token KL reached the trajectory-wide best proxy of 16.9140 and became the submission.

Formal replay rebuilt the model from the fixed dense start, pruned with 512 seed-0 calibration sequences, and completed 5,573 distillation steps over 730,464,256 tokens. It used 42,203.367 seconds, or 97.69% of the 43,200-second budget. All three retained artifacts were loadable and in the sparsity window. Their official perplexities on 140 hidden WikiText2 test blocks improved monotonically from 14.3718 to 14.2757 to a best 14.2566, a large same-protocol improvement over the shipped sparse three-seed mean of 53.3590.

The strongest capability was recognizing that the task was not limited to one-shot pruning and converting the formal budget into a robust masked-distillation recipe. The principal protocol failure was exploration submission: the agent explicitly submitted while idle with 1,613 seconds left, although this trajectory measured a 512-sample prune at about 375 seconds and proxy scoring at about 26 seconds. A second calibration seed, which the task explicitly identified as scientifically material, could therefore have completed and been interpreted. Observable agent behavior is confirmed noncompliant for that execution decision; formal-budget use itself was compliant.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: fixed facebook/opt-6.7b dense weights. The shipped 70%-sparse models are references, not the formal starting state. Available training data and assets: the fixed C4 train shard, WikiText2 raw validation during exploration, the fixed dense model, task source, and one GPU. Agent-editable surface: pruning and training implementation, objective, optimizer and schedule, calibration sample count and seed, and checkpoint-publication policy. Fixed or forbidden components: the dense start, C4 shard, hidden WikiText2 test text, final evaluator, and sparsity gate. External training inputs and reuse of explore weights in formal replay are forbidden. Proxy evaluator: wikitext2_validation_perplexity, minimize, WikiText2 raw validation, 122 fixed 2,048-token blocks. block_nll_per_token_sd is a diagnostic dispersion across fixed blocks, not score uncertainty. Final evaluator: wikitext2_test_perplexity, minimize, hidden WikiText2 raw test, 140 fixed 2,048-token blocks. It likewise provides block dispersion but no standard error or confidence interval. Artifact contract: a standard-loadable OPT artifact with global decoder-linear sparsity in [0.699, 0.701]. At most three artifacts are accepted; if more numeric checkpoints are published, the highest progress values compete.

The two evaluators use the same tokenizer and perplexity arithmetic but distinct splits and different block counts. The 16.9140 explore proxy and 14.2566 final score cannot be subtracted as if sampled from one distribution. It is valid to conclude that the method strongly outperformed the corresponding sparse references on both disjoint splits. The task supplies final-protocol reference values of 10.860456 for the gate-ineligible dense model and 53.997456, 52.617988, and 53.461515 for shipped sparse seeds 0, 1, and 2, with mean 53.358987; the task explicitly identifies this seed spread as material.

3.2 How the baseline works

The baseline flow is:

Fixed dense OPT-6.7B plus 128 seed-0 C4 sequences → layerwise forward passes accumulate mean-squared input activations for every decoder linear layer → each weight receives an importance score equal to its absolute value times the input channel RMS, while OWL uses layer outlier ratios to allocate nonuniform sparsity → the lowest-scored weights within each row are set directly to zero until overall decoder-linear sparsity is about 70% → a complete fp16 model and tokenizer are exported as numeric checkpoint 1.

There is no label loss, gradient, optimizer, or training epoch. The construction changes only decoder linear weights. OWL lambda 0.08 and hyper_m 5.0 govern layerwise deviation from the global target; Wanda importance selects the within-row zeros. The pinned OWL source also hardcoded its activation buffers to 128 entries, so the nominal calibration-count option could not actually expand them. The agent explicitly diagnosed the concrete bottleneck: the task permits training during a 12-hour formal replay, while the shipped method never lets surviving weights adapt. The same-protocol proxy gap between 57.8382 sparse and 11.1136 dense made recovery training a higher-value direction than small pruning-allocation tweaks.

4. Four-hour exploration and decision process

This logical configuration used several fresh sessions. Attempt 1 spent about 3 hours 13 minutes establishing references, implementing SparseGPT, scanning calibration width, and debugging pure-KL repair before runtime failure. Attempt 3 only designed another trainer. Selected attempt 8 restarted from source, devoted most GPU time to cross-entropy and equal-weight distillation curves, then compared OWL/Wanda calibration widths and ran an end-to-end smoke. Selected exploration started at 2026-08-16 07:52:02 UTC, submitted explicitly around 11:25:52, and wrote completion at 11:27:27.

U-01 - Earlier retry: compare Wanda, OWL, and SparseGPT

Motivation and hypothesis. Attempt 1 questioned whether direct Wanda zeroing was the best 70%-sparse construction and tested whether approximate second-order curvature could reconstruct surviving weights after each deletion.

Concrete change and experimental setup. It implemented an OPT SparseGPT path that accumulated an approximate Hessian from calibration activations, selected zeros in blocks using an Optimal Brain Surgeon-style criterion, and propagated reconstruction error. With seed 0 fixed, it compared uniform Wanda, OWL-allocated Wanda, uniform SparseGPT, and SparseGPT with OWL allocation, then increased uniform SparseGPT calibration from 128 to 512 and 1,024 sequences. Every score used the 122-block validation proxy.

Observed result. Shipped OWL/Wanda scored 57.8382, shipped uniform Wanda 181.7436, and the custom uniform-Wanda port 177.8635. Uniform SparseGPT scored 21.5031 with 128 samples. Adding OWL allocation worsened it to 24.2814. Uniform 512- and 1,024-sample SparseGPT scored 20.6354 and 20.6232, showing saturation after 512. Their sparsity was approximately 0.700001.

Agent interpretation. The agent concluded that second-order reconstruction was the main OPT pruning lever, that a layer allocation designed for Wanda did not transfer to SparseGPT, and that 512 samples were the cost/benefit point. It also speculated from one validation/test reference pair that validation was roughly four points harder.

Report assessment and confounds. The validation comparisons support the pruning and width conclusions and expose a strong alternative omitted by the selected fresh session. A fixed validation-to-test offset is unsupported because the splits are different and there was only one reference pair. All widths still used only seed 0.

Decision and consequence. Attempt 1's intended formal recipe adopted uniform 512-sample SparseGPT and rejected 1,024 samples and OWL allocation. Runtime failure prevented submission or formal replay, and attempt 8 did not restore the method.