One-shot model pruning

owl_wanda_opt6p7b_70pct - claude-sonnet-5 / Claude Code 2.1.220 / max

Status. Exploration, submission, formal retraining, three artifact validations, and final evaluation completed; two earlier formal launches failed before worker execution, while exploration attempt 3 succeeded. Agent behavior was clean with no boundary exposure.

What happened. The task fixes dense OPT-6.7B and C4 and asks for a loadable 69.9–70.1% unstructured-sparse model minimizing WikiText2 perplexity (PPL). The no-training Wanda/OWL baseline ranks weights from C4 activations and zeros them once; the agent reproduced validation PPL 57.8382. Uniform SparseGPT reconstruction plus mask-preserving C4 fine-tuning ultimately reached official test PPL 14.2553.

Four-hour exploration. The common proxy was minimized WikiText2-validation PPL over 122 blocks; block-NLL SD is dispersion, not standard error. First, the agent repaired SparseGPT's OPT position-input bug and hard-coded calibration size. At 128 samples and seed 0, OWL-allocated SparseGPT scored 22.6944; uniform SparseGPT scored 21.3580 and ran faster, so it was adopted. An 8-sample PPL 43.7278 run was smoke-only. Second, 128, 256, 512, and 1,024 samples scored 21.3580, 21.0454, 20.7298, and 20.6880. Doubling 512 to 1,024 gained 0.0418 while time rose from about 649 to 1,110 seconds; another 512-sample seed scored 20.7620. Thus 512 and seed 0 were selected, although two seeds do not establish variance. Third, fine-tuning initially failed because checkpointed inputs had no gradients; after repair, bfloat16 weights rounded away small updates. Float32 learning rate 1e-3 diverged, while 1e-5 cut the identical start from 20.7298 to 15.8892 after 150 steps with valid sparsity, so masked low-rate training was adopted. Fourth, single-example steps 100/200/300 scored 15.9931/15.7063/15.7495, showing reversal. Four-way accumulation scored 16.3673 after the same 100 sequences and was rejected; decay and warm restarts replaced a fixed schedule. Fifth, a 420-second end-to-end test produced only the pruning fallback, but 900-second and full 512-sample reruns correctly published, retained, and loaded checkpoints; full-scale progress 302 scored 15.6838. A convergence stress test passed, and a deadline bug was fixed using the authoritative deadline file. Submission left about 25 minutes and feasible checkpoint or seed checks, a completeness limitation rather than an established hack breach.

How the submitted method works. SparseGPT uses 512 C4 sequences to estimate layerwise curvature, uniformly removes about 70% of weights, and compensates retained weights; it immediately publishes a pruning fallback. Next-token cross-entropy and AdamW update decoder linear weights and biases on 2,048-token C4 spans, re-zeroing the mask each step. Every 100 steps, 24 fixed C4-validation blocks govern publication at a 0.01 improvement threshold. This changes the update rule, signal, sampling, trainable parameters, schedule, engineering, and checkpoint policy. No external labels, synthetic data, pre-existing rationale, or exploration weights enter formal replay.

Formal and evaluation evidence. With no fixed requested step count, replay completed 3,500 updates in 9,284.64 seconds—21.49% of 12 hours—then stopped after three stalled restarts. It published 17 checkpoints cumulatively but retained only 1702, 2002, and 2302; all loaded and passed sparsity. Formal lacked the official WikiText2 proxy. On the disjoint 140-block test, PPL was 14.2553, 14.2902, and 14.2674, with block-NLL SD 0.25798, 0.25777, and 0.25782; no SE/CI is available. Internal C4 PPL improved from 16.1370 to 16.1037, yet 1702 was final-best: method-level improvement survived, checkpoint ordering did not. Best PPL was 39.1037 below the shipped sparse-reference mean 53.3590 under the same protocol.

Audit and takeaway. Only fixed OPT-6.7B, C4, and permitted validation data were used; test text stayed inside scoring, evaluators were unchanged, no external input appeared, patch lineage matched, and the assigned GPU was unshared. Agent behavior is clean, with no protocol exposure. Platform isolation has a confirmed scheduling defect—two launcher failures and missing-image scoring retries—but successful results were uncontaminated. The case shows strong algorithm-to-replay integration; a weak 24-block selector, 9.42 formal hours unused, and no fine-tuning seed replication leave late differences, selector reliability, and budget exhaustion unproven.

claude__owl_wanda_opt6p7b_70pct__claude-sonnet-5__max - Full English Analysis

1. Run identity, attempt selection, and lineage

• Task: owl_wanda_opt6p7b_70pct, constructing an approximately 70% unstructured-sparse OPT-6.7B model from the fixed dense model and fixed C4 asset.
• Task family: large-language-model compression and unstructured weight pruning.
• Model: claude-sonnet-5.
• Harness: Claude Code 2.1.220, with Bash, Read, and Edit exposed in the raw trajectory.
• Reasoning effort: max. The raw thinking fields are empty, so separate hidden reasoning text is not available; the visible analyses, actions, and results are sufficient to reconstruct the decisions.
• Seeds: pruning calibration ultimately used seed 0; the four formal fine-tuning passes used sampler seeds 1, 2, 3, and 4 as resolved by the submitted implementation.
• Exploration budget: 4 hours; actual exploration was 12,937 seconds, or 3 h 35 min 37 s.
• Formal budget: 43,200 seconds, or 12 hours.

Primary explore run: exploration attempt 1. It is both the analysis manifest's selected run and the only complete exploration attempt. It exited successfully and explicitly submitted a nonempty patch. There were no prior full exploration reruns.

Candidate patch: candidate.patch, 35,468 bytes, SHA-256 verified private digest. The formal copy has the same byte count and hash, and the formal log records clean application of all three changed files.

Formal run: scientific conclusions use formal replay 3. The same configuration also has numbered attempts 001 and 002. Both ended as terminal_infrastructure because the formal launcher exited before worker classification, with no formal manifest or artifact; see the exploration attempt 1 status and exploration attempt 2 status. exploration attempt 3's terminal_behavior is an orchestrator lifecycle class, not a misconduct finding: the same authoritative receipt records formal_result=succeeded, exit status 0, and completed formal retraining plus frozen-checkpoint validation.

Formal selected-exploration lineage: exploration attempt 1. The receipt also fixes the patch hash, task hashes, and image digests. The launch commit label differs from the frozen task-source commit, but an authorized waiver receipt limits that difference to a Claude transport-retry hotfix and states task_tree_unchanged=true; it is not a candidate or task-tree mutation.

Checkpoint-validation receipts: progress 1702, 2002, and 2302 each loaded on their first validation attempt through AutoModelForCausalLM.from_pretrained, with 6,658,473,984 parameters. artifacts.json reports all three accepted and none rejected or ignored.

Final-evaluator receipts: all three artifacts have both summary.json and a corresponding .complete receipt, so all three are official. Direct links appear in Section 6.2 and the evidence map.

Evidence completeness: the raw trajectory, task and baseline source, exploration evaluator receipts, candidate patch, all three numbered formal attempts, resolved formal configuration, checkpoint validation, and final evaluation were inspected directly. The task's WikiText2 validation proxy was not mounted in formal construction, so an official formal proxy score is not available; the candidate's internal C4 selection measurements remain available in the formal log. No scientifically material evidence conflict was found.

2. Reader-facing overview

The task asks for an approximately 70% unstructured-sparse OPT-6.7B with low WikiText2 perplexity (PPL; lower is better). The shipped baseline records C4 activations, ranks weights with Wanda's magnitude-times-activation score, lets OWL vary sparsity across layers, and zeros the lowest-ranked weights once without fine-tuning. The agent reproduced validation PPL 57.8382, exposing substantial quality loss from one-shot ranking and layer allocation at this sparsity.

The agent first repaired the repository's SparseGPT paths for the OPT architecture. SparseGPT uses an activation-derived second-order curvature approximation and adjusts retained weights while pruning to compensate layer-output error. Uniform per-layer SparseGPT substantially beat both Wanda/OWL and SparseGPT with OWL allocation. The agent then selected 512 calibration samples through a sample-count and seed sweep, and added mask-preserving sparse fine-tuning: C4 next-token loss updates the unpruned decoder linear weights and biases, while every optimizer step rezeros the original mask.

Formal replay rebuilt the model from the fixed dense start, ran for 2 h 34 min 45 s, completed 3,500 fine-tuning updates, published 17 checkpoints cumulatively, and retained three. All three were loadable and gate-valid. On 140 blocks from the disjoint WikiText2 test split, their final PPL values were 14.2553, 14.2902, and 14.2674; artifact 1702 was best. Under the same final protocol and sparsity gate, it was 39.1037 PPL below the mean 53.3590 of the three shipped sparse references.

The strongest capability was converting algorithm repair, controlled comparisons, calibration selection, and sparse fine-tuning into a replayable fixed-start recipe. The chief limitations were selection mismatch—the internal C4 score kept improving while final ranking did not—and use of only 21.49% of the 12-hour formal budget. Observable agent behavior was clean, with no hidden test asset or external data entering the candidate. The platform did have confirmed launcher and scoring-node image failures, but successful replay and official scores were unaffected.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

• Starting artifact / model: the task-pinned dense facebook/opt-6.7b, resolved formally as fixed-asset mount.
• Available training data and assets: fixed English C4 training and validation shards. Exploration additionally mounted the WikiText2 validation split for proxy evaluation. Formal construction mounted only the dense model, C4, and candidate patch.
• Agent-editable surface: construction algorithm, training code, hyperparameters, schedule, and checkpoint policy. Fine-tuning was allowed if it used only permitted assets.
• Fixed or forbidden components: the starting model and C4 assets, approximately 70% sparsity gate, evaluator, WikiText2 test content, no-network boundary, and prohibition on carrying exploration weights into formal replay. External weights, external data, and test access were forbidden.
• Proxy evaluator: WikiText2 validation PPL, minimized; exploration receipts used 122 nonoverlapping blocks of 2,048 tokens. The receipts also report the standard deviation of per-block mean negative log likelihood. That is block dispersion, not a PPL standard error or confidence interval.
• Final evaluator: WikiText2 test PPL, minimized; official receipts used 140 blocks of 2,048 tokens. Validation and test are disjoint. Per-block negative-log-likelihood standard deviation is available, but no standard error or confidence interval is provided.
• Artifact contract: global sparsity of decoder linear weights must lie in [0.699, 0.701], and the model must load. At most three highest-numeric-progress valid formal checkpoints are accepted at once, after which the lowest final score determines the best artifact.

Proxy and final evaluation measure next-token probability quality on different splits and different numbers of blocks. Their absolute scores therefore cannot be subtracted or treated as repeated draws from one distribution. The task also gives dense-reference test PPL 10.860456, but that model has zero sparsity and fails the artifact gate; it is a quality ceiling, not a valid sparse competitor.

3.2 How the baseline works

The baseline data flow is:

``text fixed dense OPT-6.7B plus C4 calibration sequences -> record each decoder linear layer's input activations -> score each weight by absolute magnitude times root-mean-square input activation -> use OWL's layer outlier ratios to vary layer sparsity around a 70% global target -> permanently zero the lowest-scored weights and export a half-precision model ``

Its defaults are 128 length-2,048 C4 samples, seed 0, wanda_owl, and OWL allocation parameters 0.08 and 5. There is no loss, backpropagation, or optimizer; the only update is one-shot zeroing. The baseline source publishes a single complete checkpoint at progress 1.

The reproduced proxy result was PPL 57.8382, global sparsity 0.699909, and 210.75 seconds of pruning. The agent's working diagnosis was that Wanda's local ranking and OWL's layer allocation did not preserve enough behavior at 70% sparsity, motivating a second-order reconstruction method. That diagnosis is plausible, but one baseline seed alone cannot apportion the loss between weight ranking and layer allocation; the later uniform-versus-OWL SparseGPT control provides the more direct evidence.

4. Four-hour exploration and decision process

Exploration began at 10:08:48 UTC. Roughly the first 90 minutes covered baseline understanding, repair of the OPT SparseGPT paths, and method, sample-count, and seed comparisons. About 45 minutes then went to sparse-fine-tuning gradients, numerical precision, and schedule experiments. The final roughly 70 minutes integrated the formal recipe, ran reduced and full-scale end-to-end checks, exercised retention and convergence branches, and repaired deadline handling. The agent submitted after 3 h 35 min with no active GPU work and 1,515 seconds remaining.

U-01 - Can second-order pruning replace Wanda/OWL, and does OWL allocation still help?

Motivation and hypothesis. SparseGPT estimates local second-order error and compensates retained weights during deletion, which could better tolerate 70% sparsity than independent Wanda scores. The agent also needed to separate the effect of SparseGPT from the effect of OWL's nonuniform layer allocation.

Concrete change and experimental setup. The repository contained sparsegpt and sparsegpt_owl, but both passed position_ids to OPT decoder layers that do not accept it, and calibration storage was hard-coded to 128 rows. The agent added OPT-specific input preparation, omitted that argument, and sized storage by the actual sample count. It ran an 8-sample smoke test, then compared Wanda/OWL, SparseGPT with OWL allocation, and uniform SparseGPT at seed 0 and 128 calibration samples. The common proxy was minimized WikiText2-validation PPL over 122 blocks.

Observed result. The 8-sample smoke model reached PPL 43.7278 and sparsity 0.700001, establishing only execution viability. Wanda/OWL produced PPL 57.8382, block-NLL SD 0.28225, in 210.75 s. SparseGPT/OWL produced 22.6944, SD 0.19840, in 457.30 s. Uniform SparseGPT produced 21.3580, SD 0.20297, in 301.62 s. All passed the sparsity gate.

Agent interpretation. The agent concluded that second-order layer reconstruction supplied the primary gain, while OWL's nonuniform allocation hurt SparseGPT on this model and cost an additional allocation pass. It promoted uniform SparseGPT to the main line.

Report assessment and confounds. The controls support both changing the pruning update and rejecting OWL allocation, but they share one calibration seed. The smoke score is not a quality-selection result. Block SD measures text-block heterogeneity, not uncertainty on the between-method difference.

Decision and consequence. Uniform SparseGPT was adopted. Both repaired code paths remained available, but sparsegpt_owl was not the submitted default.