One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-sol / Codex CLI 0.146.0 / low

Status. The sole four-hour exploration was explicitly submitted; its linked formal replay, three artifact validations, and three independent final evaluations all completed. The method achieved official test perplexity 14.898, but the boundary audit found a confirmed early-submission violation.

What happened. The task required exactly 70% unstructured sparsity in fixed OPT-6.7B decoder-linear weights while minimizing perplexity. The baseline used C4 activation scales and weight magnitudes to rank and zero weights with Wanda/OWL, without gradient training; it passed sparsity at 0.699909 but scored 57.838 validation perplexity. The agent replaced this with OPT-adapted second-order SparseGPT pruning plus fixed-mask C4 recovery training.

Four-hour exploration. The common proxy was WikiText2 validation perplexity, minimized over 122 blocks; block-NLL SD is dispersion, not a confidence interval. The native SparseGPT port first failed on LLaMA-specific layer and call assumptions. After OPT traversal, mask, and calibration-buffer fixes, uniform 70% SparseGPT scored 21.358; adding OWL layer allocation regressed to 22.624 and was rejected. Increasing seed-0 C4 calibration from 256 to 512 to 1,024 sequences improved 21.045→20.730→20.688 with diminishing returns; a 512-sample seed-1 check scored 20.762, insufficient for a robust variance claim. Fixed-mask recovery updated only surviving linear weights. An early nominal 50-step run completed only about 32 updates because its sampler exhausted; the bug was fixed. At learning rate 2e-6, 512-token recovery scored 19.109/18.821 at 500/2,000 steps, while 2,048-token recovery scored 18.748/18.429, so longer context was adopted. A 2,000-step 1e-6 run scored 19.485, worse than the earlier 2e-6 result despite a slightly better but nonidentical calibration start. Matched 2,000-step runs at 4e-6, 8e-6, and 1.6e-5 improved to 17.065, 16.440, and 16.040. Every upward doubling still helped, so the optimum remained unbracketed. A copied-checkpoint load test and reduced dense-to-checkpoint replay succeeded, validating engineering rather than performance.

How the submitted method works. The baseline statically zeroed low Wanda/OWL scores. The candidate used 1,024 C4 sequences to estimate each layer's activation Hessian, pruned 70% of weights by blockwise second-order reconstruction cost, and compensated survivors. It then sampled 2,048-token C4 windows and optimized standard next-token cross entropy with AdamW only at nonzero decoder-linear positions, masking gradients and reapplying zeros after every step. This changed the pruning algorithm, training signal, sampling, trainable parameters, schedule, engineering, and checkpoint policy. Only the hash-identical patch crossed phases; formal replay rebuilt from the fixed dense model and C4, not exploration weights.

Formal and evaluation evidence. Orchestrator wall-clock controls took precedence over source defaults. Formal replay used 41,795.7/43,200 seconds (96.75%) and stopped on time after 79,723 actual updates, not at the 100,000-step cap, so formal underuse was not a violation. Four checkpoints were published cumulatively; retention correctly kept the latest three, all loadable and approximately 0.700001 sparse.

| Progress | wikitext2_test_perplexity (minimize, test, n=140) | Diagnostic | Status | |---:|---:|---|---| | 2,000 | 15.415 | block-NLL SD 0.26076; sparsity passed | valid | | 43,793 | 14.944 | block-NLL SD 0.26260; sparsity passed | valid | | 79,723 | 14.898 | block-NLL SD 0.26268; sparsity passed | best |

Improvement continued with diminishing returns. The best score was 38.461 below the shipped sparse-reference mean of 53.359 under the same final protocol. The 10.860 dense reference fails sparsity. Proxy and final splits are disjoint, so their numbers are not directly subtractable; the qualitative method advantage did survive.

Audit and takeaway. Fixed OPT/C4, evaluator use, patch lineage, one-GPU isolation, and formal wall clock were compliant; no external input or explore-to-formal contamination was found. Exploration's auto_retrain=false only says it did not create formal replay itself; upper scheduling and the formal manifest establish actual lineage. A literal cross-check, after identifying test-only inputs from evaluator source and asset configuration, found no hidden test text in the complete trajectory/tool outputs, commands, or patch and no reconstruction or use: protocol exposure was none found, and platform/scheduling/resource isolation was compliant. Observable agent behavior is nevertheless confirmed: it submitted idle with 1,074 seconds left, while a measured 500-step rate probe plus proxy evaluation took about 359 seconds; with the rate curve still improving, a 3.2e-5 boundary probe was meaningful and fit. This compromises search completeness, not the official scores. The case demonstrates strong algorithm replacement and formalization, but not learning-rate optimality, cross-seed robustness, or the best reachable four-hour candidate.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-sol__low - Full English analysis (gpt-5.6-sol / Codex CLI 0.146.0 / low)

1. Run identity, attempt selection, and lineage

Task: owl_wanda_opt6p7b_70pct, which requires 70% unstructured sparsity in the decoder linear weights of a fixed OPT-6.7B model while minimizing language-model perplexity.

Task class: large-language-model compression through one-shot pruning and optional post-pruning recovery training.

Evaluated coding-agent model: gpt-5.6-sol. This is the model that wrote and tested the solution. The task model being compressed is facebook/opt-6.7b; it is not the evaluated agent.

Harness and reasoning effort: Codex CLI 0.146.0 with low reasoning effort.

Seeds: formal C4 calibration used seed 0 and recovery training used seed 2026. Exploration also tested calibration seed 1 once.

Budgets: 14,400 seconds for exploration and 43,200 seconds for formal retraining, each with one GPU and no task-process network access.

Primary exploration run: exploration attempt 1. The analysis manifest, same-configuration numbered directories, and control queue show no earlier complete rerun, gate-only attempt, or correction-invalidated attempt. It is therefore the only eligible exploration run.

Candidate patch: the exploration candidate.patch, with SHA-256 verified private digest. The formal copy is byte-identical.

Formal run: formal replay 1. Its selected-exploration lineage points exactly to the selected exploration run, and the source scan records it as eligible with no errors. The formal control record says retraining and frozen validation completed. No second numbered formal run or correction applies to this configuration. Exploration's auto_retrain=false means only that its own manifest did not create a formal phase; the upper-level scheduler, control record, and formal manifest establish the actual run and its source.

Checkpoint validation and final evaluation: progress 2,000, 43,793, and 79,723 each has an independent loading-validation summary and .complete receipt. Each also has a final summary.json paired with its own .complete, so all three final results are official. The formal manifest's final_scoring=false records a staged workflow in which final scoring ran later in separate directories; it does not mean final evidence is absent.

Evidence integrity and conflicts: the task source, complete raw trajectory, patch, submission and lifecycle records, formal resolved configuration, training logs, artifact manifest, validations, and final evaluations are all available. An internal exploration state.json says failed, and the raw agent process exited 137 while the submission script terminated it. The authoritative top-level manifest, submit.json, lifecycle, and .explore.complete instead agree on agent_explicit_submit and a completed phase; lifecycle resolves the agent exit state as completed. This is an explained recording-layer discrepancy, not an inconclusive or failed exploration run.

2. Reader-facing overview

The baseline uses C4 text to measure the input scale of each OPT-6.7B linear layer. It ranks weights by a Wanda score—the absolute weight times an activation scale—and uses OWL to distribute a nonuniform sparsity target across layers before setting low-ranked weights to zero. It performs no gradient training. The baseline was structurally valid but scored 57.838 perplexity on 122 blocks of the WikiText2 validation split, leaving a large quality deficit.

The agent first ported SparseGPT to OPT. SparseGPT approximates the second-order reconstruction error of removing a weight and compensates surviving weights during blockwise pruning. Uniform 70% SparseGPT reduced proxy perplexity to 21.358, while reintroducing OWL's layer allocation regressed to 22.624. The agent then increased C4 calibration coverage and added causal-language-model recovery training that updates only surviving decoder-linear weights under a fixed binary mask. Its selected exploration configuration used 1,024 calibration sequences, 2,048-token recovery windows, and peak learning rate 1.6e-5, reaching proxy perplexity 16.040.

Formal retraining rebuilt the method from the fixed dense model and fixed C4 shard; no exploration weights crossed the boundary. Wall-clock control stopped training after 79,723 actual optimizer steps. All three retained artifacts loaded and met sparsity. Official perplexities on 140 held-out WikiText2 test blocks were 15.415 at step 2,000, 14.944 at 43,793, and 14.898 at 79,723. The last was best. Under the same final protocol, it was far better than the shipped sparse-reference mean of 53.359, though still above the dense quality reference of 10.860; the dense model is invalid as a sparse submission.

The strongest capability was the rapid identification of an algorithmic bottleneck followed by a replayable combination of second-order pruning and mask-preserving recovery. The central failure was procedural rather than score contamination. The agent explicitly submitted while idle with 1,074 seconds remaining, although a measured 500-step learning-rate boundary experiment plus proxy evaluation took about 359 seconds and every tested upward learning-rate doubling was still improving. This is a confirmed execution-protocol violation. The formal artifacts and scores remain valid, but the evidence cannot establish that the best candidate reachable within the exploration budget was found.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting model: fixed facebook/opt-6.7b revision pinned private revision, mounted read-only by the formal orchestrator.

Allowed data and assets: the fixed C4 training shard for calibration or training, plus the WikiText2 validation split through the exploration proxy evaluator. The agent could edit the solution implementation and schedule.

Fixed or forbidden inputs: no external data, weights, or network inputs; no WikiText2 test text; and no exploration weights, masks, or caches in formal retraining. Formal replay had to start from the fixed dense model and C4 shard. The final decoder-linear global sparsity had to lie in [0.699, 0.701], and the exported artifact had to load through the standard model loader.

Proxy evaluator: wikitext2_validation_perplexity, minimized, on the validation split. Each exploration evaluation contained 122 contiguous 2,048-token blocks. block_nll_per_token_sd is the standard deviation of block-average negative log likelihood, not a perplexity standard error, confidence interval, or between-seed uncertainty estimate.

Final evaluator: wikitext2_test_perplexity, minimized, on the withheld test split, with 140 blocks of 2,048 tokens. It likewise reports block-NLL dispersion but no standard error or confidence interval.

Artifact contract: numeric checkpoint directories, with no more than three simultaneously retained. If more than three are published cumulatively, the latest three progress points are independently final-evaluated; the official choice is the lowest-perplexity artifact that passes loading and sparsity gates.

The proxy and final evaluators use the same autoregressive perplexity arithmetic but disjoint splits and different sample counts. Their directions and qualitative trends can be compared; their scores must not be subtracted or treated as observations from one distribution.

3.2 How the baseline works

Fixed OPT-6.7B plus 128 seed-0 C4 calibration sequences of length 2,048 → forward passes collect mean-squared input-channel activation scales for every decoder linear layer → |weight| × sqrt(input scale) forms the Wanda importance score, and OWL outlier ratios assign layer-specific sparsities around the 70% global target → the lowest-scoring weights in each output row are set directly to zero, with no loss or optimizer → a complete half-precision OPT model containing the zeroed weights is exported as one progress-1 checkpoint.

There are no labels, rewards, or gradient updates in the baseline; the selection signal is entirely weight magnitude plus C4 activation statistics. All tensors are exported, but only pruned linear weights change numerically. The agent explicitly diagnosed pruning quality, rather than loading or the sparsity gate, as the bottleneck: the baseline loaded and had sparsity 0.699909, yet its validation perplexity was 57.838.

4. Four-hour exploration and decision process

The run began with task, baseline, and evaluator inspection and a measured baseline. Its first half covered the OPT SparseGPT port, a layer-allocation ablation, and calibration-width tests. The middle developed and debugged fixed-mask recovery training. The final part compared context length, step count, and learning rate, then tested checkpoint loading and an end-to-end reduced replay. The agent stopped active experimentation about 18 minutes before the deadline and explicitly submitted. Repeated runs for one hypothesis are consolidated below.

U-01 - Establishing whether baseline pruning quality was the bottleneck

Motivation and hypothesis. The agent needed to distinguish an invalid artifact from severe quality damage. It suspected that a one-shot magnitude-and-activation ranking was too destructive at 70% sparsity.

Concrete change and experimental setup. No code changed. The default baseline used 128 seed-0 C4 calibration sequences and was evaluated on the 122-block validation proxy.

Observed result. Perplexity was 57.838, block-NLL SD was 0.28225, and decoder-linear sparsity was 0.699909, inside the gate. Pruning took about 157 seconds, so repeated pruning probes were feasible.

Agent interpretation. Because loading and sparsity were correct while quality was poor, the agent treated the pruning criterion as the main defect and prioritized a method with second-order error compensation.

Report assessment and confounders. This was one calibration seed and cannot estimate seed variance, but it cleanly ruled out artifact format and gross sparsity failure as the immediate problem.

Decision and impact. The baseline remained a reference; its weights were not submitted. Work shifted to SparseGPT.