One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-sol / Codex CLI / xhigh

Status. Exploration completed and submitted; formal replay succeeded, three retained checkpoints passed frozen loading/sparsity validation, and all have official final summaries plus completion receipts. The audit found clean agent behavior, no protocol exposure, and compliant isolation.

What happened. The task requires a loadable OPT-6.7B with about 70% unstructured decoder sparsity, reconstructed from fixed dense-model/C4 assets. The baseline measures C4 activations, applies OWL/Wanda importance and layer allocation, then zeros weights without optimization; its seed-0 validation perplexity was 57.8382. The agent replaced it with curvature-compensated SparseGPT plus fixed-mask recovery and obtained a best official WikiText2 test perplexity of 13.8501 versus the disclosed sparse-baseline mean of 53.3590.

Four-hour exploration. Proxy scores below are wikitext2_validation_perplexity, minimized on 122 length-2048 validation blocks; block negative-log-likelihood (NLL) SD is not a score error bar. Repairing the OPT SparseGPT path reduced 57.8382 to 27.1112 with 32 calibration sequences and was adopted. Adding OWL's non-uniform layer allocation worsened it to 28.9880, so that tested combination was rejected. Fixed-zero-mask cross-entropy reduced perplexity from 22.3104 at five steps to 19.5503 at 200. The first FP16 scaling design failed, and one large-batch run became non-finite; FP32 master weights, mask reapplication, and non-finite skips fixed both. At approximately matched token volume, longer contexts improved about 19.26 at length 512 to 18.83 at 1024 and 18.50 at 2048, so full context was selected.

Increasing calibration at seed 0 improved raw SparseGPT from 21.3580 with 128 samples through 20.6880 with 1,024 to 20.3979 with 2,048. A post-export driver interruption did not invalidate the directly loaded 1,024 artifact. The late 2,048 result reversed the agent's tentative saturation judgment and became the formal default, although no other calibration seed or complete 2,048-sample recovery trunk was explored. Cross-entropy improved from 15.9432 at 50 steps to 15.2842 at 500. Batch-4 distillation OOMed; batch 1 succeeded and outperformed approximately token-matched pure cross-entropy. Staged dense-teacher distillation then moved 14.6020 to 14.4317; lowering the final learning rate to 1e-7 slightly regressed to 14.4326. A weight-0.25 probe scored 14.43156 versus 14.43168 for the established 0.5 path and 14.43711 for 0.75. Lacking replicates, the agent retained 0.5, so the lowest exploration point and submission differ marginally. A deadline smoke verified midpoint/final publication and loadability.

How the submitted method works. Baseline flow is dense OPT plus C4 activations to Wanda/OWL hard masking to one untrained sparse model. The candidate uses 2,048 C4 windows to estimate SparseGPT curvature, deletes 70% per row blockwise while compensating retained weights, then samples length-2048 C4 windows. Next-token labels drive cross-entropy; online dense-OPT targets drive temperature-1 KL distribution matching. AdamW updates FP32 master parameters while masked decoder positions are zeroed after every step. The patch changes the pruning algorithm, training signal, optimization, schedule, numerical engineering, and checkpoint policy. Formal replay applied only the text patch to fixed assets; exploration weights were excluded.

Formal and evaluation evidence. Replay used 42,072.117 of 43,200 seconds and exited cleanly under its deadline reserve. It completed 5,500 fixed steps, then a wall-clock tail to numeric progress 92,227; one non-finite gradient was safely skipped. Four checkpoints were cumulatively published, while retention kept the greatest three: 5,500 scored 13.941406, 48,827 scored 13.850099, and 92,227 scored 13.853095 on minimized WikiText2 test perplexity, test split, n=140. All had sparsity about 0.70000065 and loaded successfully. Reported block-NLL SDs were 0.265956, 0.267757, and 0.268064, not inferential uncertainty. The tail improved to the midpoint and then regressed by 0.0030; without an error bar, statistical ordering of the last two is inconclusive. Validation and test are disjoint, so proxy and final scores cannot be directly differenced, but the large sparse-baseline improvement survived independent final evaluation.

Audit and takeaway. Formal records confirm fixed C4, fixed dense start/teacher, a matching 23,104-byte patch hash, one isolated GPU, and runtime compliance. No explore command downloaded data or accessed test text; no hidden value was exposed, reconstructed, or embedded. For each artifact, two busy-GPU starts were gated before an idle-GPU score. Numbered-run/control scans found no second run or correction; job-level receipts resolve stale fields. The agent showed strong code-to-experiment diagnosis and replay engineering. Main limits are seed-0-only calibration, no end-to-end 2,048-sample explore replication, and no uncertainty for tiny late differences; calibration-seed robustness and the true 0.25-versus-0.5 ordering remain unproved.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-sol__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task requires a loadable OPT-6.7B whose decoder linear weights are approximately 70% zero, reconstructed from one fixed dense model using only a fixed C4 shard. The shipped baseline does no optimization: it measures C4 activations, assigns per-weight Wanda importance, uses OWL to vary sparsity between layers, and zeros weights once. Its seed-0 exploration validation perplexity was 57.8382, exposing a large one-shot pruning quality loss.

The agent first repaired the unused OPT SparseGPT path. SparseGPT uses calibration-derived second-order curvature to compensate retained weights while pruning. With only 32 calibration sequences, it reduced the same 122-block validation perplexity to 27.1112. The agent then rejected OWL's non-uniform allocation on top of SparseGPT, added fixed-mask recovery training, and studied recovery length, batch size, context length, calibration count, cross-entropy, dense-teacher distillation, learning-rate tails, distillation weight, and wall-clock checkpointing. The directly tested exploration trunk reached 14.4317 on validation. A distillation-weight-0.25 probe had a marginally lower point estimate of 14.4316, but the agent retained the better-supported 0.5 trunk because there was no uncertainty evidence resolving the tiny difference.

Formal replay rebuilt everything from the fixed dense model and C4 rather than importing exploration weights. It completed 2,048-sample SparseGPT, 5,500 fixed recovery steps, and a deadline-driven tail in 42,072.117 seconds. Four checkpoints were cumulatively published; retention kept and the frozen validator accepted progress 5500, 48827, and 92227. Their official 140-block WikiText2 test perplexities were 13.9414, 13.8501, and 13.8531, all within the sparsity gate. Progress 48827 was best. It beats the task-provided sparse-baseline test mean of 53.3590 by 39.5089 perplexity, while remaining 2.9896 above the ineligible dense reference of 10.8605.

The strongest capability was converting a code-level method opportunity into a controlled sequence of proxy ablations and then a replayable 12-hour recipe. The main scientific limitation is that every pruning calibration used seed 0 despite the task's warning that calibration-seed variation is material. In addition, the complete exploration recovery trunk used 1,024 calibration samples, while 2,048 was tested only as an untrained pruning artifact before being adopted formally. The final midpoint and endpoint differ by only 0.0030 without a standard error or confidence interval, so statistical superiority is inconclusive. The boundary audit found no agent violation or test-data exposure.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed facebook/opt-6.7b revision a45aa65…; dense and therefore ineligible at zero sparsity Available training data and assets: fixed allenai/c4 revision 1588… English shard; the same fixed dense model may be used as a teacher; WikiText2 validation is mounted only for explore-time proxy evaluation Agent-editable surface: pruning, layer allocation, calibration use, objectives, sampling, recovery optimization, schedule, and checkpoint policy under editable workspace Fixed or forbidden components: starting weights, C4 source, frozen final evaluator, and test text; no external data/weights, explore/formal test-text access, evaluation-specific lookup, resource overrun, or sparsity-gate evasion Proxy evaluator: wikitext2_validation_perplexity / minimize / validation / n=122 blocks of length 2048 / block negative-log-likelihood (NLL) SD only, not a score error bar Final evaluator: wikitext2_test_perplexity / minimize / test / n=140 blocks of length 2048 / block-NLL SD only, with no standard error or confidence interval Artifact contract: global unstructured sparsity across OPT decoder linear weights in [0.699, 0.701], loadable with AutoModelForCausalLM.from_pretrained; at most the three greatest numeric progress values are accepted, and the best valid final score is official ``

The proxy measures autoregressive perplexity on WikiText2 validation; final scoring measures it on the disjoint test split, with 0% block overlap. The metric form is the same, but the text and sample counts differ. A validation score of 14.4317 and a test score of 13.8501 therefore cannot be subtracted and interpreted as a generalization gain. block_nll_per_token_sd is dispersion across fixed text blocks, not repeated-run uncertainty.

3.2 How the baseline works

``text Fixed dense OPT-6.7B plus random C4 text windows -> forward hooks accumulate mean-squared activation for each linear-layer input channel -> |weight| × sqrt(channel activation scale) gives Wanda importance, while OWL outlier ratios allocate sparsity between layers -> the lowest-scoring target fraction in each row is set to zero; there is no label loss, optimizer, or fine-tuning -> one complete model is exported, changing only the decoder linear weights selected by the hard mask ``

The baseline defaults to 128 length-2048 C4 sequences at calibration seed 0, 0.7 unstructured sparsity, and one construction checkpoint. It neither compensates retained weights for the output error caused by deleted weights nor trains after pruning. The agent explicitly identified both bottlenecks after reading the source: an upstream SparseGPT implementation existed but lacked a working OPT route, and the baseline had no fixed-mask quality-recovery stage.

4. Four-hour exploration and decision process

The agent first inspected the task, baseline, sparsity checker, and evaluation boundary, then reproduced the baseline. It moved through pruning algorithm, layer allocation, recovery engineering and throughput, calibration scaling, objective design, and long-tail scheduling. The final portion was used for a deadline-recipe smoke test and checks of patch hash, syntax, loadability, sparsity, and remaining processes. The run lasted about 3 h 42 min before explicit submission; repetitive monitoring did not create separate scientific claims.

U-01 - Can curvature-compensated pruning replace one-shot Wanda?

Motivation and hypothesis. The agent found the SparseGPT foundation in the source but no usable OPT call path. It hypothesized that approximate-Hessian compensation of retained weights during blockwise pruning would reduce reconstruction error at the same 70% sparsity.

Concrete change and setup. It implemented OPT activation capture and layer replay, accumulated curvature per linear submodule, and pruned each row uniformly with damping 0.01 and block size 128. The first probe used only 32 length-2048 C4 sequences at calibration seed 0 and the same 122-block validation evaluator as the baseline.

Observed result. The baseline scored 57.8382 at sparsity 0.699909. The 32-sample SparseGPT artifact scored 27.1112 at sparsity 0.70000065 and was evaluable; construction took about 211 seconds.

Agent interpretation. The agent treated the more-than-30-point matched-protocol reduction as decisive evidence that uncompensated pruning, rather than merely OWL hyperparameters, was the main quality bottleneck.

Report assessment and confounds. The common model, split, evaluator, and target sparsity make the direction credible. SparseGPT changes both mask selection and retained-weight compensation, so the gain cannot be assigned to one internal operation. Only calibration seed 0 was tested.

Decision and impact. Uniform SparseGPT became the starting point for all later recovery experiments and entered the patch.