One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-sol / Codex CLI 0.146.0 / medium

Status. Exploration, submission, formal retraining, validation of four checkpoints, and final evaluation of three accepted checkpoints completed with valid lineage. The audit found no evaluation-boundary exposure, but confirmed an explore submission-timing violation.

What happened. The task required a loadable OPT-6.7B model with decoder linear-weight sparsity in [0.699,0.701], using only a fixed dense start and C4 shard. The baseline uses OWL to allocate layer sparsity and Wanda's weight-magnitude-times-activation-scale ranking to zero weights directly, without reconstructing survivors; the agent reproduced 57.8382 validation perplexity. It submitted curvature-compensated SparseGPT pruning followed by narrow dense-teacher and language-model recovery. The best official test perplexity was 15.7540.

Four-hour exploration. Proxy scores are lower-is-better WikiText2 validation perplexity over 122 complete 2,048-token blocks; reported block dispersion is not a standard error. Uniform Wanda regressed to 181.7436, while increasing OWL lambda reached only 45.8652. Standard SparseGPT reached 21.3580 from 128 calibration samples and was adopted; OWL/SparseGPT allocation, sequential grouping, manual module allocation, and damping alternatives 0.03 and 0.005 all lost their relevant comparisons. Calibration seed 0 beat seeds 1 and 2; the seed-1 chain was interrupted after artifact construction, then evaluated separately. Widening from 128 to 1,024 samples improved 21.3580 to 20.6880 with diminishing returns; 2,048 samples were estimated but not run.

Full-weight FP16 recovery diverged to about 9.06×10^110. A nearly ineffective narrow run exposed a gradient-propagation bug; after repair, training 1,712,128 normalization/bias parameters reached 17.9582 at 1,000 cumulative steps. Embedding recovery was worse, and all-weight BF16 became stable after a library-argument fix but gave negligible gain, so both were rejected. At matched token count, 2,048-token recovery windows beat 512 and 1,024. Online dense-teacher hidden-state MSE reached 16.9237; a 1,000-teacher-plus-250-LM chain improved to 16.7939, whereas a longer 4,000-plus-500 chain regressed to 16.8543. From one fixed teacher checkpoint, LM refinement improved monotonically but flattened at 16.2785 after 8,000 steps, with block NLL/token SD 0.187606 and no SE/CI; a matched recovery comparison confirmed that 1,024-sample calibration retained its advantage. Recovery-seed robustness remained untested. Finally, 90- and 360-second replays verified phase switching, loadability, sparsity, and save timing, but were engineering checks rather than twelve-hour quality evidence.

How the submitted method works. SparseGPT estimates each linear module's input curvature from 1,024 seed-0 C4 sequences and compensates surviving weights as deletions are applied. Recovery samples 2,048-token C4 windows: an online fixed dense model supplies final hidden states for mean-squared-error phases, while shifted window tokens supply next-token cross-entropy phases. AdamW with cosine decay and clipping updates only normalization/bias; sparse linear weights stay frozen. Changes span pruning rule, signal, sampling, trainable scope, hyperparameters, schedule, engineering, and checkpoint policy. Exploration weights were excluded: formal replay applied only the 20,447-byte text patch and rebuilt from fixed inputs using early teacher, short LM, long teacher, and final-20%-LM phases, not the exact best fixed-step chain.

Formal and evaluation evidence. Formal recovery ran 40,879.004 seconds, completed 201,287 updates, and the whole phase used 42,112.248 of 43,200 seconds before its time-driven stop; formal underuse was not a violation. All four artifacts loaded; accepted artifacts had sparsity 0.70000065. Progress 1 was excluded by the greatest-three rule. On WikiText2 test, lower is better and n=140: checkpoint-1271 scored 15.991484, checkpoint-1910 scored 15.753996, and checkpoint-201287 scored 15.824594. Their block NLL/token SDs were 0.262970, 0.262275, and 0.263653, with no SE/CI. The early checkpoint was best and the long endpoint slightly regressed, so the broad gain over the shipped sparse mean 53.3590 survived, but monotonic long-refinement behavior did not; validation and test values are not directly comparable.

Audit and takeaway. Fixed C4/model inputs, frozen evaluators, offline single-GPU execution, patch-only explore-to-formal transfer, hashes, and receipts were consistent; no external process shared the assigned GPU. Literal cross-checking first identified the evaluator-only WikiText2 test input, then searched the complete trajectory including tool returns, commands, and patch: no hidden test path, content hash, or text reached the trajectory, and no querying, reconstruction, or use was found. Observable agent behavior is confirmed noncompliant because it submitted idle with 865 seconds left although this trajectory measured a meaningful 2,000-step refinement plus evaluation at about 405 seconds. Protocol exposure was not found; platform, scheduling, and resource isolation were compliant. The work demonstrates strong ablation and replay engineering, but cannot establish recovery-seed robustness, long-endpoint superiority, or global optimality.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-sol__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

The formal dispatch receipt, formal manifest, and outcome receipt establish the lineage. auto_retrain=false in the explore manifest says only that the explore run did not itself create a retrain phase; the upstream formal dispatch is independently established by these receipts.

2. Reader-facing overview

The task asks for a loadable OPT-6.7B model whose decoder linear weights are approximately 70% zero, while minimizing WikiText2 perplexity from a fixed dense model and fixed C4 shard. The shipped method gathers activations from 128 C4 sequences, uses OWL to allocate nonuniform layer sparsity, and applies Wanda's weight-magnitude-times-input-scale ranking to zero weights directly. It performs no optimization or reconstruction of the surviving weights. The agent reproduced a 57.8382 validation perplexity and focused on direct-zeroing reconstruction error, calibration width, and seed sensitivity.

The agent replaced the main pruner with SparseGPT, which uses an approximate inverse input-curvature matrix to compensate the error caused by each deletion. With only 128 calibration sequences this reduced validation perplexity to 21.3580. It then tested and rejected OWL/SparseGPT allocation hybrids, sequential grouping, hand-tuned module allocations, alternative damping, embedding recovery, and all-weight recovery. The selected family used 1,024 calibration sequences with seed 0, 2,048-token windows, about 1.71 million trainable normalization and bias parameters, dense-teacher hidden-state matching, and a final language-model-loss phase. The best exploration weights reached 16.2785 on 122 validation blocks, but only source code—not those weights—was submitted.

Formal replay cleanly rebuilt the model from the fixed start. It spent 40,879 seconds in time-driven recovery and completed 201,287 updates. All four published checkpoints were loadable; the collector accepted the three greatest progress values and excluded the progress-1 pruned checkpoint. Official test perplexities were 15.9915, 15.7540, and 15.8246, with checkpoint-1910 best. The result is much better than the shipped sparse baseline's 53.3590 three-seed mean under the same final protocol, though still above the ineligible dense reference of 10.8605.

The research work was technically strong, but the explore submission timing is a separate confirmed execution-protocol violation. The agent explicitly submitted while idle with 865 seconds remaining even though this trajectory measured a complete 2,000-step refinement plus frozen evaluation at about 405 seconds. A further refinement or recovery-seed check could therefore have finished and been interpreted. This does not contaminate the patch or official scores. Formal budget use was time-driven and compliant.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: facebook/opt-6.7b at fixed revision pinned private revision Available training data and assets: fixed allenai/c4 English train shard 0; exploration may call the frozen WikiText2 validation evaluator Agent-editable surface: launch script, pruning/OWL code, calibration sampling, objective, recovery training, and checkpoint policy under editable workspace Fixed or forbidden components: dense start, C4 shard, frozen evaluators, final test text, and 70% sparsity gate; no external data/weights, exploration access to test text, or evaluator-specific lookup Proxy evaluator: wikitext2_validation_perplexity, lower is better, WikiText2 validation, n=122 complete 2,048-token blocks; block-level NLL-per-token SD is reported, but no standard error or confidence interval Final evaluator: wikitext2_test_perplexity, lower is better, WikiText2 test, n=140 complete 2,048-token blocks; likewise only block-level dispersion is reported Artifact contract: a fully loadable model with global zero fraction for OPT decoder linear weights in [0.699, 0.701]; formal accepts at most the three valid checkpoints with greatest numeric progress and the lowest final perplexity wins ``

Proxy and final evaluation use the same tokenizer, block length, loading path, sparsity recomputation, and perplexity arithmetic, but consume disjoint validation and test text. They can show whether a broad direction transfers; their numerical values must not be subtracted as though they came from one split. block_nll_per_token_sd is dispersion across fixed text blocks, not a standard error for perplexity.

3.2 How the baseline works

``text Fixed OPT-6.7B plus 128 seed-0 C4 sequences of length 2,048 -> run layerwise forward passes and accumulate mean-square input activation by linear-layer input channel -> use OWL's weight-activation outlier ratio for layer sparsity allocation and Wanda's |weight| times square-root activation scale for ranking -> zero the lowest-ranked weights per row until decoder sparsity is approximately 70% -> export the complete OPT model without optimizer updates or compensation of surviving weights ``

The default baseline is wanda_owl with lambda=0.08, hyper_m=5.0, 128 calibration sequences, and seed 0; it publishes a single constructed artifact. The agent first noted that the original calibration buffers were organized around 128 samples and made its implementation allocate them dynamically. The more consequential diagnosis came from measurement: Wanda chooses deletions but does not reconstruct the retained weights, producing 57.8382 validation perplexity. The task's three shipped sparse final scores range from 52.6180 to 53.9975, so small gains would also require seed-aware interpretation.

4. Four-hour exploration and decision process

The agent spent roughly the first ten minutes reading the launch, pruning, activation-capture, and evaluation paths, then established the baseline within about four minutes. The next hour emphasized pruner, allocation, damping, and recovery-scope choices. The middle of the run debugged numerical and gradient failures and swept calibration width, context length, and objective. The final portion tested teacher/LM mixtures, exercised the wall-clock checkpoint recipe, and studied calibration–recovery interaction. The 8,000-step final refinement ended with 16 minutes 22 seconds left; source, hash, and process checks followed, and submission occurred with about 14 minutes 25 seconds remaining.

U-01 - Was OWL allocation, rather than the Wanda mask, the main bottleneck?

Motivation and hypothesis. The agent first tested whether poor baseline quality mainly came from OWL's layer allocation. If so, widening the inter-layer sparsity range would offer a low-risk improvement without changing the pruner.

Concrete change and experimental setup. All comparisons used seed 0, 128 C4 calibration sequences, and the same 122-block validation evaluator. It reproduced default OWL/Wanda, compared uniform 70% Wanda per module, and increased OWL lambda from 0.08 to 0.10.

Observed result. Default OWL/Wanda scored 57.8382; uniform Wanda degraded to 181.7436; lambda=0.10 improved to 45.8652. All artifacts met the sparsity gate. Baseline construction/export took 181 seconds and proxy evaluation about 22 seconds.

Agent interpretation. The agent concluded that nonuniform OWL allocation was useful, but even the widened version remained far behind the reconstruction-based pruner found next.

Report assessment and confounds. The sweep covers only one calibration seed, so it does not identify an optimal lambda. The more-than-120-point uniform-Wanda regression and roughly 12-point lambda gain are nevertheless too large to plausibly be evaluator noise.

Decision and consequence. Uniform Wanda was rejected and widened OWL retained briefly as a weak fallback; the submitted main path does not use Wanda/OWL.