One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-terra / Codex CLI / xhigh

Status. Exploration, explicit submission, formal retraining, validation of three retained artifacts, and all three final evaluations completed. Each final summary.json has its .complete receipt. One numbered formal attempt succeeded; authoritative lifecycle/job receipts resolve stale low-level failed and running fields.

What happened. The task reconstructs a loadable OPT-6.7B at decoder sparsity [0.699, 0.701] from a pinned dense model and C4 shard. The baseline makes one OWL/Wanda pass: activation-weighted magnitude ranks weights, OWL varies sparsity by layer, and low-ranked weights become zero without training. Its validation perplexity was 57.8382, and its activation buffer was hard-coded to 128 calibration sequences. The agent submitted 1,024-sequence SparseGPT plus mask-preserving C4 distillation; the best official test perplexity was 16.1532607.

Four-hour exploration. Proxy evaluation minimized WikiText2 validation perplexity over 122 complete 2,048-token blocks; block-NLL SD was diagnostic, not an error bar. At the matched 128-sequence, seed-0 setting, SparseGPT used the full input-activation Hessian and retained-weight error compensation, scoring 21.3580 versus 57.8382, so it replaced OWL/Wanda; a planned uniform-Wanda control was never run. Full-model distillation initially failed before a checkpoint because the scaler could not unscale FP16 parameter gradients. BF16 native backward then preserved zero weights and reached 17.2365 in 480 seconds, so this direction was adopted. A layerwise teacher-reconstruction alternative first failed because a batch-1 attention mask was used with batch 2; a batch-1 diagnostic ran but regressed to 22.3859, so its source was removed. Seed 1 at width 128 scored 22.2979, worse than seed 0. With seed 0 fixed, unrecovered 256/512/1,024-sequence masks scored 21.0454/20.7298/20.6880; matched short recoveries scored 16.9533/16.7945/16.7193, showing diminishing gains by 1,024. Longer recovery was not monotonically useful: a 128-sequence constant-rate run ended at 17.8682, a cosine run at 17.2636, and the 1,024-sequence longer cosine path at 16.8967. A ten-minute fixed-2e-6 continuation scored 16.7132—safer but still worse than stopping. Exact run.sh replay reproduced the pruning hash and scored 16.6562940 at step 2,051. This justified the core recipe, but not the many-hour tail.

How the submitted method works. Baseline flow is C4 activations → Wanda/OWL importance and layer allocation → one-shot zero mask → sparse model. Candidate flow is 1,024 C4 calibration windows → SparseGPT second-order mask and retained-weight compensation → online dense-teacher probabilities plus true next tokens → equal cross-entropy/temperature-2 KL optimized by AdamW → atomic sparse checkpoints. All student parameters can update except zero locations in decoder linear weights, whose gradients are cleared each step. The change is mixed algorithm, training signal, sampling, hyperparameter, schedule, engineering, and checkpoint policy. Exploration weights did not cross the boundary; formal replay rebuilt from the pinned start. The formal default used 480 seconds of the higher-rate schedule, then 2e-6 to 36,000 seconds.

Formal and evaluation evidence. Formal pruning took 1,112.7 seconds; retraining totaled 37,249.953 seconds, 86.23% of the 12-hour budget, and stopped at the configured recovery duration. Six complete numeric models were cumulatively written, but retention kept only the latest three; all loaded and passed sparsity. Formal did not run the proxy. On WikiText2 test (minimize, n=140), progress 98,337 scored 16.1532607 with block SD 0.2608339; progress 131,113 tied at 16.1532607 with SD 0.2608075 and a distinct hash; progress 163,904 slightly regressed to 16.1590082 with SD 0.2607570. These SDs are not uncertainty intervals. The best is 37.205726 below the shipped sparse mean 53.358987, but 5.292805 above the gate-ineligible dense reference. Proxy and test splits are disjoint, so their values cannot be subtracted; deleted early formal checkpoints prevent testing whether the eight-minute stop was better.

Audit and takeaway. Raw commands used only pinned C4, the pinned dense model, and permitted validation text; no network input, extra model, test-text access, evaluator modification, or explore-weight contamination was found. Formal lineage, patch hash, one-GPU idle gate, wall clock, validation receipts, and paired final receipts are consistent; reading accessible evaluator code exposed only declared protocol structure, not test content or candidate scores. Observable agent behavior is clean; protocol/evaluation-boundary exposure: none found; platform, scheduling, and resource isolation: compliant. The agent showed strong matched experimentation and replay validation. Remaining limits are seed-0-only evidence at the chosen width, an under-validated low-rate tail, and no evidence that long recovery beats stopping at 480 seconds.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-terra__xhigh - Full English Analysis

1. Run identity, attempt selection, and lineage

The explore manifest's auto_retrain=false and retrain_phase_created=false mean only that this explore directory did not itself create a retrain phase. Upstream control records show a separate, successful formal run. Inspection of numbered directories and control status found only formal replay 1 for this configuration and no correction receipt or second attempt.

2. Reader-facing overview

The task is to reconstruct a loadable OPT-6.7B whose decoder linear weights are approximately 70% zero, starting only from a pinned dense model and a pinned C4 shard, while minimizing WikiText2 perplexity. The shipped baseline makes a single OWL/Wanda pruning pass: OWL allocates nonuniform layer sparsities, Wanda ranks weights by magnitude times the root-mean-square input activation, and the lowest-ranked weights are zeroed without training. Its exploration-validation perplexity was 57.8382. The code also fixed its activation buffer at 128 calibration sequences, preventing a simple increase in calibration coverage.

The agent replaced the mask constructor with SparseGPT, which uses the full input-activation Hessian and compensates retained weights after pruning. At the same 128-sequence, seed-0 setting, validation perplexity fell to 21.3580. It then added masked C4 distillation from the pinned dense model, reaching 17.2365, while a layerwise block-reconstruction alternative failed once and then regressed to 22.3859 in a runnable diagnostic. Widening calibration to 1,024 sequences and replaying the exact source path produced the exploration best, 16.6563. The submission was a reconstruction recipe, not exploration weights: 1,024-sequence SparseGPT followed by mask-preserving C4 distillation with timed checkpoints.

Formal replay rebuilt from the pinned dense start, spent 1,112.7 s pruning and the configured 36,000 s distilling, and completed retraining in 37,249.953 s, 86.23% of the 12-hour budget. The three retained checkpoints were valid. Their official WikiText2 test perplexities were 16.1532607, 16.1532607, and 16.1590082; the first two tie for best. Under the same test protocol, the best is 37.205726 lower than the shipped sparse mean of 53.358987, although it remains 5.292805 above the gate-ineligible dense reference of 10.860456.

The trajectory demonstrates strong matched experimentation, negative-result handling, and source-level replay validation. Its main scientific limitation is narrow robustness evidence: only seed 0 was used at the chosen width and in formal replay, and the many-hour low-rate tail was extrapolated from a ten-minute pilot that was itself slightly worse than stopping after eight minutes. Command, mount, hash, and receipt audits show clean observable agent behavior and no final-evaluation-boundary exposure.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: pinned facebook/opt-6.7b revision a45aa65…; formal must reconstruct from this dense model Available training data and assets: the pinned English C4 shard; WikiText2 validation is additionally available only for explore proxy evaluation Agent-editable surface: pruning, training, sampling, objective, schedule, export, and checkpoint code under editable workspace Fixed or forbidden components: starting weights, mounted data, frozen final evaluator, and sparsity gate; no external data/weights, network input, test-text access during explore/construction, or evaluation-specific lookup Proxy evaluator: wikitext2_validation_perplexity / minimize / WikiText2 validation / n=122 complete 2,048-token blocks / block NLL-per-token SD is reported but is not a score error bar Final evaluator: wikitext2_test_perplexity / minimize / WikiText2 test / n=140 complete 2,048-token blocks / the same within-text dispersion, with no standard error or confidence interval Artifact contract: decoder sparsity in [0.699, 0.701] and loadable; formal accepts up to three numeric-progress checkpoints and selects the best valid result ``

Proxy and final evaluation use the same tokenizer, 2,048-token blocking, and perplexity computation, but their splits are disjoint; final summaries record zero overlapping blocks. They can show whether the direction of improvement transfers, but subtracting 16.6563 from 16.1533 would not estimate generalization or overfitting. block_nll_per_token_sd describes dispersion among blocks of one fixed text and is explicitly not sampling uncertainty on perplexity.

3.2 How the baseline works

``text [pinned OPT-6.7B plus 128 seed-0 C4 windows of 2,048 tokens] -> [a layerwise forward pass accumulates mean-squared input activation per linear-layer channel] -> [Wanda importance is |W| × sqrt(mean-squared input activation), while OWL assigns nonuniform layer sparsities from outlier ratios] -> [the lowest-scored weights in each row are set to zero; there is no loss or optimizer] -> [decoder linear weights change and one sparse model is exported] ``

The baseline uses OWL parameters lambda=0.08 and Hyper_m=5, with calibration seed 0. It has no training loop, learning rate, or checkpoint sequence. The agent explicitly diagnosed two bottlenecks: severe quality loss at 70% sparsity, and a hard-coded 128-row buffer in prepare_calibration_input_opt that would fail if CALIBRATION_SAMPLES were simply increased. The first motivated second-order pruning and recovery; the second was later fixed by sizing the buffer to len(dataloader).

4. Four-hour exploration and decision process

Exploration consumed 14,178 s and submitted with 263 s remaining. The agent first inspected boundaries, established the shipped baseline, and implemented a matched SparseGPT control. It next validated full-model recovery, debugged FP16, and tested blockwise reconstruction. The middle and late phases measured recovery duration, learning-rate behavior, seed sensitivity, and calibration-width scaling. It finished by reproducing the selected mask hash and checkpoints through the real run.sh, testing the remaining schedule uncertainty, clearing redundant polling processes, and auditing the final patch.

U-01 - Baseline quality and the calibration-buffer bottleneck

Motivation and hypothesis. The agent needed a gate-valid measurement of the shipped method under the actual proxy protocol, both to separate model-quality problems from export failures and to decide where to invest the budget. Source inspection also suggested that calibration could not be widened because an activation buffer was fixed at 128 rows.

Concrete change and experimental setup. The first experiment made no algorithmic change: 128 C4 windows, seed 0, OWL's nonuniform layer allocation, and 70% unstructured sparsity. The later patch changed only the buffer's leading dimension from 128 to the dataloader length.

Observed result. Pruning took 161.352 s. The model loaded, had measured sparsity 0.6999092, and scored 57.8381855 on 122 validation blocks; block NLL-per-token SD was 0.2822488.

Agent interpretation. It regarded the result as confirming the large sparsification cost described by the task and chose to focus on mask quality and post-pruning recovery rather than output handling.

Report assessment and confounds. This is a valid exploration anchor, but it is not directly subtractable from the task's shipped test scores because the split differs. The 128-row source defect is real, although it does not affect this 128-sample run.

Decision and consequence. The result became the matched baseline, and dynamic buffer sizing enabled later 256/512/1,024-sample experiments. The agent mentioned a uniform-Wanda control as a plan, but never ran or scored it; no conclusion can be assigned to that control.