One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-terra / Codex CLI 0.146.0 / high

Status. Exploration, submission, formal retraining, three artifact validations, and three final evaluations completed. Lineage and scores are valid; the audit finds confirmed submission-timing noncompliance.

What happened. The task requires OPT-6.7B at 69.9%–70.1% global unstructured sparsity from fixed weights and C4. The baseline uses OWL layer allocation and Wanda activation-weight ranking without recovery; its buffer is fixed at 128 samples. The agent submitted second-order, error-compensating SparseGPT pruning plus mask-preserving dense-teacher distillation. Its best official WikiText2 test perplexity is 15.2356, versus the shipped 70%-sparse baseline's three-seed mean of 53.3590.

Four-hour exploration. The proxy was WikiText2 validation perplexity over 122 length-2048 blocks, minimized; blockwise SD is diagnostic, not an error bar.

• Replacing OWL/Wanda with 128-sample SparseGPT produced a 70%-sparse model at 21.3580 in about 312 seconds, so the agent adopted it. No same-split OWL/Wanda proxy control exists.
• Masked distillation mixed 0.8 temperature-2 KL divergence from the fixed dense teacher with 0.2 causal cross-entropy. Two longer probes left no trustworthy artifact around execution/save boundaries. A save test exposed missing safetensors; binary shards worked, and at learning rate 1e-5, 30 and 120 seconds reached 18.3988 and 17.4576, so recovery entered the patch.
• Increasing calibration to 256 samples improved seed 0 modestly to 21.0454. Seeds 0, 1, and 2 scored 21.0454, 21.5804, and 21.6548, so seed 0 was selected, although the distilled pipeline was not replicated. A post-export shell parse failure did not corrupt the scored model and was fixed.
• At 2e-5, roughly 30 and 120 seconds scored 17.1175 and the exploration best 16.5312. A 30-second 3e-5 run scored better than matched-duration 2e-5 at 16.7738, yet the agent called it a regression by comparing against the longer run. Long-horizon learning-rate superiority is therefore inconclusive.
• The agent finalized 256 samples, seed 0, and 2e-5, but submitted idle after 4,262 seconds with 10,179 seconds remaining. Measured cycles took minutes, so a 120-second 3e-5 replication or another seed could finish. This violates the explicit continue-work rule.

How the submitted method works. The baseline converts C4 activations into Wanda importance and directly zeros weights. The candidate estimates second-order statistics from 256 C4 windows, removes weights blockwise while compensating survivors, then samples fresh C4: the fixed dense model supplies logits, tokens supply next-token labels, and AdamW at 2e-5 updates surviving linear weights and other decoder-layer parameters while zero gradients are masked. This changes algorithm, signal, sampling, optimization, schedule, serialization, and checkpoint policy, without external input. Method, seed, and rate match the proxy best, but formal extrapolated 120 seconds to 40,200; exploration weights were excluded and replay began from the fixed start.

Formal and evaluation evidence. Formal replay completed about 446 seconds of pruning and 40,200 seconds with 114,513 distillation updates. The 40,766-second phase used 94.37% of budget and stopped normally, so there is no formal-underuse violation. Five publish calls included a repeated final 40200; four distinct progress values were produced, and retention three discarded 10800. The retained artifacts were loadable and sparsity-compliant. Formal proxy scores are not available because the proxy was not run.

| Progress | WikiText2 raw test perplexity, minimize, n=140 | Diagnostic fixed-block SD | Result | |---:|---:|---:|---| | 21600 | 15.2356 | 0.2621 | best | | 32400 | 15.2471 | 0.2628 | slightly worse | | 40200 | 15.2486 | 0.2616 | slightly worse |

The tiny late increase has no confidence interval or seed replication, so it does not establish significant overfitting. Proxy and final splits are disjoint. Final evidence validates the complete method, but no formal SparseGPT-only or distillation ablation separates component gains.

Audit and takeaway. Fixed C4/model, evaluator immutability, test isolation, no-network commands, one-GPU isolation, fresh formal replay, and patch hash all check out. Evaluator-only hidden identifiers and post-hoc results were identified, then literally cross-searched against the complete trajectory including tool results, patch, and experiment commands; no hidden value reached the trajectory, was used, or affected the result. Reading declared evaluator metadata is not leakage. Observable agent behavior is confirmed noncompliance solely for early submission; protocol exposure is none found, and platform/scheduling/resource isolation is compliant. One attempt-level status=running field is stale, but job-level success, exit 0, completion markers, and validations resolve usability. The case demonstrates a strong reproducible sparse-training recipe, not optimal learning rate/seed or component attribution.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-terra__high - gpt-5.6-terra / Codex CLI 0.146.0 / high - Full English Analysis

1. Run identity, attempt selection, and lineage

The central lineage records are the explore manifest, candidate patch, formal manifest, and job-level status.

2. Reader-facing overview

The task asks for a loadable OPT-6.7B artifact with global unstructured sparsity strictly inside 69.9%–70.1%, constructed from a fixed dense model and fixed C4 calibration shard while minimizing WikiText2 perplexity. The shipped method first uses OWL to allocate sparsity across layers, then applies Wanda's weight-magnitude-times-activation-scale ranking once. It has no gradient-based recovery and its upstream calibration buffer is hard-coded for 128 samples.

The agent replaced this construction with second-order, error-compensating pruning (SparseGPT), which uses input statistics to adjust surviving weights as other weights are removed. It then added masked distillation: the fixed dense model generates teacher distributions online, while the sparse student updates surviving weights and other decoder parameters without regrowing zeros. Exploration compared 128 versus 256 calibration samples, three calibration seeds, learning rates from 1e-5 to 3e-5, and short recovery durations; it also repaired a serialization dependency failure. The lowest trustworthy exploration proxy was 16.5312 validation perplexity with 256 samples, seed 0, learning rate 2e-5, and roughly 120 seconds of recovery.

Formal replay started again from the fixed dense weights rather than reusing exploration weights. It completed about 446 seconds of pruning and 40,200 seconds with 114,513 distillation updates, consuming 94.37% of the 12-hour budget. All three retained checkpoints were loadable and met the sparsity gate. Official WikiText2 test perplexities were 15.2356, 15.2471, and 15.2486; progress 21,600 was best. The shipped sparse baseline's same-protocol three-seed mean is 53.3590, so the submitted method is decisively better as a whole, although the final evidence cannot separate the contribution of SparseGPT from that of distillation.

The strongest capability was integrating a more principled pruning algorithm, a recovery objective, and robust wall-clock checkpointing into a formal-ready recipe. The central failure was submission timing. The task explicitly required continued meaningful work whenever remaining time could support a completed, interpretable experiment, yet the agent explicitly submitted idle after about 71 minutes with 10,179 seconds remaining. This trajectory measured a complete 256-sample prune-and-evaluate cycle in only several minutes. Observable agent behavior is therefore confirmed noncompliance. That finding does not alter the formal artifact lineage, sparsity, or scores, but it precludes describing the execution as protocol-compliant.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed facebook/opt-6.7b dense weights at revision pinned private revision Available training data and assets: fixed C4 English calibration shard; WikiText2 validation additionally mounted only for exploration proxy evaluation Agent-editable surface: pruning, recovery-training, launch, and checkpoint code in the solution workspace, submitted as candidate.patch Fixed or forbidden components: starting weights, C4 asset, evaluators, final WikiText2 test input, and formal mounts; network, external data, and external model weights are forbidden Proxy evaluator: wikitext2_validation_perplexity / minimize / WikiText2 validation / 122 2048-token blocks / fixed-block NLL-per-token SD is diagnostic, not a standard error or confidence interval Final evaluator: wikitext2_test_perplexity / minimize / WikiText2 raw test / 140 2048-token blocks / fixed-block dispersion only, with no sampling uncertainty Artifact contract: loadable with AutoModelForCausalLM; decoder-linear global sparsity in [0.699, 0.701]; at most three valid checkpoints retained simultaneously and best valid final score selected from up to three formal artifacts ``

The proxy and final evaluators use the same general tokenizer and perplexity arithmetic but disjoint splits. Proxy-to-final direction may be compared qualitatively; their numbers must not be subtracted or treated as repeated measurements from one distribution. The boundaries and mounts are established by task.toml, declaration.py, the proxy evaluator, and the final evaluator.

3.2 How the baseline works

``text fixed OPT-6.7B plus random C4 text windows -> collect layer inputs; OWL allocates layer sparsity and Wanda computes |weight| × sqrt(mean squared input activation) -> activation-scaled importance and the target sparsity determine which weights to remove -> remove the lowest-ranked target fraction per row, with no training loss or optimizer -> export linear weights containing the newly introduced zeros; other parameters remain unchanged ``

The baseline defaults to 128 length-2048 C4 samples with calibration seed 0 and emits one progress-1 model. It has no trainable object after pruning and no recovery schedule. The agent explicitly identified two limitations: independent Wanda deletions do not compensate for their output error, and the upstream calibration-input buffer is fixed at 128 so increasing nsamples does not actually broaden the statistics. The relevant source is run.sh, prune.py, and owl_opt.py.

4. Four-hour exploration and decision process

The agent first inspected the task, baseline, and evaluators, then produced its first SparseGPT artifact in a roughly five-to-seven-minute construction cycle. The middle of the run added distillation, investigated two short training attempts that left no trustworthy artifact, and fixed serialization. The final period compared calibration width, seeds, and learning rates before syntax, loadability, and sparsity checks. The entire exploration phase lasted only 4,262 seconds, far short of its four-hour allowance.

U-01 - Can second-order error compensation improve one-shot OWL/Wanda pruning?

Motivation and hypothesis. The agent reasoned that Wanda removes low-importance weights independently and cannot compensate for the severe output error at 70% sparsity. SparseGPT could use an input-covariance approximation to curvature, removing weights blockwise while transferring error into surviving weights.

Concrete change and experimental setup. It implemented an OPT SparseGPT adapter, corrected the calibration-buffer length, and first pruned with 128 C4 samples, seed 0, target sparsity 0.7, and no recovery. Evaluation used 122 WikiText2 validation blocks.

Observed result. Pruning took 311.953 seconds with about 21.12 GB peak memory. Global sparsity was 0.7000006411 and passed the gate. Proxy perplexity was 21.3580; fixed-block NLL-per-token SD was 0.2030.

Agent interpretation. The agent treated this as a strong, stable starting point and proceeded to investigate recovery training.

Report assessment and confounds. Exploration did not run the shipped OWL/Wanda method under the same validation protocol. The approximately 52.6–54.0 reference values in the task instructions are test scores, so they cannot be compared directly with 21.3580. This unit establishes a valid, exactly sparse, evaluable candidate but cannot by itself quantify a proxy improvement over the baseline. The later same-protocol final evaluation confirms improvement for the complete submitted method.

Decision and consequence. SparseGPT became the submitted pruner and the common starting point for all later ablations.