One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-terra / Codex CLI 0.146.0 / max

Status. Exploration, submission, formal retraining, three artifact validations, and three final evaluations completed with closed lineage. Agent behavior is clean; no protocol exposure was found; resources were compliant.

What happened. The task fixes OPT-6.7B/C4 and requires a loadable model at 0.699–0.701 decoder sparsity with low WikiText-2 perplexity. The baseline uses activations from 128 C4 sequences, Wanda importance, and OWL layer allocations to zero 70% once, without recovery. The agent found the fixed 128-row OPT buffer, an unusable LLaMA-shaped SparseGPT branch, and no repair stage. It submitted OPT SparseGPT plus mask-preserving distillation from fixed dense OPT; best official test perplexity was 15.155015.

Four-hour exploration. The proxy was lower-is-better perplexity on 122 validation blocks; block-NLL SD is dispersion, not an error bar.

After repairing SparseGPT, increasing calibration from 128 through 256 and 512 to 1,024 sequences monotonically improved 21.358021 to 20.687983, so 1,024 was adopted. At 512, calibration seed 1 was slightly worse than seed 0; each had one run. The shipped baseline was not rerun locally.

Dense-teacher recovery immediately reached 17.406253. With matched 512-sequence pruning, \(3\times10^{-5}\) reached 16.866398 early but regressed to about 17.10; \(2\times10^{-5}\) reached 17.037370. The agent kept the higher initial rate with cosine decay and early publications. The \(10^{-5}\) trial changed calibration size.

At step 800, 80% teacher/20% hard-label, pure temperature-1, and temperature-2 objectives scored 16.591889, 16.529584, and 16.995418; pure temperature 1 won. Moving from 512 to 1,024 tokens reached 16.087075; disabling dropout at 1,024 worsened it to 16.346689. A 2,048-token long run reached 15.771882 at step 2,125, motivating wall-clock recovery, though the 512/1,024 comparison also changed batch size.

Recovery seed 18 scored 15.763552 at step 1,068 versus seed 17's 15.781480 at step 1,020. Seed 18 was selected, but one progress-mismatched alternative is not robustness evidence. An abbreviated entry-point run validated loading, sparsity, metadata, and retention before the final context choice; formal replay verified the exact recipe. No scientific trial crashed or OOMed.

How the submitted method works. SparseGPT uses second-order C4 statistics to prune 70% while compensating retained weights. On new C4 spans, fixed dense OPT supplies temperature-1 token targets; the student uses pure teacher cross-entropy, AdamW, cosine decay, masked zero-entry gradients, and support reapplication. Because all student parameters enter the optimizer, unmasked embeddings, layer norms, and biases may update despite the agent's narrower description. The change spans pruning, online signal, trainable scope, schedule, engineering, and checkpoint policy. No external data or pre-existing reasoning was used. Exploration weights did not cross the boundary.

Formal and evaluation evidence. With calibration seed 0, recovery seed 18, 2,048-token contexts, and batch 1, replay completed 95,937 updates in 41,770.454 of 43,200 seconds (96.69%) by wall clock. Four artifacts were saved cumulatively; evicting progress 0 left three valid artifacts:

| Progress | WikiText-2 test perplexity, lower, n=140 | Diagnostic | Decision | |---:|---:|---|---| | 1,200 | 15.171726 | sparsity 0.700000648852; block-NLL SD 0.261695 | accepted | | 1,600 | 15.171515 | sparsity 0.700000648697; block-NLL SD 0.261677 | accepted | | 95,937 | 15.155015 | sparsity 0.700000647766; block-NLL SD 0.261764 | best |

The tail gain is small but positive. Under the same test protocol, the best artifact is 38.203972 below the shipped sparse-reference mean 53.358987. Proxy and final splits are disjoint and not numerically comparable; one formal seed cannot establish significance.

Audit and takeaway. Patch and replay used fixed OPT/C4, mounted no WikiText-2, and reused no explore weights, masks, or cache. Literal comparison of evaluator-only test input with the complete decoded trajectory, tool results, commands, and patch found no matching test row or nontrivial segment; accessible evaluator source exposed only declared metadata. No external input/extra model appeared, each phase used one unshared GPU, and source ID, patch hash, validations, and completion receipts close lineage.

Submission left 222 seconds, below measured 339-plus seconds for new pruning/evaluation or 621-plus for recovery/evaluation; formal use also complied. Agent behavior is clean, protocol exposure is none found, and resources are compliant. The run shows effective diagnosis-to-replay engineering, but the missing local baseline proxy and sparse seed coverage preclude universal or cross-seed claims.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-terra__max - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task fixes facebook/opt-6.7b, a C4 calibration shard, and a hard decoder-sparsity gate of 0.699–0.701. The shipped baseline gathers OPT activations, applies OWL layer allocations and Wanda weight importance, and zeroes roughly 70% of linear weights in one pass without recovery. The agent identified a hard-coded 128-sample OPT calibration buffer, an unusable LLaMA-shaped SparseGPT path, and the absence of post-pruning recovery.

During four hours, the agent implemented OPT-compatible SparseGPT error compensation, scaled calibration from 128 to 1,024 sequences, and added mask-preserving online distillation from the fixed dense OPT teacher on allowed C4 text. It compared learning rates and decay, mixed versus pure teacher targets, distillation temperature, context length, student dropout, calibration and recovery seeds, and end-to-end checkpoint retention. The submitted recipe uses 1,024 calibration sequences, pure temperature-1 teacher cross-entropy, 2,048-token contexts, recovery seed 18, and a long wall-clock run.

Formal replay completed 95,937 updates. All three retained artifacts loaded and remained at approximately 0.70000065 sparsity. Their official WikiText-2 test perplexities were 15.171726, 15.171515, and 15.155015; the last checkpoint was best and, under the same final protocol, substantially beat the shipped sparse-reference mean of 53.358987. Exploration used the disjoint validation split, so its proxy values cannot be subtracted from final test values.

The run demonstrates strong bottleneck diagnosis, staged ablation, and productionization. Its main scientific limits are the missing local proxy measurement of the shipped baseline and only one alternate value for each seed role. Observable agent behavior is clean; no hidden test input exposure was found, and platform/resource isolation was compliant.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: fixed facebook/opt-6.7b@a45aa65b… dense weights Available training data and assets: the designated C4 calibration shard, fixed dense model, local task source, and bundled evaluators Agent-editable surface: pruning, recovery, schedule, export, and checkpoint-selection logic under solution Fixed or forbidden components: starting weights, data source, final-test input, evaluator contract, and sparsity gate; external data/weights, network inputs, test-text access during explore/formal, and evaluation-specific lookup are forbidden Proxy evaluator: wikitext2_validation_perplexity / lower is better / WikiText-2 validation / 122 blocks of 2,048 tokens / block NLL-per-token SD only, not a standard error or confidence interval Final evaluator: wikitext2_test_perplexity / lower is better / WikiText-2 test / 140 blocks of 2,048 tokens / block NLL-per-token SD only, not a standard error or confidence interval Artifact contract: loadable model; aggregate OPT decoder-linear sparsity in [0.699, 0.701]; at most three greatest-progress checkpoints retained simultaneously ``

The proxy measures language-model perplexity on the visible validation split for exploration decisions. Final evaluation measures the hidden test split. Receipts show different block counts and zero overlap, so only qualitative survival of a method ranking can be discussed; the numerical proxy-to-final difference is not an improvement estimate. The reported block standard deviation describes between-block NLL dispersion, not uncertainty in the mean.

3.2 How the baseline works

``text fixed OPT-6.7B plus 128 C4 calibration sequences -> collect inputs to each decoder linear module -> compute Wanda importance as absolute weight times root-mean-square input activation, with OWL assigning mildly nonuniform layer sparsities -> rank weights row-wise and directly zero the least important entries to about 70% -> export one sparse model without gradient-based recovery ``

The baseline has no optimizer, training steps, or intermediate checkpoint schedule; it constructs and publishes one progress-1 artifact. The agent explicitly diagnosed the fixed 128-row OPT calibration buffer and a SparseGPT branch written for LLaMA-style module structure, then argued that one-shot pruning lacked a mechanism to recover language modeling. However, the agent never ran the shipped Wanda/OWL artifact through the local proxy. Its initial comparison of a 21.358 validation SparseGPT result with task-provided test references was protocol-confounded; only the later same-test final evaluation supports a baseline comparison.

4. Four-hour exploration and decision process

The first portion of the run established boundaries, repaired SparseGPT, and measured calibration scaling. The middle developed recovery, learning-rate scheduling, and target ablations. The final portion tested context length, dropout, and seeds, then hardened the formal entry point, metadata, checkpoint retention, and submission receipts. No scientific run failed through OOM, nonfinite loss, or a crash. Missing jq, a non-Git working directory, and polling an asynchronously launched child were inspection inconveniences rather than experimental failures.

U-01 - Does OPT SparseGPT work, and how much calibration is useful?

Motivation and hypothesis. The agent expected second-order error compensation to reduce damage from direct zeroing and expected 128 calibration sequences to estimate activation statistics poorly.

Concrete change and experimental setup. It connected Hessian accumulation and blockwise SparseGPT compensation to every OPT decoder linear module, made the calibration buffer respect arbitrary nsamples, and swept 128, 256, 512, and 1,024 fixed C4 sequences at seed 0, length 2,048, and approximately 70% sparsity. All scores used the 122-block validation proxy.

Observed result. Perplexity decreased monotonically: 21.358021, 21.045419, 20.729757, and 20.687983. Construction time rose from 315.94 to 1,085.59 seconds; the 1,024-sequence block-NLL SD was 0.203481. At 512 sequences, calibration seed 1 scored 20.761973 versus 20.729757 for seed 0.

Agent interpretation. SparseGPT was now functional, more calibration remained useful with diminishing returns, and 1,024 sequences fit the formal budget.

Report assessment and confounds. The monotonic curve supports the scale decision, but each point is a single run and only one alternate seed was tested. Without a local Wanda/OWL proxy run, this unit alone does not establish superiority to the shipped baseline.

Decision and consequence. The formal candidate adopted SparseGPT, 1,024 calibration sequences, and calibration seed 0.