One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-sol / AI4AI-Bench Codex CLI 0.146.0 / high

Status. Explore exploration attempt 2, explicit submission, formal retraining, three artifact validations, and three official final evaluations completed; exploration attempt 1 was an infrastructure failure before trial classification. Observable agent behavior was clean.

What happened. The task minimizes OPT-6.7B perplexity subject to approximately 70% unstructured decoder sparsity. Shipped OWL/Wanda uses C4 activations to rank and delete weights without recovery; its valid 122-block WikiText-2 validation perplexity was 57.8382, and the agent found that calibration storage was hard-coded to 128 sequences. It submitted uniform SparseGPT second-order reconstruction followed by mask-preserving dense-teacher distillation; the best 60,000-step formal artifact scored 13.2531 on the final test.

Four-hour exploration. All proxy scores here are lower-is-better validation perplexity on 122 fixed blocks, with no standard error or confidence interval. Uniform SparseGPT improved 57.8382 to 21.3580 and was adopted; adding OWL's nonuniform layer budgets worsened it to 22.6240 and was rejected. Initial FP16 recovery failed during gradient unscaling, so the agent trained FP32 live weights while both masking zero gradients and resetting zeros after each step. Label likelihood improved from 18.0363 at five steps to 15.9346 at 500, but dense-teacher targets reached 15.6079 at 200, 14.7224 at 2,000, and 14.5588 at 5,000 cumulative steps, so distillation was selected. Calibration seed 1 scored 22.2979 versus seed 0's 21.3580, revealing sensitivity and motivating seed 0 without proving robustness. After fixing the buffer, widths 256, 512, and 1,024 improved monotonically to 20.6880; a matched recovery check preserved the benefit, so 1,024 was adopted despite diminishing returns and about 51.18 GB peak memory. Lowering SparseGPT damping from 0.01 to 0.001 was slightly worse; an accidentally recovery-enabled launch was interrupted, then the controlled rerun justified reverting it. A staged 2,000-step run produced the exploration best, 14.3287. Enlarging the 200-step recovery pool from 256 to 16,384 blocks gained only 0.0309 but was retained. Training/export smoke, reload, sparsity, and intermediate-checkpoint checks passed. The planned 60,000 steps were extrapolated beyond the explored maximum of 5,000.

How the submitted method works. Baseline C4 activations feed Wanda/OWL scores, which directly determine zeros without optimization. The candidate instead runs uniform 70% SparseGPT on 1,024 seed-0 C4 blocks, using approximate second-order information to compensate retained weights. The sparse FP32 student then samples from a 16,384-block C4 pool with seed 17; the frozen fixed dense model produces full-vocabulary probabilities online, and fused AdamW minimizes teacher-to-student cross-entropy over all trainable student parameters. Decoder zeros remain fixed by gradient masks plus post-update reset; snapshots alone convert to FP16. This is a mixed algorithm, signal, sampling, schedule, engineering, and checkpoint-policy change. No external teacher, rationale, or exploration weights entered formal replay, which rebuilt from fixed assets using the byte-identical patch hash verified private digest.

Formal and evaluation evidence. Formal replay completed all requested 60,000 steps in 40,726.859 of 43,200 seconds, without truncation. Four checkpoints were cumulatively published at progress 1, 2,000, 20,000, and 60,000; retention three left the latter three, all loadable at sparsity about 0.700000648. Each has summary.json plus .complete. Lower-is-better test perplexity on 140 fixed blocks was 13.857927, 13.439248, and 13.253061, respectively; the best also beats the task-provided same-protocol sparse mean of 53.3590. No SE or CI exists; block-NLL SDs were 0.266692, 0.267364, and 0.267658 and are diagnostics, not error bars. Improvement continued with diminishing returns. Formal validation proxies are not available, so the exploration and disjoint-test values cannot be subtracted.

Audit and takeaway. Evidence shows only the fixed OPT model and C4 were used; final-test text was absent from exploration/formal mounts, the evaluator was unchanged, no external input succeeded, and no exploration weights crossed into formal replay. One idle assigned GPU was used per phase within wall clocks, and formal lineage names exploration attempt 2 and the identical patch. Denied background egress entries have no matching agent command or experimental input. Explore wrapper exit fields conflict, but explicit submission and independent downstream receipts establish artifact usability. Thus agent behavior was clean, protocol exposure was not found, and platform/resource isolation was compliant. The run demonstrates strong experimental closure, but one formal seed, an unreplicated small pool gain, no formal validation curve, and the unexplored 60,000-step horizon prevent claims of cross-seed robustness or absence of overfitting.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-sol__high - Full English Analysis

1. Run identity, attempt selection, and lineage

• Task: owl_wanda_opt6p7b_70pct, minimizing language-model perplexity while enforcing approximately 70% unstructured sparsity in OPT-6.7B decoder linear weights.
• Task family: post-training large-language-model pruning with mask-preserving recovery.
• Model: the research agent was gpt-5.6-sol; the compressed model was the fixed facebook/opt-6.7b revision.
• Harness: AI4AI-Bench v1.5 with Codex CLI 0.146.0.
• Reasoning effort: high.
• Seed, if applicable: formal pruning calibration used seed 0 and recovery used seed 17. Exploration also tested pruning seed 1 and used seed 29 for one continuation.
• Exploration budget: 14,400 seconds; the phase ran for 12,755 seconds and submitted with 1,688 seconds remaining.
• Formal budget: 43,200 seconds. The orchestrator reserved 600 seconds and injected a 42,600-second training limit; the formal retraining phase used 40,726.859 seconds.

• Primary explore run: exploration attempt 2.
• Why this attempt was selected: it contains the complete raw trajectory, a nonempty patch, an explicit submission receipt, and the experiment artifacts. The formal manifest also names it as selected-exploration lineage.
• Prior full-run attempts and outcomes: there was no earlier complete scientific run. exploration attempt 1 has no agent trajectory or candidate patch. Its directory manifest is stale at running, while the authoritative control receipt classifies it as terminal_infrastructure because the launcher exited before trial classification.
• Candidate patch: 14,919 bytes modifying owl/lib/prune_all.py, owl_opt.py, recover.py, and run.sh.
• Patch hash / validation: SHA-256 verified private digest. The explore and formal copies are byte-identical, and formal application succeeded.
• Formal run: formal replay 1. There are no unlinked or partial formal runs.
• Formal selected-exploration lineage: exploration attempt 2.
• Checkpoint-validation receipts: the 2,000-, 20,000-, and 60,000-step artifacts all loaded through AutoModelForCausalLM.from_pretrained, each with 6,658,473,984 parameters and a terminal validation receipt.
• Final-evaluator receipts: all three checkpoints have both summary.json and the corresponding .complete receipt, so all are official results.

Evidence completeness. The task source, raw trajectory, every exploration proxy result, candidate patch, control status, formal manifest and training metrics, three checkpoint validations, and three final evaluations are available.

Missing or conflicting evidence. No validation proxy was run during formal retraining, so formal-checkpoint validation perplexities are not available. The raw agent wrapper records exit code 137 on the final tool action, and the explore manifest simultaneously contains agent_state=failed; however, lifecycle and submission records say the explicit submission completed, the manifest exit status is 0, and formal replay consumed the same patch hash. The control job summary separately reports exit status 127. The exact cause of these wrapper-state differences is inconclusive, but formal lineage and scientific results are unaffected. The formal manifest has score phase false because final evaluation ran later under the separate final-tests tree; the three external completion receipts establish that final scoring did complete.

2. Reader-facing overview

The task requires minimizing OPT-6.7B perplexity while keeping about 70% of decoder weights exactly zero. The shipped baseline ranks weights with activation-aware weight pruning (Wanda) and changes layer budgets using outlier-weighted layer allocation (OWL). It performs no recovery training, and its calibration buffer was hard-coded to 128 sequences. The agent's valid baseline scored 57.8382 perplexity on 122 fixed WikiText-2 validation blocks, showing severe quality damage.

The agent replaced Wanda with SparseGPT's second-order local reconstruction, then investigated OWL allocation, recovery objectives, training duration, calibration seed and width, Hessian damping, recovery-pool size, and checkpoint execution. The submission combined uniform 70% SparseGPT on 1,024 C4 sequences with mask-preserving distillation from the fixed dense model. A fully matched 2,000-step staged exploration run achieved the best validation proxy, 14.3287. Exploration weights were not submitted; formal replay rebuilt the model from the fixed dense checkpoint and C4 shard.

Formal retraining completed all requested 60,000 updates. All three retained artifacts loaded and passed the sparsity gate. On 140 fixed WikiText-2 test blocks, perplexity improved with progress from 13.8579 to 13.4392 and then 13.2531, making 60,000 steps best. This is far below the task-provided same-protocol shipped sparse mean of 53.3590, although it remains above the gate-ineligible dense reference of 10.8605.

The strongest capability was a coherent sequence of controlled comparisons, negative ablations, and formal-executability checks around an explicit bottleneck. The central limitation is that the end-to-end formal recipe has only one seed, while its 60,000-step horizon extends far beyond the 5,000 steps directly explored. There is no confidence interval or formal validation curve, so cross-seed robustness and long-horizon overfitting remain unresolved. The boundary audit found no misconduct or exposure of final-test text.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

• Starting artifact / model: fixed facebook/opt-6.7b revision pinned private revision.
• Available training data and assets: the fixed C4 English calibration shard, starting model, tokenizer, task source, and the WikiText-2 validation split during exploration.
• Agent-editable surface: candidate source, pruning and recovery method, hyperparameters, schedule, and artifact-publication logic in the workspace.
• Fixed or forbidden components: starting weights, mounted data, evaluators, final-test text, sparsity gate, one-GPU limit, and wall clock. Network mode was no-network.
• Proxy evaluator: wikitext2_validation_perplexity, lower is better, WikiText-2 validation, 122 fixed 2,048-token blocks. No standard error or confidence interval is supplied. block_nll_per_token_sd is dispersion across fixed blocks, not an error bar on the mean.
• Final evaluator: wikitext2_test_perplexity, lower is better, disjoint WikiText-2 test, exactly 140 fixed 2,048-token blocks. It likewise supplies no standard error or confidence interval, only blockwise dispersion.
• Artifact contract: a model must load through standard Transformers and have actual decoder-tensor sparsity in [0.699, 0.701]. Formal replay may retain at most three valid artifacts simultaneously.

Both evaluators compute perplexity from per-token negative log likelihood, but they use different splits and block counts. They can establish whether a method direction survives; their numeric scores cannot be subtracted as if they came from the same evaluation distribution.

3.2 How the baseline works

The fixed OPT-6.7B model and 128 C4 calibration sequences are forwarded through the network. Hooks accumulate mean squared inputs for each linear layer. Wanda scores a weight by its absolute magnitude times its activation scale, while OWL uses each layer's outlier ratio to assign slightly different layer sparsities. The lowest-scored weights in each row are set to zero, producing approximately 70% unstructured sparsity. There is no loss, gradient update, or fine-tuning; a single progress-1 artifact is published.

Calibration data therefore supplies only the construction score, and the changed object is the pattern of zero decoder-linear weights. The agent explicitly identified two bottlenecks: measured perplexity 57.8382 showed that training-free pruning was highly destructive, and prepare_calibration_input_opt allocated only 128 input slots even when the public argument requested more. The baseline also had no mechanism to repair retained weights after pruning.

4. Four-hour exploration and decision process

The agent first inspected the contract and baseline and established a loadable, gate-valid anchor. It then used the GPU roughly in the order “pruning rule, recovery training, robustness and scale sweeps, formal-recipe revalidation.” A mixed-precision failure in the middle prompted a recovery implementation change; the later phase focused on calibration width, damping, sample pool, staged learning rates, and checkpoint lifecycle. It explicitly submitted after 12,755 seconds with 1,688 seconds left, having produced 19 scored proxy evaluations plus loading, sparsity, and export checks.

U-01 - Establishing the baseline and locating the no-recovery bottleneck

Motivation and hypothesis. The agent first needed to establish whether shipped OWL/Wanda was valid, whether its quality loss dominated the opportunity, and whether the public calibration-width setting actually took effect.

Concrete change and experimental setup. It ran the original unstructured OWL/Wanda path with seed 0 and 128 C4 sequences, evaluated the resulting model on the 122 fixed validation blocks, and inspected calibration-buffer construction.

Observed result. Sparsity was valid at 0.699909. Construction took 158.91 seconds and about 22.20 GB peak GPU memory. Validation perplexity was 57.8382, with block-NLL dispersion 0.28225. Source inspection confirmed the input buffer was fixed at 128.

Agent interpretation. It treated the model as “valid but very poor” and judged that a stronger reconstruction method plus post-pruning recovery was a larger opportunity than small OWL layer-budget tuning.

Report assessment and confounds. The diagnosis fits the evidence, but the baseline was measured only once at seed 0, so baseline variance is unknown.

Decision and consequence. The result became the proxy anchor, and the buffer-size defect was fixed so wider calibration could be tested for real.