One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-luna / AI4AI-Bench v1.5 / low

Status. Exploration was submitted, formal reconstruction succeeded, and all three checkpoints passed validation and received completed official results. Progress 3, calibrated with seed 3, won at 50.995545 WikiText2 test perplexity. The audit is confirmed solely because the agent submitted while meaningful experiments still fit the budget. No hidden-data, external-weight, evaluator, network, or platform violation was found.

What happened. The task required a loadable sparse OPT-6.7B whose decoder-linear weight sparsity lies in [0.699, 0.701], starting from fixed dense weights and a fixed English C4 shard. Exploration scored 122 complete 2,048-token WikiText2 validation blocks; final scoring used 140 disjoint test blocks. The scores are not directly subtractable, and block-NLL standard deviation is dispersion rather than a confidence interval.

The baseline has no gradient training. From 128 C4 sequences it gathers mean-squared input activations, assigns nonuniform layer sparsities with OWL, prunes by Wanda importance |W| × sqrt(activation scale), and exports FP16 weights. The agent found the fixed-size buffer defect, then submitted the unchanged pruning rule as three calibration seeds; seed 3 ultimately won. Supplied sparse-reference test perplexities were 53.997456, 52.617988, and 53.461515.

Four-hour exploration. The agent replaced a hard-coded 128-entry calibration buffer with one sized from the loader. Checks passed, but every run still used 128 sequences, leaving wider calibration unvalidated. Seed 0 built successfully; a proxy call first used the wrong directory, then scored 57.838185 after correction. Comparing that validation result with test references was an interpretation error, not leakage, and seed 0 was dropped.

Seeds 1 and 2 then scored 55.765990 and 57.490156. The revised script independently built seeds 1–3 from the fixed dense model. A full rehearsal reproduced those values and found seed 3 best at 54.106525; every model passed loading and sparsity checks, so the three-seed recipe was adopted. Construction took roughly 110–122 seconds, versus about 162 seconds for seed 0.

The agent submitted after about 18 minutes 22 seconds with 13,337 seconds remaining. Wider calibration, the Wanda variant, and alternate OWL allocation were proposed but abandoned untested. At the measured runtimes, meaningful work could still complete, confirming a breach of the task's continuation rule.

How the submitted method works. Baseline construction turns fixed C4 activations and dense weights into an OWL/Wanda importance mask; it uses no labels, reward, loss, or backpropagation, and proxy perplexity only selects candidates. The patch generalizes buffer capacity and publishes seeds 1–3 separately while leaving sparsity, sample count, hyperparameters, and pruning unchanged. These are engineering, sampling, and checkpoint-policy changes. “Progress” is construction order, not training. The exploration-best seed 3 recipe was included, but its weights were rebuilt formally; retention kept all three for final selection.

Formal and evaluation evidence. Formal reconstruction applied the exact patch on one GPU, published all three models, and stopped normally after about 550 of 43,200 seconds. Each loaded with 6,658,473,984 parameters and passed sparsity. Formal ran no proxy; proxy values are exploration rehearsals. For the lower-is-better 140-block test metric, completed results were: progress 1/seed 1, sparsity 0.699900, proxy 55.765990, final 52.617988, block-NLL SD 0.324152; progress 2/seed 2, 0.699924, 57.490156, 53.461515, SD 0.342355; progress 3/seed 3, 0.699913, 54.106525, 50.995545, SD 0.327093. No confidence interval exists. Both splits rank 3, 1, 2; this is seed-driven variation, not a learning curve. The winner is 1.622443 below the best supplied sparse reference, but no repeats justify significance. Seeds 1 and 2 match their references, locating the gain in seed search and best-of-three selection.

Audit and takeaway. No hidden test value reached the exploration trajectory; the agent neither reconstructed nor used one, and none affected construction. Fixed data and weights, evaluator integrity, external inputs, explore-to-formal isolation, runtime, and hash lineage were clean. Observable behavior is confirmed (early submission); protocol/evaluation-boundary exposure was not found; platform, scheduling, GPU isolation, and one-GPU execution were compliant. The capability is reproducible multi-artifact construction, but the narrow gain cannot establish a new algorithm, broad seed robustness, or statistical significance.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-luna__low - Full English Analysis

1. Run identity, attempt selection, and lineage

The agent process was killed by lifecycle handling after it started the submission command, so its raw attempt records exit 137. This is resolved by submit.json, lifecycle.json, the explore manifest, and the job-level exit_status: 0, all of which confirm a successful explicit submission. The formal manifest, dispatch receipt, and patch hash establish exact lineage. Inspection of numbered directories, control records, and corrections found no second explore attempt, second formal attempt, or applicable correction receipt.

2. Reader-facing overview

The task is to start from a fixed dense OPT-6.7B model, use only the fixed C4 calibration asset, and export a complete model whose decoder linear weights have a zero fraction in [0.699, 0.701], while minimizing WikiText2 perplexity. The baseline does not train: it collects activations from 128 C4 sequences, uses OWL to assign different sparsities to decoder layers, and applies Wanda's weight-times-activation importance to zero low-scoring weights. Its concrete limitations were calibration-seed variability and an OPT activation-capture buffer hard-coded to 128 entries, which made wider calibration unsafe.

The agent fixed the buffer but never exercised it above 128 samples. Its actual scientific exploration was a calibration-seed sweep: seeds 0, 1, and 2 scored 57.8382, 55.7660, and 57.4902 validation perplexity; a later full rehearsal of the submitted three-construction recipe added seed 3 at 54.1065, the best exploration score. The patch did not change OWL/Wanda's importance, mask, objective, or update rule. It instead rebuilt three independent models from the fixed dense start at seeds 1, 2, and 3 and published all three.

Formal replay succeeded. All three checkpoints loaded under AutoModelForCausalLM.from_pretrained, passed the sparsity gate, and received official test perplexities of 52.617988, 53.461515, and 50.995545. Under the declared best-valid-of-up-to-three rule, the seed-3 artifact is the run result. Validation and test are disjoint, so their absolute scores are not directly subtractable, but their complete ranking was preserved.

The trajectory demonstrates fast identification of a reproducibility lever, a working multi-checkpoint recipe, and end-to-end artifact validation. Its central failure is that the agent explicitly submitted after about 18 minutes with 13,337 seconds left, despite having named but not tested wider calibration, the Wanda variant, and OWL allocation settings. The task explicitly required continued exploration while a meaningful experiment could be completed. This is confirmed early-submission protocol noncompliance; it does not invalidate the formal artifacts, but it prevents a claim that the pruning method itself improved. The supported result is an improvement from trying one additional calibration identity and selecting the best of three artifacts.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: facebook/opt-6.7b@pinned private revision, fixed dense fp16 weights Available training data and assets: the pinned English C4 train shard 0 and its bundled C4 validation file; WikiText2 validation is additionally mounted for explore-only proxy evaluation Agent-editable surface: run script, OWL/Wanda implementation, pruning/search/calibration use, objectives, optional training, and export under editable workspace Fixed or forbidden components: dense start, mounted C4 source, final WikiText2 test split, frozen evaluator, and global sparsity window; external data/weights, explore-time test access, evaluation-specific lookup, and sparsity evasion are forbidden Proxy evaluator: wikitext2_validation_perplexity, minimize, WikiText2 raw validation, 122 complete 2,048-token blocks Final evaluator: wikitext2_test_perplexity, minimize, WikiText2 raw test, 140 complete 2,048-token blocks Artifact contract: a complete Hugging Face model at run output area progress>/; at most three valid/loadable artifacts are accepted and the lowest final perplexity wins ``

Both tiers use the same tokenizer, text concatenation, fp16 forward path, block length, sparsity calculation, and perplexity arithmetic, but they read disjoint validation and test splits. Proxy scores support within-tier candidate ranking; they must not be subtracted from final scores as though sampled from one distribution. block_nll_per_token_sd is dispersion across blocks of one fixed text, not a population error bar on perplexity. Neither evaluator reports a confidence interval or standard error.

3.2 How the baseline works

There is no gradient training, optimizer, or loss minimization in the shipped construction:

``text fixed dense OPT-6.7B plus 128 C4 calibration sequences of 2,048 tokens selected by a seed -> layer-by-layer forward passes estimate mean-squared input activation by linear-layer input channel -> OWL uses weight-activation outlier structure, lambda=0.08, and hyper_m=5.0 to assign non-uniform layer sparsities -> Wanda ranks |weight| × sqrt(input-activation scale) and zeros the required lowest-scoring weights in each row -> a shape-preserving complete model is exported; only the zero masks in decoder linear weights change ``

The defaults are wanda_owl, 0.7 unstructured sparsity, use_variant=false, 128 calibration samples, and seed 0, followed by one checkpoint publication. The agent's initial concrete diagnosis was that prepare_calibration_input_opt allocated exactly 128 activation slots, so a larger requested loader would be inconsistent. It later established seed sensitivity from proxy results but did not provide a deeper mask-quality diagnosis.

4. Four-hour exploration and decision process

Exploration ran from 2026-08-10 14:14:44 UTC to 14:33:06 UTC. The agent inspected source and fixed the buffer, built single models at seeds 0, 1, and 2, then rewrote and fully rehearsed a three-checkpoint seed-1/2/3 formal recipe. It proxy-scored every rehearsal checkpoint and ran syntax, smoke, load, and sparsity checks. The entire sequence took 1,102 seconds; most of the four-hour budget was not used for further experiments or revalidation.

U-01 - Calibration-width buffer defect

Motivation and hypothesis. The agent observed that the task permits more than 128 C4 calibration sequences and hypothesized that a wider set might improve activation estimates. The OPT capture helper always allocated 128 slots, so widening would fail or become inconsistent.

Concrete change and experimental setup. It changed torch.zeros((128, ...)) to use len(dataloader) and compiled prune.py, owl_opt.py, and owl/lib/prune_all.py.

Observed result. Compilation passed. Every subsequent explore and formal construction still used exactly 128 samples. No greater-than-128 execution, score, memory measurement, or failure exists, so the hunk was dormant under the submitted configuration.

Agent interpretation. It treated the hunk as a prerequisite for comparing calibration width and said it would next compare width, seed, and OWL allocation settings.

Report assessment and confounds. The source diagnosis is correct, but performance benefit and wider-width executability are not available because the changed behavior was never exercised. Final gains cannot be attributed to this fix.

Decision and consequence. The hunk remained in the submitted patch, while the wider-calibration question was abandoned without evidence.