One-shot model pruning

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-luna__high - gpt-5.6-luna / Codex CLI 0.146.0 / high

Status. Exploration, submission, formal replay, validation of three retained artifacts, three official final evaluations, and boundary audit are complete. A prior attempt stopped at GPU gating without trajectory, patch, or artifact.

What happened. The task was to construct a loadable 70%-unstructured-sparse OPT-6.7B from the fixed dense model and C4 shard. The baseline used 128 length-2,048 C4 samples to score weights by magnitude times activation scale, assigned layer sparsities with OWL, and zeroed a fixed count per row without optimization; its 122-block validation perplexity was 57.838185. The submitted method preserved OWL allocation but added mask-preserving next-token recovery, ultimately producing test perplexity 29.242228 versus the shipped sparse-baseline mean 53.358987.

Four-hour exploration. All proxy scores below are validation perplexity on the same 122 complete blocks, lower being better. Uniform Wanda (181.743562) and an alternative scoring variant (93.032322) were rejected, establishing that OWL allocation mattered. A lambda sweep moved the center from 0.08 to 0.12; varying the allocation multiplier then selected 10 at 42.227345, while 20 regressed. Lambda 0.11 improved by only 0.061 on seed 0 and was adopted without a matched multi-seed comparison, so that choice remains uncertain.

Increasing calibration to 256 first exposed a hard-coded 128-entry buffer. After the agent fixed it, 256 samples scored 43.064342 versus 42.227345 for matched 128-sample calibration and cost more time and memory, so it was rejected. Three seeds at lambda 0.12 and multiplier 10 averaged 43.042539, not the agent’s contemporaneous “about 43.38”; they support basic robustness but do not validate lambda 0.11.

Post-pruning recovery was decisive. At 1e-6, increasing from 4 to 1,024 steps improved perplexity from 41.776828 to 32.657416; two additional seeds scored 32.803609 and 32.797833, so 1,024 steps was adopted. Raising the rate to 2e-6 improved 512 steps from 34.026909 to 32.142361 and produced 30.814659/30.899132 at 1,024 steps on seeds 0/1; seed 2, neighboring rates, and longer recovery remained open. A probe’s missing one-shot flag started a second loop, interrupted after its first valid artifact; a publication-race evaluation passed on rerun. Neither affected selection. Submission nevertheless occurred with 7,576 seconds remaining after the new best, violating the explicit early-submission condition.

How the submitted method works. Baseline flow is fixed C4 activations to Wanda scores, OWL layer allocation, then per-row zeroing. The candidate prunes with lambda 0.11 and multiplier 10, then uses 128 fixed-C4 sequences as causal-language-model labels; standard shifting supplies next-token targets. SGD optimizes decoder-linear tensors under cross-entropy; per-step masking keeps zero entries fixed and all other tensors are frozen. It cycles the samples eight times for 1,024 steps at 2e-6. Changes span algorithm and training signal, sampling, hyperparameters, schedule, engineering, and checkpoint policy. Formal replay received only the matching patch, not exploration weights, and rebuilt from the fixed dense start while cycling seeds 0/1/2.

Formal and evaluation evidence. Orchestration overrode the unlimited local fallback with a 42,600-second limit and 600-second reserve, making the effective candidate guard 42,000 seconds. Replay ran 42,041.611 of 43,200 seconds; deadline salvage treated runner status 124 as success after 130 complete publications and at least 133,120 completed recovery updates. Unpublished progress 131 was excluded, and only three artifacts were retained simultaneously. All loaded with 6,658,473,984 parameters. Official test perplexity, lower better on 140 blocks, was 29.680772 at checkpoint 128/seed 1 (sparsity 0.699921; block-NLL SD 0.267536), 29.242228 at 129/seed 2 (0.699919; 0.282922, best), and 29.348080 at 130/seed 0 (0.699907; 0.280649). No standard error or confidence interval is available. These are independent reconstructions, not a learning curve. Formal weights lacked proxy evaluation, so proxy and test values cannot be subtracted; the conclusion survived because all three beat every shipped sparse baseline under the final protocol.

Audit and takeaway. Fixed C4, dense start, evaluator, sparsity gate, patch-only lineage, single-GPU isolation, and wall-clock recovery were respected. The task exposed reference scores but not hidden test text or candidate final values; no reconstruction or use of hidden assets, external training input, or explore-weight transfer was found. Observable behavior is confirmed only for the explicitly evidenced premature submission; protocol/evaluation-boundary exposure is not found, and platform, scheduling, and resource isolation are compliant. The case demonstrates a useful mechanism discovery and reproducible formalization, but early search termination and repetition of only three recipes prevent claims that lambda 0.11, 1,024 steps, or 2e-6 is optimal.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-luna__high - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: owl_wanda_opt6p7b_70pct Task family: unstructured large-language-model pruning with post-pruning recovery Model: gpt-5.6-luna Harness: Codex CLI 0.146.0 under AI4AI-Bench v1.5 Reasoning effort: high Seed: no single global seed; calibration seeds 0, 1, and 2 were used Exploration budget: 14,400 seconds; the primary run used 6,863 seconds Formal budget: 43,200 seconds; the retraining phase used 42,041.611 seconds, or 97.32%

Primary explore run: exploration attempt 2 Why this attempt was selected: it was the first attempt that actually started the agent and produced a trajectory, experimental receipts, and a candidate patch. The authoritative control status records exit_status 0, terminal_behavior, and agent_explicit_submit. Prior full-run attempts and outcomes: there was no earlier full run. A numbered exploration attempt 1 directory exists even though the analysis job manifest lists no prior attempts. It contains only three GPU idle-gate samples. GPU 7 became reserved by another container during the continuous check, and the authoritative status is terminal_infrastructure with reason “GPU failed the continuous idle/ownership gate.” It has no run manifest, agent trajectory, patch, or artifact. Patch hash / validation: SHA-256 verified private digest. The explore patch and formal retrain/candidate.patch are byte-identical. The formal log shows clean application of all four modified files. Formal run: formal replay 1 Formal the formal replay is cryptographically linked to the selected exploration Final-evaluator receipts: artifact-128, artifact-129, and artifact-130 each have out/summary.json and the corresponding .complete; all three are official results.

Evidence completeness: the task contract and baseline source, raw trajectory, 27 proxy receipts, patch, formal manifest and training log, all three checkpoint validations, and all three final evaluations are available. Missing or conflicting evidence: the analysis job manifest omits the gating-only exploration attempt 1. The explore manifest says agent_state=failed because the agent process received exit 137 while the final submit command was being captured, but submit.json, lifecycle.json, and control status jointly establish a successful explicit submission with a nonempty patch. The formal manifest is formal_only and therefore has null final fields; later, separate final-test directories contain paired summaries and completion receipts. The root formal retrain_resolved_config.json was overwritten with seed 1 when incomplete progress 131 began, whereas retrain_metrics.json still describes the last complete progress 130 at seed 0; retained-checkpoint seeds must therefore be read from the formal log. No formal checkpoint has a proxy score, standard error, or confidence interval.

2. Reader-facing overview

The task asks for a loadable OPT-6.7B whose decoder linear weights are between 69.9% and 70.1% zero while minimizing WikiText2 perplexity. Perplexity measures average predictive uncertainty and is better when lower. The shipped baseline runs one Wanda activation-aware pruning pass, with OWL assigning different sparsity levels to different layers, and performs no gradient training. Its exploration validation perplexity was 57.8382.

The agent first established that OWL’s nonuniform allocation was essential, then swept its allocation strength, calibration width, and calibration seed. Its decisive change was mask-preserving sparse recovery: after pruning, it performed 1,024 causal-language-model updates on the fixed C4 shard, optimized decoder-linear weight tensors, and reapplied the binary mask after every update so originally zero entries stayed zero. The submitted defaults were lambda 0.11, Hyper_m 10, 128 calibration sequences, SGD at 2e-6, and a formal loop cycling seeds 0, 1, and 2.

Formal replay rebuilt every model from the fixed dense start. It completed and cumulatively published 130 constructions; the wall clock cut off unpublished progress 131, and retention left only the latest three. All three loaded and passed the sparsity gate. Their official WikiText2 test perplexities were 29.680772, 29.242228, and 29.348080, with progress 129 best. That best result is 24.116759 lower than the shipped sparse baseline mean of 53.358987 under the same test protocol. The fixed dense reference scores 10.860456 but is ineligible because it has zero sparsity.

The trajectory demonstrates strong control design, parameter localization, debugging, and a useful shift from pruning-only tuning to recovery training. Its main scientific weakness is that lambda 0.11 beat the better-replicated lambda 0.12 by only 0.061 on one seed, while the formal loop repeatedly ran the same three seed configurations. Its main execution failure is clearer: it submitted with 7,576 seconds left immediately after a new learning-rate best, even though a full construction and evaluation took about six minutes. That breaches the task’s explicit early-submission condition, but it does not contaminate the formal artifacts or final scores.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

Starting artifact / model: fixed facebook/opt-6.7b dense weights at the pinned a45aa65b… revision. The dense score is a quality reference but the artifact cannot pass the 70% sparsity gate. Available training data and assets: the fixed English C4 shard and its accompanying C4 validation file; WikiText2 validation is additionally mounted only for exploration proxy evaluation. Agent-editable surface: everything under editable workspace, including pruning, search, sampling, objectives, post-pruning training, allocation, and checkpoint policy. Fixed or forbidden components: the dense start, mounted data sources, final evaluator, test text, sparsity window, single-GPU and wall-clock limits. External data or weights, network-derived experiment inputs, exploration access to test text, and evaluation-specific lookup behavior are forbidden. Proxy evaluator: wikitext2_validation_perplexity, minimize, WikiText2 validation, 122 complete 2,048-token blocks. Receipts report the standard deviation of per-token block negative log likelihood, which is fixed-text dispersion rather than a score standard error. Final evaluator: wikitext2_test_perplexity, minimize, a disjoint WikiText2 test split, 140 complete 2,048-token blocks. It likewise reports block dispersion but no confidence interval. Artifact contract: a complete Hugging Face causal language model loadable by AutoModelForCausalLM.from_pretrained, with measured decoder-linear sparsity in [0.699, 0.701]. At most three validated numeric-progress checkpoints are accepted, and the official run result is the best valid final score.

Proxy and final use the same tokenizer, block length, forward computation, and perplexity aggregation but disjoint text. Proxy values support comparisons within exploration; they cannot be subtracted from final values. The shipped sparse reference scores use the same test protocol as the new formal artifacts and are directly comparable.

3.2 How the baseline works

The baseline samples 128 length-2,048 sequences from C4 at seed 0. During a layerwise forward pass, Wanda scores each linear weight as its absolute magnitude times the square root of that input channel’s mean-squared activation. Lower-scoring weights are pruned first. OWL uses layer outlier ratios plus lambda 0.08 and Hyper_m 5 to move individual-layer sparsity around the 70% global target, after which a fixed number of the lowest-scoring weights in each row are set to zero. The complete model is exported as one progress-1 checkpoint.

There are no labels, losses, optimizer, or gradient updates in the shipped method; the only weight change is zeroing selected entries. The measured explore replay took 155.1 seconds, peaked near 22.2 GB of GPU memory, loaded successfully, and scored 57.838185 on validation. The agent initially identified the unmeasured OWL allocation strength, mask variant, and calibration sampling as concrete bottlenecks, and noted that one pruning pass did not make meaningful use of the formal wall clock.

4. Four-hour exploration and decision process

The agent spent the first few minutes inspecting the source and evaluator and establishing a loadable baseline. Roughly the next forty minutes covered mask-family controls, lambda, Hyper_m, calibration width, and seed replication. About an hour then went to implementing and extending sparse recovery, resolving a checkpoint race, validating seeds, and building the formal loop. The final minutes tested learning rate and source/artifact validity before submission. The primary run used about 1 hour 54 minutes, not the full four-hour allowance.

U-01 - Should OWL’s nonuniform allocation be retained?

Motivation and hypothesis. The agent needed to separate limitations in the Wanda weight score from limitations in OWL’s cross-layer allocation, and to test whether upstream’s cumulative-metric threshold variant improved on fixed per-row pruning counts.

Concrete change and experimental setup. All candidates used the fixed model, 128 C4 sequences, seed 0, and the 122-block validation evaluator. The shipped control used wanda_owl. A uniform control changed PRUNE_METHOD to wanda, giving every layer approximately 70% sparsity. A third run kept default OWL allocation and enabled USE_VARIANT.

Observed result. The baseline scored 57.838185 with block-NLL standard deviation 0.282249. Uniform Wanda scored 181.743562, and the threshold variant scored 93.032322. All three measured within the permitted global sparsity window; the baseline also passed a direct load check.

Agent interpretation. The agent explicitly called uniform Wanda a useful negative control and concluded that its 181.74 versus OWL’s 57.84 showed that nonuniform allocation was doing important work. It did not separately narrate the 93.03 variant result.

Report assessment and confounds. The uniform regression is far larger than observed calibration-seed variation and strongly supports retaining OWL. The variant was also substantially worse on its single seed, so rejecting it was a reasonable directional decision.

Decision and consequence. The submission retained OWL with fixed-count row pruning and rejected both uniform allocation and USE_VARIANT. Later search focused on lambda and Hyper_m.