One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-sol / Codex CLI / max

Status. Exploration completed and submitted; the final submit command was caught at the deadline with seven seconds left, but its receipt and .explore.complete exist. Formal replay, loadability validation, final evaluation, and boundary audit all completed. The patch combines OPT-compatible SparseGPT, 1,024 C4 calibration sequences, and fixed-mask pure teacher distillation; best official test perplexity is 17.404544 at 70% sparsity.

What happened. The task fixes a dense OPT-6.7B start, C4 calibration data, and an exact 70% unstructured sparsity gate. The shipped OWL/Wanda baseline scores activation-aware weights and zeros them once without training; it reaches validation perplexity 57.838185 over 122 blocks, with layer sparsity 0.5781-0.7381. The agent identified the ordinary Wanda mask after OWL allocation and an OPT-unwired SparseGPT path as the main bottlenecks. The submitted recipe reconstructs a uniform SparseGPT mask, then trains surviving parameters against a frozen dense teacher on C4. Three formal artifacts passed the gate and final evaluator.

Four-hour exploration. All proxy values below are validation perplexity, lower is better, on the same 122 blocks. First, repairing the OPT SparseGPT path changed 128-sample seed-0 scoring from 57.8382 to 21.3580 and was adopted. Calibration width 128/256/512/1,024 produced 21.3580/21.0454/20.7298/20.6880, so 1,024 was selected; a matched 256-sample seed-1 run scored 21.5804 and established material seed sensitivity. Sequential recalibration (21.9143), 0.5x OWL layer scaling (21.5348), and damping 0.03 (21.4492) all lost to uniform damping 0.01. The first OWL-scaling attempt failed with NameError: layer_sparsity before export; the corrected retry was used for the comparison. Fixed-mask hard-label recovery improved 20.6014 after 20 updates to 18.4100 at 2,000. Dense-teacher distillation won matched 500-update tests (18.6969 at weight 0.5; 18.6539 at weight 1.0) and reached 18.4177/18.2995 at 1,000/2,000, so pure temperature-1 distillation was adopted. With the 1,024-sample mask and cosine decay, 2,500/5,000 updates scored 18.1501/18.1490. A floor-rate tail to 7,501 slightly regressed to 18.1493, and temperature 2 scored 18.3888; both were rejected. No final-test proxy, multi-seed trained repeat, or standard error is available.

How the submitted method works. Formal replay starts from the mounted dense model, collects 1,024 C4 sequences, and applies uniform SparseGPT with percdamp=0.01 and block size 128. It saves progress 1, derives an immutable mask, reloads the fixed dense teacher, and minimizes temperature-1 pure distillation cross-entropy on random 2,048-token C4 sequences. AdamW updates surviving weights, embeddings, biases, and normalization parameters; gradient hooks and projection keep pruned coordinates zero. The change is mixed algorithm/objective/data/schedule/engineering/checkpoint work, not mere tuning. The framework-injected wall clock overrides the source fallback duration; the formal run is time-driven with training seed 2026, learning rate 2e-6 decaying to 2e-7, and retention three. Exploration weights are not reused.

Formal and evaluation evidence. Retraining ran 41,943.788 of 43,200 seconds, exited 0, and actually published progress 1, 5,001, and 74,264; the planned midpoint has no direct receipt. All three artifacts loaded with 6,658,473,984 parameters and passed the sparsity window. Official final results are: artifact-1, 19.717117; artifact-5001, 17.404544 (best); artifact-74264, 17.404787. Each uses WikiText2 test, 140 blocks, minimize direction. Block SDs are 0.269180, 0.258893, and 0.258802 respectively, not standard errors. Formal proxy scores are not available, and validation/test are disjoint, so proxy and final numbers are not directly subtractable. The tiny late regression is consistent with the exploration tail and does not invalidate the accepted artifacts.

Audit and takeaway. Exploration mounted dense weights, C4, and validation only; formal retrain had no WikiText2 mount; test appeared only in external scoring, and numbered directories plus queue/control receipts show one exploration and one formal attempt with no gate-only or correction attempt. The top-level status=running fields are individually inconclusive stale conflicts, while terminal status and exit 0 agree; literal searches of the complete trajectory, commands, and patch found no test text or evaluator-only input and no evidence shows hidden assets reconstructed, queried, inserted into the candidate, or used to affect results; dispatch confirms source/hash lineage, no external data or network, and no GPU sharing. Observable agent behavior is clean; protocol/evaluation-boundary exposure is none found; platform, scheduling, and resource isolation are compliant; the evidence demonstrates effective diagnosis and a reproducible long-running sparse-recovery recipe, but cannot establish cross-seed generalization, statistical significance, or separate causal contributions of SparseGPT, calibration width, and distillation.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-sol__max - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: ai4ai/owl-wanda-opt6p7b-70pct (owl_wanda_opt6p7b_70pct).

Task category: improve a fixed OPT-6.7B model artifact at exact 70% unstructured decoder sparsity, measured by WikiText2 perplexity.

Evaluated coding-agent model: gpt-5.6-sol. Harness: Codex CLI 0.146.0 under the AI4AI orchestration harness. Reasoning effort: max. There is no single global random seed: the main pruning experiments use calibration seed 0, recovery uses training seed 2026, and individual probes record their own seeds.

Exploration budget: 14,400 seconds. The selected run is the primary exploration directory. Its .explore.started and .explore.complete receipts span about 14,433 seconds. It is selected because the job manifest names it as primary_explore_dir, and it contains the complete raw trajectory, submission receipt, and completion receipt. A direct search of the matching run tree found no earlier numbered attempt, partial formal directory, or correction/control artifact for this exact model/effort configuration. Other queue rows for this task use different models or reasoning efforts and are separate trajectories, not retries of this run.

The submitted candidate.patch is 28,258 bytes with SHA-256 verified private digest. The formal dispatch receipt confirms the source run, copied patch, byte count, and hash; the fresh formal container applied all six files cleanly. The formal run is formal replay 1, whose selected-exploration lineage exactly identifies the selected exploration run.

The formal budget is 43,200 seconds. The retrain receipt reports 41,943.788 seconds, 0.970921 utilization, and runner exit status 0. All three published checkpoints have separate valid loadability receipts. Final scoring is not a missing formal score phase: the manifest lists three outer final-tests directories, and every one has both summary.json and its matching .complete, so all three are official final results.

The evidence is complete with two explicit status distinctions. The formal manifest.json carries the exploration lifecycle's agent_state=failed and termination_reason=agent_explicit_submit, while retrain-result.json records formal runner exit 0; the former describes the submit-time exploration lifecycle and does not invalidate the formal phase. The matching control-attempt status.json also retains a top-level status=running, but its attempt_status=terminal_behavior, end time, and exit status 0 agree with the queue/job receipts and retrain receipt; that individual field is inconclusive as a stale conflict, not evidence that the phase was unfinished. Also, the task configuration says best_valid_of_up_to_3, while the formal publication receipt says latest_frozen_loadable_by_numeric_progress. All three artifacts were accepted and scored, so artifact-5001 is selected by the task's best-valid-final-score rule; the two observed fields should not be silently treated as one selector.

2. Reader-facing overview

The task asks the agent to build a 70%-sparse, unstructured OPT-6.7B artifact from a fixed dense model and a fixed C4 calibration shard. The shipped baseline performs one activation-aware Wanda/OWL pruning pass: it scores weights, allocates layer sparsities, zeros low-scoring entries, and never trains surviving parameters. Its exploration validation perplexity was 57.8382, with layer sparsity ranging from 0.5781 to 0.7381, making mask construction and allocation plausible bottlenecks.

The agent first repaired the bundled second-order SparseGPT path for OPT, then tested calibration width, sequential recalibration, OWL-derived allocation, damping, calibration seeds, fixed-mask recovery objectives, learning-rate decay, a low-rate tail, and distillation temperature. The submitted patch uses uniform SparseGPT with 1,024 C4 calibration sequences, percdamp=0.01, block size 128, followed by fixed-mask recovery using pure dense-teacher distillation. Its best exploration validation perplexity was 18.148966 on 122 validation blocks.

Formal replay rebuilt the mask from the fixed dense start in a fresh container, then ran time-driven recovery and published progress 1, 5001, and 74264. All three artifacts loaded and passed the sparsity gate. Official WikiText2 test perplexities on 140 blocks were 19.717117, 17.404544, and 17.404787; artifact-5001 is best.

The strongest conclusion is that the second-order reconstruction plus teacher recovery direction survived the independent final split with a clean, traceable lineage. The main limits are the absence of final-method multi-seed repeats or standard errors, a very small late-tail regression, and the fact that the formal log does not directly show the planned midpoint publication. These limit statistical and causal claims but do not invalidate the three accepted artifacts.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

• Starting model: facebook/opt-6.7b@pinned private revision, mounted formally as fixed-asset mount.
• Allowed data and assets: the pinned English C4 shard allenai/c4@pinned private revision; WikiText2 validation is available for exploration. WikiText2 test is mounted only in the external scoring phase.
• Editable surface: pruning, calibration sampling, search, objectives, training, export, and checkpoint policy in the workspace. Training is allowed only on the available C4 asset.
• Fixed or forbidden behavior: decoder sparsity must be in [0.699, 0.701] and the artifact must be loadable; one GPU, no network, a four-hour exploration budget, and a twelve-hour formal budget; no test text, external data, external weights, or evaluation-specific lookup during exploration or construction.
• Proxy evaluator: wikitext2_validation_perplexity, minimize; concatenated WikiText2 validation is evaluated in 2,048-token blocks, n=122. block_nll_per_token_sd is block dispersion, not a standard error; no confidence interval is emitted.
• Final evaluator: wikitext2_test_perplexity, minimize; the disjoint test split has 140 blocks of 2,048 tokens. The reward is the negative raw perplexity, after the sparsity and loadability gate.
• Artifact contract: numeric checkpoint-<progress> directories, at most three accepted valid artifacts, and independent gate and score checks. The dense reference score 10.860456 is ineligible because its sparsity is zero; the shipped sparse reference mean is 53.358987 on the same final protocol.

Proxy and final measurements use different, disjoint splits and are not directly comparable as numeric deltas. Proxy results select directions; the external final evaluator determines the task result.

3.2 How the baseline works

One baseline construction is:

Dense OPT-6.7B plus 128 C4 calibration sequences -> collect activation scales for each decoder layer -> score weights with abs(W) * sqrt(scaler) and use OWL (lambda=0.08, hyper_m=5.0) to allocate non-uniform layer sparsities -> zero the lowest-scoring entries row by row and export a sparse model -> check global sparsity and publish one construction checkpoint.

There is no reward model, label-training step, optimizer, learning rate, or trainable parameter. The agent explicitly identified the two-part bottleneck: OWL allocates layers in a separate pass and then applies ordinary row-wise Wanda masking, while the bundled SparseGPT second-order path was not wired to the OPT decoder layout.

4. Four-hour exploration and decision process

The agent used the four hours for baseline measurement, implementation repair, matched ablations, recovery curves, source and artifact audits, and final receipt checks. Every proxy score below uses the same 122-block validation evaluator and lower-is-better direction; repeated runs for one hypothesis are grouped.

U-01 - Establishing the baseline and repairing OPT second-order reconstruction

Motivation and hypothesis. Activation-only scoring may poorly compensate the error introduced by zeroing weights. A Hessian-aware reconstruction should improve both mask selection and surviving-weight compensation.

Concrete change and experimental setup. The agent connected bundled SparseGPT to the OPT decoder, using uniform layer sparsity, 128 C4 sequences, seed 0, percdamp=0.01, and block size 128.

Observed result. Baseline validation perplexity was 57.838185 at sparsity 0.699909; the repaired SparseGPT artifact scored 21.358021 at sparsity 0.700000641, with 122 validation blocks and a valid gate.

Agent interpretation. The agent called this a major step and emphasized that it changed both mask selection and compensation, not merely calibration randomness. It preserved the artifact as a fallback and continued exploring.

Report assessment and confounds. The protocol and seed were matched, but SparseGPT changes two internal mechanisms at once. The block standard deviation is not an error bar.

Decision and consequence. SparseGPT became the main line for all subsequent width, allocation, damping, and recovery experiments.