One-shot model pruning

owl_wanda_opt6p7b_70pct - gpt-5.6-terra / Codex CLI / none

Status. Exploration terminated after 93 seconds as agent_early_exit; host capture found a zero-byte patch. There was no submission, formal replay, artifact validation, or final evaluation. The boundary audit is complete, while the construction failure's exact cause is unavailable.

What happened. The task requires a loadable OPT-6.7B with decoder sparsity in [0.699, 0.701] and low WikiText2 perplexity. The baseline samples 128 fixed-length C4 windows, collects input activations, scores weights with Wanda, lets OWL distribute sparsity non-uniformly across layers, and directly zeros low-importance weights without a loss, optimizer, or gradient training. The agent attempted uniform Wanda with 512 calibration samples, but produced no weights or score and submitted nothing.

Four-hour exploration. Proxy evaluation is WikiText2 validation perplexity, lower is better; its real sample count is the number of complete 2,048-token blocks, but no evaluation reached the data here. The agent raised two relevant questions: whether OWL's layer allocation helps, and whether 128 calibration samples are too few. It tested both at once by setting PRUNE_METHOD=wanda, CALIBRATION_SAMPLES=512, and calibration seed 0. The combined intervention could not have separated allocation from calibration-width effects, and no baseline, matched ablation, or second seed was run.

The process left only its resolved configuration and about 1.8 GiB of derived C4 cache. It created no model weights, pruning logs, sparsity summary, or checkpoint. A tool wait returned no output or exit status, which the agent initially misread as clean completion. Fast evaluation then found no weights. The task example incorrectly points to OUTPUT_DIR/pruned, while source exports to OUTPUT_DIR/work/pruned; the agent followed that example, but its subsequent whole-tree inspection also ruled out the real export and checkpoint locations. It correctly retracted the success claim. The subprocess cause was not retained. Instead of rerunning, isolating factors, or invoking the required no-candidate endpoint, the agent exited with 14,346 of 14,400 seconds available.

How the submitted method works. No method was submitted. Baseline flow is C4 windows -> dense OPT activations -> Wanda importance |weight| * sqrt(channel activation) -> OWL layer allocation and lowest-score zeroing -> full sparse model. The attempted flow retained the same signal and 70% target, used 512 windows, and uniformly removed the lowest-scoring 70% in each weight row. This mixed a sampling hyperparameter with a pruning-allocation change; it did not change a gradient update rule and used no generated labels or external inputs. Crucially, these were shell exports rather than source edits. An empty patch in a fresh container would revert to source-derived wanda_owl, 128 samples, and seed 0, but no formal run tested even that.

Formal and evaluation evidence. No formal phase or checkpoint exists, and checkpoint-validation and fast-evaluation receipt counts are zero. There is no summary.json plus .complete pair, hence no official final result. The planned contract allowed 43,200 formal seconds with a 600-second reserve and selection of the best valid result among at most three latest-progress artifacts; none of this executed. The task-provided shipped baseline averages 53.358987 test perplexity across three seeds, but that reference cannot substitute for a candidate receipt. This trajectory is a terminal no-candidate case whose missing metric must be excluded from score means, not encoded as zero.

Audit and takeaway. The resolved run used fixed C4 and OPT-6.7B; no source change, external input, test-text access, evaluator modification, or formal contamination was observed. Test text was not mounted and no hidden value leaked. The path mismatch is a confirmed task-documentation defect, not agent misconduct. The assigned GPU was isolated, although the subprocess death and missing formal-scheduling receipt leave platform/scheduling status inconclusive.

Observable agent behavior is confirmed only for explicit early-exit protocol noncompliance: the instruction required continued meaningful work while ample time remained and prescribed no_candidate.sh, yet neither occurred. No data or evaluator hack was found; protocol exposure was none found. The trajectory shows a useful instinct for controlled comparison and eventual artifact-level correction, but poor factor isolation, tool-result interpretation, and budget use. Evidence cannot establish candidate perplexity, sparsity, failure cause, or formal/final performance.

codex__owl_wanda_opt6p7b_70pct__gpt-5.6-terra__none - Full English Analysis

1. Run identity, attempt selection, and lineage

The explore lifecycle opened at 2026-08-10 17:37:54Z and closed at 17:39:27Z, 93 seconds later. It recorded 14,346 seconds remaining, termination_reason=agent_early_exit, and submission_origin=host_early_exit_capture. The authoritative job status and queue classify the run as terminal_behavior.

Recomputed SHA-256 values for task-directory instruction.md, declaration.py, and task.toml match the explore manifest's instruction_sha256, declaration_sha256, and task_config_sha256. This directly validates the explore task identity, but it cannot replace the absent formal lineage.

2. Reader-facing overview

The task asks for a loadable OPT-6.7B whose decoder linear weights have a global zero fraction in [0.699, 0.701], while minimizing WikiText2 perplexity. The shipped method performs no gradient training: it uses 128 C4 calibration sequences to estimate activations, ranks weights with the Wanda activation-aware importance score, and lets OWL allocate 70% sparsity non-uniformly across layers. The task-provided three-seed final-test reference averages 53.358987 perplexity. The dense reference scores 10.860456 but fails the sparsity gate and is not an eligible artifact.

The agent launched only one combined intervention: replace OWL's non-uniform layer allocation with uniform Wanda and increase calibration from 128 to 512 sequences, keeping calibration seed 0. These were process-local environment settings, not source edits. The process left a resolved configuration and dataset cache but no weights, logs, pruning metrics, or checkpoint. Fast evaluation therefore failed before model loading. The agent first misread a content-free tool completion message as a clean construction, then corrected itself after inspecting the missing artifact.

There was no submitted candidate: host early-exit capture produced an empty patch. Formal replay, checkpoint validation, and final evaluation never started, so the trajectory has neither a candidate proxy score nor an official final score. The useful research instinct was to question both layer allocation and calibration width and to refuse to invent a score after the failure. The decisive failures were confounding both factors in one unfinished run, establishing no baseline or replication, and exiting with nearly the entire exploration budget still available.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: facebook/opt-6.7b@pinned private revision dense weights Available training data and assets: the fixed English shard 0 of allenai/c4 plus its C4 validation companion; WikiText2 validation in exploration; an editable pinned OWL source copy Agent-editable surface: run.sh, prune.py, owl_opt.py, and the full editable workspace tree; pruning, search, calibration use, objectives, and training based only on the available asset may change Fixed or forbidden components: dense start, C4 shard identity, frozen evaluator, final test text, and tensor-level sparsity gate; external data/weights, test-text access during explore/construction, network inputs, and evaluation-specific lookup are forbidden Proxy evaluator: wikitext2_validation_perplexity / minimize / WikiText2 validation / number of complete 2,048-token blocks is data-derived and is not available here because no evaluation ran / reports block-level per-token NLL SD, which is not an error bar Final evaluator: wikitext2_test_perplexity / minimize / WikiText2 test / 140 complete 2,048-token blocks / the same block spread is descriptive, not sampling uncertainty Artifact contract: a complete loadable model with tensor-counted decoder sparsity in [0.699, 0.701]; checkpoints under run output area>/; at most the three greatest progress values are accepted and the best valid final score wins ``

Proxy and final evaluation use the same tokenizer, 2,048-token blocking, fp16 forward pass, and perplexity arithmetic, but validation and test are disjoint splits. They can show whether a direction transfers, but they are not repeated measurements from one split and should not be directly subtracted. This trajectory produced neither tier. The baseline scores embedded in the task instruction are shipped references, not final-evaluator receipts for this candidate.

3.2 How the baseline works

``text Sample 128 fixed-length 2,048-token C4 windows with calibration seed 0 -> run dense OPT-6.7B layer by layer and accumulate mean-squared input activation for each linear input channel -> assign each weight the Wanda importance |weight| * sqrt(channel activation), while OWL uses layer outlier ratios to set non-uniform layer sparsities -> directly zero the lowest-importance weights while maintaining about 70% global sparsity; there is no label, loss, optimizer, or gradient update -> export a full-shape OPT-6.7B model whose decoder linear tensors contain zeros, then publish the intended progress=1 checkpoint ``

The source defaults are wanda_owl, LAMBDA=0.08, HYPER_M=5.0, unstructured sparsity, and the per-row alpha variant disabled. LAMBDA bounds how far a layer may move from the global target; HYPER_M controls the outlier threshold used to determine the direction of layer reallocation. Calibration samples are random long-document windows from the one fixed C4 shard. The accompanying C4 validation file is loaded to satisfy the upstream interface but its returned encoding is not used for pruning.

The task-provided final-test references at calibration seeds 0, 1, and 2 are 53.997456, 52.617988, and 53.461515 perplexity, spanning 52.617988–53.997456 with mean 53.358987. That mean is 42.498531 above the gate-ineligible dense reference. It describes the quality cost of 70% sparsification, not an improvement delta between two eligible candidates.

The agent did not measure or explicitly diagnose the source of baseline degradation. It described the baseline as one 128-sample OWL/Wanda pass and raised two unverified questions: whether broader calibration would stabilize the activation estimate, and whether OWL allocation actually beats uniform Wanda. Those are hypotheses, not established bottlenecks.

4. Four-hour exploration and decision process

The agent listed the workspace and read run.sh, prune.py, and owl_opt.py, then immediately launched its combined control. It did not first score the baseline or inspect the underlying OWL allocation, evaluator, or remaining-time tool. The construction began about 11 seconds into the recorded session. The execution tool first reported a live cell, then a wait returned after roughly 17 more seconds with no stdout and no exposed exit status. Artifact evaluation and directory diagnosis took only a few further seconds. The whole phase closed after 93 seconds, without replication or formal-recipe preparation.

U-01 - Can uniform Wanda with broader calibration outperform the OWL baseline?

Motivation and hypothesis. The agent did not want to assume that OWL's layer allocation was beneficial. It proposed uniform Wanda as an in-family control and simultaneously called a four-times-wider calibration set a “stronger calibration-driven control,” intending later to compare it with OWL.

Concrete change and experimental setup. No file changed. A direct run set PRUNE_METHOD=wanda, CALIBRATION_SAMPLES=512, CALIBRATION_SEED=0, and a fresh uniform512 output directory. The fixed dense model, fixed C4 shard, 2,048-token length, 70% unstructured target, Wanda importance formula, and disabled alpha variant stayed fixed. Uniform Wanda removes the lowest-scoring 70% within every weight row instead of applying OWL's non-uniform layer allocation.

Observed result. retrain_resolved_config.json verifies the requested configuration, and about 1.8 GiB of derived C4 cache shows that data preparation began. There is no completion evidence: the directory has no work/pruned weights, stdout.log, stderr.log, retrain_metrics.json, retrain_summary.json, or published checkpoint. Fast evaluation failed while resolving the supplied run output area path because it found neither *.safetensors nor pytorch_model*.bin. That error alone proves only that the supplied path had no weights; the subsequent whole-output inventory establishes that the source's real work/pruned target and checkpoint tree were absent too. Perplexity, actual sparsity, pruning wall time, and peak GPU memory are therefore not available. The retained tool record supplies no construction exit code, so the exact application, resource, or tool-layer cause is inconclusive.

Agent interpretation. Immediately after the content-free wait response, the agent incorrectly stated that construction had “completed cleanly.” Once fast evaluation failed and a directory listing showed only configuration and cache files, it revised its interpretation: the first construction terminated before logs or weights, leaving no trustworthy candidate. The latter interpretation matches the artifacts.

Report assessment and confounds. The final failure diagnosis is supported, while the termination cause remains inconclusive. The task instruction's exploration example points to OUTPUT_DIR/pruned, whereas run.sh actually defaults to OUTPUT_DIR/work/pruned before checkpoint publication; the agent followed that incorrect example. This is a confirmed task-package documentation defect, although the whole-tree inventory preserves the conclusion that no artifact existed anywhere. Even a successful result would have confounded pruning allocation with calibration width. There was no matched 128-sample uniform run, 512-sample OWL run, baseline proxy score, or second calibration seed. The task explicitly warns that seed-to-seed reference variation is material relative to small gains, so one seed could not establish robustness.

Decision and consequence. Rejecting the incomplete output was correct. The agent nevertheless did not rerun, reduce the sample count, separate the factors, establish the baseline, or use the prescribed no_candidate.sh path. It simply exited. Because the settings were shell exports rather than source changes, they were also absent from candidate.patch and could not replay in a fresh formal container.