Knowledge unlearning

openunlearning_tofu_npo_llama3p2_1b - gpt-5.6-sol / Codex CLI 0.146.0 / medium

Status. Exploration, explicit submission, formal retraining, three artifact validations, three final evaluations, and the audit completed. Every official summary has its matching .complete receipt.

What happened. The task is to remove 400 TOFU forget10 answers from a fixed Llama-3.2-1B-Instruct while preserving unrelated ability. Baseline Negative Preference Optimization (NPO) lowers forget-answer likelihood against a frozen start and applies cross-entropy to paired retain90 answers; the agent diagnosed a short schedule and sparse coverage of 3,600 retain rows. It retained this update rule but changed sampling, retain weight, schedule, and checkpoints; step 896 achieved official balanced_unlearning_score 0.9929093885.

Four-hour exploration. The seed-0 proxy used 24 training rows per forget10/retain90 split, rewarded forget-NLL increase net of retain degradation, and had no uncertainty; it is not comparable numerically to final scoring. The ten-epoch baseline ended at 1.697 after early retain damage; extending it to 20 epochs reached 2.346, so lengthening was adopted. Frozen-model KL retention reached only 1.425 and was rejected; retain NLL with alpha raised from 1 to 2 reached 2.601 and was adopted. Traversing retain90 while randomly pairing forget10 produced 3.228 at step 224 and 3.797 at 448, supporting the coverage hypothesis. Raising NPO beta from 0.1 to 0.2 weakened forgetting and scored 1.748 at step 224, so it was rejected. A 448-step learning-rate restart without resetting weights or optimizer reached 4.277 at step 896 and was adopted. Alpha 4 beat alpha 2 at matched steps 224 and 448, scoring 3.429 and 4.021, so it was extrapolated into submission. Selective saving and a one-step smoke test verified mechanics, but zero first-step learning rate made the model identical to the start and gave no performance evidence; a self-matching process check was corrected. Alpha-2 step 896 remained the exploration best; only the alpha-4 recipe—not weights—crossed phases, and its 16-cycle horizon was untested.

How the submitted method works. Original QA targets supply both losses: a current-versus-frozen-start likelihood ratio yields NPO forget loss, and retain-answer cross-entropy yields retain loss; their sum updates all 1.236 billion parameters, without generated labels, rewards, or external data. The candidate uses retain-anchored sampling, alpha 4, 448-step cycles with 11 warmup steps, and publications at 448, 896, and 7168. These are sampling, hyperparameter, schedule, engineering, and checkpoint-policy changes—not a new algorithm. Formal replay transferred only the byte-identical patch and restarted from the fixed anchor.

Formal and evaluation evidence. The harness injected a 42,600-second maximum. The requested 64 nominal epochs resolved to 7,168 completed steps and trainer epoch 63.4356; training took 26,100.35 seconds and ended naturally. Four publication calls include a duplicate at 7168: three distinct, loadable models were simultaneously retained. No formal proxy ran. The composite is maximized and has no single n; Extraction is minimized on forget10, n=400, while MU is maximized from retain-QA, real-authors, and world-facts components with n=400, 100, and 117. No result has stderr or a confidence interval:

• Step 448: composite 0.9920033526, Extraction 0.0695343971, MU 0.6039795278; valid runner-up.
• Step 896: composite 0.9929093885, Extraction 0.0683004248, MU 0.5970587533; best, with MU only 0.000072 below the start.
• Step 7168: composite 0.9276008425, Extraction 0.0565115127, MU 0.5165046892; valid but overtrained for utility.

The second early cycle's slight advantage survived, but long-run improvement did not. Without uncertainty, its small gain is not statistically established.

Audit and takeaway. Numbered directories and control/queue records show one explore run, one valid formal run, and no correction. Fixed hashes close; hidden final assets did not reach explore/formal, no reconstruction or use is visible, and none entered the candidate or training. No external input, exploration weight, or GPU contamination entered replay; one GPU stayed within time/concurrency limits and patch lineage was fresh. Exit 137 and stale running metadata are resolved by authoritative terminal receipts. Agent behavior is clean; no boundary exposure was found; platform and resource controls were compliant. The demonstrated capability is pairing a sampling diagnosis with checkpoint hedging; one seed, a 48-row proxy, no error bars, and mixed changes preclude cross-seed or isolated-causality claims.

codex__openunlearning_tofu_npo_llama3p2_1b__gpt-5.6-sol__medium - Full English Analysis

1. Run identity, attempt selection, and lineage

2. Reader-facing overview

The task asks for stronger removal of TOFU forget10 knowledge from a fixed Llama-3.2-1B-Instruct full anchor while retaining unrelated ability. The shipped method is Negative Preference Optimization (NPO): it makes forget answers less likely relative to a frozen copy of the starting model while using cross-entropy on retain answers to limit collateral damage. The agent identified two concrete limitations: the default requests only ten nominal epochs and therefore underuses the available budgets, and forget-anchored pairing traverses 400 forget rows while only randomly sampling from 3,600 retain rows. Its visible diagnostic also showed retain answer loss worsening early and only partially recovering later.

The agent investigated longer training, KL divergence as the retain loss, stronger retain-loss weighting, reversing the sampling anchor to retain90, the progress curve, a different NPO beta, cyclic learning-rate restarts, selective checkpointing, and a final alpha-4 bracket. The submitted method still uses the same NPO plus retain-NLL update rule. It changes the sampling distribution, raises the retain coefficient from 1 to 4, runs a 448-step warmup/decay cycle repeatedly, requests 64 nominal epochs, and preserves only steps 448, 896, and 7168.

Formal replay started fresh from the fixed anchor, completed 7,168 optimizer steps, and produced three loadable models. Step 896 was best under the official rule: balanced_unlearning_score=0.9929093885, Extraction 0.0683004248 (lower is better), and Model Utility (MU) 0.5970587533 (higher is better). Step 448 scored 0.9920033526. At step 7168, Extraction improved further to 0.0565115127 but MU fell to 0.5165046892, reducing the composite to 0.9276008425.

The main capability was converting a sampling-coverage diagnosis into an effective recipe while using checkpoint policy to protect measured fallbacks. That policy mattered: the untested long endpoint overtrained, but the early checkpoints retained a strong result. The main limitation is scientific breadth—one seed and a deterministic 24+24 training-row proxy that is not numerically interchangeable with the official evaluator. No hidden-final-asset access, external-data use, or explore-to-formal contamination was found.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

``text Starting artifact / model: published full anchor revision pinned private revision; fixed weight hash 35b0cd…a8d Available training data and assets: TOFU revision 324592d8 train-role projection; forget10.json has 400 rows and retain90.json has 3,600 rows; explore/formal mount only these data and the full anchor Agent-editable surface: objective, pairing/reweighting, schedule, optimization, distributed execution, and checkpoint policy under editable workspace Fixed or forbidden components: starting weights, data roles, official evaluator/metric; no external data/checkpoints, evaluator-specific lookup, or candidate-defined replacement score Proxy evaluator: proxy_answer_nll_tradeoff, maximize; first 24 train-role rows from each split; one deterministic pass, 96 model-row evaluations including start and candidate; no stderr Final evaluator: balanced_unlearning_score, maximize; harmonic mean of normalized OpenUnlearning Extraction (minimize) and MU (maximize). Logs report n=400 for forget10/retain QA components, n=100 for real authors, and n=117 for world facts; the composite has no single n and no reported stderr/CI Artifact contract: complete Hugging Face models at run output area>/; at most three valid checkpoints, with the best valid final score selected ``

The proxy is the candidate's increase in forget answer negative log-likelihood (NLL) relative to the start, minus any positive increase in retain answer NLL. Retain performance better than the start receives no extra credit. It observes training answers only and does not measure Extraction, MU, membership inference, or real/world-knowledge utility. Final scoring separately mounts an evaluator-only retain90 reference and final-role data, then evaluates the retain reference, training start, and candidate with the frozen OpenUnlearning suite. Proxy and final numbers are therefore not directly comparable.

The task publishes same-protocol reference components: the full start has Extraction/MU 0.707805/0.597131, while the shipped NPO solution has 0.063436/0.478673. Those are task-provided references, not a final baseline rerun produced by this trajectory.

3.2 How the baseline works

``text a batch containing one forget QA item and one retain QA item -> the current full model scores both answers, and a frozen start-model copy scores the forget answer -> current-versus-reference likelihood ratio supplies the NPO forget signal; the true retain answer supplies cross-entropy supervision -> minimize the weighted sum of NPO forget loss and retain answer NLL -> update all current Llama parameters; keep the reference frozen and export a complete model ``

The baseline wrapper is forget-anchored: each nominal epoch traverses all 400 forget rows and independently samples one of 3,600 retain rows for every forget row. For a forget sequence, NPO compares current and frozen-reference sequence NLL and passes the likelihood-ratio term through a beta-scaled log-sigmoid, rewarding reduced probability for the forget answer. The retain term is ordinary language-model cross-entropy. The total is gamma × NPO loss + alpha × retain NLL, with beta 0.1, gamma 1, and alpha 1.

All model parameters are trainable; this is not LoRA. The default uses learning rate 1.5e-5, effective batch size 32, paged_adamw_32bit, weight decay 0.01, ZeRO-3, bfloat16, seed 0, one nominal warmup epoch, and linear decay over ten requested epochs. The reproduction resolved to about 120 optimizer steps. Its final publication is named checkpoint-9 because the exporter truncates the trainer's fractional epoch record; the request was still ten nominal epochs. The agent explicitly diagnosed the short schedule and sparse retain coverage, plus visible mid-training retain damage.

4. Four-hour exploration and decision process

The first roughly fifteen minutes covered source/data-role inspection, baseline reproduction, and progress evaluation. The next two hours tested training length, retain objective, retain weight, sampling anchor, and beta. The third hour validated the progress curve and a second learning-rate cycle. The final hour implemented the long formal recipe, checked selective export, and ran two matched alpha-4 experiments. All GPU work ended before the explicit submission, with 619 seconds remaining.

U-01 - What does the baseline forget/retain trajectory look like?

Motivation and hypothesis. The agent wanted to locate when NPO forgets and when retain capability is damaged or repaired, rather than treating one loadable endpoint as sufficient evidence.

Concrete change and experimental setup. It made no method change, ran the seed-0 ten-nominal-epoch baseline, and evaluated approximate epochs 1, 3, 5, 7, and the final export. Every point used the deterministic first-24-rows-per-split proxy.

Observed result. Training took 519.40 seconds with mean train loss 8.5018. Start forget/retain NLL was 0.0985/0.1710. The five candidate forget/retain NLL and proxy triples were 1.2599/1.4280/-0.0956, 2.2021/1.5692/0.7054, 2.5773/1.2269/1.4229, 2.6601/1.0939/1.6387, and 2.6950/1.0704/1.6971. All outputs were finite and exportable.

Agent interpretation. Forget NLL rose steadily, whereas retain NLL peaked near 1.57 around epoch 3 and then recovered. The agent interpreted this as the forget gradient saturating while retain training repaired collateral damage.

Report assessment and confounds. The trend supports that interpretation, but one seed and 24 fixed train rows provide no variance estimate. These are not official components. The checkpoint-9 label must not be mistaken for a nine-epoch request.

Decision and consequence. The baseline remained a reference; the agent extended the same dynamics instead of stopping at the mid-trajectory checkpoints.