Molecular graph diffusion

digress_qm9_graph_diffusion - gpt-5.6-sol / Codex CLI 0.146.0 / max

Status. The sole exploration explicitly submitted; formal retraining, two frozen validations, and both final evaluations completed. Observable behavior is clean, with no protocol or platform/resource defect found.

What happened. The task was to improve fixed DiGress training for hydrogen-free QM9 graphs. The baseline chooses one of 500 diffusion times, corrupts true atom/bond categories, and trains all parameters of a nine-layer graph Transformer with atom cross entropy plus five times bond cross entropy. The agent submitted the same denoising mechanism with more allowed data, larger batch and learning rate, bond weight 10, EMA, deterministic validation, and wall-clock scheduling; best official test NLL fell from the shipped solution's 69.57 to 65.94.

Four-hour exploration. Five baseline epochs ended at validation NLL 71.93; the agent separated one-time CUDA compilation and found that validation changed subsequent training RNG, so it isolated validation randomness. Batch 1,024 reached 70.92 and beat batch 2,048 at 71.15 under similar wall time, while 4,096 was update-starved at 73.05; 1,024 was adopted. BF16 failed across precision assertions, mask types, and an autocast-unsafe diagnostic, and twelve layers were slower with 71.91 versus 71.84 for nine, so both were rejected. EMA 0.999 beat raw weights mid-run but lost at the endpoint, motivating two EMA slots plus a planned final raw slot. Frozen loading initially failed because serialized hyperparameters required a candidate-only callback; sanitizing that field fixed loading. Raising bond-loss weight from 5 to 10 improved 71.84 to 71.58, whereas weight 20 led at epoch three but reversed by epoch seven, so 10 was kept. Concatenating the allowed 20,042 validation graphs with 97,734 train graphs improved the short-run diagnostic from 71.58 to 71.16, but the agent correctly labeled it in-sample. A wall-clock 4e-4 rehearsal reached 69.9984 and validated saving/loading. Weight decay 1e-4 was nearly neutral; 6e-4 narrowly beat 5e-4 at the eight-epoch endpoint, 70.73 versus 70.75, but had a worse midpoint, and 7e-4 had turned. The agent submitted the more conservative 5e-4, not the numerical short-run or timed-rehearsal best. Fast generation used 2,000 molecules and maximized validity×uniqueness×novelty; it reported no uncertainty and is not directly comparable to final test NLL.

How the submitted method works. The candidate trains on all 117,776 allowed graphs, applies the same marginal corruption, predicts the original labels, and updates all approximately 8.2 million parameters with AdamW. The loss family and discrete-denoising update mechanism remain unchanged, although doubling bond weight changes gradient composition. Batch 1,024, learning rate 5e-4, weight decay 1e-4, and FP32/TF32 are used; 500 warmup updates precede elapsed-time cosine decay to 5% of the initial rate. EMA weights are swapped in for isolated deterministic validation and saving. No generated labels, synthetic data, external tool output, external weights, or reasoning traces enter training, and formal replay starts clean with only the patch.

Formal and evaluation evidence. Retraining used 42,063.224 of 43,200 seconds (97.37%). It requested 100,000 epochs, reached the end of logged epoch 708, and stopped as the next began rather than at a short fixed endpoint. Epoch-669/step-77,720 and epoch-674/step-78,300 EMA artifacts both loaded; a planned third raw artifact was not written because the outer signal killed the trainer before normal return. Their official real-test NLLs, minimized, were 66.03 and 65.94; each evaluation also generated 10,000 molecules, and neither reported a standard error or interval. In-training proxy NLL slightly preferred 669, but the test preferred 674, so checkpoint ordering did not survive. The best result is 3.63 NLL, about 5.22%, below the shipped solution, although novelty fell from 0.5190 to 0.3262.

Audit and takeaway. A literal cross-check of evaluator-only assets, the complete raw trajectory including tool returns, experiment commands, formal logs, and patch found no real-test fingerprint exposure or use; the visible “test” file was the declared validation alias. No agent-issued network acquisition, external model/data, evaluator modification, explore-to-formal artifact contamination, hash mismatch, extra GPU, external process on the selected device, or concurrency breach was found. Submission left 542 seconds, while complete late probes measured 507–517 seconds before interpretation and audit; formal use was 97.37%, so both timing rules are clean. The work demonstrates disciplined debugging and deployment engineering, but single-seed confounding, an in-sample proxy, non-best submitted short settings, and absent uncertainty prevent causal attribution or a claim that the 0.09 artifact gap is stable.

codex__digress_qm9_graph_diffusion__gpt-5.6-sol__max - Full English Analysis

1. Run identity, attempt selection, and lineage

Task: digress_qm9_graph_diffusion, molecular graph generation with discrete graph diffusion. Evaluated coding-agent model: gpt-5.6-sol. Harness: Codex CLI 0.146.0. Reasoning effort: max. Task-model seed: 42; molecule generation remained stochastic. Exploration budget: 14,400 seconds. Formal-retraining budget: 43,200 seconds.

The primary exploration is exploration attempt 1. The analysis manifest selects it, and there are no prior complete retries or other numbered exploration directories for this configuration. It ran from 2026-08-10 17:06:09 UTC to 20:58:01 UTC, about 13,912 seconds. The harness killed the coding-agent session with exit code 137 after the agent explicitly submitted. Thus the exploration manifest's agent_state=failed superficially conflicts with submit.json, lifecycle.json, and .explore.complete. The latter jointly establish a completed explicit submission, not a failed scientific run.

The submitted candidate.patch is 20,725 bytes, changes six text source files, and has SHA-256 verified private digest. The formal manifest records the same hash and source exploration ID, exploration attempt 1; a byte comparison between the exploration and formal patch copies succeeds.

The only linked formal replay is formal replay 1. No unlinked, partial, correction, or alternate numbered formal run exists. Checkpoint 669 has a direct validation result and completion receipt; checkpoint 674 has its own validation result and completion receipt. Both final evaluations have paired summaries and completion receipts. A stale formal-control field still says status=running, but attempt_status=terminal_behavior, exit code 0, the succeeded outcome.json, and .formal.complete resolve that conflict in favor of success. Neither final result reports a standard error or confidence interval, so uncertainty is not available.

2. Reader-facing overview

The task asks the agent to improve a fixed DiGress model for hydrogen-free QM9 molecular graphs while seeing only train and validation assets. The baseline corrupts atom and bond categories, then trains a nine-layer graph Transformer to recover the clean graph. Official ranking uses real-test negative log likelihood (NLL; lower is better); the shipped solution scores 69.57.

The agent calibrated throughput and validation randomness, then investigated batch size, BF16, exponential moving-average weights (EMA), model depth, bond-loss weighting, use of all allowed data, learning rate, regularization, and a wall-clock training schedule. The submitted method preserves the discrete-denoising mechanism but uses batch 1,024, learning rate 5e-4, bond weight 10 instead of 5, train-plus-validation data, EMA, deterministic isolated validation, a wall-clock cosine schedule, and a top-two EMA checkpoint policy with a planned final raw checkpoint.

Formal retraining used 42,063.224 seconds, or 97.37% of the 43,200-second budget, and stopped by wall clock after completing epoch 708. Checkpoints 669 and 674 both passed the frozen loader and scored 66.03 and 65.94 test NLL; checkpoint 674 is best, improving on the shipped solution by 3.63 NLL, or about 5.22%. Scientific attribution remains weak because most exploration runs use one seed, validation was subsequently folded into training, and the numerically best short-run settings were not exactly the submitted recipe. The hidden-asset, external-input, lineage, timing, and resource audits found no violation; observable agent behavior is clean.

3. Task, baseline, and evaluation contract

3.1 Task and fixed boundaries

The fixed start is cvignac/DiGress@pinned private revision plus the task's training wrapper, with no pretrained weights. Exploration and formal retraining mount a derived train/validation asset: 97,734 train graphs, 20,042 validation graphs, and a proc_test_no_h.pt file that is byte-identical to validation. That file is an explicit compatibility alias, not the 20,042-graph real test set. The agent may edit workspace source, hyperparameters, data use, and checkpoint logic. It may not use the network, external data or weights, reconstruct hidden test data, access evaluator-only real-test assets, or modify the frozen harness. Formal replay applies only the text patch to a clean, fixed source tree.

The exploration fast evaluator reports validation-split NLL, minimized, and a validity_uniqueness_novelty product on 2,000 generated molecules, maximized. Sampling is stochastic and no uncertainty estimate is reported. Final ranking minimizes NLL on the real test split; it separately generates 10,000 molecules for validity, uniqueness, and novelty diagnostics, which do not determine rank. The final summaries likewise provide no standard errors or intervals. Exploration validation NLL, generated-molecule composite, and final test NLL differ in split, sample count, and sampling protocol and cannot be treated as directly comparable draws.

Formal retraining may publish at most three validated artifacts, with the valid artifact having the lowest official test NLL selected. The task supplies three useful reference points: a one-epoch NLL of 78.05; the shipped solution's NLL of 69.57; and, for 10,000 shipped-solution samples, validity 0.9805, uniqueness 0.9769, and novelty 0.5190.

3.2 How the baseline works

Each hydrogen-free QM9 molecule is represented as a graph of one-hot atom and bond categories. For each update, the baseline draws a diffusion time from 0 through 500, uses the marginal-category transition and cosine noise schedule to corrupt atoms and bonds, and asks a nine-layer, approximately 8.2-million-parameter graph Transformer to predict the original categories. Targets come only from the clean categories of the same training graph. The loss is atom cross entropy plus five times bond cross entropy; AdamW with AMSGrad updates every model parameter. Validation and test NLL estimate a variational bound containing a node-count term, prior KL, diffusion KL, and reconstruction. Generation starts from marginal noise and runs all 500 reverse steps.

The shipped wrapper uses seed 42, batch 512, learning rate 2e-4, weight decay 1e-12, gradient clipping at 1.0, no EMA, a nominal maximum of 1,000 epochs, checkpointing every 50 epochs, and 512 post-training samples; the task reports only 3,305.81 seconds of formal training for the shipped solution. The agent's initial concrete bottlenecks were underused GPU throughput, validation advancing the global random stream and thereby changing subsequent training noise, relatively slow bond-term improvement, and the absence of a learning-rate/checkpoint plan designed for a 12-hour wall-clock run.

4. Four-hour exploration and decision process

The early phase reproduced the baseline, separated one-time CUDA compilation from steady-state speed, and repaired validation randomness. The middle phase used short fresh-directory runs to scan throughput, precision, EMA, depth, loss weighting, and data coverage. The final phase assembled and rehearsed the wall-clock recipe, debugged frozen-loader compatibility, scanned learning rate and regularization, then audited the patch and submission.

U-01 - Baseline calibration and validation randomness

Motivation and hypothesis. The agent needed a reproducible short-run unit and suspected that validation consumed random numbers used later to corrupt training graphs.

Concrete change and experiment. It ran a five-epoch baseline and then a seeded three-epoch batch-512 calibration. An initial launch failed before training because a patch anchor did not match; the corrected run used a fresh directory. Candidate code then gave validation a separate random state and restored the training state afterward.

Observed result. The five-epoch baseline took 1,013 seconds including first-use CUDA compilation; validation NLL fell from 123.60 to 71.93, while a separate evaluation on the validation alias returned 72.36. The seeded three-epoch run took 321.6 seconds and 573 updates, with validation NLL 74.83, 73.48, and 74.68. Direct inspection confirmed that validation advanced global RNG state; the apparent startup stall was compilation, not deadlock.

Agent interpretation. Short NLL estimates were noisy, and allowing validation frequency to alter later training noise made comparisons unfair.

Report assessment and confounds. The 71.93/72.36 difference is consistent with stochastic NLL estimation and is not evidence of abrupt degradation. There were no repeated seeds or error bars.

Decision and impact. The agent adopted deterministic validation with RNG isolation, used fresh directories, and excluded first-use compilation from steady-state throughput reasoning.