AI4AI-Bench
Can a coding agent improve the training method an AI project already uses?
Most agents tune the settings around the existing method. Few rewrite the method itself — and that is where the gains turn out to be.
Navers Lab · Einsia.AI · 2026
Loading…
Six agents, most of them run at several settings.
Matching it scores 0.10; below it, the rewrite made things worse.
Nothing here comes close. That is the finding.
Across all 290 runs the average is 0.166 — just above the code that was already there. Claude Opus 5 leads at 0.250, GPT-5.6 Luna trails at 0.117, and in 124 runs the rewrite was worse than what it replaced. Exploration cost $5334 in all, from $4 to $626 per configuration — and the most expensive one is not the best.
What we did
Ten frozen research projects, one instruction
Each task is a working research codebase — the repository behind a paper, with the training code its authors actually ran. The agent gets that repository and the model its training starts from, and one instruction: make this project's own training method better. The ten cover deliberately different ground, so no single trick carries an agent through.
The rules
The agent writes the code; a sealed pipeline trains it and scores it
An idea can be judged in minutes, but settling it takes a full training run — so the two are split. Without that separation a high score could just mean the agent watched a run and stopped it at a lucky moment.
The agent · 4 hours · 1 GPU
A real research codebase, its training code, and the model that training starts from.
The agent edits the training code and runs its own short experiments against a cheap proxy metric.
source code only
Sealed · up to 12 hours
The submitted code trains a model for up to 12 hours in a container the agent cannot reach.
A test the agent never saw scores the model — and scores the original code the same way.
The score
How ten tasks end up on one axis
Picture quality, perplexity and puzzle solve rates cannot be averaged, and counting wins hides by how much. Each task is placed on the same 0-to-1 axis by three fixed marks.
Above 0.1 the score says how much of the remaining distance was closed. An agent that returned nothing runnable scores 0, so giving up early is never rewarded.
Why it matters
This is the part of “AI improving AI” that compounds
An agent could speed up the hardware, find better data, or improve the training method. The first two pay once. A better method is inherited by every model trained afterwards — including the one that becomes the next agent. Existing benchmarks are won by assembling data or searching over settings, and cannot tell a redesigned method from a tuned one. This one is built to.
What the agents changed
Most tuned the run. Few changed how the model learns.
A score says whether a rewrite worked, not what it did — so we read the submitted code too. Of the 263 submissions that changed anything, the changes fall into two kinds: adjusting how a run is operated (length, checkpoints, learning rate, batch size), or changing how the model learns (the objective, the supervision, the update rule, the data).
Careful engineering, original method untouched.
average score 0.126A new objective, new supervision, a different update rule, or different data.
average score 0.226The harder kind of change scores nearly twice as well, and more than half the submissions never attempt it.
What more reasoning buys
More reasoning buys nerve, not results
Every agent here can be told how much to reason before it acts. Turned up, they attempt the harder kind of change far more often: the share that touches how the model learns rises from 8% at the lowest reasoning setting to 64% at the highest. What comes back improves much less — the average score moves from 0.094 to 0.196, and no agent improves steadily as the setting rises. Every one of them peaks somewhere in the middle and falls back.
Citation
@article{ai4aibench2026,
title = {AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design
for Recursive Self-Improvement},
author = {Navers Lab and Einsia.AI},
year = {2026},
note = {Benchmark release}
}Placeholder entry; the arXiv record replaces it on publication.