Can agent leaderboards tell neighbors apart? A paired reanalysis of the HAL rollouts

October 22, 2026

Method

The data is the authors’ own release, scaffold_eval at commit e7ec464: 29,923 HAL rollouts across 9 benchmarks, 1,117 tasks, 54 models and 13 scaffolds, making 287 systems. Every comparison is paired on the tasks the two systems share, with an exact two-sided McNemar test, Benjamini-Hochberg correction across the neighbor pairs on each benchmark, and paired score intervals.

Result

On eight of nine HAL benchmarks no neighboring pair of agents shows a detectable difference on the tasks they share, and on most of them the gaps are smaller than anything the benchmark could detect.

Could not detect

An exact paired test cannot call a difference significant unless one system solves at least six more of the shared tasks than the other, even before any correction for testing many pairs, and at exactly six it only clears if the other system wins none of them back. On a 50-task benchmark that puts every gap under 12 points out of reach, however the tasks fall.

Artifacts

PDF · data release · arXiv:2610.00651

Leaderboards are how most people choose an agent. A team picking a coding agent opens SWE-bench and takes whatever sits above the one they run now, and a lab launching a model leads with the place it landed. Both are asking the board the same question: is this agent better than that one?

The board looks like it answers that. It prints a score to the decimal with a rank beside it, and a higher rank reads as a better agent. We checked whether it can actually tell, and on these boards it mostly can’t. In an exploratory redraw of one of them, 24 of 42 agents could claim first place. The public HAL rollouts behind a recent paper on leaderboard reliability contain 278 places where two agents sit next to each other on one of nine benchmarks. Compared on the tasks both of them ran, 3 of those 278 pairs show a difference a paired test can stand behind. All three are on one benchmark, USACO. On the other eight it is 0 of 265. That doesn’t make the other 275 pairs equal. It means the board’s own tasks can’t show a difference between them, and that difference is exactly what a reader assumes the rank is reporting.

This is not a claim that the leaderboards are noise. The paper we built on, Hardy, Azam, Reuel, Kochenderfer and Koyejo’s “Agent Evaluation Reliability: More Tasks Won’t (Always) Fix An Agent Leaderboard”, finds that rankings of whole systems (a model plus its scaffold) are highly reliable, with reliability coefficients between 0.935 and 0.994. We reproduced every HAL count in the paper from the authors’ own released data, and nothing here disputes their numbers. A reliability coefficient asks whether the whole ordering would come out the same again. It does not ask whether the agent at rank four beats the agent at rank five. A board can answer the first question well and the second hardly at all, and that is what these boards do. Our question sits where people actually decide: given the task outcomes a board already has, which neighbors does it separate?

What this means if you read leaderboards

The coarse position on a board is information. The rank number mostly is not. Systems far apart usually do differ: on GAIA 373 of 561 pairs separate, and on CORE-Bench Hard 521 of 1,540 do. You just have to go a long way down. Among the top half of each board, the typical distance before a lower system is reliably worse is 7 places on GAIA, 23 on τ-bench Airline and 24 on CORE-Bench Hard. On USACO it is 2.

Most neighbor gaps are smaller than the best case the test allows. An exact paired test cannot call a difference significant unless one system solves at least six more of the shared tasks than the other, even before any correction for testing many pairs, and at exactly six it only clears if the other system wins none of them back. On a 50-task benchmark that puts every gap under 12 points out of reach, however the tasks fall. 262 of the 278 neighbor gaps are under that floor. The floor is a limit, not a promise: a gap of six tasks can still fail the test. The paired detection limit in Figure 2, which accounts for how often two systems disagree task by task, sits well above the floor on every board where it exists.

Two of the nine benchmarks cannot rank anything. On AssistantBench and SciCode pass rates sit near 3%, and no pair of systems at any distance separates on either board, 0 of 435 pairs and 0 of 666. The largest neighbor gap on AssistantBench is 3 points against an 18-point floor. On SciCode it is 1.5 points against 9.

Task count and pass rates near the middle are what buy resolution here. USACO has 307 tasks and pass rates between 25% and 70%, and it is the only board where neighbors separate. This is a different target from the one in the paper’s title, which concerns ranking underlying models across scaffolds, where task-by-scaffold interaction caps reliability. For telling two fixed systems apart on shared tasks, more tasks help, and most of these benchmarks have too few.

The same boards, redrawn

If a board printed only what its data supports, each agent would get a range of ranks instead of one number. The best rank an agent can claim is one plus the number of agents that clearly beat it. The worst is the size of the board minus the number of agents it clearly beats. This view is exploratory: it re-displays the same pre-registered tests and is not a new pre-registered result. A range is the set of places the pairwise tests can’t rule out, not a confidence interval on a true rank. Pairwise results don’t chain neatly, so the ranges are a picture of the tests rather than a separate check on them.

On SWE-bench Verified Mini, three agents can claim rank 1, all running sweagent, and none of the other 23 can. Below them, every agent’s range overlaps the ranges of both its neighbors, and ten agents share the same worst rank of 26. The board can tell its top from its bottom and not much in between. (Handling the repeated runs the other way, with each system’s latest run, six agents can claim rank 1.)

On other boards the top is more crowded. On τ-bench Airline 24 of 42 agents could be ranked first. On ScienceAgentBench it is 15 of 25, and on Online-Mind2Web 10 of 23. On AssistantBench and SciCode every agent’s range is the whole board. USACO is the exception again: its top two agents are the only ones on any board pinned to a single rank.

That is what is broken about these boards, and it is a format problem more than a data problem. A board prints “1” for USACO’s leader, whose place the data pins down, and prints the same “1” for τ-bench Airline’s leader, which is one of 24 agents the data cannot order. Nothing on the page tells a reader which kind of “1” they are looking at. Every rank gets the same typeface and the same confidence, and the people choosing agents read them that way.

Rerun the same agent

Part of the reason is that one run is a noisy measurement. HAL ran 16 of its 287 systems more than once on the same tasks. Across those 18 pairs of runs, 14.5% of task outcomes flipped between runs (95% interval 12.7% to 16.5%). On the 6 run pairs whose ids differ only by a timestamp, which come from 4 systems, it was 18.1%. Four systems is a small base, so treat this as an order of magnitude and not a constant. At that rate, rerunning alone is worth roughly 7 points of standard error on the gap between two systems on a 50-task benchmark. A 3-point gap between two single runs is well inside what rerunning either agent could produce.

Scaffolds

A score gap and a difference in behavior are separate things, and a board shows neither well. The paper’s sharpest warning is about scaffolds: the same model ranks differently depending on the harness around it. The task-level data shows the same thing from below. Of 122 swaps where one model was run under two scaffolds on the same tasks, 42 moved many tasks in both directions while the net score barely changed. After subtracting what rerunning alone does, 22 still do, which is the pre-registered count. As sensitivities, it is 25 if the three default-versus-high-effort run pairs and the one ambiguous pair are left out of the rerun baseline, and 13 if only the six timestamp-only reruns count.

The gap between two models can also change with the scaffold, and that effect is real but concentrated. The direct test asks whether the task-by-task difference between two models is the same under both scaffolds, with correction across the model pairs on each benchmark. It confirms a change on three scaffold contrasts: GAIA’s hal_generalist against hf_open_deep_research (46 or 47 of 78 model pairs, depending on the test), Online-Mind2Web’s browser-use against seeact (24 of 45), and SWE-bench Verified Mini’s hal_generalist against sweagent (1 of 1). On the other six benchmarks it is confirmed nowhere.

What a leaderboard needs to answer that question

People bring one question to a board: does this agent beat that one. Answering it takes the outcome on every task, not only the score, and an honest statement of how small a gap the board can see. Most boards publish neither. Each fix below comes straight from a result above.

1. Publish per-task outcomes. The three pairs that separate here were found by comparing two agents on the same tasks, and a headline score cannot be compared that way.
2. Print the smallest gap each benchmark can detect next to its scores. 262 of the 278 neighbor gaps here sit below that floor, and a reader should see the floor before reading the rank.
3. Print a range of ranks for each agent instead of a single number. In the exploratory redraw above, that turns rank 1 on SWE-bench Verified Mini into a three-way claim, and on τ-bench Airline into a 24-way one.
4. Run the top entries more than once and publish every run. When 14.5% to 18.1% of task outcomes flip on a rerun, a gap between two single runs carries noise the board never shows.
5. Size a benchmark for the gaps it gets used to rank. USACO, at 307 tasks, is the only board here where neighbors separate.

Most of this costs little next to running the evaluation in the first place. Rerunning costs compute, and it is still cheaper than a team choosing an agent because it sits one place higher.

What we ran

The data is the authors’ own release, scaffold_eval at commit e7ec464: 29,923 HAL rollouts across 9 benchmarks, 1,117 tasks, 54 models and 13 scaffolds, making 287 systems. Every count in the paper’s HAL tables came back identical. Harbor Index, the paper’s second dataset, ran as a corroboration panel of 18 systems on 82 shared tasks, where 0 of 17 neighbor pairs separate.

Every comparison is paired on the tasks the two systems share, with an exact two-sided McNemar test, Benjamini-Hochberg correction across the neighbor pairs on each benchmark, and paired score intervals. Uncorrected, 5 of 278 neighbor pairs clear p ≤ 0.05, and correcting across every pair on a benchmark instead of only the neighbors still gives 3. The tests ran through Poisson Prove, our paired-comparison tool. The few calculations Prove does not do yet, such as the paired detection limit and the scaffold interaction test, ran in the study code.

The analysis was pre-registered and frozen in a signed commit before any outcome was analyzed. Changes made after the data was reconciled but before any test ran are dated in the spec. Changes after the results are logged with their reasons, and none changes a verdict. About 4% of task cells hold more than one rollout. The primary analysis takes each system’s earliest run, and repeating everything with the latest run gives 1 of 278 instead of 3.

A second model recomputed every headline number from the raw rows with its own code. Everything reproduced. The one count that depends on the choice of test, the GAIA scaffold interactions, is reported both ways above.

The whole read cost nothing to run. The study code rebuilds every table and figure from the released data with one command.

What this is a measurement of

The quantity tested is the difference between two systems on the tasks they both ran. Carrying it to a wider population of similar tasks takes the usual assumption that the benchmark’s tasks are a fair sample of that population. Each comparison is on the shared tasks of two systems, one rollout per task and system, so it mixes which tasks were sampled with how a single run happened to go. It cannot separate the two. The rerun figure comes from 16 systems, and three of those 18 run pairs are default-effort and high-effort runs that the paper’s matrix codes as one system. AssistantBench’s metric is partial credit, and the pass/fail outcomes are the paper’s thresholding of it. 39 of the 278 neighbor pairs are the same model at two effort levels or two scaffolds, distinct systems by the paper’s definition, and none of them separates. The typical distance before systems separate is a description of the boards, not a tested quantity.

Three sentences

A leaderboard can be reliable as a whole and still unable to tell rank four from rank five. On eight of nine HAL benchmarks no neighboring pair of agents shows a detectable difference on the tasks they share, and on most of them the gaps are smaller than anything the benchmark could detect. Read the board in ranges, not ranks.

Cite this

@misc{kolasinski2026neighbors,
  author       = {Kolasinski, Taylor},
  title        = {Can Agent Leaderboards Tell Neighbors Apart? A Paired Reanalysis of the {HAL} Rollouts},
  year         = {2026},
  month        = oct,
  organization = {Poisson Labs},
  howpublished = {\url{https://poissonlabs.ai}},
  note         = {Pre-registered paired reanalysis of the HAL rollouts released with Hardy et al. (2026), arXiv:2610.00651}
}

Edited 2026-11-02: the figure captions now describe the figures as drawn, and Figure 3’s table adds HAL’s own score for each named entry. No number or finding changed.