45.0% and 44.0%
- Method
- τ²-bench airline, 50 tasks with 4 replicates each, two arms on identical config and seeds run back to back inside twenty minutes, compared task by task and call by call; the exact McNemar test on the 200 outcome pairs, with a task-clustered interval on the net swing.
- Result
- Two runs of τ²-bench airline with identical config and seeds scored 45.0% and 44.0%, while 44 of 200 task outcomes flipped and every pair took a different path. Divergence does not accumulate with trajectory length, it varies by task, and matching seeds did not stop it.
- Could not detect
- A rate for one model pin, one provider and one twenty-minute window, not a constant: 44 of 200, with a task-clustered 95% interval of 0.155 to 0.285. A non-significant association here is not evidence of none, since the design could not detect a small one, and nothing here speaks to a week.
We ran the same agent benchmark twice. Same configuration, same tasks, same seeds, nothing changed in between.
The first run scored 45.0%. The second scored 44.0%.
A point apart. On any leaderboard that is the same number twice, and the reasonable conclusion is that the benchmark is stable.
Then we compared the two runs task by task instead of score to score.
Forty-four of the 200 matched tasks came out differently — solved in one run, unsolved in the other. And every one of the 200 matched pairs, including the 156 that agreed on the final answer, took a different path to get there. Not one trajectory was reproduced.
156 trajectory diverged, outcome agreed 44 trajectory diverged, outcome disagreed
200 / 200 trajectories diverged 44 / 200 outcomes disagreed 0 / 200 identical pairs in run order
The 44 outcome disagreements are the discordant pairs that the exact McNemar test in the text is run on; the 200 trajectory divergences are counted, not tested. Runs 20260915T210140Z-4847 and 20260915T211148Z-965d.
Two things happened here and they are worth keeping separate. Outcome disagreement is the 44: pairs where the benchmark’s verdict flipped. Trajectory divergence is the 200: pairs where the sequence of model calls differed, whether or not the verdict changed. The first is a fifth of the tasks. The second is all of them.
The flips cancelled
Here is the whole result in four numbers.
| R′ solved | R′ unsolved | ||
|---|---|---|---|
| R solved | 67 | 23 | 90 |
| R unsolved | 21 | 89 | 110 |
| 88 | 112 | 200 |
Twenty-three tasks went solved to unsolved. Twenty-one went the other way. Exact McNemar gives p = 0.880; the net swing is +1.0 points with a task-clustered interval running from −5.5 to +7.5.
So the two pass rates sit a point apart, not because little changed underneath, but because the 23 and 21 were close enough to cancel out.
You cannot see any of this from the scores. Two numbers a point apart look like agreement. Finding the 44 requires comparing the runs case by case.
The symmetry is a fact about this pair of runs, not a rule. Nothing forces flips to cancel, and with 23 and 21 out of 200 the interval on the difference is roughly ±6 points. Another pair of arms could land several points apart on noise alone.
What we ran
τ²-bench airline, 50 tasks, 4 replicates per task, two arms run back to back inside a twenty-minute window. 400 episodes, 200 matched pairs. Within each pair the config hash and the seed are identical by construction, and we asserted that per pair rather than assuming it.
Arms 20260915T210140Z-4847 and 20260915T211148Z-965d, sharing salt d2ebb5be1d394d12.
All 400 episodes terminated either solved or unsolved — zero errored, zero timeouts, none of τ²’s eight termination-reason labels occurred. The outcome is binary in this dataset, so solved is the pass rate quoted above.
The experiment cost $5.45.
Definitions were filed before anything ran: what counts as agreement, which pairs are eligible, what would stop the run. That matters for a reason two sections down.
Divergence happens throughout the run
The obvious objection is drift. Agent trajectories are long, small differences compound, and of course two runs end up apart eventually.
That is not what the data shows. We measured the conditional hazard — among pairs still identical entering call i, the fraction that first diverge at i. Over the first three calls it runs 0.470, 0.594, 0.488, with overlapping intervals.
First-divergence hazard by call index
Among pairs still identical entering each call. Risk set shown below the axis.
Points are the observed hazard; bars are 95% Clopper–Pearson intervals. n is the risk set: pairs still identical entering that call. Axis stops at call 4. Beyond it fewer than 10 pairs remain at risk: 6 at call 5, 2 at call 6, 1 at call 7.
No increase, so divergence is not something that builds toward the end of a trajectory. No decrease either, so it is not concentrated at the start. The estimates are consistent with divergence occurring at a substantial rate throughout the run.
We are not claiming the calls are independent, and we have not tested that. A constant-looking hazard over the calls where we have support is not the same as a constant process, and it is certainly not fifteen independent draws. What it does rule out is the drift story: nothing in this data suggests divergence accumulates with trajectory length.
One reading we withdrew: the median first divergence sits at call index 1, which we initially read as divergence being immediate. It is not. The user simulator makes call 0 in all 400 episodes and the agent makes call 1 in all 400, so index 1 means by the agent’s first turn. That is speaking order, not elapsed time.
Where the first difference shows up
All 200 first divergences look the same: the request was byte-identical and the response differed. The first observable difference is at the model-response boundary.
No pair diverged first on a tool response. None diverged first on ordering. Both of those cases existed in the classifier before the run — we did not go looking for the model boundary and find it. Three entry points were checked and two never fired.
It happens on both sides of the conversation: 114 pairs diverge first in the user simulator, 86 in the agent.
And 87 of the 200 diverge while the provider reports the same backend configuration. Same config, same request, temperature zero, same seed, different response.
Divergence point clusters by task
If divergence is not governed by position in the trajectory, the next question is what it is governed by.
Task, at least in part. The between-task share of variance in divergence point is 0.399, against a permutation null averaging 0.247, p = 0.0018. Sixteen of the fifty tasks have all four of their pairs first diverging at exactly the same call index.
Position, meanwhile, tells you nothing about the verdict. Whether a pair diverged early or late has no detectable relationship to whether the two runs disagreed — Spearman rho of +0.155, p = 0.281, against a threshold declared at 0.30 before looking.
Four pairs per task is thin, and we are not building anything further on this.
Matching seeds did not help here
We ran the comparison again with different seeds instead of matched ones.
Same seed: 22.0% of pairs disagree. Different seed: 23.3%. The intervals overlap almost entirely.
Matching the seed did not measurably reduce disagreement in this comparison. That is the whole claim. It is not an argument that seeds are useless — they pin what they pin, and the provider documents the parameter as best-effort. It is an observation that here, on this substrate and this provider, matching them made no measurable difference.
One reading we withdrew
Before the run we declared that if two paired runs were served by different backend configurations — the provider publishes a fingerprint for this — that would name a cause for the divergence.
We withdrew it.
The problem is the missing comparison. At the very first span the request is identical by construction, so any fingerprint difference there has nothing to do with the run; it is whatever rate the provider happens to route differently. That rate is 49.5%. At the first divergence it is 56.5%. Fisher’s exact gives p = 0.193.
That difference is not distinguishable from the base rate, so it does not support the causal reading we had declared. We are not concluding the reverse either — a non-significant association is not evidence of no association, and this design could not detect a small one.
What we can say is that we were about to read 56.5% as a finding, and the only reason we did not is that we built the base-rate control. Our own pre-registration declared a comparison without declaring what it would be compared against. That is filed as a defect in the pre-registration rather than as a footnote in the analysis.
A related miss, in one of the tests above: the effect-size floor we declared in advance sat below the permutation null’s own mean, meaning data with less structure than chance would have cleared it. The verdict held because a second criterion also fired, but the threshold did no work. A declared threshold with no stated null is not yet a test.
The correction that changed the numbers
The provider mints a fresh random identifier for every tool call and echoes it into the next request. Our first analysis compared spans byte-for-byte, so two episodes making the identical tool call with identical arguments scored as divergent purely on that identifier.
Correcting it moved the index distribution substantially and dropped the apparent hazard at call 3 from 1.00 to 0.14.
We normalised those call identifiers and nothing else. Other identifiers in the payload are domain data — a user id that differs between two runs is a real difference, and normalising it would hide exactly what we are looking for.
What did not move: all 200 pairs still diverge, all 200 first divergences are still response-only at a byte-identical request, 94 of 200 still diverge at call 0, and the 44 flips come from stored outcomes rather than from spans, so they were never affected.
What this is a measurement of
200 pairs, τ²-bench airline, one model pin, one provider, one twenty-minute window.
Outcome disagreement was 44/200 = 22.0%, with a task-clustered 95% interval of 0.155 to 0.285. That is a rate for that setup. It is not a constant and not a number to quote about agent evaluation generally. The absence of tool and ordering divergences is an absence in this harness. The seed result is a result about this provider under these conditions. And we do not know the rate over longer windows — twenty minutes is what we measured, and this experiment says nothing about a week.
What we would expect to travel is the shape rather than the magnitude: that a stable aggregate can sit on top of unstable task outcomes, that matching seeds may not stop it, that divergence is not a function of trajectory length, and that it varies by task. The magnitude is yours to measure on your own system.
What is still open
Why the model returns different responses under identical conditions is inside the provider. Decoding, routing, batching, backend state — all live candidates, and none separable from outside.
Closing it needs a different experiment: direct API replay outside the benchmark, one request repeated many times.
We know where the first difference appears. We do not know what happens on the other side of it.
Why this matters for anything built on top
You already know your test suite has a flake rate. You have hit rerun on a red CI job, watched it go green, and shipped.
The version of that problem here is harder to notice, because a noisy score at least looks noisy. Two scores a point apart look like agreement, and the instability is one level down in the individual task outcomes.
Which means a difference that was really a lopsided draw can become the new baseline. The next change gets measured against it, and the next. It ends up as an entry in the experiment log, then a regression target, then a decision.
Others have found related things
On Randomness in Agentic Evals (arXiv:2602.07150) collected 60,000 agent trajectories on SWE-Bench-Verified across three models and two scaffolds. Single-run pass@1 varied by 2.2 to 6.0 percentage points depending on which run was selected, with standard deviations above 1.5 points at temperature zero.
Verdent’s SWE-bench Verified technical report found that running the same model through different API providers, under an identical scaffold and evaluation setup, moved pass@1 by as much as 1.2 points.
Quantifying Variance in Evaluation Benchmarks (arXiv:2406.10229) makes the general point across 280 models: variance in evaluation benchmarks is rarely quantified, even though it decides whether an observed difference means anything.
Those are score-level results. What we have added is the level underneath — the same comparison run task by task and call by call rather than score to score.
Paired comparison, confidence intervals and power analysis all predate agent benchmarks by decades. The machinery is not the problem.
What it cost
Episodes cost $0.0136 each. The whole experiment — 400 episodes across two arms, every number above — came to $5.45, measured from the provider’s own per-call cost accounting rather than a rate card.
Scaling that: a three-arm comparison at 2,200 episodes per arm on this substrate is about $91 at the same measured rate. Whether that is enough episodes to detect any particular effect is a separate question with its own assumptions, and we are not claiming it here.
Three sentences
Two runs of the identical thing scored 45.0% and 44.0% while 44 of 200 task outcomes flipped, and the flips cancelled.
Divergence does not accumulate with trajectory length, it varies by task, and matching seeds did not stop it.
The first difference appears at the model-response boundary, and what happens beyond that boundary is not visible from outside the provider.