11% to 33%

September 28, 2026

Method

Four stored sets where one unchanged agent ran the same inputs more than once (two τ²-bench airline agents on gpt-4.1-mini, a HumanEval loop on gpt-4o-mini, the Tasks calibration harness on gpt-5-nano), and one synthetic workload with a planted change sealed by digest before any run. The re-run rule was fixed before any outcome was read: a first-run failure is "fixed" when the same input did not fail on a re-run, once or within three tries, with 95% intervals clustered by task. Trace's comparison is the log-rank test stratified by task, unpaired, at a ceiling of 1.3 on the minimum detectable hazard ratio; 200 independent null pairs for the false-alarm rate.

Result

On four real agents, re-running a failing input with nothing changed brought it back "fixed" 11% to 33% of the time on one re-run and 28% to 45% within three; on a synthetic agent that fails by pure chance, 73% and 98%. Trace's comparison read too small to tell or no significant difference on every unchanged pair, found a sealed planted change at p = 1.65e-9 and put it on the right step, and called 11 of 200 null pairs different.

Could not detect

Four sets is not agents in general: three OpenAI models, two tool-using agents and one thin code loop, at temperatures 0, 0.7 and 1.0; a different agent can sit outside 11% to 33%. The false-alarm rate is measured on the synthetic null only. Trace's pooled comparison could not have detected a hazard ratio below 1.113 on the HumanEval pair and did not flag the one problem that broke between the runs; Prove's per-cell diff did.

Artifacts

the Trace page

Teams check a fix by re-running the inputs that failed and counting how many now pass. LangChain’s Engine v2 post (2026-09-24) describes that method: reproduce the failure, re-run the same inputs against the candidate, keep adjusting until the failure stops. It states no sample size, no confidence level and no test.

We ran that rule, and Trace’s comparison, on the same stored runs: every set on this machine where one agent, unchanged, ran the same inputs more than once, and one synthetic workload we built to have a known answer. The rule was fixed before any outcome was read: take the first run’s failing inputs, look at the same input in the second run, and call the failure “fixed” when it did not fail, once or within three tries.

The short answer

On real agents, re-running a failing input with nothing changed brought it back “fixed” 11% to 33% of the time on one re-run and 28% to 45% within three, across four agents. Real failures are partly to mostly consistent: an input that failed once failed again 66% to 81% of the time. So the rule does not call most failures fixed on these agents. It calls one in nine to one in three fixed with nothing changed, the share differs by agent and by kind of task, and nothing in the rule can see it.

On a synthetic agent that fails by pure chance, the same rule “fixed” 73% of failures on one re-run and 98% within three, with nothing changed. That is the ceiling: a failure that happens with probability p comes back “fixed” with probability 1 − p on one re-run and 1 − p³ within three, whatever was or was not changed.

Fig. 1. Of the first run’s failing inputs, the share that did not fail on one re-run of the same, unchanged agent (ink), with a 95% interval clustered by task, and within three re-runs (hollow). The last row is the synthetic agent. Runs 20260915T210140Z-4847 and 20260915T211148Z-965d; t4a-reps00-21 and t4a-reps22-43 (halves of one run); 20260905T185407Z-0313 and 20260905T195009Z-18ac; card-attempts0-1 and card-attempts2-4 (halves of one run); synthetic 20260928T195532Z-4e4e and 20260928T195535Z-d088.

What was measured

Every stored set of real agent runs on this machine where the same agent, unchanged, ran the same inputs more than once. Four passed the rule that was written before any of them was read:

On each: the re-run rule exactly as defined, with a 95% interval clustered by task because real failures cluster by task; Trace’s comparison at the study’s settings; and a per-task consistency measure, C, the chance that a task which failed once fails again on another run, beside the plain failure rate p and κ = (C − p) / (1 − p), which is 0 when failures are pure chance and 1 when every task always or never fails.

The results on real agents

By temperature on HumanEval: at 0.0, 8.8% “fixed” on one re-run (κ 0.89); at 0.7, 14.2% (κ 0.67). Warmer runs “fix” more by chance, as expected.

The synthetic control behaves as built: C equals p (0.267 against 0.264, κ 0.005), because every synthetic failure is pure chance, and the rule’s 73.3% is 1 − p. Every real agent sits far from that: C is 0.66 to 0.81 while p ranges from 0.13 to 0.58.

The reading was fixed before any of this was computed. “Re-run validation calls most failures fixed on an unchanged agent”: on one re-run the interval lies entirely below 0.5 on all four sets, so the claim does not hold on real data. “Re-run validation reports fixes that do not exist”: on one re-run the interval lies above 0.10 on three sets and contains 0.10 on HumanEval; within three re-runs it lies above 0.10 on all four, the lowest lower bound being 19.7%.

Is re-run validation misleading on real agents?

Less than the synthetic study alone would say, and not in the way we first put it. On these four agents most failures come back failing, so re-run validation does not call most failures fixed. But on every one of them it calls one failure in nine to one in three “fixed” when nothing changed, and up to nearly half if it keeps trying three times, and nothing in the method says which of those numbers applies to the agent in front of it. A team that “fixes” a handful of failing inputs and watches them pass on re-run cannot tell its fix from that background without running the unchanged agent the same way. That is the control Trace’s comparison is built on.

Trace said nothing false on any real unchanged pair: too small to tell on 400 episodes, no significant difference with detection limits of 1.175 and 1.113 on the larger sets.

The synthetic study

The synthetic agent is where the answer is known. The workload is Tasks’ scripted three-call chain (review, sample, measure, then a reply) with a policy that fails each call independently at a fixed rate for its stage, with fresh randomness on every run; a failed call is made wrongly, the engine rejects it, and the episode ends wrong_state. Three arms on the same 50 tasks:

• A, the baseline: every stage fails at 0.10, so 27.1% of episodes fail.
• A′, A again with nothing changed, under a fresh salt.
• B, A with one stage’s failure rate doubled to 0.20. Which stage was drawn from the operating system’s random source and sealed by digest before any run; the pre-registration carries the digest, never the name, and the seal was opened last.

Two sizes: small, 2 runs per task (100 per arm), underpowered by design; large, 40 runs per task (2,000 per arm). Then 200 more independent A vs A′ pairs at the large size, for the false-alarm rate. Everything ran on a laptop: the six main runs in 26 seconds, the 200 pairs in about half an hour, no paid call.

What the re-run rule concluded.

On an agent that was made worse it still “fixed” two thirds of the failures on the first try and 96% within three.

What Trace said. Large A vs A′: no significant difference (p = 0.597), could not have detected a hazard ratio below 1.188. Large A vs B: different, runs die earlier in B (p = 1.65e-09), could not have detected a hazard ratio below 1.174; per step, review q = 1.67e-20, measure 0.422, sample 0.865. Small A vs A′: too small to tell, could not have detected a hazard ratio below 2.245, and 1.3 was asked for (p = 0.397). Small A vs B: different (p = 9.52e-04), which the pre-registration gave a 17% to 31% chance; a detection at low power is real but it is not what the design promised, and the small A vs A′ line is the one to read.

Fig. 2. The planted change, as Trace placed it: deaths by step in each arm of the large pair, 2,000 runs per arm. The seal, opened after every comparison, named review, the only step under q = 0.05. Runs 20260928T195532Z-4e4e and 20260928T195539Z-d1d7.

The false-alarm rate. Two hundred independent A vs A′ pairs, nothing changed: called different in 11 of 200 = 5.5%, Wilson 95% interval 3.1% to 9.6%, which contains the nominal 5%; called no significant difference in 189; too small to tell in 0, with the detection limit between 1.180 and 1.192 on every pair. The 200 p-values are flat across twenty bins. In 12 of 200 pairs some step’s Benjamini–Hochberg q fell below 0.05, 6.0% against a nominal 5% per-pair ceiling.

Fig. 3. The p-values of the 200 null comparisons in twenty bins of 0.05; the first bin is the 11 pairs called different with nothing changed.

The predictions, scored. Six were pre-registered. Five hit: the large planted change detected; the large unchanged pair cleared with a limit tighter than the planted change; the re-run rule “fixing” about 73% and 98% on the unchanged synthetic pair (predicted 72.9% and 98.0%); the false-alarm interval containing 0.05; the sealed stage being the one step under q = 0.05. One half-missed: both small pairs were predicted too small to tell, and the small A vs B pair came out different, the outcome the pre-registration gave 17% to 31%. It stays in the record.

Why a re-run of the failing inputs cannot validate a fix

A stochastic agent fails a given input with some probability p. Re-run that input and it passes with probability 1 − p, fix or no fix. Re-run it until it passes and it passes with probability 1 − pᵏ after k tries, which is nearly one for any k above three. On an agent that fails by pure chance that is 73% and 98%; on the real agents above, whose failures repeat, it is 11% to 33% and 28% to 45%. Both are numbers the method cannot see.

What the method lacks is a control, a count and a limit. A control: the unchanged agent, re-run the same way, which shows how many failures “fix themselves”. A count: how many independent runs the comparison rests on, which sets what it could have seen. A limit: the smallest effect the design could have detected, printed with every verdict, so that “no difference” is never confused with “could not tell”. Trace prints all three, and its small-size answer on this study is the honest one: 100 runs per arm could not have detected a hazard ratio below about 2, so it says too small to tell rather than pass or fail.

What Trace missed

One HumanEval problem broke between the two runs of an agent nobody touched: HumanEval/129 went from 20 of 32 failing to 32 of 32 at temperature 0.0, and from 12 of 32 to 32 of 32 at 0.7. Trace’s compare, a pooled test over the whole distribution of failure times across 328 cells, read no significant difference, p = 0.109, and could not have detected a hazard ratio below 1.113; a one-problem break is far below that limit. Prove’s per-cell diff caught it: Fisher’s exact test per cell, corrected across all 328, flags the two cells and nothing else. Trace’s fix check, built after this study, now flags the same two inputs with the same numbers, and reads the whole pair as different between the runs (p = 0.036) with the inputs not moving as a body (by input, p = 0.127): a true statement about two runs and a false one about two versions, which is why every report closes by saying that a difference between two stored runs is a difference between the runs.

How a critic would attack this, and the answers

• “Four sets is not agents in general.” Correct. Three OpenAI models, two tool-using agents and one thin code loop, temperatures 0, 0.7 and 1.0. The range 11% to 33% is what they showed; a different agent can sit outside it. The synthetic study is the upper bound (pure chance), and a fully deterministic agent would be the lower (0%).
• “You knew the τ²-bench answer before you wrote the rules.” Yes, and the pre-registration says so: the R.1 study had published the 2 × 2 (21 of 110). The set was kept by the rule, and the other three sets were not computed before the rules were committed.
• “Temperature 0 is not how agents run.” Half the data is warmer: HumanEval at 0.7 (14.2%) and the card at 1.0 (33.0%).
• “Two of the sets are halves of one run, not two runs.” They are the same agent on the same inputs under fresh seeds, which is what a re-run is. The τ²-bench halves (21.7%, fresh seeds) and the τ²-bench two-run pair (19.1%, identical seeds) agree, which is what the R.1 study found too: the seed parameter buys almost nothing.
• “You wrote the synthetic agent, so of course the numbers match.” Yes: the arms are synthetic and planted by construction, and the study says so in its first line. The point is that both methods were run on the same stored rows with rules fixed in advance, and the re-run rule’s “fixed” share is what its own arithmetic predicts on any stochastic agent.
• “The re-run method is a strawman.” It is the post’s description computed literally: the failing inputs, re-run against the candidate, counted as fixed when they pass, once and within three tries. Engine’s actual loop is not public. If it applies a test, a control or a limit, the post does not say, and the study measures what the post describes.
• “Prediction 1 missed; you are reporting a failed prediction.” Yes, and it stays. The pre-registration gave that outcome a 17% to 31% chance, and it happened here and once more in the reproduction test.
• “Trace’s own τ²-bench verdict is ‘too small to tell’, which is a non-result.” That is the result. The design of 400 real episodes could not detect a 45% change in the hazard, and a tool that says so is the alternative to a tool that says “validated”.
• “The false-alarm check only tests Trace on your synthetic null.” Correct; the log-rank’s null calibration on this workload is what it measures, and that is what it claims. It does not measure calibration on any real agent’s traces.

Where the evidence is

Every output above is the tools’ own, as written: the re-run rule’s counts, the consistency script’s counts, Trace’s reports and the false-alarm summary, filed with the pre-registration that was sealed before any run and the addendum that was pre-registered before any real-data outcome. The τ²-bench R.1 pair is public in Trace’s fixtures; the calibration card’s records are in the Tasks repository; the T.4 and HumanEval stores are private. The study folder’s public home is being decided, and this page will link it when it exists.

Empirical evidence under a stated test envelope, not formal verification. Conditions outside the envelope were not tested; conditions inside it can only resolve effects at or above the reported detection limit.