How we count.

Each condition gets its own test. Test 400 conditions at p < 0.05 and about 20 will pass by luck alone, with nothing changed. A map with 20 blue squares on it would look like a finding.

So the tests are corrected together, across every condition in the run, with the Benjamini–Hochberg procedure at q < 0.05. It controls the share of the conditions we call changed that are false alarms. A condition is coloured only if it survives that.

Which test sits underneath matters, and we say which. On the Go1 runs, Fisher's exact test calls 60 conditions safer, McNemar's test calls 55 conditions safer and the log-rank test calls 88 conditions safer. All three call 0 worse. The published figure uses the first; the spread is in the report, because a reader should see how much the answer depends on the choice.

Every verdict carries a second number: the smallest change that condition had the power to detect, given how many trials it had and how often the system was already failing there. It is printed per condition, and the worst one is printed in the headline.

It works before the run as well as after. Prove computes what a design can resolve before it spends anything, and refuses to start one that cannot resolve the effect you asked about.

On the figures on this page and on Prove's page it appears in the legend as “blind under N pp”. Trace states its limit in its caption and its report instead, as a hazard ratio or in points of failure share (the three lines, below). That is the figure the top of this page is about.

A stored run carries its trace: the spans the agent's framework emitted, nested as the framework nests them. Trace puts every run on one clock by counting leaf spans, the spans that are the parent of no other span, in the order they started. An agent turn or a workflow is a container and is not a step; the model call, the tool call and the guardrail check inside it are. A run with L leaves ran L steps, and the kind of a step is the leaf's kind: model call, tool call, retrieval, agent, and so on.

Each run is then placed, in this order: excluded when it has no readable trace or no outcome (and counted by reason in the header); censored at its last step when it timed out or hit the capture cap, because a run whose trace stops early has an unknown death step even if its outcome says it failed; died at its last step when its outcome is one of the events named on the command line; censored at its last step otherwise, because it ended. The survival curve over step index is the Kaplan–Meier estimator with Greenwood's variance for its 95% band, the same estimator as Fig. 3 on a different clock. Where runs die, by kind of step, is the Aalen–Johansen cumulative incidence, which treats the kinds as competing causes so that the parts sum exactly to one minus survival, with a seeded bootstrap band (1,000 resamples). The tool's own worked example, on the recorded customer-service demo, is on its documentation page: 24 episodes, 7 deaths, and a band at step 40 that spans a quarter of the axis, which is why the report calls one run a description.

Two stored runs, a before and an after, are compared with the log-rank test stratified by condition over the conditions both runs carry: within each condition and at each step where a run died, the deaths observed in the first run against the deaths expected if the two shared one hazard, summed over steps and conditions, and tested as chi-square on one degree of freedom. Stratifying is what makes two runs of fifty tasks comparable at all, because the tasks differ in length; a pooled test would mix “this task is long” with “runs die later after the change”. The test uses every death at every step with no step chosen after looking; a difference at a chosen step is printed on request, with a Wald interval, and is never the test. It is unpaired: the two runs share their seeds, but the log-rank does not use that fact, and the header says so.

The direction is stated as runs die earlier or later after the change, never better or worse. What the comparison could not have detected is the minimum detectable hazard ratio by Schoenfeld's formula at 80% power, from the deaths over the common conditions, and it prints on every comparison. On the tool's own example, the same agent stored twice on the same seeds, 200 episodes each, the test reads p = 0.521 and could not have detected a hazard ratio below 1.457: 400 real episodes of one agent could not have seen a 45% change in its hazard, and the report says so instead of calling the pair unchanged. Per kind of step, the cause-specific log-rank is run with the other kinds censored, corrected across kinds with Benjamini–Hochberg.

The question a team asks after a fix is whether the new version fails less on the inputs that failed. Re-running those inputs and counting the passes has no control: on an unchanged agent some of them pass on a re-run anyway, and the launch study measured how many. The fix check runs the same inputs on both versions, side by side, and reads the old version's re-run as the control. The inputs are chosen before either arm runs, and a run that was used to choose them is refused as an arm, because its failures on those inputs are there by selection.

The test is the exact conditional Cochran–Mantel–Haenszel test, stratified by input (Cochran 1954, Mantel and Haenszel 1959; Birch 1964 for the conditional form). Each input is a two-by-two table, version against failed or not; given each input's total failures, the new version's count on it is hypergeometric under the null whatever that input's failure rate, and the distribution of the sum is computed in full, so the p-value is exact and two-sided with no normal approximation. An input that always fails or never fails contributes nothing, no signal and no noise, which is why consistent failures shrink the variance a comparison has to beat rather than inflating it. With one input it is Fisher's exact test, the number Prove's diff prints for that cell; with one run per arm per input it is the exact McNemar test, the arithmetic behind the paired test above. It is the one test that contains both.

The difference is the Mantel–Haenszel risk difference in points of failure share, new minus old, with Sato's variance (1989) for a large-sample 95% interval that is labelled so; the line follows the test, never the interval, and the report says so when the two disagree. A second reading by input treats each input as one observation and flips its sign, so it does not assume that repeat runs of one input are independent draws; it prints beside the test and has no part in the line. How consistent the failures were is printed as κ, the chance a failure repeats beyond what its rate alone would give: a comparison inside inputs needs 1 − κ times the runs a pooled one needs, which on the tool's τ²-bench example (κ = 0.531) is under half.

What it could not have detected is the smallest proportional drop in every input's failure rate that the test would have found four times in five, stated in points of failure share, with the rise beside it. On the τ²-bench pair, 200 runs per arm, that limit is 11.1 points; on the HumanEval pair, 328 inputs at 32 repeats, it is 0.64 points. Which inputs moved is Fisher's exact test per input with Benjamini–Hochberg across every input compared, number for number what Prove's diff does, and every report states what the design could not have flagged: on the τ²-bench pair no change to a single task could have been flagged at all, since a lone changed input needs a p below 0.001 and the smallest p a 4-against-4 table can give is 0.0286; at least 29 tasks would have to flip together.

The fix check reads the outcome alone, so it works on a run stored with no trace, and it matches nothing by seed on purpose. On the HumanEval pair, two runs of one unchanged agent 56 minutes apart, it reads different (p = 0.036, +0.47 points) while the inputs did not move as a body (by input, p = 0.127), and the flags name the one problem that broke, HumanEval/129; the step-clock comparison on the same pair read p = 0.109 with a limit of 1.113. That is a true statement about two runs and a false one about two versions, which is why every report closes by saying that a difference between two stored runs is a difference between the runs, and why the recipe says side by side.

A knob level in Tasks is a candidate until a model has been run against it. A card measures each level on 60 generated tasks from the eval split, 5 independent attempts at each, through the null harness, which gives the model the task's prompt and tool block and nothing else, with the decoding declared in full. Strict pass@1 is first in every table: the attempt reached the goal state exactly and ended with a reply, no stage past the goal, no other record touched; it is the mean over tasks of each task's pass rate, so every task weighs the same. Leaky pass@1, beside it, also accepts going past the goal or touching another record, and the overshoot share says how often that happened. Depth is read on reaching the goal, the leaky rate, with stopping shown beside it, because below the top level the strict rate also asks the model to stop at the goal and the two calibrated models do not fail those two things the same way.

The interval is a bootstrap that resamples tasks, each task's attempts moving together, 2,000 resamples at a fixed seed. Two neighbouring levels are distinguishable only when their paired interval excludes zero and they differ by at least 0.15, a resolution declared before any run; otherwise the card says the difference is too small for this design to call. A level at or below 0.05 or at or above 0.95 on the rate being compared is saturated and left out of the comparison. Each row also prints the median model calls, tokens and episode seconds; time is printed and never compared. Attempts that failed for a reason that is not the model's leave the denominator and are counted apart; a cell whose share of them is above 0.02 is not published. A cell in which more than 10% of attempts made no tool call is not published either, and neither is the card that holds it, because it would measure whether the model starts rather than how hard the level is. A card is pinned to the generator's hash and says so when it is stale. The rules in full are in how to read a card.

The code that makes every number here is Prove for the conditions, the seeds, the diff and the limit, Trace for the step clock and the fix check, and Tasks for the calibration card. All three are open source; Tasks is released, and Prove and Trace are not yet.