Poisson Labs · Methods
How we count.
Every result on this site is a comparison between two runs of the same system, and every one is stated with its limit. This page is how those numbers are made. The worked example throughout is the Go1 study: 400 conditions, 32 trials in each, 12,800 rollouts a side.
The point of all of it
“Nothing got worse” is not a finding until you say what the test could have seen.
In the Go1 study, 0 of 400 conditions got significantly worse after the retrain. That sentence is true, and on its own it is nearly empty. With 32 trials in a cell, a small rise in the fall rate looks exactly like no rise at all.
The least-powered cell that showed no significant change could not have detected a rise under 34pp. So the claim is: nothing got worse by more than that. Below it, the run was blind, and the report says so on the same line.
one run, no comparison — this scale carries no verdict
Fig. 1Not the result: the blind spot. The same 400 Go1 conditions, shaded by the smallest effect each one could have detected. Darker is blinder. The worst is friction 0.95, push 100%, n=32. Achromatic on purpose: this is a magnitude, not a verdict. report.md
Seeds, and when the matching is used
Every condition is run many times, and each of those runs has a seed. The seeds are derived from one salt, so a second run with the same salt gets the same seeds. The third trial of a condition before the change and the third trial of that condition after it start from the same random draw. The system is compared with itself, not with a fresh draw of luck.
Prove can use that matching (a paired test) or ignore it (the default). The matching only helps when a seed tends to do the same thing both times. For the Go1 walking policy it does not, because contact physics on a GPU amplifies tiny differences, so the default loses nothing. For an agent on a sampling model the seeds do less still: the same seeds do not give the same run. So the first thing we measure is the floor. We ran one code agent twice with nothing changed: 328 conditions, 32 episodes each. 1 came back significantly worse and 0 safer. A real change has to clear that.
1 worse 327 no significant change 0 safer coloured only at q<0.05, corrected across every condition
A correction across every condition
Each condition gets its own test. Test 400 conditions at p < 0.05 and about 20 will pass by luck alone, with nothing changed. A map with 20 green squares on it would look like a finding.
So the tests are corrected together, across every condition in the run, with the Benjamini–Hochberg procedure at q < 0.05. It controls the share of the conditions we call changed that are false alarms. A condition is coloured only if it survives that.
Which test sits underneath matters, and we say which. On the Go1 runs, Fisher's exact test calls 60 conditions safer, McNemar's test calls 55 conditions safer and the log-rank test calls 88 conditions safer. All three call 0 worse. The published figure uses the first; the spread is in the report, because a reader should see how much the answer depends on the choice.
The smallest effect a condition could detectMDE, the “blind under” figure
Every verdict carries a second number: the smallest change that condition had the power to detect, given how many trials it had and how often the system was already failing there. It is printed per condition, and the worst one is printed in the headline.
It works before the run as well as after. Prove computes what a design can resolve before it spends anything, and refuses to start one that cannot resolve the effect you asked about.
On every figure here it appears in the legend as “blind under N pp”. That is the figure the top of this page is about.
Runs that did not finish
“Fell at step 40” and “was still standing when the clock ran out” are different kinds of number. Averaging them is wrong. A run that outlives the test is censored: we know it lasted at least that long and nothing more, and the estimate treats it that way.
Agents add a second problem: a run can end by completing, by breaking a rule, or by timing out, and those compete. Each is counted against the others rather than alone.
ScopeWhat a result here does and does not claim.
Empirical evidence under a stated test envelope, not formal verification. Conditions outside the envelope were not tested; conditions inside it can only resolve effects at or above the reported MDE.
- It is a map of the conditions that were named.
- Nothing here says a system is safe. A result covers the conditions in the run and is silent about every condition outside them. Sweeping the wrong axes carefully gives a precise answer to the wrong question, and choosing the axes stays with the people who know the system.
- It is about the system that was tested, where it was tested.
- The Go1 result is one policy in the simulator it was trained in. It is not a claim about a robot on a floor. An agent result is about that agent, that model snapshot and those tasks.
- Finding and confirming are kept apart.
- Anything that goes looking for failures chooses where to look, and a result found that way is a lead. It becomes a finding when it is run again on fresh seeds, with the conditions declared before the run.
- What Prove is not.
- Not a simulator, an environment framework, a benchmark, a training loop or an observability product. It never reaches into your process and it never phones home. It brings the conditions, the seeds, the tests and the limit, and nothing else.
The code that makes every number here is Prove: open source, not released yet.