News · 2026-10-29

Poisson Prove is out

Whether a change to your agent changed how often it fails, condition by condition, with error bounds and the smallest change each could have shown.

Poisson Prove 0.1.0, 41 seconds, no sound.

The video’s text

0:00$ pip install poisson-prove · poisson labs · prove 0.1.0 · Evals give you a score. Prove tells you what changed. · Same cases, same seeds, before and after a change. · A verdict per condition, and what it couldn't detect.
0:06Same agent. Same config. Same seeds. · It scored 45.0%, then 44.0%. · 44 of 200 tasks flipped. · τ²-bench airline · two runs back to back · one square per matched task, lit where the outcome flipped
0:12Poisson Prove · Find out what a change actually did. · Condition by condition: what got better, what got worse, what the run couldn't detect.
0:16compare: same conditions, same seeds, before and after · test: a paired test per condition, corrected across all of them · verdict: safer, worse, or no significant change · limit: the smallest change each condition could have caught · setup: your coding agent writes the adapter
0:21Fall rate by condition, before and after a retrain · Microduck walking policy · checkpoint 2250 → 5999 100 rollouts per condition, same seeds · falls, all 14 conditions: 0.505 → 0.208 · 8 safer · 5 worse · 1 no significant change · push 0.600 · friction 0.6: 0 → 16 of 100, worse · the unmoved one could have caught a rise of 0.123 · Better on average. Worse in 5 of 14 conditions. · paired test on shared seeds · Holm over 14 tests, 0.05
0:30Prove tells you when a gap is too small to tell. · HAL · 9 agent benchmarks · neighbouring ranks on the tasks they share · exact paired test, BH per benchmark, as published · AssistantBench: 0 of 29 · CORE-Bench Hard: 0 of 55 · GAIA: 0 of 33 · Online-Mind2Web: 0 of 22 · SciCode: 0 of 36 · ScienceAgentBench: 0 of 24 · SWE-bench Verified Mini: 0 of 25 · τ-bench Airline: 0 of 41 · USACO: 3 of 13 · 3 of 278 neighbouring pairs can be told apart.
0:36Fix or noise. Prove which.
0:38poisson labs · prove 0.1.0 · pip install poisson-prove · poissonlabs.ai/prove

We ran the same τ²-bench airline agent twice, with the same config and the same seeds. It scored 45.0% and then 44.0%, and 44 of its 200 task outcomes came out differently. A one-point move after a change to an agent can be that kind of noise, and a pass rate on its own does not say which.

Poisson Prove is built for that question. It replays the same cases before and after a change, on the same seeds, and compares the two runs condition by condition. A condition is one combination of the things you vary, such as the case, the model or how often a tool fails. For each condition Prove says whether failures went up, went down, or did not move by more than that condition could have detected. The comparison then gets one verdict, safer, worse, trade or no significant change, with stated error bounds and detection limits.

When the runs are too small to tell, Prove says too small to tell and does not report the comparison as no change. Before a sweep runs, its plan states the smallest change one condition could be flagged for, and it refuses a design too small for the change you accept, naming the number of replicates that would do.

The first run needs no key:

pip install poisson-prove
poisson-prove init --example
poisson-prove sweep --yes
poisson-prove sweep --yes --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest
poisson-prove diff

The example sweeps a mock agent and a patched version of it, 194 episodes in each of nine conditions, and the comparison reads trade: the patch makes one kind of failure rarer and, in one condition where tools fail most, another more common. The mock agent is there to learn the commands on, and its numbers measure nothing.

For your own agent you write one function that runs one case and says how it ended. Or run poisson-prove init --agent, which writes a skill and a one-line prompt for your coding agent. The agent reads your code, writes the function and the sweep spec, checks them with a dry run that runs no episode, and stops for you to review. Prove itself makes no model call.

You read the results in a terminal app, which poisson-prove opens with no verb, or in a dashboard of static pages built from the store. A live dashboard comes next. Sweeps run on your machine, in Modal containers on your own account, or on your own Burla cluster in your own AWS, Google Cloud or Azure account, which is experimental. Prove never phones home.

We have used Prove in our own research. In the study linked below, the later of two checkpoints of a walking policy was safer in 8 of 14 conditions and worse in 5, which its average fall rate hides.

Prove needs Python 3.11 to 3.14, on macOS or Linux. It is open source, under the MIT licence.

Changelog entry Where a 14-DoF Biped Falls Prove docs github.com/poisson-labs/prove pypi.org/project/poisson-prove