Tools · open source

Prove

Prove tells you whether a change to an AI agent really changed how often it fails.

It replays the same cases before and after the change, on the same seeds, and compares the two runs condition by condition. The verdict is safer, worse, trade or no significant change, with stated error bounds and detection limits.

It works alongside the tracer and the test runner you already have, uses no model to grade your agent, and needs no hosted account.

When no change counted and at least one condition could not have shown a change of the size you accept, the verdict is too small to tell instead. Before a sweep runs, its plan refuses a design that small and names the number of replicates that would do.

GitHub · PyPI · Docs · Changelog · Release notes

Poisson Prove 0.1.0, 41 seconds, no sound.

The video’s text

0:00$ pip install poisson-prove · poisson labs · prove 0.1.0 · Evals give you a score. Prove tells you what changed. · Same cases, same seeds, before and after a change. · A verdict per condition, and what it couldn't detect.
0:06Same agent. Same config. Same seeds. · It scored 45.0%, then 44.0%. · 44 of 200 tasks flipped. · τ²-bench airline · two runs back to back · one square per matched task, lit where the outcome flipped
0:12Poisson Prove · Find out what a change actually did. · Condition by condition: what got better, what got worse, what the run couldn't detect.
0:16compare: same conditions, same seeds, before and after · test: a paired test per condition, corrected across all of them · verdict: safer, worse, or no significant change · limit: the smallest change each condition could have caught · setup: your coding agent writes the adapter
0:21Fall rate by condition, before and after a retrain · Microduck walking policy · checkpoint 2250 → 5999 100 rollouts per condition, same seeds · falls, all 14 conditions: 0.505 → 0.208 · 8 safer · 5 worse · 1 no significant change · push 0.600 · friction 0.6: 0 → 16 of 100, worse · the unmoved one could have caught a rise of 0.123 · Better on average. Worse in 5 of 14 conditions. · paired test on shared seeds · Holm over 14 tests, 0.05
0:30Prove tells you when a gap is too small to tell. · HAL · 9 agent benchmarks · neighbouring ranks on the tasks they share · exact paired test, BH per benchmark, as published · AssistantBench: 0 of 29 · CORE-Bench Hard: 0 of 55 · GAIA: 0 of 33 · Online-Mind2Web: 0 of 22 · SciCode: 0 of 36 · ScienceAgentBench: 0 of 24 · SWE-bench Verified Mini: 0 of 25 · τ-bench Airline: 0 of 41 · USACO: 3 of 13 · 3 of 278 neighbouring pairs can be told apart.
0:36Fix or noise. Prove which.
0:38poisson labs · prove 0.1.0 · pip install poisson-prove · poissonlabs.ai/prove

Start

Needs Python 3.11 to 3.14, on macOS or Linux. The example needs no key and no account.

python3 -m venv .venv && source .venv/bin/activate
pip install poisson-prove
poisson-prove init --example
poisson-prove sweep --yes
poisson-prove sweep --yes \
    --target poisson_prove.examples.mock_agent:rollout_patched \
    --pair-with latest
poisson-prove diff

The example sweeps a mock agent, then a patched version of it on the same seeds, and compares the two. --yes answers the question each sweep asks after its plan.

The first run

The example reads trade.

$ poisson-prove diff
trade  timeout: … safer, … worse · violated: … safer, … worse (paired test
       on shared seeds); …
  method  paired test on shared seeds (McNemar), the default for runs
          paired at sweep time
  before  mock-agent <date> a
  after  mock-agent <date> b
  trade  …: violated fell, timeout rose
  verdict rule  …
  limits  …
  intervals  …
  fragility  …
…
  next: poisson-prove report --x task_id --y tool_fail_rate

trade means some conditions got safer and at least one got worse. The patched agent takes a forbidden fallback (violated) less often, and in one condition where tools fail most it runs out of tool calls (timeout) more often. Under the word, diff names the test, the two runs, the conditions that traded, the rule the word was decided by, each condition's detection limit, and an interval on each change. The next: line is the command to run next, whole.

The mock agent is a fixture for learning the commands, and its numbers measure nothing. They also move from one pair of sweeps to the next, so this page shows the verdict and the shape of the output and leaves the counts out.

How it works

Sweep

Every condition, many times, each episode on its own seed. A sweep after the change, paired with the first, gets the same seeds, so each episode is compared with its twin.

Compare

A test per condition and outcome. The flags are held to a false discovery rate (Benjamini–Hochberg), and the verdict counts only the changes that also hold at the family-wise level (Holm).

State the limit

Every condition that did not move says how large a change it would have caught 4 times in 5, computed for the test the comparison used, or that it could not have caught one at all.

The sweep spec

target = "poisson_prove.examples.mock_agent:rollout"
target_id = "mock-agent"
t_unit = "tool_calls"
replicates = 194

[axes.task_id]
type = "categorical"
values = ["search", "checkout", "refund"]

[axes.tool_fail_rate]
type = "continuous"
values = [0.0, 0.3, 0.55]

[declare]
failures = ["timeout", "violated"]

The example's prove.toml, as init --example writes it: 9 conditions, 194 episodes each.

The one function you write

from poisson_prove import Episode
from support_bot import TICKETS, handle  # your agent


def rollout(cfg, seed):
    ticket = TICKETS[cfg["ticket"]]
    reply = handle(ticket, temperature=cfg["temperature"],
                   seed=seed)
    if reply.timed_out:
        outcome = "timeout"
    elif reply.resolved:
        outcome = "resolved"
    else:
        outcome = "unresolved"
    return Episode(outcome=outcome, t=reply.steps, horizon=20,
                   meta={"case_id": cfg["ticket"]})

One condition and one seed in, an outcome and a time out. support_bot stands for your own agent.

Set up by your coding agent

init --agent writes a skill for your coding agent, which writes the setup for you to review.

$ poisson-prove init --agent
ok  wrote the setup for your coding agent (4 files)
  wrote  .agents/skills/poisson-prove-setup/SKILL.md
  wrote  .agents/skills/poisson-prove-setup/prompt.txt
  wrote  .claude/skills/poisson-prove-setup/SKILL.md
  wrote  .claude/skills/poisson-prove-setup/prompt.txt
  prompt  Use the poisson-prove-setup skill in this repository: set
          Poisson Prove up for me to review, then stop.
  next: give your coding agent the prompt above, then review what it
        wrote

The skill is written where Claude Code, Codex and Cursor look for skills. Given the prompt, your agent reads your code, writes prove_rollout.py and prove.toml, checks them with poisson-prove sweep --dry-run, which runs no episode, and stops with a list of what you need to confirm. You run the sweep yourself.

Prove makes no model call and reads no key. The coding agent is yours, and so is whatever it costs to run.

The terminal app and the dashboard

The terminal app and the dashboard read the same store.

poisson-prove with no verb, on a terminal, opens the terminal app: your runs, your comparisons and their verdicts, a sweep of the folder's prove.toml, and every condition and episode behind a verdict, all by keys. Every number on its screens is the one the commands print.

$ poisson-prove ui
ok  dashboard at file:///…/prove-runs/.poisson/ui/index.html

poisson-prove ui builds the dashboard: pages over the store that you open from disk, with nothing served and nothing loaded from the network. They show the runs, each comparison saved by report, serve or diff --save with its verdict and charts, each condition, and the episodes behind an outcome. In 0.1.0 the pages are static: once built, they are rebuilt by the commands that write to the store, and a page shows the store as it was when it was built.

Next: a live local dashboard over the same store, then setting up and running sweeps from either the terminal app or the dashboard.

The terminal app, at 80 × 24

The terminal app's home at 80 by 24: the store, the latest run, the saved comparison and its verdict trade, the next command, and every run and comparison — Home

The terminal app's comparison screen at 80 by 24: the verdict trade with its sentence, the method, the two runs, the condition that traded, and a count per event — The comparison

The terminal app's condition screen at 80 by 24 for refund at a 0.55 tool failure rate: timeout rose, each run's share with its interval, the change and its interval, the test, and the rise and fall it could have detected — A condition

The dashboard

The dashboard's page for one condition, refund at a 0.55 tool failure rate: each run's share with its interval, the change, the test and the limits — A condition

The dashboard's heatmap of timeout over the nine conditions: each cell's verdict, its change and its q — A chart: the timeout heatmap

One run of the keyless example, from a fresh install: the app drawn from its own terminal output, the pages as poisson-prove ui wrote them, the top of each. The agent is a mock, so its counts measure nothing, and they move from one pair of sweeps to the next.

Where it runs, and what leaves your machine

Prove never phones home.

Sweeps run on your machine by default. With pip install "poisson-prove[modal]" they run in parallel Modal containers on your own Modal account. Given the seconds an episode takes, the plan estimates the compute cost from Modal's prices before anything starts, and a cap you set refuses a sweep estimated above it; your model's tokens are not in the estimate. With pip install "poisson-prove[burla]" they run on your own Burla cluster, in your own AWS, Google Cloud or Azure account; the extra pins Burla 1.8.2, and this backend is experimental. Your rollout's code and conditions go to the account you choose, and the episodes come back to the same local store.

Beyond the backend you choose, Prove sends nothing anywhere: no telemetry, no analytics, no update check. poisson-prove serve listens on 127.0.0.1 only.

What the numbers do not claim

It compares two runs, under the conditions you swept.

no significant change means no change counted, and every condition could have caught one of the size you accepted. A smaller change can still be there.

Prove does not claim that the change caused anything: it compares two runs. It says nothing about conditions you did not sweep, and it cannot check that an outcome label means what you meant by it. It is not a benchmark, an agent framework or an observability tool, and it never edits your code.

On an agent

1 higher 327 no significant change, 49 of them blind at any size 0 lower coloured only at q<0.05, corrected across every condition

failed_assertion: 0 lower, 1 higher, 327 no significant change over 328 conditions. Hover any point.

Fig. 1One code agent against itself, nothing changed: 328 conditions, 32 episodes each, McNemar's test per condition. The verdict is higher: one condition, HumanEval/129 at 0.7, failed more often, a problem that broke between the runs. Of the other 327, the weakest that has a limit could only have detected a 50pp change, and 49 could not have detected one of any size. This is the floor a real change has to clear. report.md · receipt.html

On a robot policy

0 higher 340 no significant change, 72 of them blind at any size 60 lower coloured only at q<0.05, corrected across every condition

PUSH, % BODYWEIGHT → ↑ FRICTION μ 32 TRIALS PER CELL PER VERSION

Go1 locomotion, v0 → v1. Hover any cell, or focus the grid and use the arrow keys.

Fig. 2Go1 locomotion, before and after a retrain: 400 conditions, 32 trials each, Fisher's exact test per condition. 60 conditions fell significantly less often and 0 more often; 34 of the 60 also hold at the family-wise level, which is what the verdict, lower, counts. Of the 340 with no significant change, the weakest that has a limit could only have detected a 47pp change, and 72 could not have detected one of any size. report.md · receipt.html

5 higher 1 no significant change 8 lower coloured only at q<0.05, corrected across every condition

LATERAL PUSH, m/s → ↑ FRICTION μ 100 TRIALS PER CELL PER VERSION

Microduck walking, checkpoint 2250 → checkpoint 5999. Hover any cell.

Fig. 3A biped at two checkpoints: 14 conditions, 100 trials each, McNemar's test per condition. The verdict is mixed: falls went down in 8 conditions and up in 5, all 13 at the family-wise level too, and the one condition with no significant change could have detected a change of 0.123. The later checkpoint is better under a hard push and worse under a gentle one, which an average would hide. report.md · receipt.html

2026.09
Where a 14-DoF Biped Falls Two checkpoints of a Microduck walking policy under a lateral push: a cliff and a slope, with a crossover between 0.600 and 0.625 m/s. A retrain meant to widen the boundary was worse in 14 of 14 cells. Microducklocomotion read report receipt retrain report
2026.08
Map the Failure Boundary A Go1 locomotion policy across 400 conditions of floor friction and lateral push. Retrained on the two gaps the map exposed: 60 conditions significantly safer, none worse. Go1locomotion read report receipt code

Each figure is its study's stored pair compared by Prove 0.1.0, with the test the study published. The runs were made before outcomes could be declared as failures, so Prove names the directions lower and higher, and a comparison with both mixed; for a declared failure it says safer, worse and trade.

Empirical evidence under a stated test envelope, not formal verification. Conditions outside the envelope were not tested; conditions inside it can only resolve effects at or above the reported MDE.

The release

Prove 0.1.0

Released 2026-10-29, from 3dc29cc, tagged v0.1.0. Open source, under the MIT licence.

Changelog entry · Release notes · github.com/poisson-labs/prove · pypi.org/project/poisson-prove · Docs