Tools · open source

Prove

Did the change move the failure rate, and where?

Name the conditions. Prove runs your system across them before and after a change, on the same seeds, and says which conditions got safer, which got worse, and what was too small for the run to detect.

Open source, in private preview. Docs · Talk to us

The commands you run

pip install poisson-prove   # private preview: not public yet

# 1. sweep the conditions, before the change
poisson-prove sweep quickstart.toml \
    --target poisson_prove.examples.mock_agent:rollout --yes

# 2. sweep again after it, on the same seeds
poisson-prove sweep quickstart.toml \
    --target ...mock_agent:rollout_patched --pair-with <run-a> --yes

# 3. diff the pair into report.md, heatmap.svg, receipt.html
poisson-prove report <run-a> <run-b> \
    --event violated --x tool_fail_rate --y task_id

# in CI: exit 1 if any condition came back significantly worse
poisson-prove check <run-a> <run-b> --event violated --fail-on worse

How it works

Sweep

Every condition, many times, each run on a seed. A second sweep gets the same seeds.

Diff

A test per condition, corrected across all of them. Safer, worse, or no significant change.

State the limit

Every verdict carries the smallest effect that condition could have detected.

The sweep spec you write

target_id = "mock-agent@v0"
t_unit = "tool_calls"
replicates = 100

[axes.task_id]
type = "categorical"
values = ["search", "checkout", "refund"]

[axes.tool_fail_rate]
type = "continuous"
values = [0.0, 0.3, 0.55]

quickstart.toml: 9 conditions, 100 replicates each. No API key, no network.

The one function you write

import poisson_prove

def rollout(cfg: dict, seed: int) -> poisson_prove.Episode:
    result = run_my_system(task=cfg["task_id"], seed=seed)
    return poisson_prove.Episode(
        outcome=result.terminal_state,  # your label
        t=result.steps,
        horizon=20,
    )

One condition and one seed in, an outcome and a time out.

On an agent

1 worse 327 no significant change 0 safer coloured only at q<0.05, corrected across every condition

failed_assertion: 0 safer, 1 worse, 327 no significant change over 328 conditions. Hover any point.

Fig. 1One code agent against itself, nothing changed: 328 conditions, 32 episodes each. 327 show no significant change; 1 is worse, HumanEval/129 at 0.7, a problem that broke between the runs. This is the floor a real change has to clear. report.md · receipt.html

On a robot policy

0 worse 340 no significant change 60 safer coloured only at q<0.05, corrected across every condition

PUSH, % BODYWEIGHT → ↑ FRICTION μ 32 TRIALS PER CELL PER VERSION

Go1 locomotion, v0 → v1. Hover any cell, or focus the grid and use the arrow keys.

Fig. 2Go1 locomotion, before and after a retrain: 400 conditions, 32 trials each. 60 significantly safer, 0 worse. The weakest unchanged condition could only have detected a 34pp rise. report.md · receipt.html

5 worse 1 no significant change 8 safer coloured only at q<0.05, corrected across every condition

LATERAL PUSH, m/s → ↑ FRICTION μ 100 TRIALS PER CELL PER VERSION

Microduck walking, checkpoint 2250 → checkpoint 5999. Hover any cell.

Fig. 3A biped at two checkpoints: 8 of 14 conditions safer, 5 worse. The later checkpoint is better under a hard push and worse under a gentle one, which an average would hide. report.md · receipt.html

2026.09
Where a 14-DoF Biped Falls Two checkpoints of a Microduck walking policy under a lateral push: a cliff and a slope, with a crossover between 0.600 and 0.625 m/s. A retrain meant to widen the boundary was worse in 14 of 14 cells. Microducklocomotion read report receipt retrain report
2026.08
Map the Failure Boundary A Go1 locomotion policy across 400 conditions of floor friction and lateral push. Retrained on the two gaps the map exposed: 60 conditions significantly safer, none worse. Go1locomotion read report receipt code

Empirical evidence under a stated test envelope, not formal verification. Conditions outside the envelope were not tested; conditions inside it can only resolve effects at or above the reported MDE.