Prove

Prove runs your system across the conditions you name, before and after a change, on the same seeds. It reports which conditions got significantly safer, which got significantly worse, and the smallest effect each condition could have detected.

You bring one function that runs your system once. Prove brings the sweep, the seeds, the store and the statistics. It never loads your agent or your simulator itself.

Install

pip install poisson-prove

Prove is in private preview; the package is not on PyPI yet. The command it installs is poisson-prove.

The loop

# 1. sweep the conditions, before the change
poisson-prove sweep quickstart.toml \
    --target poisson_prove.examples.mock_agent:rollout --yes

# 2. sweep again after it, on the same seeds
poisson-prove sweep quickstart.toml \
    --target ...mock_agent:rollout_patched --pair-with <run-a> --yes

# 3. diff the pair into report.md, heatmap.svg, receipt.html
poisson-prove report <run-a> <run-b> \
    --event violated --x tool_fail_rate --y task_id

# in CI: exit 1 if any condition came back significantly worse
poisson-prove check <run-a> <run-b> --event violated --fail-on worse

quickstart.toml ships with the tool and needs no API key and no network.

The one function you write

import poisson_prove

def rollout(cfg: dict, seed: int) -> poisson_prove.Episode:
    result = run_my_system(task=cfg["task_id"], seed=seed)
    return poisson_prove.Episode(
        outcome=result.terminal_state,  # your label
        t=result.steps,
        horizon=20,
    )

Prove hands it one condition and one seed, and takes back an outcome and a time.

What a report says

For each condition: the event rate in each run, a test of the difference corrected across every condition, a verdict (safer, worse, or no significant change), and the smallest effect that condition could have detected. “None worse” is a claim only above that number.

Before a sweep starts, Prove works out the smallest effect the design can detect and refuses a design that cannot resolve the effect you asked for.

Looking at a pair

poisson-prove --store ./prove-runs serve A B --event NAME --x AXIS

serve holds a read-only page for one stored pair on a local port. Reading a serve page explains each section.

Where it runs

--backend local (the default) or modal, or a cluster in your own cloud account, set up during onboarding. The run’s manifest records which.

More

• Defining outcomes: what an outcome label must mean before the first run
• What capture stores: what is kept from a trace, and what is not
• Methods: the statistics, in plain words
• Changelog