Run Prove on your machine

Install

pip install poisson-prove

Accounts and keys

None. The mock agent calls no model and the sweep runs in Prove's own process.

What a run costs

Stated by the plan: $0 (local). Spent: $0.

Tools

Prove, The quickstart, Backends

Source

docs/quickstart.md · the mock agent at 83c1f84

This run

poisson-prove 0.2.1 (Python 3.11.7), local, 2026-11-06. Stored output.

This is the quickstart run end to end on one machine. Prove sweeps a mock agent over three tasks and three tool failure rates, sweeps a patched version of it on the same seeds, compares the two, and writes the report. Nothing here needs a key, an account or the network.

Before you start: Install and Your machine.

1. Write the example’s spec

init --example writes prove.toml: the keyless mock agent over three tasks and three tool failure rates, 194 replicates of each of the nine conditions.

poisson-prove init --example

Output2 lines

ok  wrote prove.toml: a sweep of the keyless mock agent, 9 conditions x 194 replicates
  next: poisson-prove sweep

2. Sweep the agent

sweep reads ./prove.toml, prints the plan, and with --yes runs it without asking. The plan says what will run, what it costs (nothing, here), and the smallest change in one condition’s failure rate this design would catch 4 times in 5.

poisson-prove sweep --yes

Outputexcerpt, 2 of 15 lines

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
⋯ 8 lines
done  1,746 rollouts in run "mock-agent 2026-11-06 a" (complete) in 2.3 s
⋯ 5 lines

All output15 lines

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
  target  poisson_prove.examples.mock_agent:rollout (imports, takes cfg and seed)
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: local · target: mock-agent · t_unit: tool_calls
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
1,729/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-06 a" (complete) in 2.3 s
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  2.3 s
  next: poisson-prove sweep --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

3. Sweep the patched agent on the same seeds

The patched agent retries a failed tool call once. --pair-with latest gives its run the same seeds as the last one, so the two can be compared episode by episode.

poisson-prove sweep --yes --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

Outputexcerpt, 1 of 15 lines

⋯ 9 lines
done  1,746 rollouts in run "mock-agent 2026-11-06 b" (complete) in 1.2 s, paired with "mock-agent 2026-11-06 a"
⋯ 5 lines

All output15 lines

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
  target  poisson_prove.examples.mock_agent:rollout_patched (imports, takes cfg and seed)
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: local · target: mock-agent · t_unit: tool_calls
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
1,580/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-06 b" (complete) in 1.2 s, paired with "mock-agent 2026-11-06 a"
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  1.2 s
  next: poisson-prove diff

4. Compare the two runs

diff with no runs named compares the previous run with the latest. Each condition gets a verdict, with the smallest change the run could have flagged beside it.

poisson-prove diff

Outputexcerpt, 1 of 35 lines

trade  timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds)
⋯ 34 lines

All output35 lines

trade  timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  mock-agent 2026-11-06 a
  after  mock-agent 2026-11-06 b
  trade  task_id refund · tool_fail_rate 0.55: violated fell, timeout rose
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 2 rows that did not count have no limit on their fall (none detectable: 2), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 97.778%, false coverage (Benjamini-Yekutieli) over the 8 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  30 outcomes would change the word (the search is not exhaustive: it may be fewer): 30 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise

inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared

condition                               event     before  after  change  verdict    rise   fall
task_id checkout · tool_fail_rate 0     timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id checkout · tool_fail_rate 0     violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id checkout · tool_fail_rate 0.3   timeout   0.026   0.000  -0.026  unchanged  0.104  none detectable
task_id checkout · tool_fail_rate 0.3   violated  0.294   0.000  -0.294  safer      0.195  0.163
task_id checkout · tool_fail_rate 0.55  timeout   0.057   0.036  -0.021  unchanged  0.129  none detectable
task_id checkout · tool_fail_rate 0.55  violated  0.897   0.273  -0.624  safer      0.089  0.152
task_id refund · tool_fail_rate 0       timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id refund · tool_fail_rate 0       violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id refund · tool_fail_rate 0.3     timeout   0.428   0.036  -0.392  safer      0.200  0.189
task_id refund · tool_fail_rate 0.3     violated  0.325   0.010  -0.314  safer      0.198  0.170
task_id refund · tool_fail_rate 0.55    timeout   0.186   0.490  +0.304  worse      0.178  0.129
task_id refund · tool_fail_rate 0.55    violated  0.814   0.304  -0.510  safer      0.129  0.178
task_id search · tool_fail_rate 0       timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0       violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.3     timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.3     violated  0.191   0.005  -0.186  safer      0.180  0.131
task_id search · tool_fail_rate 0.55    timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.55    violated  0.773   0.227  -0.546  safer      0.143  0.186

  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound

  next: poisson-prove report --x task_id --y tool_fail_rate

5. Write the report

report writes the comparison as three files, with a heatmap of the change per condition.

poisson-prove report --x task_id --y tool_fail_rate

Outputexcerpt, 1 of 12 lines

ok  wrote prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b (3 files)
⋯ 11 lines

All output12 lines

ok  wrote prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b (3 files)
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/report.md
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/heatmap.svg
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/receipt.html
  saved  comparison e04d93fdf1b9

inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared
timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 18 tests, paired)
Trade: violated fell in 1 cell where timeout rose.

  next: poisson-prove ui

The comparison

The same comparison, drawn by this site from the run’s diff --json in the site’s verdict scale: cobalt where the patch made the agent fail significantly less often, ink with a hatch where it failed significantly more often, and the no-change cells in grey. Hover a cell for its rates and limit.

timeout · 1 safer, 1 worse, 7 no significant change · mock-agent 2026-11-06 a → mock-agent 2026-11-06 b

tool_fail_rate ↑ 0 · 0.3 · 0.55

+39pp · more often no significant change · blind under 6pp −39pp · less often

1 worse 7 no significant change, 2 of them blind at any size 1 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION

timeout: hover any condition, or focus the grid and use the arrow keys.

violated · 6 safer, 0 worse, 3 no significant change · mock-agent 2026-11-06 a → mock-agent 2026-11-06 b

tool_fail_rate ↑ 0 · 0.3 · 0.55

+62pp · more often no significant change · blind under 6pp −62pp · less often

0 worse 3 no significant change 6 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION

violated: hover any condition, or focus the grid and use the arrow keys.

Stored output

Every output above is what the command printed, stored in the site repository under src/data/examples/run-prove-on-your-machine/ with the run’s diff.json and the three files report wrote, as Prove 0.2.1 wrote them. The page cannot show an output that is not there.