Skip to content
Poisson Labs Docs
GitHub poissonlabs.ai

Examples

Check a change on runs you already have

Every Prove command but sweep reads runs that already exist. A store (./prove-runs by default, or --store PATH) holds the runs earlier sweeps wrote, and runs, diff, check and report read them without running anything. This example starts from the store Run Prove on your machine left behind: two runs of the keyless mock agent on the same seeds, before and after a patch. The commands are the ones runs, diff, check and the Python API document.

Before you start: Install and Your machine.

1. See what the store holds

Section titled “1. See what the store holds”

runs lists every run, newest first, with its number, name, id and row counts, so the next command can name one. latest and previous name the two newest complete runs.

Terminal window
poisson-prove runs
Outputexcerpt, 4 of 5 lines
ok  2 runs in prove-runs
⋯ 1 line
#  name                     id                     status    backend  target      rows   rows without an outcome  created
1  mock-agent 2026-11-06 b  20261106T205551Z-cc11  complete  local    mock-agent  1,746  0                        2026-11-06T20:55:51Z
2  mock-agent 2026-11-06 a  20261106T205547Z-95a7  complete  local    mock-agent  1,746  0                        2026-11-06T20:55:47Z
All output5 lines
ok  2 runs in prove-runs


#  name                     id                     status    backend  target      rows   rows without an outcome  created
1  mock-agent 2026-11-06 b  20261106T205551Z-cc11  complete  local    mock-agent  1,746  0                        2026-11-06T20:55:51Z
2  mock-agent 2026-11-06 a  20261106T205547Z-95a7  complete  local    mock-agent  1,746  0                        2026-11-06T20:55:47Z

2. Compare the two runs

Section titled “2. Compare the two runs”

diff with two runs named compares them in that order, baseline then candidate. Each condition gets a verdict, with the smallest change the run could have flagged beside it.

Terminal window
poisson-prove diff previous latest
Outputexcerpt, 3 of 35 lines
trade  timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds)
⋯ 1 line
  before  mock-agent 2026-11-06 a
  after  mock-agent 2026-11-06 b
⋯ 31 lines
All output35 lines
trade  timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  mock-agent 2026-11-06 a
  after  mock-agent 2026-11-06 b
  trade  task_id refund · tool_fail_rate 0.55: violated fell, timeout rose
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 2 rows that did not count have no limit on their fall (none detectable: 2), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 97.778%, false coverage (Benjamini-Yekutieli) over the 8 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  30 outcomes would change the word (the search is not exhaustive: it may be fewer): 30 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise


inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared


condition                               event     before  after  change  verdict    rise   fall
task_id checkout · tool_fail_rate 0     timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id checkout · tool_fail_rate 0     violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id checkout · tool_fail_rate 0.3   timeout   0.026   0.000  -0.026  unchanged  0.104  none detectable
task_id checkout · tool_fail_rate 0.3   violated  0.294   0.000  -0.294  safer      0.195  0.163
task_id checkout · tool_fail_rate 0.55  timeout   0.057   0.036  -0.021  unchanged  0.129  none detectable
task_id checkout · tool_fail_rate 0.55  violated  0.897   0.273  -0.624  safer      0.089  0.152
task_id refund · tool_fail_rate 0       timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id refund · tool_fail_rate 0       violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id refund · tool_fail_rate 0.3     timeout   0.428   0.036  -0.392  safer      0.200  0.189
task_id refund · tool_fail_rate 0.3     violated  0.325   0.010  -0.314  safer      0.198  0.170
task_id refund · tool_fail_rate 0.55    timeout   0.186   0.490  +0.304  worse      0.178  0.129
task_id refund · tool_fail_rate 0.55    violated  0.814   0.304  -0.510  safer      0.129  0.178
task_id search · tool_fail_rate 0       timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0       violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.3     timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.3     violated  0.191   0.005  -0.186  safer      0.180  0.131
task_id search · tool_fail_rate 0.55    timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.55    violated  0.773   0.227  -0.546  safer      0.143  0.186


  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound


  next: poisson-prove report "mock-agent 2026-11-06 a" "mock-agent 2026-11-06 b" --x task_id --y tool_fail_rate

3. Gate on one event

Section titled “3. Gate on one event”

check is the same comparison as a gate: it reads the events you name and exits 0 when the comparison passes and 4 when it fails, so a pipeline can stop on a change for the worse. A gate on violated passes here.

Terminal window
poisson-prove check --event violated
Outputexcerpt, 1 of 12 lines
passed  violated: 0 worse, 6 safer in 9 conditions; fail on worse (paired test on shared seeds)
⋯ 11 lines
All output12 lines
passed  violated: 0 worse, 6 safer in 9 conditions; fail on worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  comparison  safer
  before  mock-agent 2026-11-06 a
  after  mock-agent 2026-11-06 b
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 9 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 9 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 9 tests)
  intervals  on each condition's change: Tango at 96.667%, false coverage (Benjamini-Yekutieli) over the 6 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  7 outcomes would change the word: 7 runs of B in task_id refund, tool_fail_rate 0.0 made to end in violated


violated: 6 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 9 tests, paired)

A gate on timeout fails, names the condition, and exits 4.

Terminal window
poisson-prove check --event timeout
Outputexcerpt, 1 of 12 lines
failed  timeout: 1 worse, 1 safer in 9 conditions; fail on worse; failed in task_id refund · tool_fail_rate 0.55 (paired test on shared seeds)
⋯ 11 lines
All output12 lines
failed  timeout: 1 worse, 1 safer in 9 conditions; fail on worse; failed in task_id refund · tool_fail_rate 0.55 (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  comparison  trade
  before  mock-agent 2026-11-06 a
  after  mock-agent 2026-11-06 b
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 9 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 9 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 9 tests); 2 rows that did not count have no limit on their fall (none detectable: 2), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 98.889%, false coverage (Benjamini-Yekutieli) over the 2 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  29 outcomes would change the word: 23 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise; 6 runs of A in task_id refund, tool_fail_rate 0.55 made to end in timeout


timeout: 1 safer, 1 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 9 tests, paired)

4. The same from Python

Section titled “4. The same from Python”

poisson_prove.compare gives the result objects the command line prints: diff returns a Comparison, check a CheckResult with passed. The script is the one on the Python API.

compare_runs.py
from poisson_prove import compare


result = compare.diff("prove-runs")
print(result.verdict, result.method, result.method_chosen)
for row in result.rows[:3]:
    print(row["event"], row["config"], row["delta"], row["ci_lo"], row["ci_hi"])


gate = compare.check("prove-runs", "previous", "latest", events=["violated"], fail_on="any")
print(gate.verdict, gate.passed)
Terminal window
python compare_runs.py
Outputexcerpt, 2 of 5 lines
trade mcnemar_bh default
⋯ 3 lines
failed False
All output5 lines
trade mcnemar_bh default
timeout {'task_id': 'checkout', 'tool_fail_rate': 0.3} -0.02577319587628866 -0.06674551352597285 0.0011459292273588821
timeout {'task_id': 'refund', 'tool_fail_rate': 0.3} -0.3917525773195876 -0.4793931425951903 -0.3024518865527003
timeout {'task_id': 'refund', 'tool_fail_rate': 0.55} 0.30412371134020616 0.2052539219628603 0.399258153685234
failed False

5. Write the report

Section titled “5. Write the report”

report writes the comparison as three files, from the same two runs.

Terminal window
poisson-prove report --x task_id --y tool_fail_rate
Outputexcerpt, 1 of 12 lines
ok  wrote prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b (3 files)
⋯ 11 lines
All output12 lines
ok  wrote prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b (3 files)
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/report.md
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/heatmap.svg
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/receipt.html
  saved  comparison e04d93fdf1b9


inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared
timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 18 tests, paired)
Trade: violated fell in 1 cell where timeout rose.


  next: poisson-prove ui

The comparison

Section titled “The comparison”

The comparison, drawn by this site from the run’s diff --json in the site’s verdict scale: cobalt where the patch made the agent fail significantly less often, ink with a hatch where it failed significantly more often, and the no-change cells in grey. Hover a cell for its rates and limit.

timeout · 1 safer, 1 worse, 7 no significant change · mock-agent 2026-11-06 a → mock-agent 2026-11-06 b
tool_fail_rate ↑ 0 · 0.3 · 0.55

1 worse 7 no significant change, 2 of them blind at any size 1 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION
timeout: hover any condition, or focus the grid and use the arrow keys.
violated · 6 safer, 0 worse, 3 no significant change · mock-agent 2026-11-06 a → mock-agent 2026-11-06 b
tool_fail_rate ↑ 0 · 0.3 · 0.55

0 worse 3 no significant change 6 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION
violated: hover any condition, or focus the grid and use the arrow keys.

Stored output

Section titled “Stored output”

Every output above is what the command printed, stored in the site repository under src/data/examples/check-a-change-on-runs-you-already-have/ with the run’s diff.json and the three files report wrote, as Prove 0.2.1 wrote them. The page cannot show an output that is not there.