Gate a pull request with Prove

Install

pip install poisson-prove

Accounts and keys

None to run it here: the mock agent calls no model. In CI, GitHub Actions on your repository; a model key your own agent needs goes in the CI's secrets.

What a run costs

Stated by the plan: $0 (local). Spent: $0.

Tools

Prove, check, In CI, The quickstart

Source

docs/recipes/in-ci.md · docs/commands/check.md · docs/quickstart.md at 83c1f84

This run

poisson-prove 0.2.1 (Python 3.11.7), local, 2026-11-07. Stored output.

check is Prove’s gate. It compares a baseline run with a candidate over the events you name and exits 4 when a condition got significantly worse, 0 when none did, so a CI step can stop the build on it. This example runs the gate on your machine first, on the quickstart’s two runs, the way In CI does: one gate passes and one fails. Then it shows the workflow that runs the same gate on a pull request, in full, as Prove’s docs give it.

Before you start: Install and Your machine.

1. Make a baseline and a candidate

A gate needs two runs on the same seeds. init --example writes the keyless mock agent’s spec; the first sweep is the baseline, and the patched agent swept with --pair-with latest is the candidate, as on the quickstart.

poisson-prove init --example

Output2 lines · exit 0

ok  wrote prove.toml: a sweep of the keyless mock agent, 9 conditions x 194 replicates
  next: poisson-prove sweep

poisson-prove sweep --yes

Outputexcerpt, 2 of 18 lines · exit 0

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
⋯ 11 lines
done  1,746 rollouts in run "mock-agent 2026-11-07 a" (complete) in 6.3 s
⋯ 5 lines

All output18 lines

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
  target  poisson_prove.examples.mock_agent:rollout (imports, takes cfg and seed)
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: local · target: mock-agent · t_unit: tool_calls
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
653/1,746
961/1,746
1,290/1,746
1,601/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-07 a" (complete) in 6.3 s
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  6.3 s
  next: poisson-prove sweep --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

poisson-prove sweep --yes --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

Outputexcerpt, 1 of 19 lines · exit 0

⋯ 13 lines
done  1,746 rollouts in run "mock-agent 2026-11-07 b" (complete) in 6.2 s, paired with "mock-agent 2026-11-07 a"
⋯ 5 lines

All output19 lines

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
  target  poisson_prove.examples.mock_agent:rollout_patched (imports, takes cfg and seed)
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: local · target: mock-agent · t_unit: tool_calls
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
385/1,746
769/1,746
1,089/1,746
1,334/1,746
1,554/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-07 b" (complete) in 6.2 s, paired with "mock-agent 2026-11-07 a"
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  6.2 s
  next: poisson-prove diff

2. A gate that passes

check with no runs named compares the previous run with the latest. A gate on violated passes: the patch made the agent violate its rules significantly less often in six conditions and more often in none. The exit code is 0.

poisson-prove check --event violated

Outputexcerpt, 2 of 12 lines · exit 0

passed  violated: 0 worse, 6 safer in 9 conditions; fail on worse (paired test on shared seeds)
⋯ 1 line
  comparison  safer
⋯ 9 lines

All output12 lines

passed  violated: 0 worse, 6 safer in 9 conditions; fail on worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  comparison  safer
  before  mock-agent 2026-11-07 a
  after  mock-agent 2026-11-07 b
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 9 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 9 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 9 tests)
  intervals  on each condition's change: Tango at 96.667%, false coverage (Benjamini-Yekutieli) over the 6 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  7 outcomes would change the word: 7 runs of B in task_id refund, tool_fail_rate 0.0 made to end in violated

violated: 6 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 9 tests, paired)

3. A gate that fails

A gate on timeout fails. The patch retries a failed tool call, so where tools fail most it runs out of tool calls more often, and check names that condition. The exit code is 4, which fails a CI step.

poisson-prove check --event timeout

Outputexcerpt, 1 of 12 lines · exit 4

failed  timeout: 1 worse, 2 safer in 9 conditions; fail on worse; failed in task_id refund · tool_fail_rate 0.55 (paired test on shared seeds)
⋯ 11 lines

All output12 lines

failed  timeout: 1 worse, 2 safer in 9 conditions; fail on worse; failed in task_id refund · tool_fail_rate 0.55 (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  comparison  trade
  before  mock-agent 2026-11-07 a
  after  mock-agent 2026-11-07 b
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 9 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 9 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 9 tests); 1 row that did not count has no limit on its fall (none detectable: 1), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 98.333%, false coverage (Benjamini-Yekutieli) over the 3 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  34 outcomes would change the word: 20 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise; 14 runs of A in task_id refund, tool_fail_rate 0.55 made to end in timeout

timeout: 2 safer, 1 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 9 tests, paired)

Each exit code means one thing, so a CI step can tell them apart: 0 passed, 4 failed, 1 refused (nothing was compared, such as an event the runs do not have), 2 a wrong command line, 3 an unexpected failure.

4. The result for a comment

With --json, check prints the same result as one document, with passed, the verdict, its sentence and every condition. Written to a file, it is what a bot turns into the comment on the pull request. Nothing prints on the terminal, and the exit code is still 4.

poisson-prove check --event timeout --json > check.json

Outputnone on the terminal · exit 4

5. The workflow

This is the GitHub Actions workflow In CI gives, in full. GitHub runs a workflow from any file under .github/workflows/; the docs give it without a name, and prove.yml follows its name:. GitHub Actions is the setup page for it.

.github/workflows/prove.yml

name: prove
on: pull_request
jobs:
  prove:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install poisson-prove -e .
      - uses: actions/cache/restore@v4
        with:
          path: prove-runs
          key: prove-baseline-${{ github.event.pull_request.base.sha }}
      - run: poisson-prove sweep --yes --pair-with latest
      - run: poisson-prove check --event unresolved --event timeout --json > check.json
      - if: always()
        run: python make_comment.py check.json

On a pull request it installs Prove and your package, restores the main branch’s runs from the cache, sweeps the pull request’s code paired with them (--yes, because CI is piped), and gates. Three things in it are yours, not Prove’s:

• The events. Its check names unresolved and timeout, the outcomes of the support bot on Your own agent. Name your agent’s own failures; an event the runs do not have is refused with exit 1.
• The baseline. The main branch’s job is the same without the pull request’s steps: it sweeps and saves prove-runs with actions/cache/save under its own commit. Keep the spec, the target and the replicates the same on both sides, or the runs will not have the same conditions to compare.
• The comment. make_comment.py is your script. In CI gives the few lines that read check.json and print the comment.

A passing gate says no condition counted as worse at the confidence the test states, and each condition’s limit says how large a rise it would have caught 4 times in 5. It does not say nothing got worse.

The comparison

The comparison the gates read, drawn by this site from the run’s diff --json in the site’s verdict scale: cobalt where the patch made the agent fail significantly less often, ink with a hatch where it failed significantly more often, and the no-change cells in grey. The hatched cell is the one the timeout gate failed on. Hover a cell for its rates and limit.

timeout · 2 safer, 1 worse, 6 no significant change · mock-agent 2026-11-07 a → mock-agent 2026-11-07 b

tool_fail_rate ↑ 0 · 0.3 · 0.55

+44pp · more often no significant change · blind under 6pp −44pp · less often

1 worse 6 no significant change, 1 of them blind at any size 2 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION

timeout: hover any condition, or focus the grid and use the arrow keys.

violated · 6 safer, 0 worse, 3 no significant change · mock-agent 2026-11-07 a → mock-agent 2026-11-07 b

tool_fail_rate ↑ 0 · 0.3 · 0.55

+59pp · more often no significant change · blind under 6pp −59pp · less often

0 worse 3 no significant change 6 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION

violated: hover any condition, or focus the grid and use the arrow keys.

Stored output

Every output above is what the command printed, with the exit code it returned, stored in the site repository under src/data/examples/gate-a-pull-request-with-prove/ with the run’s diff.json, as Prove 0.2.1 wrote them. The workflow is stored there too, byte for byte as docs/recipes/in-ci.md gives it at Prove’s commit, with its sha256. The page cannot show an output that is not there.