Run on

GitHub Actions

In CI, Prove sweeps the pull request’s code on the same seeds as the main branch’s run and gates on the comparison. check is the gate: it exits 4 when a condition got significantly worse and 0 when none did, so the step fails the build. In CI is Prove’s own page on it.

Install

Nothing on your machine. The workflow installs Prove and your package on the runner, in the step pip install poisson-prove -e ..

Credentials

None for Prove. A model key your agent needs goes in the CI’s secrets, as for any test that calls the model.

Config

One workflow file. GitHub runs a workflow from any file under .github/workflows/. Prove’s docs give this one without a name, and prove.yml follows its name:.

name: prove
on: pull_request
jobs:
  prove:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install poisson-prove -e .
      - uses: actions/cache/restore@v4
        with:
          path: prove-runs
          key: prove-baseline-${{ github.event.pull_request.base.sha }}
      - run: poisson-prove sweep --yes --pair-with latest
      - run: poisson-prove check --event unresolved --event timeout --json > check.json
      - if: always()
        run: python make_comment.py check.json

Three things in it are yours. The events its check names are the outcomes of the docs’ own example agent, so name your agent’s failures. The baseline is a job on your main branch that sweeps and saves prove-runs with actions/cache/save, which the docs describe and do not give as a file. And make_comment.py is your script: In CI gives the lines that read check.json.

Check

Prove’s docs give no command that checks the workflow from your machine. What can be checked before the first pull request is the step the workflow gates on. These are the quickstart’s two runs, a baseline and a patched agent on the same seeds.

poisson-prove init --example

Output2 lines · exit 0

ok  wrote prove.toml: a sweep of the keyless mock agent, 9 conditions x 194 replicates
  next: poisson-prove sweep

poisson-prove sweep --yes

Outputexcerpt, 1 of 16 lines · exit 0

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
⋯ 15 lines

All output16 lines

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
  target  poisson_prove.examples.mock_agent:rollout (imports, takes cfg and seed)
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: local · target: mock-agent · t_unit: tool_calls
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
833/1,746
1,353/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-09 a" (complete) in 4.7 s
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  4.7 s
  next: poisson-prove sweep --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

poisson-prove sweep --yes --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

Outputexcerpt, 1 of 17 lines · exit 0

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
⋯ 16 lines

All output17 lines

plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on local, $0 (local)
  target  poisson_prove.examples.mock_agent:rollout_patched (imports, takes cfg and seed)
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: local · target: mock-agent · t_unit: tool_calls
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
677/1,746
1,153/1,746
1,551/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-09 b" (complete) in 3.6 s, paired with "mock-agent 2026-11-09 a"
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  3.6 s
  next: poisson-prove diff

A gate on time-outs fails on them, names the condition, and exits 4. On the runner the same exit code fails the job.

poisson-prove check --event timeout

Outputexcerpt, 1 of 12 lines · exit 4

failed  timeout: 1 worse, 1 safer in 9 conditions; fail on worse; failed in task_id refund · tool_fail_rate 0.55 (paired test on shared seeds)
⋯ 11 lines

All output12 lines

failed  timeout: 1 worse, 1 safer in 9 conditions; fail on worse; failed in task_id refund · tool_fail_rate 0.55 (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  comparison  trade
  before  mock-agent 2026-11-09 a
  after  mock-agent 2026-11-09 b
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 9 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 9 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 9 tests); 1 row that did not count has no limit on its fall (none detectable: 1), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 98.889%, false coverage (Benjamini-Yekutieli) over the 2 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  21 outcomes would change the word: 12 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise; 9 runs of A in task_id refund, tool_fail_rate 0.55 made to end in timeout

timeout: 1 safer, 1 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 9 tests, paired)

Each exit code means one thing, so a step can tell them apart: 0 passed, 4 failed, 1 refused (nothing was compared, such as an event the runs do not have), 2 a wrong command line, 3 an unexpected failure. Gate a pull request with Prove walks the gate through, passing and failing.