Skip to content
Poisson Labs Docs
GitHub poissonlabs.ai

Examples

Run Prove on Modal

--backend modal runs a sweep’s rollouts in parallel containers on Modal, on your own account: Prove builds the image, sends the episodes in batches, and writes the rows to your local store as they come back. The comparison is the same as on one machine, because the seeds come from the run, not from where it ran. What leaves your machine is on Backends.

This run’s virtual environment was Python 3.11, the image’s default. The two must match: a serialized function cannot be defined with one Python and run in another.

Before you start: Install and Modal.

1. Write the example’s spec

Section titled “1. Write the example’s spec”

The same spec as on one machine: the keyless mock agent, nine conditions, 194 replicates each.

Terminal window
poisson-prove init --example
Output2 lines
ok  wrote prove.toml: a sweep of the keyless mock agent, 9 conditions x 194 replicates
  next: poisson-prove sweep

2. See the plan and its cost first

Section titled “2. See the plan and its cost first”

--dry-run prints the plan and stops. With --est-seconds the plan carries a cost estimate, from Modal’s prices as Prove carries them; --max-cost refuses the sweep before anything starts if the estimate is above it.

Terminal window
poisson-prove sweep --backend modal --dry-run --est-seconds 0.5 --max-cost 1
Outputexcerpt, 2 of 11 lines
plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on modal, ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
⋯ 8 lines
  Cost: ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
⋯ 1 line
All output11 lines
plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on modal, ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
  target  poisson_prove.examples.mock_agent:rollout
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: modal · target: mock-agent · t_unit: tool_calls
  Image: python 3.11 · 0 pip · gpu: none · import: poisson_prove.examples.mock_agent:rollout
  Target: poisson_prove.examples.mock_agent:rollout · resolved inside the container, not on this machine
  Containers: up to 16 · batch: 64 · episode timeout: 300s
  Secrets: none
  Cost: ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
  smallest rise one condition alone would be flagged for  0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)

3. Sweep the agent on Modal

Section titled “3. Sweep the agent on Modal”

The same plan, run. Prove builds the image once, then the batches come back as the containers finish.

Terminal window
poisson-prove sweep --backend modal --yes --est-seconds 0.5 --max-cost 1
Outputexcerpt, 1 of 23 lines
⋯ 17 lines
done  1,746 rollouts in run "mock-agent 2026-11-06 a" (complete) in 14.3 s
⋯ 5 lines
All output23 lines
plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on modal, ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
  target  poisson_prove.examples.mock_agent:rollout
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: modal · target: mock-agent · t_unit: tool_calls
  Image: python 3.11 · 0 pip · gpu: none · import: poisson_prove.examples.mock_agent:rollout
  Target: poisson_prove.examples.mock_agent:rollout · resolved inside the container, not on this machine
  Containers: up to 16 · batch: 64 · episode timeout: 300s
  Secrets: none
  Cost: ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
  smallest rise one condition alone would be flagged for  0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
384/1,746
768/1,746
1,170/1,746
1,426/1,746
1,618/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-06 a" (complete) in 14.3 s
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  14.3 s
  next: poisson-prove sweep --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

4. Sweep the patched agent on the same seeds

Section titled “4. Sweep the patched agent on the same seeds”

--pair-with latest gives the patched agent’s run the same seeds as the first, on Modal as anywhere else.

Terminal window
poisson-prove sweep --backend modal --yes --est-seconds 0.5 --max-cost 1 --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest
Outputexcerpt, 1 of 23 lines
⋯ 17 lines
done  1,746 rollouts in run "mock-agent 2026-11-06 b" (complete) in 14.7 s, paired with "mock-agent 2026-11-06 a"
⋯ 5 lines
All output23 lines
plan  1,746 rollouts of mock-agent: 9 cells x 194 replicates on modal, ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
  target  poisson_prove.examples.mock_agent:rollout_patched
  outcomes  failures: timeout, violated
  Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
  Backend: modal · target: mock-agent · t_unit: tool_calls
  Image: python 3.11 · 0 pip · gpu: none · import: poisson_prove.examples.mock_agent:rollout_patched
  Target: poisson_prove.examples.mock_agent:rollout_patched · resolved inside the container, not on this machine
  Containers: up to 16 · batch: 64 · episode timeout: 300s
  Secrets: none
  Cost: ~$0.011 (best-effort, Modal's 2026-09-03 prices, which cover compute, not model tokens)
  smallest rise one condition alone would be flagged for  0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
512/1,746
832/1,746
1,024/1,746
1,362/1,746
1,618/1,746
1,746/1,746
done  1,746 rollouts in run "mock-agent 2026-11-06 b" (complete) in 14.7 s, paired with "mock-agent 2026-11-06 a"
  episodes with an outcome  1,746
  episodes with no outcome  0
  episodes timed out  0
  wall time  14.7 s
  next: poisson-prove diff

5. Compare the two runs

Section titled “5. Compare the two runs”

The rows landed in the local store as they came back, so diff reads them as it reads any pair.

Terminal window
poisson-prove diff
Outputexcerpt, 1 of 35 lines
trade  timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds)
⋯ 34 lines
All output35 lines
trade  timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  mock-agent 2026-11-06 a
  after  mock-agent 2026-11-06 b
  trade  task_id refund · tool_fail_rate 0.55: violated fell, timeout rose
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 2 rows that did not count have no limit on their fall (none detectable: 2), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 97.778%, false coverage (Benjamini-Yekutieli) over the 8 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  24 outcomes would change the word (the search is not exhaustive: it may be fewer): 24 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise


inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared


condition                               event     before  after  change  verdict    rise   fall
task_id checkout · tool_fail_rate 0     timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id checkout · tool_fail_rate 0     violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id checkout · tool_fail_rate 0.3   timeout   0.010   0.000  -0.010  unchanged  0.085  none detectable
task_id checkout · tool_fail_rate 0.3   violated  0.289   0.031  -0.258  safer      0.195  0.161
task_id checkout · tool_fail_rate 0.55  timeout   0.062   0.021  -0.041  unchanged  0.132  none detectable
task_id checkout · tool_fail_rate 0.55  violated  0.835   0.278  -0.557  safer      0.121  0.173
task_id refund · tool_fail_rate 0       timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id refund · tool_fail_rate 0       violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id refund · tool_fail_rate 0.3     timeout   0.469   0.041  -0.428  safer      0.198  0.193
task_id refund · tool_fail_rate 0.3     violated  0.268   0.000  -0.268  safer      0.192  0.156
task_id refund · tool_fail_rate 0.55    timeout   0.139   0.397  +0.258  worse      0.166  0.109
task_id refund · tool_fail_rate 0.55    violated  0.861   0.340  -0.521  safer      0.109  0.166
task_id search · tool_fail_rate 0       timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0       violated  0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.3     timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.3     violated  0.196   0.010  -0.186  safer      0.181  0.133
task_id search · tool_fail_rate 0.55    timeout   0.000   0.000  0.000   unchanged  0.064  no room
task_id search · tool_fail_rate 0.55    violated  0.768   0.242  -0.526  safer      0.145  0.187


  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound


  next: poisson-prove report --x task_id --y tool_fail_rate

6. Write the report

Section titled “6. Write the report”
Terminal window
poisson-prove report --x task_id --y tool_fail_rate
Outputexcerpt, 1 of 12 lines
ok  wrote prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b (3 files)
⋯ 11 lines
All output12 lines
ok  wrote prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b (3 files)
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/report.md
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/heatmap.svg
  wrote  prove-report-mock-agent-2026-11-06-a-vs-mock-agent-2026-11-06-b/receipt.html
  saved  comparison 508933ec8811


inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared
timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 18 tests, paired)
Trade: violated fell in 1 cell where timeout rose.


  next: poisson-prove ui

The comparison

Section titled “The comparison”

The comparison, drawn by this site from the run’s diff --json in the site’s verdict scale: cobalt where the patch made the agent fail significantly less often, ink with a hatch where it failed significantly more often, and the no-change cells in grey. Hover a cell for its rates and limit.

timeout · 1 safer, 1 worse, 7 no significant change · mock-agent 2026-11-06 a → mock-agent 2026-11-06 b
tool_fail_rate ↑ 0 · 0.3 · 0.55

1 worse 7 no significant change, 2 of them blind at any size 1 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION
timeout: hover any condition, or focus the grid and use the arrow keys.
violated · 6 safer, 0 worse, 3 no significant change · mock-agent 2026-11-06 a → mock-agent 2026-11-06 b
tool_fail_rate ↑ 0 · 0.3 · 0.55

0 worse 3 no significant change 6 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION
violated: hover any condition, or focus the grid and use the arrow keys.

Stored output

Section titled “Stored output”

Every output above is what the command printed, stored in the site repository under src/data/examples/run-prove-on-modal/ with the run’s diff.json and the three files report wrote, as Prove 0.2.1 wrote them. The page cannot show an output that is not there.