Skip to content
Poisson Labs Docs
GitHub poissonlabs.ai

Examples

Run Prove in your AWS account

--backend cloud runs a sweep’s rollouts on a cluster in your own AWS account. Prove starts the nodes, runs the sweep on them, and stops them when the sweep ends, however it ends. You never start or stop a node yourself. The comparison is the same as on one machine, because the seeds come from the run, not from where it ran.

The cluster software is Burla, and the extra pins it to the one version Prove runs (burla==1.8.4). What leaves your machine is on Backends.

This run’s virtual environment was Python 3.11, the nodes’ image (python:3.11 on the plan’s Cluster line). Prove checks that the two match before anything starts.

Before you start: Install and AWS.

The same spec as on one machine: the keyless mock agent, nine conditions, 194 replicates each.

Terminal window
poisson-prove init --example
Output2 lines · exit 0
ok wrote prove.toml: a sweep of the keyless mock agent, 9 conditions x 194 replicates
next: poisson-prove sweep

where cloud --yes writes the [cloud] table into the spec, with Prove’s defaults for what no option gives, and says whether this machine is signed in. From here, sweep runs in your account; --backend cloud below says so on the command line too.

Terminal window
poisson-prove where cloud --yes
Output4 lines · exit 0
where your AWS account, us-east-2, 4 × m7i.2xlarge; signed in, ready
signed in AWS 6030…4933
wrote [cloud] into prove.toml
next: poisson-prove sweep --dry-run

--dry-run prints the plan and stops. It names the account, masked, the nodes Prove will start and what they cost, how soon they stop themselves if this machine stops first, and who carries your code. The ledger is what sweeps from this spec have spent so far. The stop is the spend at which Prove stops the sweep itself, and the cap is the estimate above which it refuses to start.

Terminal window
poisson-prove sweep --backend cloud --dry-run
Outputexcerpt, 4 of 13 lines · exit 0
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00
⋯ 4 lines
Account: AWS 6030…4933, signed in with the AWS CLI
Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run
⋯ 2 lines
Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine
⋯ 3 lines
All output13 lines
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00
target poisson_prove.examples.mock_agent:rollout (imports, takes cfg and seed)
outcomes failures: timeout, violated
Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
Backend: cloud · AWS · target: mock-agent · t_unit: tool_calls
Account: AWS 6030…4933, signed in with the AWS CLI
Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run
Bundle: poisson_prove 0.3.0 · 0.5 MB · sha256 194ed182028e…
Worker view: job 0 matches this machine
Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine
Cost: ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00
Chaos: none
smallest rise one condition alone would be flagged for 0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)

The same plan, run. Prove starts the nodes, waits until they are ready, runs the rollouts, and stops the cluster before it prints done. The cluster line says how long the nodes were up and what the ledger was charged for them.

Terminal window
poisson-prove sweep --backend cloud --yes
Outputexcerpt, 6 of 27 lines · exit 0
⋯ 13 lines
starting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 s
ready 4 nodes · 32 vCPU · 45.9 s
⋯ 3 lines
stopping the cluster
stopped 4 nodes, up 1 min 26 s
done 1,746 rollouts in run "mock-agent 2026-11-13 a" (complete) in 7.7 s
⋯ 4 lines
cluster 4 × m7i.2xlarge up 1 min 26 s, stopped 21:12:30Z · ledger $0.36
⋯ 1 line
All output27 lines
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00
target poisson_prove.examples.mock_agent:rollout (imports, takes cfg and seed)
outcomes failures: timeout, violated
Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
Backend: cloud · AWS · target: mock-agent · t_unit: tool_calls
Account: AWS 6030…4933, signed in with the AWS CLI
Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run
Bundle: poisson_prove 0.3.0 · 0.5 MB · sha256 194ed182028e…
Worker view: job 0 matches this machine
Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine
Cost: ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00
Chaos: none
smallest rise one condition alone would be flagged for 0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
starting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 s
ready 4 nodes · 32 vCPU · 45.9 s
256/1,746
512/1,746
1,746/1,746
stopping the cluster
stopped 4 nodes, up 1 min 26 s
done 1,746 rollouts in run "mock-agent 2026-11-13 a" (complete) in 7.7 s
episodes with an outcome 1,746
episodes with no outcome 0
episodes timed out 0
wall time 7.7 s
cluster 4 × m7i.2xlarge up 1 min 26 s, stopped 21:12:30Z · ledger $0.36
next: poisson-prove sweep --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest

5. Sweep the patched agent on the same seeds

Section titled “5. Sweep the patched agent on the same seeds”

--pair-with latest gives the patched agent’s run the same seeds as the first. Prove starts a fresh cluster for it and stops that one too. The plan’s ledger now counts the first sweep.

Terminal window
poisson-prove sweep --backend cloud --yes --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest
Outputexcerpt, 8 of 27 lines · exit 0
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0.36 so far · stop $2.00 · cap $3.00
⋯ 12 lines
starting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 s
ready 4 nodes · 32 vCPU · 46.5 s
⋯ 2 lines
stopping the cluster
stopped 4 nodes, up 1 min 31 s
1,746/1,746
done 1,746 rollouts in run "mock-agent 2026-11-13 b" (complete) in 6.2 s, paired with "mock-agent 2026-11-13 a"
⋯ 4 lines
cluster 4 × m7i.2xlarge up 1 min 31 s, stopped 21:14:16Z · ledger $0.36
⋯ 1 line
All output27 lines
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0.36 so far · stop $2.00 · cap $3.00
target poisson_prove.examples.mock_agent:rollout_patched (imports, takes cfg and seed)
outcomes failures: timeout, violated
Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts
Backend: cloud · AWS · target: mock-agent · t_unit: tool_calls
Account: AWS 6030…4933, signed in with the AWS CLI
Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run
Bundle: poisson_prove 0.3.0 · 0.5 MB · sha256 194ed182028e…
Worker view: job 0 matches this machine
Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine
Cost: ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0.36 so far · stop $2.00 · cap $3.00
Chaos: none
smallest rise one condition alone would be flagged for 0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
starting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 s
ready 4 nodes · 32 vCPU · 46.5 s
256/1,746
1,490/1,746
stopping the cluster
stopped 4 nodes, up 1 min 31 s
1,746/1,746
done 1,746 rollouts in run "mock-agent 2026-11-13 b" (complete) in 6.2 s, paired with "mock-agent 2026-11-13 a"
episodes with an outcome 1,746
episodes with no outcome 0
episodes timed out 0
wall time 6.2 s
cluster 4 × m7i.2xlarge up 1 min 31 s, stopped 21:14:16Z · ledger $0.36
next: poisson-prove diff

The rows came back to the local store, so diff reads them as it reads any pair.

Terminal window
poisson-prove diff
Outputexcerpt, 1 of 36 lines · exit 0
trade timeout: 2 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds); counted at the family-wise level: timeout 1 rose and 1 fell
⋯ 35 lines
All output36 lines
trade timeout: 2 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds); counted at the family-wise level: timeout 1 rose and 1 fell
method paired test on shared seeds (McNemar), the default for runs paired at sweep time
before mock-agent 2026-11-13 a
after mock-agent 2026-11-13 b
trade task_id refund · tool_fail_rate 0.55: violated fell, timeout rose
verdict rule the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests
flagged only 1 flagged at the false discovery rate does not count at the family-wise level
limits the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 2 rows that did not count have no limit on their fall (none detectable: 2), so a fall of any size cannot be excluded there
intervals on each condition's change: Tango at 97.5%, false coverage (Benjamini-Yekutieli) over the 9 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
fragility 29 outcomes would change the word (the search is not exhaustive: it may be fewer): 29 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise
inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared
condition event before after change verdict rise fall
task_id checkout · tool_fail_rate 0 timeout 0.000 0.000 0.000 unchanged 0.064 no room
task_id checkout · tool_fail_rate 0 violated 0.000 0.000 0.000 unchanged 0.064 no room
task_id checkout · tool_fail_rate 0.3 timeout 0.010 0.000 -0.010 unchanged 0.085 none detectable
task_id checkout · tool_fail_rate 0.3 violated 0.258 0.000 -0.258 safer 0.191 0.153
task_id checkout · tool_fail_rate 0.55 timeout 0.057 0.010 -0.046 safer 0.129 none detectable
task_id checkout · tool_fail_rate 0.55 violated 0.851 0.268 -0.582 safer 0.114 0.169
task_id refund · tool_fail_rate 0 timeout 0.000 0.000 0.000 unchanged 0.064 no room
task_id refund · tool_fail_rate 0 violated 0.000 0.000 0.000 unchanged 0.064 no room
task_id refund · tool_fail_rate 0.3 timeout 0.428 0.046 -0.381 safer 0.200 0.189
task_id refund · tool_fail_rate 0.3 violated 0.340 0.036 -0.304 safer 0.198 0.174
task_id refund · tool_fail_rate 0.55 timeout 0.160 0.448 +0.289 worse 0.172 0.119
task_id refund · tool_fail_rate 0.55 violated 0.840 0.340 -0.500 safer 0.119 0.172
task_id search · tool_fail_rate 0 timeout 0.000 0.000 0.000 unchanged 0.064 no room
task_id search · tool_fail_rate 0 violated 0.000 0.000 0.000 unchanged 0.064 no room
task_id search · tool_fail_rate 0.3 timeout 0.000 0.000 0.000 unchanged 0.064 no room
task_id search · tool_fail_rate 0.3 violated 0.191 0.005 -0.186 safer 0.180 0.131
task_id search · tool_fail_rate 0.55 timeout 0.000 0.000 0.000 unchanged 0.064 no room
task_id search · tool_fail_rate 0.55 violated 0.804 0.247 -0.557 safer 0.133 0.181
more --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound
next: poisson-prove report --x task_id --y tool_fail_rate
Terminal window
poisson-prove report --x task_id --y tool_fail_rate
Outputexcerpt, 1 of 12 lines · exit 0
ok wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b (3 files)
⋯ 11 lines
All output12 lines
ok wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b (3 files)
wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b/report.md
wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b/heatmap.svg
wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b/receipt.html
saved comparison 221cfb936583
inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared
timeout: 2 safer, 1 worse · violated: 6 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 18 tests, paired)
Trade: violated fell in 1 cell where timeout rose.
next: poisson-prove ui

where with no word says where sweeps run and whether that is ready. It also reads the instances the last sweep launched and terminates any still running, with a note: line for each; here there was none. Read from the account itself with the AWS CLI, nothing is pending, running or shutting down in the region.

Terminal window
poisson-prove where
Output3 lines · exit 0
where your AWS account, us-east-2, 4 × m7i.2xlarge; signed in, ready
signed in AWS 6030…4933
next: poisson-prove sweep
Terminal window
aws ec2 describe-instances --region us-east-2 --filters Name=instance-state-name,Values=pending,running,shutting-down --query 'Reservations[].Instances[].InstanceId'
Output1 line · exit 0
[]

The comparison, drawn by this site from the run’s diff --json in the site’s verdict scale: cobalt where the patch made the agent fail significantly less often, ink with a hatch where it failed significantly more often, and the no-change cells in grey. Hover a cell for its rates and limit.

timeout · 2 safer, 1 worse, 6 no significant change · mock-agent 2026-11-13 a → mock-agent 2026-11-13 b
tool_fail_rate ↑ 0 · 0.3 · 0.55

1 worse 6 no significant change, 1 of them blind at any size 2 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION
timeout: hover any condition, or focus the grid and use the arrow keys.
violated · 6 safer, 0 worse, 3 no significant change · mock-agent 2026-11-13 a → mock-agent 2026-11-13 b
tool_fail_rate ↑ 0 · 0.3 · 0.55

0 worse 3 no significant change 6 safer coloured only at q<0.05, corrected across every condition

TASK_ID: CHECKOUT · REFUND · SEARCH → ↑ TOOL_FAIL_RATE 194 TRIALS PER CELL PER VERSION
violated: hover any condition, or focus the grid and use the arrow keys.

Every output above is what the command printed, stored in the site repository under src/data/examples/run-prove-in-your-aws-account/ with the run’s diff.json and the three files report wrote, as Prove 0.3.0 wrote them. The page cannot show an output that is not there.