Run Prove in your AWS account
--backend cloud runs a sweep’s rollouts on a cluster in your own AWS account. Prove starts the
nodes, runs the sweep on them, and stops them when the sweep ends, however it ends. You never
start or stop a node yourself. The comparison is the same as on one machine, because the seeds
come from the run, not from where it ran.
The cluster software is Burla, and the extra pins it to the one version
Prove runs (burla==1.8.4). What leaves your machine is on
Backends.
This run’s virtual environment was Python 3.11, the nodes’ image (python:3.11 on the plan’s
Cluster line). Prove checks that the two match before anything starts.
Before you start: Install and AWS.
1. Write the example’s spec
Section titled “1. Write the example’s spec”The same spec as on one machine: the keyless mock agent, nine conditions, 194 replicates each.
poisson-prove init --exampleok wrote prove.toml: a sweep of the keyless mock agent, 9 conditions x 194 replicates next: poisson-prove sweep2. Point the spec at your AWS account
Section titled “2. Point the spec at your AWS account”where cloud --yes writes the [cloud] table into the spec, with Prove’s defaults for what no
option gives, and says whether this machine is signed in. From here, sweep runs in your
account; --backend cloud below says so on the command line too.
poisson-prove where cloud --yesOutput4 lines · exit 0
where your AWS account, us-east-2, 4 × m7i.2xlarge; signed in, ready signed in AWS 6030…4933 wrote [cloud] into prove.toml next: poisson-prove sweep --dry-run3. See the plan and its cost first
Section titled “3. See the plan and its cost first”--dry-run prints the plan and stops. It names the account, masked, the nodes Prove will start
and what they cost, how soon they stop themselves if this machine stops first, and who carries
your code. The ledger is what sweeps from this spec have spent so far. The stop is the spend at
which Prove stops the sweep itself, and the cap is the estimate above which it refuses to start.
poisson-prove sweep --backend cloud --dry-runplan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00⋯ 4 lines Account: AWS 6030…4933, signed in with the AWS CLI Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run⋯ 2 lines Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine⋯ 3 linesAll output13 lines
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00 target poisson_prove.examples.mock_agent:rollout (imports, takes cfg and seed) outcomes failures: timeout, violated Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts Backend: cloud · AWS · target: mock-agent · t_unit: tool_calls Account: AWS 6030…4933, signed in with the AWS CLI Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run Bundle: poisson_prove 0.3.0 · 0.5 MB · sha256 194ed182028e… Worker view: job 0 matches this machine Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine Cost: ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00 Chaos: none smallest rise one condition alone would be flagged for 0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)4. Sweep the agent in your account
Section titled “4. Sweep the agent in your account”The same plan, run. Prove starts the nodes, waits until they are ready, runs the rollouts, and
stops the cluster before it prints done. The cluster line says how long the nodes were up and
what the ledger was charged for them.
poisson-prove sweep --backend cloud --yes⋯ 13 linesstarting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 sready 4 nodes · 32 vCPU · 45.9 s⋯ 3 linesstopping the clusterstopped 4 nodes, up 1 min 26 sdone 1,746 rollouts in run "mock-agent 2026-11-13 a" (complete) in 7.7 s⋯ 4 lines cluster 4 × m7i.2xlarge up 1 min 26 s, stopped 21:12:30Z · ledger $0.36⋯ 1 lineAll output27 lines
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00 target poisson_prove.examples.mock_agent:rollout (imports, takes cfg and seed) outcomes failures: timeout, violated Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts Backend: cloud · AWS · target: mock-agent · t_unit: tool_calls Account: AWS 6030…4933, signed in with the AWS CLI Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run Bundle: poisson_prove 0.3.0 · 0.5 MB · sha256 194ed182028e… Worker view: job 0 matches this machine Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine Cost: ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0 so far · stop $2.00 · cap $3.00 Chaos: none smallest rise one condition alone would be flagged for 0.199 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)starting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 sready 4 nodes · 32 vCPU · 45.9 s256/1,746512/1,7461,746/1,746stopping the clusterstopped 4 nodes, up 1 min 26 sdone 1,746 rollouts in run "mock-agent 2026-11-13 a" (complete) in 7.7 s episodes with an outcome 1,746 episodes with no outcome 0 episodes timed out 0 wall time 7.7 s cluster 4 × m7i.2xlarge up 1 min 26 s, stopped 21:12:30Z · ledger $0.36 next: poisson-prove sweep --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest5. Sweep the patched agent on the same seeds
Section titled “5. Sweep the patched agent on the same seeds”--pair-with latest gives the patched agent’s run the same seeds as the first. Prove starts a
fresh cluster for it and stops that one too. The plan’s ledger now counts the first sweep.
poisson-prove sweep --backend cloud --yes --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latestplan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0.36 so far · stop $2.00 · cap $3.00⋯ 12 linesstarting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 sready 4 nodes · 32 vCPU · 46.5 s⋯ 2 linesstopping the clusterstopped 4 nodes, up 1 min 31 s1,746/1,746done 1,746 rollouts in run "mock-agent 2026-11-13 b" (complete) in 6.2 s, paired with "mock-agent 2026-11-13 a"⋯ 4 lines cluster 4 × m7i.2xlarge up 1 min 31 s, stopped 21:14:16Z · ledger $0.36⋯ 1 lineAll output27 lines
plan 1,746 rollouts of mock-agent: 9 cells x 194 replicates on your AWS account, ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0.36 so far · stop $2.00 · cap $3.00 target poisson_prove.examples.mock_agent:rollout_patched (imports, takes cfg and seed) outcomes failures: timeout, violated Sweep: 2 axes · 9 cells · N=194 → 1,746 rollouts Backend: cloud · AWS · target: mock-agent · t_unit: tool_calls Account: AWS 6030…4933, signed in with the AWS CLI Cluster: Prove starts 4 × m7i.2xlarge in us-east-2 (8 vCPU each, 32 in all, ceiling 32) running python:3.11, and stops them when the sweep ends · 8 slots each · 32 at once · if this machine stops first, the nodes stop themselves within 5 min · this machine is kept from idle sleep while they run Bundle: poisson_prove 0.3.0 · 0.5 MB · sha256 194ed182028e… Worker view: job 0 matches this machine Traffic: your code, its inputs and its results go between this machine and the nodes through Burla's relay (the cluster software Prove uses); the store stays on this machine Cost: ~$0.45 this run (4 × m7i.2xlarge at $0.4687/node-hour, Prove's AWS prices for us-east-2 of 2026-11-06, each node's disk priced at 52 GB) · ledger cloud-ledger.jsonl $0.36 so far · stop $2.00 · cap $3.00 Chaos: none smallest rise one condition alone would be flagged for 0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)starting 4 × m7i.2xlarge in us-east-2 0 of 4 ready · 0.0 sready 4 nodes · 32 vCPU · 46.5 s256/1,7461,490/1,746stopping the clusterstopped 4 nodes, up 1 min 31 s1,746/1,746done 1,746 rollouts in run "mock-agent 2026-11-13 b" (complete) in 6.2 s, paired with "mock-agent 2026-11-13 a" episodes with an outcome 1,746 episodes with no outcome 0 episodes timed out 0 wall time 6.2 s cluster 4 × m7i.2xlarge up 1 min 31 s, stopped 21:14:16Z · ledger $0.36 next: poisson-prove diff6. Compare the two runs
Section titled “6. Compare the two runs”The rows came back to the local store, so diff reads them as it reads any pair.
poisson-prove difftrade timeout: 2 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds); counted at the family-wise level: timeout 1 rose and 1 fell⋯ 35 linesAll output36 lines
trade timeout: 2 safer, 1 worse · violated: 6 safer, 0 worse (paired test on shared seeds); counted at the family-wise level: timeout 1 rose and 1 fell method paired test on shared seeds (McNemar), the default for runs paired at sweep time before mock-agent 2026-11-13 a after mock-agent 2026-11-13 b trade task_id refund · tool_fail_rate 0.55: violated fell, timeout rose verdict rule the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests flagged only 1 flagged at the false discovery rate does not count at the family-wise level limits the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 2 rows that did not count have no limit on their fall (none detectable: 2), so a fall of any size cannot be excluded there intervals on each condition's change: Tango at 97.5%, false coverage (Benjamini-Yekutieli) over the 9 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own fragility 29 outcomes would change the word (the search is not exhaustive: it may be fewer): 29 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise
inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declared
condition event before after change verdict rise falltask_id checkout · tool_fail_rate 0 timeout 0.000 0.000 0.000 unchanged 0.064 no roomtask_id checkout · tool_fail_rate 0 violated 0.000 0.000 0.000 unchanged 0.064 no roomtask_id checkout · tool_fail_rate 0.3 timeout 0.010 0.000 -0.010 unchanged 0.085 none detectabletask_id checkout · tool_fail_rate 0.3 violated 0.258 0.000 -0.258 safer 0.191 0.153task_id checkout · tool_fail_rate 0.55 timeout 0.057 0.010 -0.046 safer 0.129 none detectabletask_id checkout · tool_fail_rate 0.55 violated 0.851 0.268 -0.582 safer 0.114 0.169task_id refund · tool_fail_rate 0 timeout 0.000 0.000 0.000 unchanged 0.064 no roomtask_id refund · tool_fail_rate 0 violated 0.000 0.000 0.000 unchanged 0.064 no roomtask_id refund · tool_fail_rate 0.3 timeout 0.428 0.046 -0.381 safer 0.200 0.189task_id refund · tool_fail_rate 0.3 violated 0.340 0.036 -0.304 safer 0.198 0.174task_id refund · tool_fail_rate 0.55 timeout 0.160 0.448 +0.289 worse 0.172 0.119task_id refund · tool_fail_rate 0.55 violated 0.840 0.340 -0.500 safer 0.119 0.172task_id search · tool_fail_rate 0 timeout 0.000 0.000 0.000 unchanged 0.064 no roomtask_id search · tool_fail_rate 0 violated 0.000 0.000 0.000 unchanged 0.064 no roomtask_id search · tool_fail_rate 0.3 timeout 0.000 0.000 0.000 unchanged 0.064 no roomtask_id search · tool_fail_rate 0.3 violated 0.191 0.005 -0.186 safer 0.180 0.131task_id search · tool_fail_rate 0.55 timeout 0.000 0.000 0.000 unchanged 0.064 no roomtask_id search · tool_fail_rate 0.55 violated 0.804 0.247 -0.557 safer 0.133 0.181
more --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound
next: poisson-prove report --x task_id --y tool_fail_rate7. Write the report
Section titled “7. Write the report”poisson-prove report --x task_id --y tool_fail_rateok wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b (3 files)⋯ 11 linesAll output12 lines
ok wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b (3 files) wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b/report.md wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b/heatmap.svg wrote prove-report-mock-agent-2026-11-13-a-vs-mock-agent-2026-11-13-b/receipt.html saved comparison 221cfb936583
inferred: no event was given, so the event set is timeout, violated, the failures and successes the runs declaredtimeout: 2 safer, 1 worse · violated: 6 safer, 0 worse(aalen_johansen + mcnemar_bh, BH q<0.05 over 18 tests, paired)Trade: violated fell in 1 cell where timeout rose.
next: poisson-prove ui8. Check that nothing is left running
Section titled “8. Check that nothing is left running”where with no word says where sweeps run and whether that is ready. It also reads the
instances the last sweep launched and terminates any still running, with a note: line for each;
here there was none. Read from the account itself with the AWS CLI, nothing is pending, running or
shutting down in the region.
poisson-prove whereOutput3 lines · exit 0
where your AWS account, us-east-2, 4 × m7i.2xlarge; signed in, ready signed in AWS 6030…4933 next: poisson-prove sweepaws ec2 describe-instances --region us-east-2 --filters Name=instance-state-name,Values=pending,running,shutting-down --query 'Reservations[].Instances[].InstanceId'Output1 line · exit 0
[]The comparison
Section titled “The comparison”The comparison, drawn by this site from the run’s diff --json in the site’s verdict scale:
cobalt where the patch made the agent fail significantly less often, ink with a hatch where
it failed significantly more often, and the no-change cells in grey. Hover a cell for its
rates and limit.
1 worse 6 no significant change, 1 of them blind at any size 2 safer coloured only at q<0.05, corrected across every condition
0 worse 3 no significant change 6 safer coloured only at q<0.05, corrected across every condition
Stored output
Section titled “Stored output”Every output above is what the command printed, stored in the site repository under
src/data/examples/run-prove-in-your-aws-account/ with the run’s diff.json and the three files
report wrote, as Prove 0.3.0 wrote them. The page cannot show an output that is not there.