Run Prove on your own agent

Prove runs your agent through a rollout: a function you write, rollout(cfg, seed), that runs one case and returns how it ended. A sweep calls it once per condition and replicate and stores every ending. This is the example on Your own agent, run end to end: a support bot, a rollout around it, a spec, a sweep, a change to the bot’s prompt swept on the same seeds, and the comparison. The bot is a stand-in that calls no model, so the page runs as it is; in your repository the rollout calls your own agent.

Before you start: Install and Your machine.

1. Save the agent

Two files stand in for your agent: its prompt, and a bot whose chance of leaving a ticket unresolved depends on the ticket, the temperature and the prompt. Save them in one folder, as the docs give them.

SYSTEM = "Answer the customer's ticket. Ask before refunding."

"""A stand-in for your own agent, so this page runs as it is."""
import random
from dataclasses import dataclass

from prompts import SYSTEM

#: Each ticket, as the share of attempts on it that end unresolved.
TICKETS = {"refund": 0.30, "password": 0.10, "shipping": 0.20}


@dataclass
class Reply:
    resolved: bool
    timed_out: bool
    steps: int


def handle(ticket, temperature, seed):
    rng = random.Random(seed)
    steps = rng.randint(2, 20)
    gives_up = ticket * (1 + temperature) * (0.5 if "step by step" in SYSTEM else 1.0)
    return Reply(resolved=steps < 20 and rng.random() >= gives_up, timed_out=steps == 20, steps=steps)

2. Write the rollout

The rollout runs the case cfg names, with every random choice seeded from seed, maps how it ended to an outcome, and returns an Episode with the case’s id in meta. This is the one file that knows your agent.

from poisson_prove import Episode
from support_bot import TICKETS, handle  # your agent and its cases


def rollout(cfg, seed):
    ticket = TICKETS[cfg["ticket"]]
    reply = handle(ticket, temperature=cfg["temperature"], seed=seed)
    if reply.timed_out:
        outcome = "timeout"
    elif reply.resolved:
        outcome = "resolved"
    else:
        outcome = "unresolved"
    return Episode(outcome=outcome, t=reply.steps, horizon=20, meta={"case_id": cfg["ticket"]})

3. Write the spec

The spec names the rollout as module:function, the axes that make the conditions (three tickets, two temperatures), 200 replicates of each, and which outcomes count against the agent and which for it.

target = "my_rollout:rollout"
target_id = "support-bot"
t_unit = "steps"
replicates = 200

[axes.ticket]
type = "categorical"
values = ["refund", "password", "shipping"]

[axes.temperature]
type = "continuous"
values = [0.0, 0.7]

[declare]
outcomes = ["resolved", "unresolved", "timeout"]
failures = ["unresolved", "timeout"]
successes = ["resolved"]

--dry-run checks all of it and runs nothing: it imports the rollout in a process of its own, checks it takes (cfg, seed) without calling it, and prints the plan.

poisson-prove sweep support.toml --dry-run

Outputexcerpt, 3 of 7 lines · exit 0

plan  1,200 rollouts of support-bot: 6 cells x 200 replicates on local, $0 (local)
  target  my_rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: timeout, unresolved; successes: resolved
⋯ 4 lines

All output7 lines

plan  1,200 rollouts of support-bot: 6 cells x 200 replicates on local, $0 (local)
  target  my_rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: timeout, unresolved; successes: resolved
  Sweep: 2 axes · 6 cells · N=200 → 1,200 rollouts
  Backend: local · target: support-bot · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.194 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)

4. Sweep it

poisson-prove sweep support.toml --yes

Outputexcerpt, 2 of 13 lines · exit 0

plan  1,200 rollouts of support-bot: 6 cells x 200 replicates on local, $0 (local)
⋯ 7 lines
done  1,200 rollouts in run "support-bot 2026-11-07 a" (complete) in 0.7 s
⋯ 4 lines

All output13 lines

plan  1,200 rollouts of support-bot: 6 cells x 200 replicates on local, $0 (local)
  target  my_rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: timeout, unresolved; successes: resolved
  Sweep: 2 axes · 6 cells · N=200 → 1,200 rollouts
  Backend: local · target: support-bot · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.194 at its worst base rate, 0.400 (Fisher, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
1,200/1,200
done  1,200 rollouts in run "support-bot 2026-11-07 a" (complete) in 0.7 s
  episodes with an outcome  1,200
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.7 s

5. Change the prompt and sweep again

Save the new prompt over prompts.py, then sweep with --pair-with latest, so the changed bot meets the same seeds as the first run, episode by episode.

SYSTEM = "Answer the customer's ticket step by step. Ask before refunding."

poisson-prove sweep support.toml --yes --pair-with latest

Outputexcerpt, 1 of 14 lines · exit 0

⋯ 8 lines
done  1,200 rollouts in run "support-bot 2026-11-07 b" (complete) in 0.5 s, paired with "support-bot 2026-11-07 a"
⋯ 5 lines

All output14 lines

plan  1,200 rollouts of support-bot: 6 cells x 200 replicates on local, $0 (local)
  target  my_rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: timeout, unresolved; successes: resolved
  Sweep: 2 axes · 6 cells · N=200 → 1,200 rollouts
  Backend: local · target: support-bot · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.197 at its worst base rate, 0.400 (the paired test, q 0.05 over 18 tests, 4 times in 5; you accept up to 0.200)
1,200/1,200
done  1,200 rollouts in run "support-bot 2026-11-07 b" (complete) in 0.5 s, paired with "support-bot 2026-11-07 a"
  episodes with an outcome  1,200
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.5 s
  next: poisson-prove diff

6. Compare the two runs

poisson-prove diff

Outputexcerpt, 3 of 34 lines · exit 0

safer  resolved: 6 safer, 0 worse · unresolved: 6 safer, 0 worse · timeout: 0 safer, 0 worse (paired test on shared seeds)
⋯ 1 line
  before  support-bot 2026-11-07 a
  after  support-bot 2026-11-07 b
⋯ 30 lines

All output34 lines

safer  resolved: 6 safer, 0 worse · unresolved: 6 safer, 0 worse · timeout: 0 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  support-bot 2026-11-07 a
  after  support-bot 2026-11-07 b
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 6 rows that did not count have no limit on their fall (none detectable: 6), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 96.667%, false coverage (Benjamini-Yekutieli) over the 12 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  8 outcomes would change the word (the search is not exhaustive: it may be fewer): 8 runs of A in temperature 0.7, ticket refund that ended in timeout made to end otherwise

inferred: no event was given, so the event set is timeout, unresolved, resolved, the failures and successes the runs declared

condition                          event       before  after  change  verdict    rise   fall
temperature 0 · ticket password    resolved    0.865   0.920  +0.055  safer      0.106  0.161
temperature 0 · ticket password    unresolved  0.080   0.025  -0.055  safer      0.139  0.075
temperature 0 · ticket password    timeout     0.055   0.055  0.000   unchanged  0.126  none detectable
temperature 0 · ticket refund      resolved    0.720   0.845  +0.125  safer      0.157  0.191
temperature 0 · ticket refund      unresolved  0.225   0.100  -0.125  safer      0.183  0.141
temperature 0 · ticket refund      timeout     0.055   0.055  0.000   unchanged  0.126  none detectable
temperature 0 · ticket shipping    resolved    0.760   0.825  +0.065  safer      0.146  0.185
temperature 0 · ticket shipping    unresolved  0.185   0.120  -0.065  safer      0.175  0.127
temperature 0 · ticket shipping    timeout     0.055   0.055  0.000   unchanged  0.126  none detectable
temperature 0.7 · ticket password  resolved    0.810   0.885  +0.075  safer      0.129  0.176
temperature 0.7 · ticket password  unresolved  0.140   0.065  -0.075  safer      0.163  0.108
temperature 0.7 · ticket password  timeout     0.050   0.050  0.000   unchanged  0.122  none detectable
temperature 0.7 · ticket refund    resolved    0.465   0.690  +0.225  safer      0.195  0.190
temperature 0.7 · ticket refund    unresolved  0.475   0.250  -0.225  safer      0.195  0.191
temperature 0.7 · ticket refund    timeout     0.060   0.060  0.000   unchanged  0.129  none detectable
temperature 0.7 · ticket shipping  resolved    0.630   0.830  +0.200  safer      0.177  0.196
temperature 0.7 · ticket shipping  unresolved  0.325   0.125  -0.200  safer      0.194  0.168
temperature 0.7 · ticket shipping  timeout     0.045   0.045  0.000   unchanged  0.119  none detectable

  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound

  next: poisson-prove report --x temperature --y ticket

safer: the new prompt left significantly fewer tickets unresolved, and no condition got worse at the confidence the test states. To fail a build on a change for the worse instead, see Gate a pull request with Prove.

7. Start on your own rollout

For your own agent, init --target writes a commented spec with your rollout as its target. Set its axes to what you vary and its outcomes to how your cases end, check it with --dry-run, and sweep.

poisson-prove init --target my_rollout:rollout --out mine.toml

Output2 lines · exit 0

ok  wrote mine.toml: a template; set its axes to what you vary
  next: poisson-prove sweep mine.toml

The comparison

The comparison, drawn by this site from the run’s diff --json in the site’s verdict scale, one grid per outcome: cobalt where the new prompt made a failure significantly rarer (or a success significantly more common), ink with a hatch where it went the other way, and the no-change cells in grey. Hover a cell for its rates and limit.

timeout · 0 safer, 0 worse, 6 no significant change · support-bot 2026-11-07 a → support-bot 2026-11-07 b

temperature ↑ 0 · 0.7

0 worse 6 no significant change, 6 of them blind at any size 0 safer coloured only at q<0.05, corrected across every condition

TICKET: PASSWORD · REFUND · SHIPPING → ↑ TEMPERATURE 200 TRIALS PER CELL PER VERSION

timeout: hover any condition, or focus the grid and use the arrow keys.

unresolved · 6 safer, 0 worse, 0 no significant change · support-bot 2026-11-07 a → support-bot 2026-11-07 b

temperature ↑ 0 · 0.7

0 worse 0 no significant change 6 safer coloured only at q<0.05, corrected across every condition

TICKET: PASSWORD · REFUND · SHIPPING → ↑ TEMPERATURE 200 TRIALS PER CELL PER VERSION

unresolved: hover any condition, or focus the grid and use the arrow keys.

resolved · 6 safer, 0 worse, 0 no significant change · support-bot 2026-11-07 a → support-bot 2026-11-07 b

temperature ↑ 0 · 0.7

0 worse 0 no significant change 6 safer coloured only at q<0.05, corrected across every condition

TICKET: PASSWORD · REFUND · SHIPPING → ↑ TEMPERATURE 200 TRIALS PER CELL PER VERSION

resolved: hover any condition, or focus the grid and use the arrow keys.

Stored output

Every output above is what the command printed, with the exit code it returned, stored in the site repository under src/data/examples/run-prove-on-your-own-agent/ with the run’s diff.json, as Prove 0.2.1 wrote them. The five files are stored there too, byte for byte as docs/your-own-agent.md gives them at Prove’s commit, each with its sha256. The page cannot show an output that is not there.