Poisson Break
Break finds where an AI agent fails. It runs the agent you already have, many times, while changing how its tools behave and what the user asks, and keeps the conditions under which the agent fails more often than it does with no change. Each one is a finding: a named condition you can replay, with the numbers behind it, that a fix can later be checked against.
Try it on an example agent. This needs no key and no network once it is installed.
python -m venv .venv. .venv/bin/activatepip install poisson-breakpoisson-break init --example deskpoisson-break hunt --yespoisson-break confirm --yesinit --example desk writes a small help desk into the folder: an agent with two tools and a
scripted model, six requests to try it on, and break.toml, the settings. hunt looks for
conditions that make it fail. confirm tests what the hunt found on fresh runs. To run Break on
an agent of your own, go on to On your own agent.
Break calls your tools for real. Run it against a test environment. It refuses a tool that changes something outside until you say how that tool is made safe to repeat.
How Break changes a tool
Section titled “How Break changes a tool”The desk’s model is a plain Python function, scripted_model, that picks the next step from the
request and from what the tools have returned so far. That is why the example needs no key and
no network. Break makes no model call of its own; on your own agent, the model calls are the
ones your agent makes.
Break starts no app and no server. It imports the function break.toml names, desk_agent:desk
here, and calls it once for each run, in Break’s own process. One run is one call of that
function, on one case, with one seed, under one condition. The control runs the same cases with no
change, and every condition is compared with it; each condition the desk’s hunt tries is one
change.
Each tool is wrapped once with wrap, which puts Break between the agent and the tool. On the
call a condition names, such as the first call to lookup_order, Break applies the condition’s
fault. A timeout, an error or a rate limit is raised where the tool’s own failure would arrive,
and the tool does not run, so a timeout reaches the agent as a TimeoutError. Other faults run
the tool and change what comes back: empty hands back an empty result, malformed breaks its
structure, and stale hands back the previous call’s result. lost_reply lets the tool act and
then reports an error. Every other call runs the real tool, as every call does in the control.
What you see
Section titled “What you see”hunt shows its plan, then runs the agent with no change (the control), then tries one change
at a time:
plan a hunt of desk on 6 cases, at most 120 runs target desk_agent:desk (rollout) tools lookup_order (read), check_policy (read) may change tool behaviour (4 faults), inputs (2 operators) strategy screen stages control 12 runs, search and shrink up to 108 cap 120 runs, 2 hours estimate at most 120 runs and 2 hours, with no prices to put in dollars, from the settings' limitsnote: Break calls your tools for real: run it against a test environment, never against anything that mattersfound 3 candidates in 42 runs control 12 runs · 0 failed (0.000) search 30 runs · 10 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=timeout 3 of 3 failed failed 2 tool:lookup_order#1=rate_limited 3 of 3 failed failed 3 tool:lookup_order#1=error 3 of 3 failed failed not looked at: faults turned off: slow, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 33 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmThe desk never failed in its 12 control runs. Of the 10 conditions the hunt tried, three made
it fail on every case they were tried on: a timeout, a rate limit and an error on the first
call to lookup_order. The other seven, among them every fault on check_policy and both
changes to the request, did not. The line not looked at says what this hunt left untried.
Those three are candidates: a few runs each, and chosen because they failed. confirm runs
each of them again on every case, three times, beside a control arm. The control arm is the
set of runs on the same cases with no change, and each candidate’s runs are compared with it.
confirm then asks Trace whether the difference holds. Trace is the Poisson Labs tool that
checks whether a fix held:
plan confirm 3 candidates against a control design 6 cases × 3 repeats, once with no change and once under each candidate · 72 runs reference up to 54 more runs, 18 for each candidate that is confirmed cap 120 runs, 2 hours Trace will say if this is too small to see a difference of 0.200, and how many repeats would be enough estimate 126 runs, about 0 s; no prices, so no dollars, from the control stage's 0 s a runnote: Break calls your tools for real: run it against a test environment, never against anything that mattersconfirmed 3 of 3 candidates fail more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=timeout confirmed 1.000 +1.000 2 tool:lookup_order#1=rate_limited confirmed 1.000 +1.000 3 tool:lookup_order#1=error confirmed 1.000 +1.000 wrote break-runs/desk-1/findings/1.toml, 2.toml, 3.toml reference 54 runsnext: poisson-break findings 1All three hold, and each is now a finding. Read one, and run it once more to see what the agent did:
poisson-break findings 1poisson-break replay 1note: Break calls your tools for real: run it against a test environment, never against anything that mattersfailed finding 1 on case status-4821, seed 56661626: failed tool call change real tool ran 1 lookup_order 1 tool:lookup_order#1=timeout noThe desk has no second try: one failed look-up ends the request. Reading a finding goes through every line a finding prints.
The hunt called the desk’s function 42 times, once for each run: 12 control runs, two on each of
the six cases, and 30 runs for the ten conditions it tried. confirm called it 126 more times:
72 runs with no change and under the three candidates, and 54 for the reference, the runs
recheck later compares a fix with.
Everything is kept in break-runs/desk-1/, one folder for each hunt. hunt.json holds the
settings as resolved and the stages run, candidates.json the candidates hunt handed to
confirm, findings/1.toml and the two beside it one finding each, and store/ every run, as
a Prove store. Prove is the Poisson Labs tool that tells you whether a difference is real, and
its own commands read the store.
A stored run is Break’s record of the run: the case by its id, the change, the outcome and the
kind of failure, and each tool call with whether the real tool ran. It holds none of the agent’s
prompt, reply or tool results. replay is how you see what the agent did: it runs a finding
once more, stores nothing, and shows each tool call with the change on it.
On your own agent
Section titled “On your own agent”A way in is how Break runs your agent and reaches its tools. This version has one, rollout:
your agent as a Python function that Break calls once for each run, with each tool wrapped
once.
The function takes cfg and seed and returns an Episode. cfg is one case from your case
file, as a mapping that holds the request under input. seed is the run’s seed. Episode is
Prove’s record of one run, and Break reads the agent’s own outcome from it.
from poisson_prove import Episode
from poisson_break.apply import wrap
lookup_order = wrap("lookup_order", lookup_order) # one line a tool, where it is defined
def refund_agent(cfg, seed): reply = run_my_agent(cfg["input"]) # your agent, as it is return Episode(outcome="resolved", t=1.0, horizon=1.0, meta={"reply": reply})The wrap line is what this way in asks of your code. You write it yourself, once for each
tool, where the tool is defined, at the top of its module, and not at each call. init does not
add it: it finds the tools already wrapped by importing your module. The line makes your module
import poisson_break. Outside a run the wrapped tool is the tool itself, so the line can stay
in. A tool that is async def is refused. Adapters for agent frameworks and MCP servers are not
in this version (see Status, below).
Then, in the folder that holds it, poisson-break init asks where the agent is, where your
test cases are, and, for each tool it finds, whether calling it changes anything. It chooses
everything else, says what it chose, and writes break.toml, where you can change any of it. poisson-break doctor checks that everything loads. hunt and confirm work as
above. The rollout way in has the details, and the
catalogue lists every change a hunt can make.
When you have changed the agent, poisson-break recheck runs every finding again and says
whether each now fails less: Checking a fix.
The examples
Section titled “The examples”Each is written into the folder by poisson-break init --example NAME, and each runs with no
key and no network against a scripted model.
| Example | What it is | What it ends with |
|---|---|---|
desk | a help desk with two read tools | 3 findings: the desk gives up when its order look-up fails |
full-loop | an order desk, and the same desk with a fix | 3 findings; after the fix is copied in, recheck reads all 3 as fails less |
stand-in | a refund desk whose refund tool moves money, with a stand-in for it | 1 finding: after a lost reply the desk refunds twice (duplicate_write) |
own-check | a desk that always answers, with a check of the team’s own | 3 findings judged by that check (own:made_up) |
own-fault | a desk, with a fault the team added to the catalogue | 1 finding: the desk falls over on an error page (crashed) |
The examples page walks through each.
The docs
Section titled “The docs”- The catalogue: every change Break can make, and how each is named.
- Reading a finding: what each line and number means.
- Checking a fix: hunt, confirm, fix, recheck.
- The rollout way in: your agent as a function, its tools and their classes.
- Cases from the requests your users already sent: LangSmith, Langfuse, Phoenix.
- What Break does not claim.
- The reference: the commands, the settings file, the stored runs, the library.
- Extending Break: a fault, a way in, a strategy, a check of your own.
What Break does not claim
Section titled “What Break does not claim”A hunt that finds nothing has shown that the conditions it tried, on the cases you gave it, did not fail more than the control in the runs it had. It has not shown more than that, and Break does not say it has. A finding says the agent fails more under one named condition; it does not say how often that condition happens to you. Every test, interval and p-value Break prints beside a finding is computed by Trace or Prove; Break itself only counts runs and divides. Each finding prints the command that gives the full report. What Break does not claim sets the limits out.
Status
Section titled “Status”This is version 0.1.0, an alpha. It has one way in, rollout: an agent Break can call as a
Python function. It changes tool behaviour (ten faults) and the request (five operators), and
searches with two strategies that do not learn. Adapters for agent frameworks and MCP servers,
changes to the conversation and to the model’s own calls, and a search that learns are not in
this version; the catalogue names what waits for them. It runs on macOS and Linux, on Python
3.11 or later.
Break depends on Prove and Trace, each at one exact version, and pip installs both from
PyPI with it. Trace is a pre-release there, 0.1.0a1.
The reference has the commands, the settings file, how runs are stored, and the library under the command.