breakways-in
The rollout way in
A way in is how Break runs your agent and reaches its tools. This version has one, rollout:
your agent as a Python function that Break calls once for each run. If you can call your agent
from Python, you can hunt it this way, whatever it is built with.
A whole agent to start from, at the end of this page, is two files you can copy and edit, and the commands that take them to a hunt.
The function
Section titled “The function”from poisson_prove import Episode
from poisson_break.apply import wrap
def _lookup_order(order: str) -> str: ... # your tool, as it is
lookup_order = wrap("lookup_order", _lookup_order)
def refund_agent(cfg, seed): reply = run_my_agent(cfg["input"]) # your agent, calling lookup_order as it always does return Episode(outcome="resolved", t=1.0, horizon=1.0, meta={"reply": reply})Two things make a function a rollout target.
It takes a case and a seed and returns an Episode. cfg is one case from your case file,
as a mapping: its input, and any other key you put on the line. seed is the run’s seed; an
agent that draws random numbers should draw them from it, so that a run can be replayed.
Episode is Poisson Prove’s record of one run. Break reads two things from it:
| Field | What Break reads |
|---|---|
outcome | your agent’s own word for how the run ended. List the words that mean failure in [fails] outcomes |
meta | optional facts: reply (what the agent answered), tokens_in and tokens_out (for prices and caps), and ended = "limit" when the agent stopped at its own step limit |
t and horizon are Prove’s own fields: the time at which the run ended, and the longest a
run could go. Break’s verdicts do not use them, and an agent with no such measure passes
t=1.0, horizon=1.0.
Each tool is wrapped once. wrap(name, function) returns the tool with a hook in it. Outside
a run the wrapped tool is the tool itself, so the line can stay in your code. Inside a run,
each call goes through Break, which either passes it on or puts a fault in its place. A fault
arrives where the tool’s own failure would: an error is an exception raised from the call,
and a timeout is a TimeoutError. Wrap a tool where it is defined, at the top of a module,
so that init can find it by importing that module.
An async def tool is refused; wrap a function that is not async.
The settings
Section titled “The settings”init asks where the agent is, where the cases are and, for each tool it finds, whether
calling it changes anything, and writes break.toml; every other setting takes its default,
and init lists what it chose. A flag answers its own question, and --yes takes the default
for every question left, so init --entry my_app.agents:refund_agent --yes asks nothing. These
are the lines that describe the target:
[target]kind = "rollout"entry = "my_app.agents:refund_agent" # module:name, importable from the folder you run incases = "cases.jsonl"
[tools]lookup_order = "read"issue_refund = { class = "write", stand_in = "my_app.fakes:issue_refund" }send_email = "off"The command puts the folder it runs in first on the import path, so a module beside
break.toml is found.
Tool classes
Section titled “Tool classes”Break calls your tools for real, many times. So every tool your agent calls has a line in
[tools] that says what calling it does.
| Line | Meaning |
|---|---|
"read" | calling it changes nothing; Break may call it any number of times |
{ class = "write", repeatable = true } | it changes something, and your test environment absorbs repeats |
{ class = "write", stand_in = "module:name" } | it changes something; Break calls the stand-in in its place in every run, with the same arguments |
"off" | Break never touches it: no fault is put on it, and its calls pass straight through |
A write tool with neither repeatable nor a stand-in is refused before any run. A tool your
agent calls that has no line stops the hunt, and Break prints the init command that classes
it off so that the hunt can go on.
A write tool with its stand-in
Section titled “A write tool with its stand-in”The stand-in example is a refund desk whose issue_refund asks a payments service to pay a
customer back. Its break.toml names a stand-in for that tool, so the real function never
runs.
poisson-break init --example stand-inpoisson-break hunt --yespoisson-break confirm --yespoisson-break replay 1The hunt ends with one candidate:
found 1 candidate in 27 runs control 12 runs · 0 failed (0.000) search 15 runs · 5 conditions tried on 3 cases each condition failed kind 1 tool:issue_refund#1=lost_reply 3 of 3 failed duplicate_write not looked at: faults turned off: slow, rate_limited, empty, cut_off, malformed, stale, planted_instruction; 0 single changes and 6 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/refund-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmlost_reply lets the tool act and then reports an error, as a network does when a reply is
lost on the way back. This desk answers a failed refund by sending it again, so the customer is
refunded twice. Break’s duplicate_write check sees the same write tool run again with the
same arguments after a lost reply. The replay shows the two calls:
note: Break calls your tools for real: run it against a test environment, never against anything that mattersfailed finding 1 on case damaged-4821, seed 32408516: duplicate_write tool call change real tool ran 1 lookup_order 1 none yes 2 issue_refund 1 tool:issue_refund#1=lost_reply yes 3 issue_refund 2 none yesIn the last column, the tool that ran for issue_refund is the stand-in. The example’s real
issue_refund would open a connection to the payments service, and in a hunt it is never
called.
The case file is JSON Lines, one case a line. A case needs only input:
{"input": "Where is order 7?"}{"id": "refund-late", "input": "Refund order 9.", "expect": {"calls": ["lookup_order", "issue_refund"], "reply_contains": "refunded"}}{"id": "gold", "input": "Where is order 12?", "customer_tier": "gold"}A case with no id is numbered by its place in the file. expect adds rules for that case: a
run is wrong when the reply lacks reply_contains or when the tools in calls were not
called in that order, and a run that calls a tool in never_calls fails as forbidden_call. Any other key is your agent’s own
and reaches it in cfg. To make cases from requests your users already sent, see the recipes
for LangSmith, Langfuse and
Phoenix.
Check it before you hunt
Section titled “Check it before you hunt”poisson-break doctordoctor reads the settings and the cases, imports the agent, lists its tools and names the
first problem it finds. doctor --run also runs the first case once with no change, after
asking, and stores nothing. In the example’s folder it prints:
note: Break calls your tools for real: run it against a test environment, never against anything that mattersready refund_desk:refund_desk loads, with 2 tools and 6 cases tools lookup_order, issue_refundnext: poisson-break huntWhat rollout cannot do
Section titled “What rollout cannot do”Break reaches a rollout target’s tools and its request. It does not sit between the agent and
its model, and it does not play a conversation’s turns. So two kinds of change in the catalogue
are refused for a rollout target before any run: the conversation moves and the model faults.
They wait for adapters that are not in this version. init never turns them on, and when
[vary] conversation or [vary] model in break.toml names one, doctor, hunt, confirm
and recheck each say which line to empty and print the init command that writes the file
with it emptied. An agent that is not reachable as a Python
function, such as one behind an HTTP service, also needs a way in this version does not have.
Adding a way in gives the interface one is written against.
A whole agent to start from
Section titled “A whole agent to start from”This is a whole agent you can copy and edit: a bookshop desk with one tool that reads the stock and one that holds a copy, each wrapped once. It has no model, so it needs no key. Make a folder for it:
mkdir shopcd shopSave this as shop.py:
"""A bookshop desk: one tool reads the stock, one holds a copy. No model, no key."""
import re
from poisson_prove import Episode
from poisson_break.apply import wrap
STOCK = {"dune": 3, "emma": 0, "ulysses": 1}
def _check_stock(title): return STOCK.get(title)
def _hold_copy(title): return f"hold-{title}"
check_stock = wrap("check_stock", _check_stock)hold_copy = wrap("hold_copy", _hold_copy)
def shop(cfg, seed): title = re.search(r"hold (\w+) for me", cfg["input"]).group(1) if not check_stock(title): reply = f"Sorry, {title} is out of stock." return Episode(outcome="out_of_stock", t=1.0, horizon=1.0, meta={"reply": reply}) ticket = hold_copy(title) reply = f"I have held {title} for you: {ticket}." return Episode(outcome="held", t=1.0, horizon=1.0, meta={"reply": reply})Save these three cases as cases.jsonl. Each says what its reply must hold, so a run whose
reply lacks it fails as wrong:
{"id": "dune", "input": "Can you hold dune for me?", "expect": {"reply_contains": "hold-dune"}}{"id": "emma", "input": "Can you hold emma for me?", "expect": {"reply_contains": "out of stock"}}{"id": "ulysses", "input": "Can you hold ulysses for me?", "expect": {"reply_contains": "hold-ulysses"}}Then write break.toml with init, given everything by its flags, check that the agent loads,
and hunt:
poisson-break init --entry shop:shop --cases cases.jsonl \ --tool 'check_stock = "read"' \ --tool 'hold_copy = { class = "write", repeatable = true }' --yespoisson-break doctorpoisson-break hunt --yescheck_stock is classed read, since looking at the stock changes nothing. hold_copy changes
something, and repeatable = true says your test environment absorbs a hold made twice; the
other lines a tool can have are under Tool classes. To make it your own agent,
put your tools in place of the two here, keep one wrap line for each, and have shop call your
agent and hand back its reply in the Episode.