Find where an agent fails with Break
Break ships five small agents to try it on. Each has a scripted model in place of a real one: a
function that decides what the agent does next from the request and from what its tools
returned. So each runs with no key and no network, and does the same thing every time.
poisson-break init --example NAME writes one into the folder you are in. init will not write
over a file that is already there, so each example below is made in a new folder of its own.
Before you start: Install and Your machine.
1. desk: a first hunt and confirm
Section titled “1. desk: a first hunt and confirm”This is a help desk for an online shop. It looks an order up, and for a refund request it also reads the refund policy. When the look-up fails it gives up; when the policy cannot be read it answers without it.
mkdir deskcd deskpoisson-break init --example deskhunt prints its plan, runs the desk with no change (the control), then tries one change at a
time and keeps the ones under which the desk failed.
poisson-break hunt --yes⋯ 9 linesfound 3 candidates in 42 runs control 12 runs · 0 failed (0.000) search 30 runs · 10 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=timeout 3 of 3 failed failed 2 tool:lookup_order#1=rate_limited 3 of 3 failed failed 3 tool:lookup_order#1=error 3 of 3 failed failed not looked at: faults turned off: slow, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 33 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmAll output20 lines
plan a hunt of desk on 6 cases, at most 120 runs target desk_agent:desk (rollout) tools lookup_order (read), check_policy (read) may change tool behaviour (4 faults), inputs (2 operators) strategy screen stages control 12 runs, search and shrink up to 108 cap 120 runs, 2 hours estimate at most 120 runs and 2 hours, with no prices to put in dollars, from the settings' limitsnote: Break calls your tools for real: run it against a test environment, never against anything that mattersfound 3 candidates in 42 runs control 12 runs · 0 failed (0.000) search 30 runs · 10 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=timeout 3 of 3 failed failed 2 tool:lookup_order#1=rate_limited 3 of 3 failed failed 3 tool:lookup_order#1=error 3 of 3 failed failed not looked at: faults turned off: slow, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 33 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmThe desk never failed in its 12 control runs. Of the 10 conditions the hunt tried, three made it
fail on every case they were tried on, and all three are on the first call to lookup_order.
Those three are candidates, kept because they failed in a few runs each. confirm runs each of
them again on every case, three times, beside a control arm that runs the same cases with no
change, and asks Trace whether the difference holds.
poisson-break confirm --yes⋯ 7 linesconfirmed 3 of 3 candidates fail more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=timeout confirmed 1.000 +1.000 2 tool:lookup_order#1=rate_limited confirmed 1.000 +1.000 3 tool:lookup_order#1=error confirmed 1.000 +1.000 wrote break-runs/desk-1/findings/1.toml, 2.toml, 3.toml reference 54 runsnext: poisson-break findings 1All output15 lines
plan confirm 3 candidates against a control design 6 cases × 3 repeats, once with no change and once under each candidate · 72 runs reference up to 54 more runs, 18 for each candidate that is confirmed cap 120 runs, 2 hours Trace will say if this is too small to see a difference of 0.200, and how many repeats would be enough estimate 126 runs, about 0 s; no prices, so no dollars, from the control stage's 0 s a runnote: Break calls your tools for real: run it against a test environment, never against anything that mattersconfirmed 3 of 3 candidates fail more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=timeout confirmed 1.000 +1.000 2 tool:lookup_order#1=rate_limited confirmed 1.000 +1.000 3 tool:lookup_order#1=error confirmed 1.000 +1.000 wrote break-runs/desk-1/findings/1.toml, 2.toml, 3.toml reference 54 runsnext: poisson-break findings 1All three hold, so the desk ends with three findings: a timeout, a rate limit and an error on the
order look-up each end the request. findings 1 reads the first of them.
poisson-break findings 1ok finding 1: tool:lookup_order#1=timeout fails 1.000 under it, 0.000 under the control difference +1.000 p 3.1e-08, adjusted 3.1e-08, q 0.050 kinds failed 18 signals gave_up 18, retry_storm 0, over_budget 0 judged by break flagged none left out not re-run on the old version 0, not re-run on the new version 0, no outcome on the old version 0, no outcome on the new version 0 cases 6 cases × 3 repeats, in confirm stage 1 arms desk-1-confirm-1 against desk-1-confirm-control reference desk-1-reference-1next: poisson-break replay 1 --hunt desk-1replay 1 runs that finding once more and stores nothing. It lists every tool call the desk made,
with the change that landed on it. Reading a finding goes through every
line of both.
poisson-break replay 1⋯ 1 linefailed finding 1 on case status-4821, seed 56661626: failed tool call change real tool ran 1 lookup_order 1 tool:lookup_order#1=timeout noAll output4 lines
note: Break calls your tools for real: run it against a test environment, never against anything that mattersfailed finding 1 on case status-4821, seed 56661626: failed tool call change real tool ran 1 lookup_order 1 tool:lookup_order#1=timeout no2. full-loop: hunt, confirm, put a fix in, recheck
Section titled “2. full-loop: hunt, confirm, put a fix in, recheck”agent.py is an order desk that looks an order up once, and agent_fixed.py is the same desk
with a failed look-up tried again. This example goes round the whole loop.
mkdir full-loopcd full-looppoisson-break init --example full-looppoisson-break hunt --yes⋯ 9 linesfound 3 candidates in 27 runs control 12 runs · 0 failed (0.000) search 15 runs · 5 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=error 3 of 3 failed failed 2 tool:lookup_order#1=timeout 3 of 3 failed failed 3 tool:lookup_order#1=rate_limited 3 of 3 failed failed not looked at: faults turned off: slow, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 4 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/order-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmAll output20 lines
plan a hunt of order_desk on 6 cases, at most 120 runs target agent:order_desk (rollout) tools lookup_order (read) may change tool behaviour (4 faults), inputs (1 operator) strategy screen stages control 12 runs, search and shrink up to 108 cap 120 runs, 2 hours estimate at most 120 runs and 2 hours, with no prices to put in dollars, from the settings' limitsnote: Break calls your tools for real: run it against a test environment, never against anything that mattersfound 3 candidates in 27 runs control 12 runs · 0 failed (0.000) search 15 runs · 5 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=error 3 of 3 failed failed 2 tool:lookup_order#1=timeout 3 of 3 failed failed 3 tool:lookup_order#1=rate_limited 3 of 3 failed failed not looked at: faults turned off: slow, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 4 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/order-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmpoisson-break confirm --yes⋯ 7 linesconfirmed 3 of 3 candidates fail more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=error confirmed 1.000 +1.000 2 tool:lookup_order#1=timeout confirmed 1.000 +1.000 3 tool:lookup_order#1=rate_limited confirmed 1.000 +1.000 wrote break-runs/order-desk-1/findings/1.toml, 2.toml, 3.toml reference 54 runsnext: poisson-break findings 1All output15 lines
plan confirm 3 candidates against a control design 6 cases × 3 repeats, once with no change and once under each candidate · 72 runs reference up to 54 more runs, 18 for each candidate that is confirmed cap 120 runs, 2 hours Trace will say if this is too small to see a difference of 0.200, and how many repeats would be enough estimate 126 runs, about 0 s; no prices, so no dollars, from the control stage's 0 s a runnote: Break calls your tools for real: run it against a test environment, never against anything that mattersconfirmed 3 of 3 candidates fail more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=error confirmed 1.000 +1.000 2 tool:lookup_order#1=timeout confirmed 1.000 +1.000 3 tool:lookup_order#1=rate_limited confirmed 1.000 +1.000 wrote break-runs/order-desk-1/findings/1.toml, 2.toml, 3.toml reference 54 runsnext: poisson-break findings 1confirm ends with three findings, as the help desk’s does. The fix is the copy of
agent_fixed.py over the agent.
cp agent_fixed.py agent.pyrecheck runs each finding’s condition again on the agent as it is now. With --check the exit
code carries the answer: the command exits 4 when a finding fails more than it did.
poisson-break recheck --yes --check⋯ 6 linesrechecked 3 of 3 findings now fail less condition verdict numbers 1 tool:lookup_order#1=error fails less 1.000 before · 0.000 now 2 tool:lookup_order#1=timeout fails less 1.000 before · 0.000 now 3 tool:lookup_order#1=rate_limited fails less 1.000 before · 0.000 now gap 0.0 days since the reference runsAll output12 lines
plan recheck 3 findings on the agent as it is now findings 1 tool:lookup_order#1=error, 2 tool:lookup_order#1=timeout, 3 tool:lookup_order#1=rate_limited runs 54: each finding over the cases and repeats of its reference run cap 120 runs, 2 hours estimate 54 runs, about 0 s; no prices, so no dollars, from the control stage's 0 s a runnote: Break calls your tools for real: run it against a test environment, never against anything that mattersrechecked 3 of 3 findings now fail less condition verdict numbers 1 tool:lookup_order#1=error fails less 1.000 before · 0.000 now 2 tool:lookup_order#1=timeout fails less 1.000 before · 0.000 now 3 tool:lookup_order#1=rate_limited fails less 1.000 before · 0.000 now gap 0.0 days since the reference runsEach of the three findings failed in every run of its reference run and in none now, and the recheck exited 0. Checking a fix explains each part of it.
3. stand-in: a write tool that never runs
Section titled “3. stand-in: a write tool that never runs”This is a refund desk. Its issue_refund tool asks a payments service to pay a customer back,
which must not happen in a test. break.toml classes the tool write and names a stand-in,
payments_fake:issue_refund, and Break calls the stand-in in the tool’s place in every run.
mkdir stand-incd stand-inpoisson-break init --example stand-inpoisson-break hunt --yes⋯ 9 linesfound 1 candidate in 27 runs control 12 runs · 0 failed (0.000) search 15 runs · 5 conditions tried on 3 cases each condition failed kind 1 tool:issue_refund#1=lost_reply 3 of 3 failed duplicate_write not looked at: faults turned off: slow, rate_limited, empty, cut_off, malformed, stale, planted_instruction; 0 single changes and 6 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/refund-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmAll output18 lines
plan a hunt of refund_desk on 6 cases, at most 120 runs target refund_desk:refund_desk (rollout) tools lookup_order (read), issue_refund (write, stand-in) may change tool behaviour (3 faults), inputs (0 operators) strategy screen stages control 12 runs, search and shrink up to 108 cap 120 runs, 2 hours estimate at most 120 runs and 2 hours, with no prices to put in dollars, from the settings' limitsnote: Break calls your tools for real: run it against a test environment, never against anything that mattersfound 1 candidate in 27 runs control 12 runs · 0 failed (0.000) search 15 runs · 5 conditions tried on 3 cases each condition failed kind 1 tool:issue_refund#1=lost_reply 3 of 3 failed duplicate_write not looked at: faults turned off: slow, rate_limited, empty, cut_off, malformed, stale, planted_instruction; 0 single changes and 6 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/refund-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmpoisson-break confirm --yes⋯ 7 linesconfirmed 1 of 1 candidate fails more than the control (0.000) condition verdict numbers 1 tool:issue_refund#1=lost_reply confirmed 1.000 +1.000 wrote break-runs/refund-desk-1/findings/1.toml reference 18 runsnext: poisson-break findings 1All output13 lines
plan confirm 1 candidate against a control design 6 cases × 3 repeats, once with no change and once under each candidate · 36 runs reference up to 18 more runs, 18 for each candidate that is confirmed cap 120 runs, 2 hours Trace will say if this is too small to see a difference of 0.200, and how many repeats would be enough estimate 54 runs, about 0 s; no prices, so no dollars, from the control stage's 0 s a runnote: Break calls your tools for real: run it against a test environment, never against anything that mattersconfirmed 1 of 1 candidate fails more than the control (0.000) condition verdict numbers 1 tool:issue_refund#1=lost_reply confirmed 1.000 +1.000 wrote break-runs/refund-desk-1/findings/1.toml reference 18 runsnext: poisson-break findings 1There is one finding. lost_reply lets the refund go through and then reports an error. The desk
answers an error by sending the refund again, so the customer is paid twice, and Break’s
duplicate_write check sees the second call. The replay shows the two refund calls.
poisson-break replay 1⋯ 1 linefailed finding 1 on case damaged-4821, seed 32408516: duplicate_write tool call change real tool ran 1 lookup_order 1 none yes 2 issue_refund 1 tool:issue_refund#1=lost_reply yes 3 issue_refund 2 none yesAll output6 lines
note: Break calls your tools for real: run it against a test environment, never against anything that mattersfailed finding 1 on case damaged-4821, seed 32408516: duplicate_write tool call change real tool ran 1 lookup_order 1 none yes 2 issue_refund 1 tool:issue_refund#1=lost_reply yes 3 issue_refund 2 none yesThe last column reads yes for both issue_refund calls, although the write tool never runs.
The column says whether a call was passed on to the tool or answered by the fault alone. Where a
tool has a stand-in, Break passes the call to the stand-in and records the stand-in’s run as the
tool’s. So yes on these two rows means the stand-in ran. Under lost_reply the stand-in runs
first and the desk is then given an error in place of the receipt. The example’s own
issue_refund, which would ask the payments service to pay, is not called in any run.
The rollout way in says how a tool is classed.
4. own-check: a failure only your own rule can see
Section titled “4. own-check: a failure only your own rule can see”This order desk always answers. When its look-up fails it sends a confident reply anyway and
reports resolved, so none of Break’s own checks sees anything wrong. The folder’s checks.py
holds the team’s own rule, named in break.toml as [fails] own: a reply that says where the
order is, sent after every look-up failed, is made up.
mkdir own-checkcd own-checkpoisson-break init --example own-checkpoisson-break hunt --yes⋯ 9 linesfound 3 candidates in 24 runs control 12 runs · 0 failed (0.000) search 12 runs · 4 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=error 3 of 3 failed own:made_up 2 tool:lookup_order#1=timeout 3 of 3 failed own:made_up 3 tool:lookup_order#1=rate_limited 3 of 3 failed own:made_up not looked at: faults turned off: slow, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 0 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/order-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmAll output20 lines
plan a hunt of order_desk on 6 cases, at most 120 runs target desk:order_desk (rollout) tools lookup_order (read) may change tool behaviour (4 faults), inputs (0 operators) strategy screen stages control 12 runs, search and shrink up to 108 cap 120 runs, 2 hours estimate at most 120 runs and 2 hours, with no prices to put in dollars, from the settings' limitsnote: Break calls your tools for real: run it against a test environment, never against anything that mattersfound 3 candidates in 24 runs control 12 runs · 0 failed (0.000) search 12 runs · 4 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=error 3 of 3 failed own:made_up 2 tool:lookup_order#1=timeout 3 of 3 failed own:made_up 3 tool:lookup_order#1=rate_limited 3 of 3 failed own:made_up not looked at: faults turned off: slow, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 0 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/order-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmpoisson-break confirm --yes⋯ 7 linesconfirmed 3 of 3 candidates fail more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=error confirmed 1.000 +1.000 2 tool:lookup_order#1=timeout confirmed 1.000 +1.000 3 tool:lookup_order#1=rate_limited confirmed 1.000 +1.000 wrote break-runs/order-desk-1/findings/1.toml, 2.toml, 3.toml reference 54 runsnext: poisson-break findings 1All output15 lines
plan confirm 3 candidates against a control design 6 cases × 3 repeats, once with no change and once under each candidate · 72 runs reference up to 54 more runs, 18 for each candidate that is confirmed cap 120 runs, 2 hours Trace will say if this is too small to see a difference of 0.200, and how many repeats would be enough estimate 126 runs, about 0 s; no prices, so no dollars, from the control stage's 0 s a runnote: Break calls your tools for real: run it against a test environment, never against anything that mattersconfirmed 3 of 3 candidates fail more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=error confirmed 1.000 +1.000 2 tool:lookup_order#1=timeout confirmed 1.000 +1.000 3 tool:lookup_order#1=rate_limited confirmed 1.000 +1.000 wrote break-runs/order-desk-1/findings/1.toml, 2.toml, 3.toml reference 54 runsnext: poisson-break findings 1The kind own:made_up is the word the check returned, and the three findings rest on it.
findings 1 says so in its judged by line.
poisson-break findings 1ok finding 1: tool:lookup_order#1=error fails 1.000 under it, 0.000 under the control difference +1.000 p 3.1e-08, adjusted 3.1e-08, q 0.050 kinds own:made_up 18 signals gave_up 18, retry_storm 0, over_budget 0 judged by your check flagged none left out not re-run on the old version 0, not re-run on the new version 0, no outcome on the old version 0, no outcome on the new version 0 cases 6 cases × 3 repeats, in confirm stage 1 arms order-desk-1-confirm-1 against order-desk-1-confirm-control reference order-desk-1-reference-1next: poisson-break replay 1 --hunt order-desk-1Adding a check of your own explains how a check is written.
5. own-fault: a fault of your own
Section titled “5. own-fault: a fault of your own”This is an order desk, with a fault the team wrote in faults.py: error_page, under which a
tool returns a page of HTML with no error raised. The agent’s module imports faults.py, which
registers the fault, and break.toml names it in [vary] tools beside three of Break’s own.
mkdir own-faultcd own-faultpoisson-break init --example own-faultpoisson-break hunt --yes⋯ 9 linesfound 1 candidate in 24 runs control 12 runs · 0 failed (0.000) search 12 runs · 4 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=error_page 3 of 3 failed crashed not looked at: faults turned off: slow, empty, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 0 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/order-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmAll output18 lines
plan a hunt of order_desk on 6 cases, at most 120 runs target shop_agent:order_desk (rollout) tools lookup_order (read) may change tool behaviour (4 faults), inputs (0 operators) strategy screen stages control 12 runs, search and shrink up to 108 cap 120 runs, 2 hours estimate at most 120 runs and 2 hours, with no prices to put in dollars, from the settings' limitsnote: Break calls your tools for real: run it against a test environment, never against anything that mattersfound 1 candidate in 24 runs control 12 runs · 0 failed (0.000) search 12 runs · 4 conditions tried on 3 cases each condition failed kind 1 tool:lookup_order#1=error_page 3 of 3 failed crashed not looked at: faults turned off: slow, empty, cut_off, malformed, stale, lost_reply, planted_instruction; 0 single changes and 0 pairs of them untried these are candidates, not findings: a few runs each, chosen for having failed wrote break-runs/order-desk-1/hunt.json, proposals.jsonl, candidates.json and every run in store/next: poisson-break confirmpoisson-break confirm --yes⋯ 7 linesconfirmed 1 of 1 candidate fails more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=error_page confirmed 1.000 +1.000 wrote break-runs/order-desk-1/findings/1.toml reference 18 runsnext: poisson-break findings 1All output13 lines
plan confirm 1 candidate against a control design 6 cases × 3 repeats, once with no change and once under each candidate · 36 runs reference up to 18 more runs, 18 for each candidate that is confirmed cap 120 runs, 2 hours Trace will say if this is too small to see a difference of 0.200, and how many repeats would be enough estimate 54 runs, about 0 s; no prices, so no dollars, from the control stage's 0 s a runnote: Break calls your tools for real: run it against a test environment, never against anything that mattersconfirmed 1 of 1 candidate fails more than the control (0.000) condition verdict numbers 1 tool:lookup_order#1=error_page confirmed 1.000 +1.000 wrote break-runs/order-desk-1/findings/1.toml reference 18 runsnext: poisson-break findings 1There is one finding, and it names the team’s fault. The hunt puts each fault on the first
call to lookup_order only. Break’s error, timeout and rate_limited each reach the desk
as an exception raised from that call. The desk catches an exception from a look-up and calls
the tool again, up to three times, and the second call has no fault on it and returns the order
record. So those three do it no harm. error_page raises nothing: the first call returns a page
of HTML as its result, and the desk has nothing to catch. It takes the page for an order record
and splits it into the record’s two fields. That line fails on the page, it sits outside the
desk’s try, and the run ends crashed. The replay shows the call.
poisson-break replay 1⋯ 1 linefailed finding 1 on case status-4821, seed 56661626: crashed tool call change real tool ran 1 lookup_order 1 tool:lookup_order#1=error_page noAll output4 lines
note: Break calls your tools for real: run it against a test environment, never against anything that mattersfailed finding 1 on case status-4821, seed 56661626: crashed tool call change real tool ran 1 lookup_order 1 tool:lookup_order#1=error_page noAdding a fault explains how a fault is written.
Stored output
Section titled “Stored output”Every output above is what the command printed, stored in the site repository under
src/data/examples/find-where-an-agent-fails-with-break/ as poisson-break 0.1.0 printed it from
commit 37da225, each example in a new folder of its own. The run
left out 5 next: lines, each the poisson-trace fix-check command for one
finding’s full report, because each holds a full path on the machine the run was made on. Those
lines are not in the stored output, so the blocks above do not show them.