Catch a planted regression in a fintech back-office agent
The change on this page is planted. We broke the agent on purpose, in a way we chose and
know, so that you can see what catching a regression looks like when the answer is known in
advance. Nobody should make this change. On your own agent you will not know which step broke,
or whether anything did; this page shows what Prove and Trace print when something has.
The agent does fintech back-office operations, the casework a payments company’s operations
team does by hand, on Claude Sonnet 5.5 in a synthetic payments back office. It works four kinds
of case: compliance requests for information (RFIs), reconciliation breaks, late payouts and
failing payment rails. It reads what it needs through six read-only tools and ends each case with one
action, and every pick is scored by code against the answer the world’s generator knows:
right, wrong, harmful or no_action. The world is synthetic: the transactions, end users,
cases and six providers are invented; the rails’ clocks, banking calendars and FX fixings copy
public rules and official series.
The planted change: read_identity_record, the tool an RFI is answered from, times out on every
call, as if the service behind it had gone down. The agent’s loop stops an episode at a tool that
times out, as a harness does with a tool that hangs. The other three kinds of case never read
the identity record, so they are the control: they should not move.
The world is a function of its seed. ensure generates it, about six minutes, and checks every
table, the answer key and the context documents against the digests the repository pins.
Terminal window
python -m backoffice_world ensure world
Output1 line · exit 0
ok world generated at world, seed 2026, digest 4f6b38229f4a, equal to the pin
The agent prices every model call from the usage the API returns and writes it to a ledger.
plan states what the two sweeps will cost, from per-case figures measured on an earlier run of
this agent; at the cap an episode makes no call.
Terminal window
python -m backoffice_agent plan planted-regression.toml --sweeps sonnet planted
Output6 lines · exit 0
plan 2 sweeps of 400 episodes each: 4 action types x 5 cases x 20 replicates
sonnet claude-sonnet-5-5 about $14.90
planted claude-sonnet-5-5 about $14.90
all about $29.80; already spent $0.00 (spend.jsonl)
cap $35.00: at the cap an episode makes no call and raises (BACKOFFICE_CAP_USD)
per case: measured on an earlier run of this agent, by action type
Four kinds of case by five fixed cases each, twenty conditions, twenty replicates. Prove’s dry
run checks the design against the change you accept, 0.20 by default, and refuses it:
The two next: lines are Prove’s two ways forward: 238 replicates a condition, about $180 a
sweep, or accept that one condition alone can only be flagged for a larger change. For the paired
sweep Prove’s limit is 0.729, so both sweeps run with --max-mde 0.729. The power check stays on
at that size, and each sweep prints the change it accepts. A tool that stops every RFI is far
larger than that. A subtle regression, a few points in one kind of case, would go unseen at this
size; that is what this design could not have detected.
poisson-prove sweep planted-regression.toml --yes --max-mde 0.729 --name before
Outputexcerpt, 4 of 22 lines · exit 0
⋯ 6 lines
smallest rise one condition alone would be flagged for 0.670 at its worst base rate, 0.135 (Fisher, q 0.05 over 80 tests, 4 times in 5; you accept up to 0.729)
⋯ 10 lines
done 400 rollouts in run "before" (complete) in 37 min 36 s
episodes with an outcome 395
episodes with no outcome 5
⋯ 2 lines
All output22 lines
plan 400 rollouts of back-office-agent-planted: 20 cells x 20 replicates on local, $0 (local)
target backoffice_agent.agent:rollout_sonnet (imports, takes cfg and seed)
outcomes failures: harmful, no_action, wrong; successes: right
Sweep: 2 axes · 20 cells · N=20 → 400 rollouts
Backend: local · target: back-office-agent-planted · t_unit: tool_calls
Cost: $0 (local)
smallest rise one condition alone would be flagged for 0.670 at its worst base rate, 0.135 (Fisher, q 0.05 over 80 tests, 4 times in 5; you accept up to 0.729)
40/400
80/400
120/400
160/400
200/400
240/400
280/400
320/400
360/400
400/400
done 400 rollouts in run "before" (complete) in 37 min 36 s
episodes with an outcome 395
episodes with no outcome 5
episodes timed out 0
wall time 37 min 36 s
6. The agent with the planted change, on the same seeds
smallest rise one condition alone would be flagged for 0.728 at its worst base rate, 0.135 (the paired test, q 0.05 over 80 tests, 4 times in 5; you accept up to 0.729)
⋯ 10 lines
done 400 rollouts in run "planted" (complete) in 34 min 40 s, paired with "before"
episodes with an outcome 395
episodes with no outcome 5
⋯ 3 lines
All output23 lines
plan 400 rollouts of back-office-agent-planted: 20 cells x 20 replicates on local, $0 (local)
target backoffice_agent.planted:rollout_planted (imports, takes cfg and seed)
outcomes failures: harmful, no_action, wrong; successes: right
Sweep: 2 axes · 20 cells · N=20 → 400 rollouts
Backend: local · target: back-office-agent-planted · t_unit: tool_calls
Cost: $0 (local)
smallest rise one condition alone would be flagged for 0.728 at its worst base rate, 0.135 (the paired test, q 0.05 over 80 tests, 4 times in 5; you accept up to 0.729)
40/400
80/400
120/400
160/400
200/400
240/400
280/400
320/400
360/400
400/400
done 400 rollouts in run "planted" (complete) in 34 min 40 s, paired with "before"
episodes with an outcome 395
episodes with no outcome 5
episodes timed out 0
wall time 34 min 40 s
next: poisson-prove diff
Five episodes in each run, all of them reroutes, are stored with no outcome. The model asked for
more tool calls in one turn than the episode had left, and the example’s loop, at the commit that
ran, counted the calls it refused, so the episode’s step count passed its horizon and Prove
rejected it. That was a bug in the example, not in the agent or the tools; it is fixed in the
repository, where such an episode now stops at its twelfth call with no action. The reroutes are
the control here, and Prove compares the twins it has.
method paired test on shared seeds (McNemar), the default for runs paired at sweep time
before before
after planted
trade action rfi · case 2: wrong fell, no_action rose
verdict rule the verdict counts changes at the family-wise level (Holm, q 0.05 over 80 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 80 tests
limits the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 80 tests); 2 rows that did not count have no limit either way (none detectable: 2), so no change of any size can be excluded there; 3 rows that did not count have no limit on their fall (none detectable: 3), so a fall of any size cannot be excluded there; 3 rows that did not count have no limit on their rise (none detectable: 3), so a rise of any size cannot be excluded there
intervals on each condition's change: Tango at 99.375%, false coverage (Benjamini-Yekutieli) over the 10 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
fragility 9 outcomes would change the word (the search is not exhaustive: it may be fewer): 9 runs of A in action rfi, case 2 that ended in wrong made to end otherwise
inferred: no event was given, so the event set is harmful, no_action, wrong, right, the failures and successes the runs declared
condition event before after change verdict rise fall
action break · case 1 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 1 right 1.000 1.000 0.000 unchanged no room 0.616
action break · case 1 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 1 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 2 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 2 right 1.000 1.000 0.000 unchanged no room 0.616
action break · case 2 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 2 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 3 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 3 right 1.000 1.000 0.000 unchanged no room 0.616
action break · case 3 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 3 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 4 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 4 right 1.000 1.000 0.000 unchanged no room 0.616
action break · case 4 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 4 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 5 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 5 right 1.000 1.000 0.000 unchanged no room 0.616
action break · case 5 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action break · case 5 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action reroute · case 1 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action reroute · case 1 right 1.000 1.000 0.000 unchanged no room 0.616
action reroute · case 1 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action reroute · case 1 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 2 harmful 0.000 0.000 0.000 unchanged 0.663 no room
action stuck · case 3 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 3 right 1.000 1.000 0.000 unchanged no room 0.616
action stuck · case 3 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 3 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 4 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 4 right 1.000 1.000 0.000 unchanged no room 0.616
action stuck · case 4 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 4 harmful 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 5 no_action 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 5 right 1.000 1.000 0.000 unchanged no room 0.616
action stuck · case 5 wrong 0.000 0.000 0.000 unchanged 0.616 no room
action stuck · case 5 harmful 0.000 0.000 0.000 unchanged 0.616 no room
no outcome 5 episodes of the first run and 5 of the second ended with none, and are in neither the shares nor the tests
more --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound
next: poisson-prove report before planted --x action --y case
Every RFI case got worse: no episode stopped without an action before, every one does now, and
the share right fell with it. In case 2 the agent was wrong on every episode before, so there
wrong fell as no_action rose; that is why the first word is trade, Prove’s word for changes
both ways. The breaks, late payouts and reroutes show no significant change, each at the
detection limit in its row.
ok 119 of 395 tries failed; their last steps: 100 at read_identity_record and 19 at read_payment (not necessarily the cause)
# Poisson Trace · where runs die · planted
## Summary
- Read: planted (back-office-agent-planted): 395 tries, each placed at the step where it stopped.
- Question: At which step do failing tries stop?
- Answer: 119 of 395 failed. Their last steps: 100 at read_identity_record and 19 at read_payment (not necessarily the cause), between step 5 and step 12.
- Next: See whether a change moved this: `poisson-trace compare before planted --store prove-runs --event no_action --max-mde-hr X`, choosing X before you look.
## Before you act on this
- The step is where a try stopped, not where it went wrong; the kind of step is that of its last step, not the cause. Check what passing tries end on before reading it as one part's fault.
- 5 rows are not placed (no trace 5), so they are in none of these counts.
- One run shows where tries stop, not whether a change moved anything.
## Run
- run: planted (20261109T203356Z-35dd) · target: back-office-agent-planted · t_unit: tool_calls · status: complete · no cap
- Prove's tally: rows 400 · with outcome 395 · no outcome 5 · timed out 0 · infra 0 · other 5
- placement: placed 395 · died 119 · ended 276 · timed out 0 · capped 0 · excluded 5 (no trace 5)
- events: no_action
## Share not yet failed, by step
Aalen–Johansen over leaf-span index: finishing without failing and timing out are ends of their own, and only capped tries are censored; band: bootstrap 95%, resampling inputs, B = 1,000, seed = 0.
| step | at risk | failed | not yet failed | 95% band |
|---|---|---|---|---|
| 5 | 395 | 3 | 0.992 | 0.977 – 1.000 |
| 6 | 156 | 81 | 0.787 | 0.591 – 0.949 |
| 7 | 58 | 16 | 0.747 | 0.545 – 0.901 |
| 12 | 35 | 19 | 0.699 | 0.465 – 0.896 |
median step of the failed tries: 6
the share failed stayed under one half: 0.301 by the last failure
## Where they die, by step name
Aalen–Johansen: each kind's share of tries failed by step 12, and the share that timed out; band: bootstrap 95%, resampling inputs, B = 1,000, seed = 0.
the kinds' shares add to the share failed, 1 − not yet failed = 0.301 (1 − 0.699); timed out is an end of its own and not among them
## Declared steps
Reached by k of N placed runs, from Prove's declaration; the set of steps never reached is Prove's own.
| step | reached by | note |
|---|---|---|
| draft_reroute | 76 of 395 | undeclared |
| episode | 395 of 395 | undeclared |
| flag_late_payout | 100 of 395 | undeclared |
| read_break | 100 of 395 | undeclared |
| read_case | 100 of 395 | undeclared |
| read_context | 395 of 395 | undeclared |
| read_identity_record | 100 of 395 | undeclared |
| read_payment | 135 of 395 | undeclared |
| read_rail_health | 95 of 395 | undeclared |
| resolve_break | 100 of 395 | undeclared |
## What this cannot say
- The step index is where the run stopped, not why. A run that died at a tool step may have gone wrong three steps earlier.
- Capped runs are censored at their last recorded step. If they would have died soon after, the curve is optimistic by that much. A run that timed out ends where it stopped, counted beside the kinds and not as a death; the header counts both.
- Excluded rows are not on the curve; the header counts them by reason.
- On one run, this is a description. Only a comparison of two stored runs on the same cells says whether anything moved, and only within what it could not have detected.
next: poisson-trace compare before planted --store prove-runs --event no_action
compare tests each step on its own: among episodes still going at a step, did more of them stop
there after the change? Trace asks for the smallest change you want to see before it looks; here,
a 1.5-fold change in a step’s rate.
# Poisson Trace · compare · before (before) and planted (after)
## Summary
- Compared: before before against after planted, on the same 20 inputs: 395 tries each. A try counts as failed when it ended no_action.
- Question: Step by step, among tries still going, did the after version fail more or less often? You asked to see that rate change 1.5-fold, either way.
- Answer: Failures come sooner. 119 of 395 tries failed after and 22 of 395 before; if each input failed equally often under both versions, a gap at least this big, either way, would turn up less than once in 1,000 (counting tries as independent); after against before, the per-step failure rate is 24.68 times as high (15.05 to 40.46).
- Next: Compare how often each version fails overall: `poisson-trace fix-check before planted --store prove-runs --event no_action --inputs all --max-mde X`, choosing X before you look.
## Before you act on this
- They started 37 min apart. Trace can't tell whether they were run side by side, so anything else that changed between them (the provider, the model behind the API, the day) may be part of the difference, in either direction.
- Both runs name the same target (back-office-agent-planted); Trace can't tell what differs between them, only what you tell it.
- A try that finished without failing counts as still going up to its last step, so this mixes how often and how early tries fail. Tries of the same input are counted as separate chances; if they are near copies, this is more confident than it should be.
- A 2.03-fold change means the per-step failure rate among tries still going rises 2.03 times, or falls to 0.49 of what it was; it is not the ratio of the failure counts.
- This asks whether failures come sooner or later among tries still going. Whether a version fails more often overall is fix-check's question, and on the same two runs the two can read differently.
- Before putting the difference down to the version, run both versions side by side.
## Run A (before)
- run: before (20261109T195619Z-efd7) · target: back-office-agent-planted · t_unit: tool_calls · status: complete · no cap
- Prove's tally: rows 400 · with outcome 395 · no outcome 5 · timed out 0 · infra 0 · other 5
- placement: placed 395 · died 22 · ended 373 · timed out 0 · capped 0 · excluded 5 (no trace 5)
- events: no_action
## Run B (after)
- run: planted (20261109T203356Z-35dd) · target: back-office-agent-planted · t_unit: tool_calls · status: complete · no cap
- Prove's tally: rows 400 · with outcome 395 · no outcome 5 · timed out 0 · infra 0 · other 5
- placement: placed 395 · died 119 · ended 276 · timed out 0 · capped 0 · excluded 5 (no trace 5)
- events: no_action
## Comparison
log-rank, stratified by config cell over 20 common cells (excluded: 0 in A only, 0 in B only), unpaired, though the runs share a run_salt; α = 0.05.
| arm | placed | died | observed | expected |
|---|---|---|---|---|
| A (before) | 395 | 22 | 22 | 72.373 |
| B (after) | 395 | 119 | 119 | 68.627 |
statistic: χ² = 161.488 on 1 d.f., variance 15.713, p = 5.35e-37
direction: runs die later in A (before) than in B (after): runs die earlier in B
hazard ratio, after against before: 24.68 (15.05 to 40.46, 95%); Peto's one-step estimate from the observed, expected and variance above
The test compares, step by step, the failure rate among tries still going; the rows by step below and the curves `poisson-trace where` draws give the share of tries failed by a step. A version whose tries finish sooner can differ in one and not the other.
### Per step
cause-specific log-rank, stratified by config cell, deaths of other steps censored at their step; Benjamini–Hochberg at q = 0.05.
| step | deaths A | deaths B | χ² | p | q | could not have detected a hazard ratio below |
could not have detected a hazard ratio below 2.027 (from the log-rank's own information, its variance 15.713 over the common cells; 80% power, α = 0.05)
**different: runs die earlier in B (after) than in A (before) (log-rank, stratified by cell, p = 5.35e-37)**
## Failures by kind
Each failed try's kind of failure, by the built-in rules; the same rules classified both runs. Counts, not a test: the verdict above is about all failures together.
of 400 tries of A and 400 of B; a try has one kind
Tries with no outcome: 5 of A's 400 and 5 of B's 400; the counts above include them. The verdict counts only inputs both runs carry with an outcome, so it counts 395 tries of each.
Signals seen on tries, failed or not:
| signal | A | A share | B | B share |
|---|---|---|---|---|
| over_budget | 34 | 8.5% | 135 | 33.8% |
For the share of runs that fail, input by input, and the inputs that changed: poisson-trace fix-check A B --inputs all
## What this cannot say
- The step index is where the run stopped, not why. A run that died at a tool step may have gone wrong three steps earlier.
- Capped runs are censored at their last recorded step. If they would have died soon after, the curve is optimistic by that much. A run that timed out ends where it stopped, counted beside the kinds and not as a death; the header counts both.
- Excluded rows are not on the curve; the header counts them by reason.
- On one run, this is a description. Only a comparison of two stored runs on the same cells says whether anything moved, and only within what it could not have detected.
next: poisson-trace fix-check before planted --store prove-runs --event no_action --inputs all
read_identity_record went from no stopped episodes to 100, the planted step. The 22 and 19
episodes that stop after read_payment are late-payout cases where the agent read the payout and
gave up without an action, before and after alike; Trace finds no significant change there, and
the last column says how large a change it could have missed.
A, before: the agent as it isB, planted: read_identity_record times outsignificantly more episodes stopped here, q < 0.05
The steps as Trace’s compare --json gave them, drawn by this site: episodes that ended without
an action, by the step they stopped at, in each run, with Trace’s q for each step.
fix-check asks the overall question across all twenty cases at once, case by case:
Terminal window
trace-env/bin/poisson-trace fix-check before planted --store prove-runs --event no_action --inputs all --max-mde 0.10
Outputexcerpt, 2 of 120 lines · exit 0
different the new version failed more often
⋯ 7 lines
- Answer: More failures. The new version failed 119 of 395 tries and the old 22 of 395 (25 more per 100 tries); 100 of the net 98 extra failures come from 5 inputs.
⋯ 111 lines
All output120 lines
different the new version failed more often
# Poisson Trace · fix check · before (old) and planted (new)
## Summary
- Compared: old before against new planted, on the same 20 inputs: 395 tries of each. A try counts as failed when it ended no_action.
- Question: Did the new version fail more or less often than the old? You asked that the run be able to catch a 10-point drop in how often it fails.
- Answer: More failures. The new version failed 119 of 395 tries and the old 22 of 395 (25 more per 100 tries); 100 of the net 98 extra failures come from 5 inputs.
- Next: See which inputs moved: `poisson-trace changes before planted --store prove-runs --event no_action`.
## Before you act on this
- They started 37 min apart. Trace can't tell whether they were run side by side, so anything else that changed between them (the provider, the model behind the API, the day) may be part of the difference, in either direction.
- Both runs name the same target (back-office-agent-planted); Trace can't tell what differs between them, only what you tell it.
- 5 tries of the old and 5 of the new had no outcome, and they are left out.
- These results fit anything from 20 to 30 more failures per 100 tries, but that range is only a rough guide here and may be off either way (fewer than 10 inputs vary): don't use it to rule sizes in or out.
- Tries of the same input are counted as separate chances; here they are fairly alike (agreement 0.50, where 0 is no more alike than chance and 1 is identical), and if they are near copies the answer is more confident than it should be.
- The 10 points you asked about is close to the most these inputs could drop (the old version failed 5.6% of tries).
- If each input failed equally often under both versions, a gap at least this big, either way, would turn up less than once in 1,000 (counting tries as independent).
- Before putting the difference down to the version, run both side by side.
- Prove's tally (new): rows 400 · with outcome 395 · no outcome 5 · timed out 0 · infra 0 · other 5
- the two runs started 37 min apart
- events: no_action
- inputs: every input either run carries · selected 20 · compared 20 · left out 0
- rows used: old 395 (timed out 0, no outcome 5) · new 395 (timed out 0, no outcome 5)
- inputs short of the declared repeats: old 1 · new 2
- inputs with mixed outcomes across their repeats: old 2 · new 2
## The two versions
The old version and the new on the same inputs.
| version | runs | failed | failure share | did not fail |
|---|---|---|---|---|
| old | 395 | 22 | 5.6% | 94.4% |
| new | 395 | 119 | 30.1% | 69.9% |
The old version did not fail on 94.4% of its episodes on these inputs.
## Failures by kind
Each failed try's kind of failure, by the built-in rules; the same rules classified both runs. Counts, not a test: the verdict above is about all failures together.
| kind | old | old share | new | new share | change |
of 400 tries of the old version and 400 of the new; a try has one kind
Tries with no outcome: 5 of the old version's 400 and 5 of the new version's 400; the counts above include them. The verdict counts only inputs both runs carry with an outcome, so it counts 395 tries of each.
Signals seen on tries, failed or not:
| signal | old | old share | new | new share |
|---|---|---|---|---|
| over_budget | 34 | 8.5% | 135 | 33.8% |
## The comparison
exact conditional Cochran-Mantel-Haenszel test, stratified by input over 20 inputs (7 informative), two-sided; α = 0.05.
new failures: observed 119, expected 70.119, variance 14.613; p = 8.48e-45
difference: +24.8 points, new minus old (+19.8 to +30.0; Mantel-Haenszel weights; range: stratified Miettinen-Nurminen score interval, 95%, for these inputs as run; rough here: fewer than 10 inputs vary)
by input: p = 0.047 (sign-flip over the 7 inputs that moved, exact); not part of the line below
## Consistency
failure share 0.178 · fails again 0.587 · κ 0.497 · within-input variance 0.075 against pooled 0.147
a comparison that ignored the inputs would need 1/(1 − κ) = 1.99 times the runs
## What it could not have detected
proportional change on every input, 80% power, α = 0.05, continuity-corrected normal approximation to the stratified statistic.
- a drop in failure share: could not have detected one below 5.7 points
- a rise in failure share: could not have detected one below 6.5 points
**different: the new version failed more on these inputs (exact Cochran-Mantel-Haenszel, stratified by input, p = 8.48e-45)**
## Inputs that changed
Fisher's exact test per input, Benjamini-Hochberg at q = 0.05 over 20 inputs; 5 flagged.
| input | old failed/runs | new failed/runs | change | p | q | flag |
|---|---|---|---|---|---|---|
| {"action":"rfi","case":"1"} | 0/20 | 20/20 | +100.0 points | 1.45e-11 | 5.80e-11 | failed more in new |
| {"action":"rfi","case":"3"} | 0/20 | 20/20 | +100.0 points | 1.45e-11 | 5.80e-11 | failed more in new |
| {"action":"rfi","case":"5"} | 0/20 | 20/20 | +100.0 points | 1.45e-11 | 5.80e-11 | failed more in new |
| {"action":"rfi","case":"4"} | 0/20 | 20/20 | +100.0 points | 1.45e-11 | 5.80e-11 | failed more in new |
| {"action":"rfi","case":"2"} | 0/20 | 20/20 | +100.0 points | 1.45e-11 | 5.80e-11 | failed more in new |
10 of 20 inputs shown, by ascending p; --json lists all.
A lone changed input is flagged only if its p is below α / M = 0.0025, and the smallest p an input of this design can give is 1.45e-11: a lone change can be flagged; a lone complete flip needs 6 repeats per arm. An input with no flag is not an input that did not change.
## What this cannot say
- A difference between two stored runs is a difference between the runs. The version, the provider and the hour changed together unless the two were run side by side, and this test cannot separate them.
- Repeat runs of one input are read as separate draws and never as replays: a served model is not bit-identical on repeat, so nothing here matches a run of the old version to a run of the new by seed.
- If repeat runs of one input are near copies of one another, the run-level test overstates its confidence. The by-input line does not depend on that.
- Rows with no outcome are left out of both arms and counted in the header. A new version that times out where the old one failed does not show as a failure here.
- Trace reads the runs it is given. If the inputs were chosen by the old arm's own outcomes, or the more favourable of several runs was handed over, the confidence printed does not hold.
- The detection limit and the run counts are computed from these runs' own failure rates, which are estimates. They are a guide to the next design, not a promise about it.
next: poisson-trace changes before planted --store prove-runs --event no_action
claude-sonnet-5-5 $25.73 1,829 calls, 10,114,047 tokens in, 550,561 out
The plan stated $29.80 for the two sweeps. The ledger, priced by the agent from the usage the API
returned at Anthropic’s published rates, came to $25.73: $13.85 for the agent as it is and $11.88
with the planted change, cheaper because the broken RFIs stop early.
That Trace finds causes. A step is where an episode stopped, not where it went wrong. Here the
two are the same step because we planted them that way; on your own agent, a failure that
stops at a step may have gone wrong several steps before.
Anything about a small regression. At twenty replicates a condition, Prove could only flag one
condition alone for a change of about 0.73; the planted change is near the largest a change can
be.
Anything about a real system. The world, the cases and the providers are synthetic.
Every output above is what the command printed, stored in the site repository under
src/data/examples/catch-a-planted-regression/ with the run’s diff.json, the three files
report wrote, and Trace’s compare --json, as Prove 0.2.2 and Trace 0.1.0rc2 wrote them. The
page cannot show an output that is not there.