Can a fintech back-office agent move to a cheaper model?
The agent on this page does fintech back-office operations: the casework a payments company’s
operations team does by hand. It runs on Claude and works four kinds of case in a synthetic
payments back office: compliance requests for information (RFIs), reconciliation breaks, late
payouts and failing payment rails. It reads what it needs through six read-only tools and ends each case with one action. Every case has a correct answer
the world’s generator knows, so every pick is scored by code: right, wrong, harmful (a
pick that would pay twice, cite an item the record does not hold, carry a wrong number to a
provider, or reroute onto a degraded rail) or no_action.
Claude Haiku 5.5 costs about a twentieth of Claude Sonnet 5.5 per case here. The question is
the one a team asks before it cuts an agent’s model bill: does the agent fail more often on the
cheaper model, and on which kind of case? Prove runs the same agent, prompt and tools on both
models, on the same cases and the same seeds, and compares them condition by condition, with the
smallest change each condition could have caught. Trace reads the same two runs. The basic shape
of a comparison is on Check a change on runs you already have;
the rollout contract is on Run Prove on your own agent.
The world is synthetic: the transactions, end users, cases and six providers are invented. The
rails’ clocks, banking calendars and FX fixings copy public rules and official series.
The world is a function of its seed. ensure generates it, about six minutes, and checks every
table, the answer key and the context documents against the digests the repository pins.
Terminal window
python -m backoffice_world ensure world
Output1 line · exit 0
ok world generated at world, seed 2026, digest 4f6b38229f4a, equal to the pin
The agent prices every model call from the usage the API returns and writes it to a ledger.
plan states what the two sweeps will cost, from per-case figures measured on an earlier run of
this agent; at the cap an episode makes no call.
Terminal window
python -m backoffice_agent plan cheaper-model.toml --sweeps sonnet haiku
Output6 lines · exit 0
plan 2 sweeps of 760 episodes each: 4 action types x 1 case x 190 replicates
sonnet claude-sonnet-5-5 about $28.31
haiku claude-haiku-5-5 about $1.29
all about $29.60; already spent $0.00 (spend.jsonl)
cap $40.00: at the cap an episode makes no call and raises (BACKOFFICE_CAP_USD)
per case: measured on an earlier run of this agent, by action type
Prove’s power check refuses a sweep in which one condition alone could not be flagged for the
change you accept, 0.20 by default. A condition is one fixed case, so more cases need more
replicates each: one case per kind of work, four conditions, needs 190 replicates for both the
first sweep and the paired one. That is the smallest design that still covers all four kinds of
work. The dry run accepts it:
Terminal window
poisson-prove sweep cheaper-model.toml --dry-run
Outputexcerpt, 2 of 8 lines · exit 0
note: the axis "case" has one value, so it varies nothing
⋯ 6 lines
smallest rise one condition alone would be flagged for 0.198 at its worst base rate, 0.400 (Fisher, q 0.05 over 16 tests, 4 times in 5; you accept up to 0.200)
All output8 lines
note: the axis "case" has one value, so it varies nothing
plan 760 rollouts of back-office-agent: 4 cells x 190 replicates on local, $0 (local)
target backoffice_agent.agent:rollout_sonnet (imports, takes cfg and seed)
outcomes failures: harmful, no_action, wrong; successes: right
Sweep: 2 axes · 4 cells · N=190 → 760 rollouts
Backend: local · target: back-office-agent · t_unit: tool_calls
Cost: $0 (local)
smallest rise one condition alone would be flagged for 0.198 at its worst base rate, 0.400 (Fisher, q 0.05 over 16 tests, 4 times in 5; you accept up to 0.200)
done 760 rollouts in run "sonnet" (complete) in 1 h 6 min
⋯ 4 lines
All output22 lines
plan 760 rollouts of back-office-agent: 4 cells x 190 replicates on local, $0 (local)
target backoffice_agent.agent:rollout_sonnet (imports, takes cfg and seed)
outcomes failures: harmful, no_action, wrong; successes: right
Sweep: 2 axes · 4 cells · N=190 → 760 rollouts
Backend: local · target: back-office-agent · t_unit: tool_calls
Cost: $0 (local)
smallest rise one condition alone would be flagged for 0.198 at its worst base rate, 0.400 (Fisher, q 0.05 over 16 tests, 4 times in 5; you accept up to 0.200)
76/760
152/760
228/760
304/760
380/760
456/760
532/760
608/760
684/760
760/760
done 760 rollouts in run "sonnet" (complete) in 1 h 6 min
episodes with an outcome 760
episodes with no outcome 0
episodes timed out 0
wall time 1 h 6 min
6. The same agent on Claude Haiku 5.5, on the same seeds
--target names the Haiku rollout; --pair-with sonnet gives it the first run’s seeds, so each
episode has a twin on the same case. The paired sweep checks the design again, for the paired
test:
smallest rise one condition alone would be flagged for 0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 16 tests, 4 times in 5; you accept up to 0.200)
⋯ 10 lines
done 760 rollouts in run "haiku" (complete) in 57 min 48 s, paired with "sonnet"
⋯ 5 lines
All output23 lines
plan 760 rollouts of back-office-agent: 4 cells x 190 replicates on local, $0 (local)
target backoffice_agent.agent:rollout_haiku (imports, takes cfg and seed)
outcomes failures: harmful, no_action, wrong; successes: right
Sweep: 2 axes · 4 cells · N=190 → 760 rollouts
Backend: local · target: back-office-agent · t_unit: tool_calls
Cost: $0 (local)
smallest rise one condition alone would be flagged for 0.200 at its worst base rate, 0.400 (the paired test, q 0.05 over 16 tests, 4 times in 5; you accept up to 0.200)
76/760
152/760
228/760
304/760
380/760
456/760
532/760
608/760
684/760
760/760
done 760 rollouts in run "haiku" (complete) in 57 min 48 s, paired with "sonnet"
no significant change every condition that did not count could have shown a change of 0.200: the largest limit it read is 0.078 (the paired test (McNemar), Holm q 0.05 over 16 tests, 4 times in 5); harmful: 0 safer, 0 worse · no_action: 0 safer, 0 worse · right: 0 safer, 0 worse · wrong: 0 safer, 0 worse (paired test on shared seeds)
⋯ 11 lines
condition event before after change verdict rise fall
action break · case 1 harmful 0.000 0.000 0.000 unchanged 0.059 no room
action break · case 1 no_action 0.000 0.000 0.000 unchanged 0.059 no room
action break · case 1 right 1.000 1.000 0.000 unchanged no room 0.059
action break · case 1 wrong 0.000 0.000 0.000 unchanged 0.059 no room
action reroute · case 1 no_action 0.000 0.000 0.000 unchanged 0.065 no room
action reroute · case 1 right 0.995 0.958 -0.037 unchanged none detectable 0.078
action reroute · case 1 wrong 0.000 0.011 +0.011 unchanged 0.065 no room
action rfi · case 1 harmful 0.000 0.000 0.000 unchanged 0.059 no room
action rfi · case 1 no_action 0.000 0.000 0.000 unchanged 0.059 no room
action rfi · case 1 right 1.000 1.000 0.000 unchanged no room 0.059
action rfi · case 1 wrong 0.000 0.000 0.000 unchanged 0.059 no room
action stuck · case 1 harmful 0.000 0.000 0.000 unchanged 0.059 no room
action stuck · case 1 no_action 0.000 0.000 0.000 unchanged 0.059 no room
action stuck · case 1 right 1.000 1.000 0.000 unchanged no room 0.059
action stuck · case 1 wrong 0.000 0.000 0.000 unchanged 0.059 no room
⋯ 4 lines
All output33 lines
no significant change every condition that did not count could have shown a change of 0.200: the largest limit it read is 0.078 (the paired test (McNemar), Holm q 0.05 over 16 tests, 4 times in 5); harmful: 0 safer, 0 worse · no_action: 0 safer, 0 worse · right: 0 safer, 0 worse · wrong: 0 safer, 0 worse (paired test on shared seeds)
method paired test on shared seeds (McNemar), the default for runs paired at sweep time
threshold 0.200 (recorded at sweep time by the runs)
before sonnet
after haiku
verdict rule the verdict counts changes at the family-wise level (Holm, q 0.05 over 16 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 16 tests
limits the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 16 tests); 1 row that did not count has no limit on its fall (none detectable: 1), so a fall of any size cannot be excluded there; 1 row that did not count has no limit on its rise (none detectable: 1), so a rise of any size cannot be excluded there
intervals on each condition's change: Tango at 99.688%, simultaneous (Bonferroni) over the 16 tests
fragility 3 outcomes would change the word (the search is not exhaustive: it may be fewer): 3 runs of B in action reroute, case 1 that ended in right made to end otherwise
inferred: no event was given, so the event set is harmful, no_action, wrong, right, the failures and successes the runs declared
condition event before after change verdict rise fall
action break · case 1 harmful 0.000 0.000 0.000 unchanged 0.059 no room
action break · case 1 no_action 0.000 0.000 0.000 unchanged 0.059 no room
action break · case 1 right 1.000 1.000 0.000 unchanged no room 0.059
action break · case 1 wrong 0.000 0.000 0.000 unchanged 0.059 no room
action reroute · case 1 no_action 0.000 0.000 0.000 unchanged 0.065 no room
action reroute · case 1 right 0.995 0.958 -0.037 unchanged none detectable 0.078
action reroute · case 1 wrong 0.000 0.011 +0.011 unchanged 0.065 no room
action rfi · case 1 harmful 0.000 0.000 0.000 unchanged 0.059 no room
action rfi · case 1 no_action 0.000 0.000 0.000 unchanged 0.059 no room
action rfi · case 1 right 1.000 1.000 0.000 unchanged no room 0.059
action rfi · case 1 wrong 0.000 0.000 0.000 unchanged 0.059 no room
action stuck · case 1 harmful 0.000 0.000 0.000 unchanged 0.059 no room
action stuck · case 1 no_action 0.000 0.000 0.000 unchanged 0.059 no room
action stuck · case 1 right 1.000 1.000 0.000 unchanged no room 0.059
action stuck · case 1 wrong 0.000 0.000 0.000 unchanged 0.059 no room
more --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound
next: poisson-prove report sonnet haiku --x action --y case
No condition counted as a change. What that means differs by kind of case:
RFIs, breaks and late payouts. Both models were right on all 190 episodes of each. Each
could have shown a rise of 0.059 in any failure, 4 times in 5. Within that, the cheaper model
held on these three cases.
Reroute drafts. Sonnet drafted one harmful reroute in 190, Haiku six, each a note carrying
a number the world contradicts; Haiku also drafted two that left payouts off the list. Prove
calls none of it a change. But its limits for a rise in harmful and a fall in right read
none detectable: at these counts this run cannot rule out a rise of any size there. The
cheaper model did not clear this case.
Trace reads Prove’s store as it is. Trace 0.1.0rc2 pins Prove 0.1.3 exactly, so it goes in an
environment of its own. Trace refuses an event a run does not carry, and harmful is the one
failure both runs hold.
Every harmful episode on Haiku stopped at the reroute draft, the step that wrote the note:
Terminal window
trace-env/bin/poisson-trace where haiku --store prove-runs --event harmful --by step
Outputexcerpt, 4 of 73 lines · exit 0
ok 6 of 760 tries failed; each one's last step was draft_reroute (not necessarily the cause)
⋯ 39 lines
| step | tries | share by step 7 | 95% band |
|---|---|---|---|
| draft_reroute | 6 | 0.008 | 0.000 – 0.024 |
⋯ 30 lines
All output73 lines
ok 6 of 760 tries failed; each one's last step was draft_reroute (not necessarily the cause)
# Poisson Trace · where runs die · haiku
## Summary
- Read: haiku (back-office-agent): 760 tries, each placed at the step where it stopped.
- Question: At which step do failing tries stop?
- Answer: 6 of 760 failed. Each failure's last step was draft_reroute (not necessarily the cause), between step 4 and step 7.
- Next: See whether a change moved this: `poisson-trace compare sonnet haiku --store prove-runs --event harmful --max-mde-hr X`, choosing X before you look.
## Before you act on this
- The step is where a try stopped, not where it went wrong; the kind of step is that of its last step, not the cause. Check what passing tries end on before reading it as one part's fault.
- One run shows where tries stop, not whether a change moved anything.
## Run
- run: haiku (20261109T210244Z-1267) · target: back-office-agent · t_unit: tool_calls · status: complete · no cap
- Prove's tally: rows 760 · with outcome 760 · no outcome 0 · timed out 0 · infra 0 · other 0
- placement: placed 760 · died 6 · ended 754 · timed out 0 · capped 0 · excluded 0
- events: harmful
## Share not yet failed, by step
Aalen–Johansen over leaf-span index: finishing without failing and timing out are ends of their own, and only capped tries are censored; band: bootstrap 95%, resampling inputs, B = 1,000, seed = 0.
| step | at risk | failed | not yet failed | 95% band |
|---|---|---|---|---|
| 4 | 738 | 4 | 0.995 | 0.984 – 1.000 |
| 5 | 351 | 1 | 0.993 | 0.980 – 1.000 |
| 7 | 4 | 1 | 0.992 | 0.976 – 1.000 |
median step of the failed tries: 4
the share failed stayed under one half: 0.008 by the last failure
## Where they die, by step name
Aalen–Johansen: each kind's share of tries failed by step 7, and the share that timed out; band: bootstrap 95%, resampling inputs, B = 1,000, seed = 0.
| step | tries | share by step 7 | 95% band |
|---|---|---|---|
| draft_reroute | 6 | 0.008 | 0.000 – 0.024 |
| timed out | 0 | 0.000 | 0.000 – 0.000 |
the kinds' shares add to the share failed, 1 − not yet failed = 0.008 (1 − 0.992); timed out is an end of its own and not among them
## Declared steps
Reached by k of N placed runs, from Prove's declaration; the set of steps never reached is Prove's own.
| step | reached by | note |
|---|---|---|
| answer_rfi | 190 of 760 | undeclared |
| draft_reroute | 190 of 760 | undeclared |
| episode | 760 of 760 | undeclared |
| flag_late_payout | 190 of 760 | undeclared |
| read_break | 190 of 760 | undeclared |
| read_case | 190 of 760 | undeclared |
| read_context | 760 of 760 | undeclared |
| read_identity_record | 190 of 760 | undeclared |
| read_payment | 223 of 760 | undeclared |
| read_rail_health | 190 of 760 | undeclared |
| resolve_break | 190 of 760 | undeclared |
## What this cannot say
- The step index is where the run stopped, not why. A run that died at a tool step may have gone wrong three steps earlier.
- Capped runs are censored at their last recorded step. If they would have died soon after, the curve is optimistic by that much. A run that timed out ends where it stopped, counted beside the kinds and not as a death; the header counts both.
- Excluded rows are not on the curve; the header counts them by reason.
- On one run, this is a description. Only a comparison of two stored runs on the same cells says whether anything moved, and only within what it could not have detected.
- Compared: before sonnet against after haiku, on the same 4 inputs: 760 tries each. A try counts as failed when it ended harmful.
- Question: Step by step, among tries still going, did the after version fail more or less often? You asked to see that rate change 1.5-fold, either way.
- Answer: Failures come sooner. 6 of 760 tries failed after and 1 of 760 before; if each input failed equally often under both versions, a gap at least this big, either way, would turn up about 3 times in 100 (counting tries as independent); after against before, the per-step failure rate is 5.59 times as high (1.22 to 25.72).
- Next: Compare how often each version fails overall: `poisson-trace fix-check sonnet haiku --store prove-runs --event harmful --inputs all --max-mde X`, choosing X before you look.
## Before you act on this
- They started 1 h 6 min apart. Trace can't tell whether they were run side by side, so anything else that changed between them (the provider, the model behind the API, the day) may be part of the difference, in either direction.
- Both runs name the same target (back-office-agent); Trace can't tell what differs between them, only what you tell it.
- A try that finished without failing counts as still going up to its last step, so this mixes how often and how early tries fail. Tries of the same input are counted as separate chances; if they are near copies, this is more confident than it should be.
- A 8.85-fold change means the per-step failure rate among tries still going rises 8.85 times, or falls to 0.11 of what it was; it is not the ratio of the failure counts.
- This asks whether failures come sooner or later among tries still going. Whether a version fails more often overall is fix-check's question, and on the same two runs the two can read differently.
- Before putting the difference down to the version, run both versions side by side.
## Run A (before)
- run: sonnet (20261109T195618Z-6fa5) · target: back-office-agent · t_unit: tool_calls · status: complete · no cap
- Prove's tally: rows 760 · with outcome 760 · no outcome 0 · timed out 0 · infra 0 · other 0
- placement: placed 760 · died 1 · ended 759 · timed out 0 · capped 0 · excluded 0
- events: harmful
## Run B (after)
- run: haiku (20261109T210244Z-1267) · target: back-office-agent · t_unit: tool_calls · status: complete · no cap
- Prove's tally: rows 760 · with outcome 760 · no outcome 0 · timed out 0 · infra 0 · other 0
- placement: placed 760 · died 6 · ended 754 · timed out 0 · capped 0 · excluded 0
- events: harmful
## Comparison
log-rank, stratified by config cell over 4 common cells (excluded: 0 in A only, 0 in B only), unpaired, though the runs share a run_salt; α = 0.05.
| arm | placed | died | observed | expected |
|---|---|---|---|---|
| A (before) | 760 | 1 | 1 | 3.841 |
| B (after) | 760 | 6 | 6 | 3.159 |
statistic: χ² = 4.892 on 1 d.f., variance 1.650, p = 0.027
direction: runs die later in A (before) than in B (after): runs die earlier in B
hazard ratio, after against before: 5.59 (1.22 to 25.72, 95%); Peto's one-step estimate from the observed, expected and variance above
The test compares, step by step, the failure rate among tries still going; the rows by step below and the curves `poisson-trace where` draws give the share of tries failed by a step. A version whose tries finish sooner can differ in one and not the other.
### Per step
cause-specific log-rank, stratified by config cell, deaths of other steps censored at their step; Benjamini–Hochberg at q = 0.05.
| step | deaths A | deaths B | χ² | p | q | could not have detected a hazard ratio below |
could not have detected a hazard ratio below 8.854 (from the log-rank's own information, its variance 1.650 over the common cells; 80% power, α = 0.05)
**different: runs die earlier in B (after) than in A (before) (log-rank, stratified by cell, p = 0.027)**
## Failures by kind
Each failed try's kind of failure, by the built-in rules; the same rules classified both runs. Counts, not a test: the verdict above is about all failures together.
of 760 tries of A and 760 of B; a try has one kind
For the share of runs that fail, input by input, and the inputs that changed: poisson-trace fix-check A B --inputs all
## What this cannot say
- The step index is where the run stopped, not why. A run that died at a tool step may have gone wrong three steps earlier.
- Capped runs are censored at their last recorded step. If they would have died soon after, the curve is optimistic by that much. A run that timed out ends where it stopped, counted beside the kinds and not as a death; the header counts both.
- Excluded rows are not on the curve; the header counts them by reason.
- On one run, this is a description. Only a comparison of two stored runs on the same cells says whether anything moved, and only within what it could not have detected.
A: Claude Sonnet 5.5B: Claude Haiku 5.5significantly more harmful episodes stopped here, q < 0.05
The step as Trace’s compare --json gave it, drawn by this site: harmful episodes by the step
they stopped at, in each run, with Trace’s q.
Trace reads one step at a time and counts each try as its own chance, so on the reroute step
alone 1 against 6 reads as different, at q 0.027. Prove tests each condition and outcome and
corrects over all sixteen, so the same counts do not count there. fix-check, which asks how much
more often across all four cases, cannot tell at this size, and says what would settle it:
- Compared: old sonnet against new haiku, on the same 4 inputs: 760 tries of each. A try counts as failed when it ended harmful.
- Question: Did the new version fail more or less often than the old? You asked that the run be able to catch a 5-point drop in how often it fails.
- Answer: Can't tell. The new version failed 6 of 760 tries and the old 1 of 760, and with these tries Trace can't reliably catch a drop of any size.
- Next: Run each version 525 times on each of these 4 inputs with your own harness, then run `poisson-trace fix-check` on the two new runs.
## Before you act on this
- They started 1 h 6 min apart. Trace can't tell whether they were run side by side, so anything else that changed between them (the provider, the model behind the API, the day) may be part of the difference, in either direction.
- Both runs name the same target (back-office-agent); Trace can't tell what differs between them, only what you tell it.
- These results fit anything from 0.03 fewer failures per 100 tries to 1.7 more, but that range is only a rough guide here and may be off either way (fewer than 10 inputs vary): don't use it to rule sizes in or out.
- Tries of the same input are counted as separate chances, and if they are near copies the answer is more confident than it should be.
- The 5 points you asked about is close to the most these inputs could drop (the old version failed 0.1% of tries).
- The 525 times per input is an estimate from these runs' failure rates. 12 inputs like these at the present tries would do the same, more than the 4 here. Even with good estimates, about half of runs built to it end up a little too small. Make it a fresh run, judged on its own: don't add tries to these and test again.
- Prove's tally (new): rows 760 · with outcome 760 · no outcome 0 · timed out 0 · infra 0 · other 0
- the two runs started 1 h 6 min apart
- events: harmful
- inputs: every input either run carries · selected 4 · compared 4 · left out 0
- rows used: old 760 (timed out 0, no outcome 0) · new 760 (timed out 0, no outcome 0)
- inputs short of the declared repeats: old 0 · new 0
- inputs with mixed outcomes across their repeats: old 1 · new 1
## The two versions
The old version and the new on the same inputs.
| version | runs | failed | failure share | did not fail |
|---|---|---|---|---|
| old | 760 | 1 | 0.1% | 99.9% |
| new | 760 | 6 | 0.8% | 99.2% |
The old version did not fail on 99.9% of its episodes on these inputs.
## Failures by kind
Each failed try's kind of failure, by the built-in rules; the same rules classified both runs. Counts, not a test: the verdict above is about all failures together.
| kind | old | old share | new | new share | change |
of 760 tries of the old version and 760 of the new; a try has one kind
## The comparison
exact conditional Cochran-Mantel-Haenszel test, stratified by input over 4 inputs (1 informative), two-sided; α = 0.05.
new failures: observed 6, expected 3.500, variance 1.722; p = 0.122
difference: +0.7 points, new minus old (-0.0 to +1.7; Mantel-Haenszel weights; range: stratified Miettinen-Nurminen score interval, 95%, for these inputs as run; rough here: fewer than 10 inputs vary)
by input: p = 1.000 (sign-flip over the 1 input that moved, exact); not part of the line below
## Consistency
failure share 0.005 · fails again 0.016 · κ 0.011 · within-input variance 0.005 against pooled 0.005
a comparison that ignored the inputs would need 1/(1 − κ) = 1.01 times the runs
## What it could not have detected
proportional change on every input, 80% power, α = 0.05, continuity-corrected normal approximation to the stratified statistic.
- a drop in failure share: no limit; even every failure gone, a drop of 0.46 points, would have been detected less than four times in five
- a rise in failure share: could not have detected one below 1.8 points
**too small to tell: could not have detected a drop in failure share below or at 0.46 points, which is every failure gone, and 5 was asked for**
## Guidance
- repeats: 525 per arm on these 4 inputs would bring the limit to 5 points
- inputs: 12 inputs like these, at the present repeats, would do the same
- ceiling: no value of --max-mde accepts this design as it stands
These numbers assume the next runs fail at the rates these did, input by input, and that new inputs resemble the present ones: for failing inputs chosen by a rule, chosen by the same rule. It is a guide to the next design and not a promise about it: a design built to the number has an expected limit at the ceiling asked for, so about half the time its own limit will land a little above.
## Inputs that changed
Fisher's exact test per input, Benjamini-Hochberg at q = 0.05 over 4 inputs; 0 flagged.
| input | old failed/runs | new failed/runs | change | p | q | flag |
A lone changed input is flagged only if its p is below α / M = 0.0125, and the smallest p an input of this design can give is 1.99e-113: a lone change can be flagged; a lone complete flip needs 5 repeats per arm. An input with no flag is not an input that did not change.
## What this cannot say
- A difference between two stored runs is a difference between the runs. The version, the provider and the hour changed together unless the two were run side by side, and this test cannot separate them.
- Repeat runs of one input are read as separate draws and never as replays: a served model is not bit-identical on repeat, so nothing here matches a run of the old version to a run of the new by seed.
- If repeat runs of one input are near copies of one another, the run-level test overstates its confidence. The by-input line does not depend on that.
- Rows with no outcome are left out of both arms and counted in the header. A new version that times out where the old one failed does not show as a failure here.
- Trace reads the runs it is given. If the inputs were chosen by the old arm's own outcomes, or the more favourable of several runs was handed over, the confidence printed does not hold.
- The detection limit and the run counts are computed from these runs' own failure rates, which are estimates. They are a guide to the next design, not a promise about it.
next: run 525 repeats per arm on these 4 inputs, then poisson-trace fix-check on the two new runs
The three readings agree on what matters: the reroute drafts are where the cheaper model may
cost something, and this run is too small to say how much.
claude-haiku-5-5 $1.12 1,757 calls, 7,900,646 tokens in, 654,985 out
claude-sonnet-5-5 $24.38 1,751 calls, 9,715,076 tokens in, 494,556 out
The plan stated $29.60. The ledger, priced by the agent from the usage the API returned at
Anthropic’s published rates, came to $25.49: $24.38 on Sonnet and $1.12 on Haiku for the same
760 cases.
That Haiku is as good as Sonnet for this agent. Each kind of case was run on one case; another
RFI, break or payout can behave differently, and the reroute drafts are not cleared.
How large a rise in harmful reroute drafts the cheaper model brings. Trace’s fix check names
the size of run that would answer it.
Anything about a real system. The world, the cases and the providers are synthetic, and a
model’s behaviour can change behind its name; these runs were made on 2026-11-09.
Every output above is what the command printed, stored in the site repository under
src/data/examples/can-the-agent-move-to-a-cheaper-model/ with the run’s diff.json, the three
files report wrote, and Trace’s compare --json, as Prove 0.2.2 and Trace 0.1.0rc2 wrote them.
The page cannot show an output that is not there.