Poisson Tasks is out. Find where your agent breaks before your users do. Read the launch
Poisson Labs
Talk to us

Tools · open source

Trace

Where runs die, and whether a change made them fail less.

Re-running a failing input is not proof that a fix worked. On an agent nobody changed, some failures pass on a re-run anyway. Trace re-runs the old version beside the new one as the control, and says different, no significant difference or too small to tell.

Open source, in private preview. Docs · Talk to us

The commands you run

pip install poisson-trace
pip install poisson-prove

# where do runs die, over the step index
poisson-trace where RUN --store ROOT --event NAME

# do two runs' survival curves differ
poisson-trace compare A B --store ROOT --event NAME --max-mde-hr X

# did the change make runs fail less, with the old version as the control
poisson-trace fix-check OLD NEW --store ROOT --event NAME \
    ( --failing-from RUN | --inputs all ) --max-mde X

# which inputs moved, corrected across all of them
poisson-trace changes A B --store ROOT --event NAME

The four verbs

where

What share of runs is still going at each step, and at which kind of step they die.

compare

Whether two runs die at different steps, and the smallest difference it could have detected.

fix-check

Whether the new version fails less on the same inputs, with the old version re-run as the control.

changes

Which inputs moved, corrected across all of them.

The launch study

On four real agents, re-running a failing input with nothing changed "fixed" it 11% to 33% of the time.

Within three re-runs, 28% to 45%. A re-run has no control, so it cannot tell a fix from that background. Read the study.

τ²-bench airline, R.1 pair 50 tasks × 4, two runs 10 min apart 19% τ²-bench airline, T.4 arm A 47 tasks × 44 in one run, read as halves 22% HumanEval, golden pilot 328 cells × 32, two runs 56 min apart 11% Tasks calibration card, model A 840 tasks × 5 attempts, read as halves 33% Synthetic, planted by construction 50 tasks × 40, two runs, nothing changed 73% 0%25%50%75%100% OF THE FIRST RUN'S FAILING INPUTS, CALLED FIXED WITH NOTHING CHANGED
on one re-run, with its task-clustered 95% interval within three re-runs synthetic, pure chance: the ceiling
Fig. 1Of a first run's failing inputs, the share that passed on a re-run of the same, unchanged agent. The last row is a synthetic agent that fails by pure chance: 73.3%, the ceiling.
review q = 1.67e-20 A · 206 B · 422 sample q = 0.865 A · 180 B · 157 measure q = 0.422 A · 146 B · 114 RUNS THAT DIED AT THIS STEP, OF 2,000 PER ARM
A, the baseline B, one stage's failure rate doubled significantly more deaths in B, q < 0.05
Fig. 2A planted change, found. One stage's failure rate was doubled and sealed before any run. Trace called the pair different (p = 1.65e-9) and put it on review, the stage the seal named. On the same pair a re-run called 66.0% of failures fixed.
10 PER BIN IF THE LEVEL HOLDS 11 8 4 18 9 12 6 13 10 10 7 16 9 2 5 6 15 15 6 18 0 0.25 0.5 0.75 1 p, LOG-RANK STRATIFIED BY TASK · 200 PAIRS, NOTHING CHANGED
called different at α = 0.05: 11 of 200 the other 189
Fig. 3False alarms. On 200 pairs of an unchanged synthetic agent, Trace called 11 different: 5.5%, against a nominal 5%.

What it missed.

One HumanEval problem broke between two runs of an unchanged agent. compare read no significant difference (p = 0.109): a one-problem break is below what a pooled test can see. Prove's per-condition diff caught it, and changes now flags the same two inputs.

Four real agents and one synthetic workload, not agents in general. Empirical evidence under a stated test envelope, not formal verification.

Works with what you have

Already scoring with DeepEval or Ragas?

poisson-trace-import reads the result files they wrote, and fix-check says whether the difference between two versions is real. Integrations.