Trace

Trace reads stored runs and answers two questions, each with a stated confidence and with what the runs could not have detected:

• Where do runs die? where and compare put every run on one clock, the step it was executing, and say what share is still going at each step.
• Did the change make runs fail less? fix-check and changes compare two versions on the same inputs, with the old version re-run as the control.

Trace runs nothing and captures nothing. It reads a store that Prove wrote, or one its importer wrote from another tool’s results.

Install

pip install poisson-trace
pip install poisson-prove

Neither package is released yet. Until then, install both from a clone:

git clone https://github.com/poisson-labs/trace
cd trace
python -m venv .venv && . .venv/bin/activate
pip install .
pip install -e /path/to/prove

First run

The repository ships stored runs under tests/fixtures/. This reads one of them:

poisson-trace where 20260925T234602Z-3ea0 --store tests/fixtures/cs \
  --event forbidden_action --event unresolved --event max_turns --event app_error \
  --at 20 --at 40

placement: placed 24 · died 7 · ended 17 · timed out 0 · capped 0 · excluded 0

median step: 50
S(20) = 1.000 (1.000 – 1.000)
S(40) = 0.721 (0.484 – 0.959)

| kind | deaths | F at step 58 | 95% band |
|---|---|---|---|
| llm | 4 | 0.343 | 0.056 – 0.679 |
| agent | 3 | 0.441 | 0.000 – 0.871 |

Seven of 24 runs died, four at a model call and three at a guardrail agent. S(40) is the share of runs still going at step 40, with its 95% band.

The four commands

poisson-trace where RUN --store ROOT --event NAME [--event NAME ...]
                   [--by kind|step] [--at STEP ...] [--boot B] [--seed S]
                   [--svg FILE] [--json]

poisson-trace compare A B --store ROOT --event NAME [--event NAME ...] --max-mde-hr X
                     [--by kind|step] [--at STEP ...] [--alpha A] [--json]

poisson-trace fix-check OLD NEW --store ROOT --event NAME [--event NAME ...]
                       ( --failing-from RUN [--min-failures M] | --inputs all )
                       --max-mde X [--alpha A] [--seed S] [--top N] [--json]

poisson-trace changes A B --store ROOT --event NAME [--event NAME ...]
                     [--alpha A] [--top N] [--json]

Command	What it prints
where	The survival curve over step index with a 95% band, the median step, and deaths by kind of step
compare	Whether two runs’ curves differ (log-rank, stratified by condition), and the smallest hazard ratio it could have detected
fix-check	Whether the new version fails less on the same inputs (exact Cochran–Mantel–Haenszel, stratified by input), the difference in points with its interval, and the smallest change it could have detected
changes	Which inputs moved (Fisher’s exact test per input, Benjamini–Hochberg across all of them)

--store and --event are required. --event names the outcomes that count as a failure and can be repeated. --json prints the same report as a structure.

Reading the verdict

compare and fix-check end with one of three lines:

Line	Meaning	Exit code
different	The test rejected. The line says which way	0
no significant difference	The test did not reject, and the design could have detected an effect at or under your ceiling	0
too small to tell	The test did not reject, and the design could not have detected an effect as small as you asked for. The report says how many repeats or inputs would settle it	2

A refusal, such as an unknown run or event, exits 1 with one sentence.

The ceiling is yours: --max-mde-hr (a hazard ratio) for compare and --max-mde (a share, 0.10 is ten points) for fix-check. Neither has a default.

Checking a change

1. A stored run of the old version has failures. --failing-from that run names its failing inputs.
2. Run those inputs again with Prove, once per version, side by side, with repeats.
3. Compare the two new runs:

poisson-trace fix-check OLD NEW --store ROOT --event NAME --failing-from S --max-mde 0.10

The run used to choose the inputs cannot also be an arm: both arms must be fresh re-runs.

Results from other tools

poisson-trace-import writes another tool’s saved results as a run that fix-check can read. See DeepEval and Ragas.

More

• README: every option, the placement rules, and worked examples on real runs
• Methods: the statistics, in plain words
• Changelog