Poisson LabsEvaluation for AI systems · open source + cloud
An average can’t tell you what broke.
Poisson tests your AI system before and after a change, condition by condition, on matched seeds, and tells you what improved, what regressed, and what remains uncertain, before the change ships.
significantly safersignificantly worseno significant changethe average
microduck-walking@v1
5 of 14 conditions got worse.
8 better · 5 worse · 1 too close to call, below a 8-point limit
5 of 14 conditions significantly worse.
McNemar’s test per condition on 100 paired trials, Benjamini–Hochberg at q < 0.05. The unchanged condition could have shown a rise of 8 points or more.
Published on AI agentswalking robotsa vision encodera multi‑agent benchmarktraining stability
01The problem
Change nothing. Run it twice.
We ran the same airline agent twice on τ²-bench: identical configuration, identical seed, temperature zero. The score barely moved. Underneath, 44 of 200 tasks flipped: 23 passed only in the first run and 21 only in the second, and they cancelled out.
pairs still identical after each model calln = 200
one task, both runs, call by callidentical, then not
Both runs, identical
- 0 user message · 27 tokens
- 1 agent message · 46 tokens
- 2 user message · 38 tokens
- 3 agent
get_reservation_details
Call 4: the runs part. The agent is given the same 3,659 tokens in both runs. It writes 96 tokens in one and 129 in the other.
Run R 20260915T210140Z-4847
- 4 agent message · 96 tokens
- 5 user message · 59 tokens
- 6 agent
transfer_to_human_agents - 7 agent message · 18 tokens
- 8 user message · 3 tokens
Passed solved · COMMUNICATE 1 · DB 1
Run R′ 20260915T211148Z-965d
- 4 agent message · 129 tokens
- 5 user message · 62 tokens
- 6 agent
cancel_reservation - 7 agent message · 111 tokens
- 8 user message · 95 tokens
- 9 agent
transfer_to_human_agents - 10 agent message · 18 tokens
- 11 user message · 3 tokens
Failed unsolved · COMMUNICATE 1 · DB 0
Top-line scores stay flat while dozens of tasks flip underneath. Measure the noise before you call anything a change.
02Proof
A safer policy. Measured, condition by condition.
The same method on a different kind of system: the thing under test changes, the way its failures are found and measured does not. We mapped a Go1 walking policy across 400 conditions of floor friction and lateral push. The map showed two conditions its training had never covered; we retrained on those, then ran both policies through every condition again, 32 times each on the same seeds.
- 60
- conditions significantly safer
- 0
- conditions significantly worse
- 12,800
- rollouts per version, 32 per condition on the same seeds
- 400
- conditions tested
340 showed no significant change. For the least-powered of them, the run could not have detected a rise in the fall rate under 34 percentage points. That limit is reported with the result.
60 conditions significantly safer after the retrain0 significantly worse, anywhere in the 400
03The tools
Five tools. One evidence trail.
Find where runs fail. Measure whether a change helped. Map how difficulty changes outcomes. Search for the conditions that break your system. Train an agent on its own failures. Use them on your own machines, or in CI with Poisson Cloud.
poisson-trace fix-check OLD NEW --store ROOT --event NAME \ ( --failing-from RUN | --inputs all ) --max-mde X
Show more commands
pip install poisson-tracepip install poisson-prove# where do runs die, over the step indexpoisson-trace where RUN --store ROOT --event NAME# do two runs' survival curves differpoisson-trace compare A B --store ROOT --event NAME --max-mde-hr X# did the change make runs fail less, with the old version as the controlpoisson-trace fix-check OLD NEW --store ROOT --event NAME \ ( --failing-from RUN | --inputs all ) --max-mde X# which inputs moved, corrected across all of thempoisson-trace changes A B --store ROOT --event NAME
poisson-prove check <run-a> <run-b> --event violated --fail-on worse Show more commands
pip install poisson-prove # private preview: not public yet# 1. sweep the conditions, before the changepoisson-prove sweep quickstart.toml \ --target poisson_prove.examples.mock_agent:rollout --yes# 2. sweep again after it, on the same seedspoisson-prove sweep quickstart.toml \ --target ...mock_agent:rollout_patched --pair-with <run-a> --yes# 3. diff the pair into report.md, heatmap.svg, receipt.htmlpoisson-prove report <run-a> <run-b> \ --event violated --x tool_fail_rate --y task_id# in CI: exit 1 if any condition came back significantly worsepoisson-prove check <run-a> <run-b> --event violated --fail-on worse
| Depth | Pass@1 | 95% interval | vs. previous level |
|---|---|---|---|
| d1 | 54.7% | baseline | |
| d2 | 39.7% | −15pp · distinct | |
| d3 | 23.7% | −16pp · distinct | |
| d5 | 17.7% | too small to call | |
| d8 | 87.0% | +69pp · distinct |
Break runs your agent many times while it changes what the agent meets in production, keeps only the changes that fail more often than an untouched control, and hands each one over as a replayable condition a fix can be checked against.
- What breaks it before a user does?A hunt before a release, and a short list of the conditions that fail, worst first.
- What happens when my tools misbehave?Errors, timeouts, slow answers, rate limits, empty or malformed results, and an instruction planted in a tool result.
- I fixed it. Did I?Each finding re-run on the changed agent, with Trace’s answer: fails less, fails more, or too small to tell.
- Keep what we found.Findings are files, and re-run in CI as a standing check.
The gym trains an agent against checks it cannot see, draws its practice from where it currently fails, and reports for each checkpoint whether training helped, at a stated confidence.
- TasksSeeded worlds, the tools the agent acts through, and the hidden checks that give the reward.
- BreakSearches for where the current checkpoint fails. Those failures become the curriculum.
- TraceReads every recorded episode and says at which step the remaining failures happen.
- ProveCompares each checkpoint with its starting point on held-out conditions training never touched.
04In your CI
Catch regressions before they ship.
Establish a baseline, rerun the same conditions on every pull request, and block the merge when a condition is significantly worse.
- 1
Establish a baseline
Run your agent across the conditions you care about, each run on a seed.
- 2
Rerun on every pull request
The same conditions on the same seeds, so each one is compared with itself.
- 3
Review the evidence
Improved, regressed, or too small to call, condition by condition. A regression fails the check.
It fails on real regressions, not on noise. A condition fails the check only when it is significantly worse than its own baseline, with the correction for testing every condition at once built in. You get a pass or a fail on the pull request; the full receipt (the test per condition, Benjamini–Hochberg at q < 0.05, and each condition’s detection limit) is one click away. A regression smaller than that limit can pass, and the report says so.
on: pull_requestjobs: agent: steps: - name: Check the agent run: | poisson-prove sweep agent.toml \ --target my_agent:rollout \ --pair-with <main-run> --yes poisson-prove check <main-run> <this-run> \ --event violated --fail-on worse
05Integrations
Bring the stack you already run.
Your evals, your agent framework and your compute plug in as they are. We onboard a small number of teams at a time and set up every integration your stack needs with you, before your first run.
Your evals and traces
- DeepEval
- Ragas
- promptfoo
- Inspect
- OpenAI Evals
- Braintrust
- Weave
- +5 more
Your agent
- OpenAI Agents SDK
- LangGraph
- CrewAI
Your cloud
- AWS
- GCP
- Azure
- Your machine
- Modal
Poisson
- Break hunts for the conditions that make your agent fail, with Monte pointing it where failure is likely.
- Trace reads the runs you already have, step by step.
- Prove reruns every condition before and after a change, on the same seeds, with a test per condition corrected for testing them all at once.
What you get
A verdict per condition
- Improved
- Regressed
- Too small to tell, with the limit
In CI, the check fails on a regression.
At scale, in your own cloud. Prove runs a sweep across a cluster in your AWS, GCP or Azure account. How it runs there
06What it leaves you
Nothing we find is thrown away.
A failure doesn’t end as a line in a report. It becomes a test on every pull request, training data aimed at exactly that failure, and the starting point of the next search. The first is in private preview; the other two are being built.
-
1Into your CIPrivate preview
Every failure becomes a test.
Each condition that broke the agent is saved as a file: the change, the seed and the failure rate under it. It re-runs on every pull request as a standing check, so a fixed failure stays fixed.
-
2Into trainingIn development
Every fix becomes training data.
The same condition on the same seed, failing before a change and passing after, is a preference pair. So are passing and failing runs within one condition, each with the prompt, retry or tool-schema patch behind it. For a robot policy, training ranges shift toward where it broke, as in the Go1 retrain.
-
3Into MonteIn research
Every hunt sharpens the next.
Labelled failures, each with its trace and the exact condition, are what Monte is built to learn from: a model of where agents fail that starts the next hunt where failure is most likely. It will learn only from records whose owner turned training on.
Nothing counts as fixed until it is re-run. A retrained agent goes back through the same conditions on fresh seeds, and the result is a receipt like any other. The Go1 retrain is this loop, run once: the map showed two conditions training had missed, and the policy was retrained on exactly those.
07How we count
Know what changed, and what your eval missed.
- i.
Measure the noise.
Establish how much the system varies when nothing was changed on purpose.
- ii.
Compare like with like.
Match conditions and seeds across the baseline and the new version.
- iii.
Show where it moved.
Report changes by condition, not just as an average. An average can improve while 5 conditions get worse.
- iv.
Publish what you couldn’t detect.
State the smallest effect the study was equipped to find.
Gate every merge on what actually moved.
We’re working with a small number of teams to validate Poisson against real AI systems and release workflows. They shape the product, test it on real workloads, and pick the integrations and regression checks that matter most.