Poisson Labs Log in Contact

Poisson Labs · Platform

Sangfroid is in pre-release.

The lab’s three open-source tools against your agent on every pull request: the conditions that break it, the step that failed, and a check that fails when a change makes things worse.

We’re opening it to a small number of teams first. In the meantime the tools run on your own machines, and the lab runs audits.

Request early access Or have the lab run it for you

sangfroid / prove Merge blocked

5 conditions got worse.

  • 8 safer
  • 5 worse
  • 1 no significant change

checkpoint 2250 against checkpoint 5999, 14 conditions, 100 trials each, same seeds on both sides. Too small to see: anything under 8pp where nothing moved.

report.md microduck-walking@v1

A real run and real numbers. The pull-request check around them is in pre-release.

What we runThree open tools, in order.

An eval gives you one number. These three give you a place to look and a way to know whether looking helped. Each is open source and works on its own; Sangfroid runs them as one pipeline so you do not have to.

  1. 1Break

    Find the failures you didn’t think to test.

    Your test suite covers the cases you imagined. This throws tool errors, timeouts and truncated outputs at your agent, then searches for the combinations that break it, so compute goes where the failures are. What it finds is confirmed on fresh seeds.

    Tool returns 503Tool times outOutput truncatedMalformed JSONSchema changedSlow responseRetry stormEmpty result
    Illustration. The fault types are the ones the tool injects; no run is shown.

  2. 2Trace

    See the step that failed, not just the run.

    “The agent failed” is not something you can fix. Trace reads every run over OpenTelemetry or LiteLLM and points at the steps where runs are being lost, so the fix starts in the right file.

    τ²-bench airline task 28 seed 1207528101 temperature 0 gpt-4.1-mini

    Both runs, identical

    1. 0 user message
    2. 1 agent message
    3. 2 user message
    4. 3 agent get_reservation_details

    Call 4: the runs part. The agent is given the same 3,659 tokens in both runs. It writes 96 tokens in one and 129 in the other.

    Run R 20260915T210140Z-4847

    1. 4 agent message
    2. 5 user message
    3. 6 agent transfer_to_human_agents
    4. 7 agent message
    5. 8 user message

    Passed solved · COMMUNICATE 1 · DB 1

    Run R′ 20260915T211148Z-965d

    1. 4 agent message
    2. 5 user message
    3. 6 agent cancel_reservation
    4. 7 agent message
    5. 8 user message
    6. 9 agent transfer_to_human_agents
    7. 10 agent message
    8. 11 user message

    Failed unsolved · COMMUNICATE 1 · DB 0

    A real pair from the R.1 study: one τ²-bench task, same seed, run twice. The runs part at call 4, and the one that fails makes a tool call the other never makes. Drawn from the stored spans by a script, not by Trace, which is still being built.

  3. 3Prove

    Find out whether the fix worked.

    A fix that helps one case can quietly hurt five others. Prove reruns the same conditions on the same seeds, before and after, and shows every condition that got better, every one that got worse, and what was too small for the run to detect.

    About Prove →

    5 worse 1 no significant change 8 safer coloured only at q<0.05, corrected across every condition

    LATERAL PUSH, m/s → ↑ FRICTION μ 100 TRIALS PER CELL PER VERSION
    checkpoint 2250 → checkpoint 5999. Hover any cell.
    A real run: a walking policy at two checkpoints, 14 conditions, on the same seeds in both, tested by McNemar's test per condition and corrected across all 14. The later one is better in 8 (green) and worse in 5 (purple, hatched). report.md

What is sold

The running of it. The three tools are open source and stay that way: you can run all of this yourself, on your own machines, and nothing is sent home.

Sangfroid is for teams who would rather not. It sells the hosted pipeline: the compute, the search that spends it well, the check on your pull requests, and the record your team keeps.

If you would rather hand over the whole job, the lab runs audits.

Request early access

Try it on a robotLive. One rollout per world.

The same loop, on a system you can watch fail. This is the quadruped from the Go1 study: 400 worlds of floor friction and sideways push, mapped, retrained, mapped again. Pick a world and shove it. The map on the left is the study’s result, and it marks the cell you landed in.

0 worse 340 no significant change 60 safer coloured only at q<0.05, corrected across every condition

PUSH, % BODYWEIGHT → ↑ FRICTION μ 32 TRIALS PER CELL PER VERSION
Go1 locomotion, baseline → retrained. Hover any cell.
Load the simulator, pick a world, and the cell you landed in is marked here.

Push it yourself.

Pick a floor friction and a sideways push, and watch one rollout of the policy in that world. Worlds someone has already probed replay instantly. A new one runs on a GPU, about 3 seconds warm and 25 cold.

A 3D viewer, about 53 MB. It loads only when you ask.

The two maps use the same seeds, and the comparison is corrected across all 400 conditions. The replay is one rollout at one fixed seed; the counts beside it are over all 32 trials in that cell. A policy in a simulator, not a robot on a floor.

What we do for you while we run it

We run the compute.
A real test is thousands of runs of your system. We send them out in batches, collect them as they finish, and never rewrite a result once it is stored.
PLAN ORDER · 200 BATCHES OF 64 · COMPLETE BEFORE ANYTHING RUNS ARRIVAL ORDER · APPEND-ONLY · STILL FILLING next part READ ONCE, AT THE END, OVER THE WHOLE STACK

Out — 400 conditions × 32 trials is 12,800 rollouts, chunked into 200 batches of 64. Every shade in the upper strip is one batch, tinted by its place in that plan. Back — the same batches in the same tints, in the order they actually finished. The gradient is gone, and that scramble is the whole point: no batch waits for another and none of them share state, so the store fills in an order nothing chose. Each one is appended as a part and never rewritten. Only at the end, over the whole stack, does anything read it — Aalen–Johansen per cell, then Fisher exact and Benjamini–Hochberg across all 400, giving 60 safer · 0 worse.

Batch count, batch size and rollout count are this run's. Arrival order is illustrative — the golden fixture's wall_time is synthetic, so there is no real completion order to draw.

One run of the Go1 study, 12,800 rollouts. The figure says which of its numbers are the run’s own and which part is illustrative.
We decide where to look.
Testing everything evenly spends most of the budget where nothing happens. The search puts runs where your agent is likely to break, and anything it finds is rerun on fresh seeds before it is reported.
We guard every pull request.
The same test runs on each change. If any condition comes back significantly worse, the check fails and names it. If the test was too small to tell, it says that instead of passing. 
- name: Crash-test the agent
  run: |
    proofload sweep agent.toml \
        --target my_agent:rollout \
        --pair-with <main-run> --yes
    proofload check <main-run> <this-run> \
        --event violated --fail-on worse
What that check is underneath. The commands and flags are Prove’s own and work today; angle brackets are placeholders.
We keep the record.
Every result is kept as a receipt: what was run, on which seeds, what moved, and what the run could not have detected. Here is a real one.