News · 2026-10-22
Poisson Tasks is out
Tool-use tasks on your own tools, at a difficulty you set, each checked by code. For teams building agents, and for people training models.
The video’s text
- 0:00$ pip install poisson-tasks · poisson labs · tasks 0.1.0 · Evals measure what you test. · Tasks varies the conditions. · Your schemas. No live backend calls.
- 0:06The eval passed. · The tool call failed. · The task didn't.
- 0:10Poisson Tasks · Controlled failure tests. · Introduce a failure. Measure what the agent does next.
- 0:13tools: OpenAI, MCP, OpenAPI · responses: simulated from schemas · backend: never called · difficulty: one variable at a time · grading: deterministic code
- 0:18Pass@1 by injected fault count · gpt-5-nano · petstore · faults:f0 0.900 [0.700, 1.000] · faults:f1 0.800 [0.500, 1.000] · faults:f2 0.800 [0.500, 1.000] · faults:f4 0.300 [0.000, 0.600] · 10 tasks per level · 1 attempt · 95% intervals · task-level bootstrap
- 0:25Vary one condition at a time. · Tool failures · History length · Competing goals · Chain depth
- 0:28Find the failure. Before the release.
- 0:31poisson labs · tasks 0.1.0 · pip install poisson-tasks · poissonlabs.ai/tasks
Your agent passes every eval. Then a tool call fails once in production and it quits.
Most evals grade the happy path. They hand you a pass rate and no way to move it on purpose, so you never see what breaks it. Poisson Tasks is built to show you that.
It builds tool-use tasks on your own tools, from an OpenAI tools file, an MCP server or an OpenAPI document. Responses are simulated from your schemas, so your backend never gets touched. Every task has one right end state, and code checks it. No judge model decides what counts as a pass.
Then you turn a dial. Each dial changes one thing about the task: tool calls that fail and have to be retried, long conversation histories, several goals at once, longer chains of steps. The card shows what each turn of the dial did to your pass rate.
Here is one real run, on the Petstore API with gpt-5-nano and 10 tasks at each level. With no failed calls the pass rate was 0.90. With four, it was 0.30. That is the kind of thing you want to find before your users do.
We measured our own dials before shipping them. Four make tasks harder on the models we tested: depth, failing tools, long histories and several goals at once. Four more did not on those two models, so they ship labelled “available, retested in v2”, and the decoy-tools dial is marked experimental. The full results, and how the measurement ran, are in the docs.
If you train models, the same tasks export to verifiers, checkable at the difficulty you pick.
pip install poisson-taskspoisson-tasksIt needs Python 3.11 to 3.13. Run it with no arguments and it walks you through setup, a preview, a first run and your first card. It’s open source.
This is the first piece of what Poisson Labs is building. A score only means something if you control what changed between runs. Tasks controls the change. Prove tells you whether the result held.
Changelog entryTasks docsgithub.com/poisson-labs/taskspypi.org/project/poisson-tasks