Poisson Tasks is out. Find where your agent breaks before your users do. Read the launch
Poisson Prove is out. It tells you whether a change to your agent really changed how often it fails. Read the launch
Poisson Labs
Talk to us

Poisson Labs · Contribute

Contribute

Engineers and statisticians who want to work on this are welcome. If you read the methods section first, you'll fit in.

Why

Math as infrastructure

Decisions about AI systems rest on numbers: a pass rate, a benchmark score, a before and after. Most of them arrive without the statistics that say how far to trust them. Run the same agent twice and the score moves with nothing changed. Re-run a failing input and it sometimes passes on its own.

A rating without a proof test is a claim, not evidence. We build the layer between a stochastic run and an engineering decision: exact tests, detection floors and runs anyone can repeat, written as production code and tested like infrastructure, so anyone can check the arithmetic.

The code

How the code is built

Typed and linted on every change.
Prove and Tasks run mypy --strict and ruff in CI on every pull request.
The statistics have their own suite.
Prove's math suite checks that each test holds the error rate it states. It runs on any change that touches the statistics, and nightly on every supported Python.
The docs are run, not read.
Every command in Tasks' README and docs runs from the wheel built from the commit, and its output is compared with what the page prints. Prove runs every docs page again from the built wheel in a fresh environment.
Tests are written first, and stay.
Each Tasks module was built against tests written before it, and a test's assertions are not changed to make a build pass. One test regenerates every task on the published cards and compares each with the recorded one.
No model grades another.
Tasks checks every task with code from the same generator. Prove names the test it ran (Fisher's exact, McNemar's, the log-rank) and corrects across every condition.

Each line is what the Prove and Tasks repositories' CI and contributing guide say at their release commits, 83c1f84 and 474ce46.

The values

What we hold to

  • Every claim carries a receipt.

    Each number in a study traces to a stored run. This site's build fails when a page's number drifts from the run it came from.

  • The limit is part of the claim.

    Every study is published with what it couldn't detect, and with its misses: The Same Run Twice names a threshold of ours that did no work.

  • The tools are open.

    Prove and Tasks are open source and on PyPI today, and the methods are on one page.

  • You control and own your data.

    The tools run beside your code, on your machines and keys.

The tools

Try the tools

Prove and Tasks are out and open source. Install one, point it at your own agent, and tell us what broke. Docs

shell · the tools that are out
# Prove 0.2.1 · github.com/poisson-labs/provepip install poisson-prove# Tasks 0.4.1 · github.com/poisson-labs/taskspip install poisson-tasks

Ways in

  • Open an issue

    Issues are open to anyone, on any of our repos. A bug, a number that looks wrong, a test that should exist, an integration you need. GitHub

  • Become a collaborator

    Pull requests come from collaborators we invite. If you want to work on the code, open an issue about what you'd take on, or write to us at work@poissonlabs.ai.

  • Openings

    We're hiring a founding engineer and a statistics advisor. work@poissonlabs.ai

MATH ASINFRASTRUCTURE.

Talk to us