Skip to content
Poisson Labs Docs
GitHub poissonlabs.ai

Examples

Generate tasks, then check them with Prove

Prove is one of the two routes a generated task can be run down. This is Under Prove’s scripted path, run end to end from a pip install: one script writes a pre-registration, builds a registry module from it, writes a Prove sweep spec, and runs three real poisson-prove sweep processes, one per arm, driven by the reference policy, the one that plays each task correctly. Every episode should come back passed: the point of the scripted path is to exercise the plumbing with the model taken out, so a later failure is a model’s and not the instrument’s. The script lives in the Tasks repository, not in the package; it needs only the two installed packages, so it is shown here in full, as saved.

Before you start: Install and Your machine.

1. Install Tasks with the Prove extra

Section titled “1. Install Tasks with the Prove extra”

The prove extra installs poisson-prove beside Tasks; poisson_tasks.adapter is the only module that needs it. Pin both to the versions this run used.

Terminal window
python3.11 -m venv .venv && source .venv/bin/activate
pip install "poisson-tasks[prove]==0.4.0" "poisson-prove==0.2.1"
Output1 line
Successfully installed … poisson-prove-0.2.1 poisson-tasks-0.4.0 …

2. Fetch the tokenizer file

Section titled “2. Fetch the tokenizer file”

Generating a task needs the tokenizer file, a 3.6 MB file the package fetches once, because it counts tokens the way the published cards do (From Python). On a fresh install the script fails without it, so fetch it first.

Terminal window
python -m poisson_tasks.fetch_encoding
Outputexcerpt, 1 of 3 lines
o200k_base: fetched from https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken
⋯ 2 lines
All output3 lines
o200k_base: fetched from https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken
  path   /Users/Shared/tasks-under-prove/.venv/lib/python3.11/site-packages/poisson_tasks/data/o200k_base.tiktoken
  sha256 446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d

3. Save the script

Section titled “3. Save the script”

This is examples/prove_sweep/run.py from the Tasks repository at the commit the side box names, unchanged. Save it as run.py in an empty folder; it writes its store beside itself.

run.py
"""Run the depth knob's arms through Prove, end to end, and read the outcomes.


    python examples/prove_sweep/run.py [--store DIR]


This is the whole scripted path: a pre-registration file, a registry module built from
it, a Prove sweep spec, and three real `poisson-prove sweep` runs (two arms of depth and the
control for the longer one) driven by the reference policy -- the one that plays each task
correctly. Every episode should come back
`passed`, because the policy is the answer the generator shipped with the task. That is
the point of the scripted path: it exercises the plumbing with the model taken out, so a
later failure is a model's and not the instrument's.


It needs Prove installed (`pip install "poisson-tasks[prove]"`), so that `poisson_prove` is
importable and `python -m poisson_prove` runs. Nothing else, and no network: the tasks are
generated in process and the policy is a function.


Files it writes, under `--store` (default: ./example-store beside this file):


    registration.toml   the pre-registration: knob, arms, slack, horizon, split
    registry.py         `load_registry(...)` plus one module-level callable per arm
    sweep.toml          Prove's own spec: the cfg axes, held at one value each
    runs/               the store the sweep writes to
"""


from __future__ import annotations


import argparse
import hashlib
import json
import os
import pathlib
import re
import subprocess
import sys


from poisson_tasks.knobs import ArmSpec, arm_id
from poisson_tasks.tasks import generate


HERE = pathlib.Path(__file__).resolve().parent


GENERATOR_VERSION = "0"  # the version the published cards were measured at
SLACK = 4  # N16: the turn budget is the turns the arm needs, plus this
SPLIT = "eval"
STRICTNESS = "strict"
REPLICATES = 8
MAX_MDE = "0.99"  # this run is about the plumbing, not about detecting an effect
SALT = "0123456789abcdef"


# Two arms of one knob -- depth 1 and depth 3 -- and depth 1's control arm. Same
# scenario, same tools, same prompt shape; the only difference between d1 and d3 is how
# many chained stage calls the goal is away.
#
# `depth:d1+h3` is the control: a depth-1 task whose prompt is seeded with the history a
# depth-3 episode would have produced, so it is as *long* as d3 without being as *deep*.
# Leaving it out is a pre-registration error, and `validate_preregistration` says so --
# try deleting it from this list and running again.
ARMS = [
    ArmSpec("depth", 1, None, STRICTNESS, SPLIT, GENERATOR_VERSION),
    ArmSpec("depth", 3, None, STRICTNESS, SPLIT, GENERATOR_VERSION),
    ArmSpec("depth", 1, 3, STRICTNESS, SPLIT, GENERATOR_VERSION),
]




def registration_text() -> str:
    """The pre-registration, as TOML.


    `horizon` is N17's declared number: the largest turn budget any arm in this
    registration has. `validate_preregistration` recomputes it and refuses a document
    that states a different one, so it cannot drift from the arms beside it.
    """
    horizon = max(arm.max_turns(SLACK) for arm in ARMS)
    arms = json.dumps([arm_id(a) for a in ARMS])
    return (
        f'knob = "depth"\n'
        f'strictness = "{STRICTNESS}"\n'
        f'split = "{SPLIT}"\n'
        f'generator_version = "{GENERATOR_VERSION}"\n'
        f"slack = {SLACK}\n"
        f"horizon = {horizon}\n"
        f"arms = {arms}\n"
    )




def registry_text(registration: pathlib.Path, digest: str) -> str:
    """A registry module is three lines: load the registration under its hash, then put
    the targets it built into the module's globals, where `--target` can reach them."""
    return (
        "from poisson_tasks.adapter import load_registry\n\n"
        f"REGISTRY = load_registry({str(registration)!r}, sha256={digest!r})\n"
        "globals().update(REGISTRY.targets)\n"
    )




def sweep_text(horizon: int) -> str:
    """Prove's spec. Every cfg axis is held at one value, because this run is about
    the plumbing and not about a contrast; `t_unit = "turns"` is what the horizon counts.


    `replicates` and `--max-mde` are a real gate: Prove refuses a design it judges
    underpowered for the effect you say you want to detect, before it runs anything. At 8
    replicates it will only accept a loose `--max-mde`, which is honest for a run whose
    outcome is known in advance."""
    generator_hash = generate(ARMS[0], 0).generator_hash
    return (
        'target_id = "poisson-tasks-example"\n'
        't_unit = "turns"\n'
        f"replicates = {REPLICATES}\n\n"
        f'[axes.slack]\ntype = "continuous"\nvalues = [{SLACK}]\n\n'
        f'[axes.horizon]\ntype = "continuous"\nvalues = [{horizon}]\n\n'
        f'[axes.strictness]\ntype = "categorical"\nvalues = ["{STRICTNESS}"]\n\n'
        f'[axes.generator_hash]\ntype = "categorical"\nvalues = ["{generator_hash}"]\n'
    )




def sweep(store: pathlib.Path, spec: pathlib.Path, target: str, modules: pathlib.Path) -> str:
    env = dict(os.environ)
    env["PYTHONPATH"] = os.pathsep.join(
        [str(modules)] + ([env["PYTHONPATH"]] if env.get("PYTHONPATH") else [])
    )
    done = subprocess.run(
        [sys.executable, "-m", "poisson_prove", "--store", str(store), "sweep", str(spec),
         "--target", target, "--max-mde", MAX_MDE, "--salt", SALT, "--yes", "--json"],
        env=env, capture_output=True, text=True, timeout=600,
    )
    # The text form of `sweep` names the run by its name; the id is in the JSON document.
    try:
        run_id = json.loads(done.stdout)["run_id"]
    except (ValueError, KeyError, TypeError):
        run_id = None
    if done.returncode != 0 or not isinstance(run_id, str):
        raise SystemExit(
            f"poisson-prove sweep --target {target} exited {done.returncode}\n"
            f"--- stdout\n{done.stdout}\n--- stderr\n{done.stderr}"
        )
    return run_id




def episodes(store: pathlib.Path, run_id: str) -> list[dict[str, object]]:
    """The stored episodes, read from the store's parquet files.


    `outcome`, `t` and `seed` are Prove's own columns. `meta` is what the adapter
    wrote beside them -- the arm, the knob and level, the scenario and task ids, the
    pin record, and the calls the policy actually made. Prove's public verbs do not
    print an episode's `meta`, which is why this reads the files; that is filed as PG-17
    in `gap-log.md`.
    """
    import pyarrow.parquet as pq


    rows: list[dict[str, object]] = []
    for part in sorted((store / run_id / "episodes").glob("part-*.parquet")):
        rows.extend(pq.read_table(part).to_pylist())
    for row in rows:
        row["meta"] = json.loads(str(row["meta"]))
    return sorted(rows, key=lambda r: int(r["replicate_index"]))




def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--store", type=pathlib.Path, default=HERE / "example-store")
    args = parser.parse_args()
    out = args.store
    out.mkdir(parents=True, exist_ok=True)


    registration = out / "registration.toml"
    registration.write_text(registration_text())
    digest = hashlib.sha256(registration.read_bytes()).hexdigest()
    print(f"registration {registration}\n  sha256 {digest}")
    print("  " + registration_text().replace("\n", "\n  ").rstrip())


    modules = out / "modules"
    modules.mkdir(exist_ok=True)
    (modules / "registry.py").write_text(registry_text(registration, digest))


    from poisson_tasks.adapter import load_registry


    registry = load_registry(registration, sha256=digest)
    print(f"\nregistry: slack {registry.slack}, horizon {registry.horizon}")
    print(f"  targets: {', '.join(sorted(registry.targets))}")


    spec = out / "sweep.toml"
    spec.write_text(sweep_text(registry.horizon))


    store = out / "runs"
    run_ids: dict[str, str] = {}
    for arm in ARMS:
        # The target name is the policy, then the arm id with every run of
        # non-alphanumeric characters collapsed to one underscore: `--target module:attr`
        # will not take a colon or a plus in an attribute name.
        target = "reference_" + re.sub(r"[^0-9A-Za-z]+", "_", arm_id(arm))
        print(f"\n--- poisson-prove sweep --target registry:{target} ---")
        run_id = sweep(store, spec, f"registry:{target}", modules)
        rows = episodes(store, run_id)
        labels: dict[str, int] = {}
        for row in rows:
            label = str(row["outcome"])
            labels[label] = labels.get(label, 0) + 1
        print(f"  run_id   {run_id}")
        print(f"  episodes {len(rows)}  outcomes {labels}")
        for row in rows[:3]:
            meta = row["meta"]
            print(f"    seed {row['seed']:>10}  t={row['t']}  {row['outcome']:<10} "
                  f"arm {meta['arm_id']}  {meta['sampled_calls']} calls")
        print(f"    ... {len(rows) - 3} more")
        run_ids[arm_id(arm)] = run_id


    # `poisson-prove diff` is how a knob is read: the same seeds under two arms. `--event` is
    # needed here because every episode passed, so there is no failure event for Prove
    # to infer. The number worth reading is `truncated`, and it should be 0: N17 is the claim
    # that the declared horizon cuts no arm of this registration short, and this is where it
    # is measured rather than asserted. `--json` carries it in every Prove the extra allows,
    # so this reads the document and prints that number and the verdict.
    print("\n--- poisson-prove diff: depth:d1 against depth:d3 (head) ---")
    diff = subprocess.run(
        [sys.executable, "-m", "poisson_prove", "--store", str(store), "diff",
         run_ids["depth:d1"], run_ids["depth:d3"], "--event", "passed", "--json"],
        capture_output=True, text=True, timeout=300,
    )
    if diff.returncode != 0:
        raise SystemExit(f"poisson-prove diff exited {diff.returncode}\n{diff.stderr}")
    compared = json.loads(diff.stdout)
    print(f"  {compared['verdict']}  {compared['sentence']}")
    print(f"  truncated {compared['truncated']}  (episodes the horizon cut short)")
    print("  ... (run the command yourself, without --json, for the report)")


    print(f"\nthe store is at {store}.")
    print(f"  poisson-prove --store {store} inspect {run_ids['depth:d3']} --event passed")




if __name__ == "__main__":
    main()

4. Run it

Section titled “4. Run it”

The registration, the registry, the sweep spec, three sweeps of eight episodes each (two arms of the depth knob and the control for the longer one), and a first comparison. It takes a few seconds and writes about 100 KB under example-store.

Terminal window
python run.py
Outputexcerpt, 2 of 44 lines
⋯ 37 lines
--- poisson-prove diff: depth:d1 against depth:d3 (head) ---
  no significant change  every condition that did not count could have shown a change of 0.990: the largest limit it read is 0.801 (the paired test (McNemar), Holm q 0.05 over 1 tests, 4 times in 5); passed: 0 lower, 0 higher (paired test on shared seeds)
⋯ 5 lines
All output44 lines
registration /Users/Shared/tasks-under-prove/example-store/registration.toml
  sha256 642874648e05868eab18a6de12e2412ec9dd54dc045df29622973f3248411dc6
  knob = "depth"
  strictness = "strict"
  split = "eval"
  generator_version = "0"
  slack = 4
  horizon = 8
  arms = ["depth:d1", "depth:d3", "depth:d1+h3"]


registry: slack 4, horizon 8
  targets: reference_depth_d1, reference_depth_d1_h3, reference_depth_d3


--- poisson-prove sweep --target registry:reference_depth_d1 ---
  run_id   20261106T231318Z-cdf9
  episodes 8  outcomes {'passed': 8}
    seed  366625197  t=2.0  passed     arm depth:d1  1 calls
    seed 3234607431  t=2.0  passed     arm depth:d1  1 calls
    seed 1210394339  t=2.0  passed     arm depth:d1  1 calls
    ... 5 more


--- poisson-prove sweep --target registry:reference_depth_d3 ---
  run_id   20261106T231321Z-4ba1
  episodes 8  outcomes {'passed': 8}
    seed  366625197  t=4.0  passed     arm depth:d3  3 calls
    seed 3234607431  t=4.0  passed     arm depth:d3  3 calls
    seed 1210394339  t=4.0  passed     arm depth:d3  3 calls
    ... 5 more


--- poisson-prove sweep --target registry:reference_depth_d1_h3 ---
  run_id   20261106T231322Z-3382
  episodes 8  outcomes {'passed': 8}
    seed  366625197  t=2.0  passed     arm depth:d1+h3  1 calls
    seed 3234607431  t=2.0  passed     arm depth:d1+h3  1 calls
    seed 1210394339  t=2.0  passed     arm depth:d1+h3  1 calls
    ... 5 more


--- poisson-prove diff: depth:d1 against depth:d3 (head) ---
  no significant change  every condition that did not count could have shown a change of 0.990: the largest limit it read is 0.801 (the paired test (McNemar), Holm q 0.05 over 1 tests, 4 times in 5); passed: 0 lower, 0 higher (paired test on shared seeds)
  truncated 0  (episodes the horizon cut short)
  ... (run the command yourself, without --json, for the report)


the store is at /Users/Shared/tasks-under-prove/example-store/runs.
  poisson-prove --store /Users/Shared/tasks-under-prove/example-store/runs inspect 20261106T231321Z-4ba1 --event passed

5. Read the store it wrote

Section titled “5. Read the store it wrote”

The three runs are an ordinary Prove store, so every Prove command reads them. runs lists them.

Terminal window
poisson-prove --store example-store/runs runs
Outputexcerpt, 5 of 6 lines
ok  3 runs in example-store/runs
⋯ 1 line
#  name                                id                     status    backend  target                 rows  rows without an outcome  created
1  poisson-tasks-example 2026-11-06 c  20261106T231322Z-3382  complete  local    poisson-tasks-example  8     0                        2026-11-06T23:13:22Z
2  poisson-tasks-example 2026-11-06 b  20261106T231321Z-4ba1  complete  local    poisson-tasks-example  8     0                        2026-11-06T23:13:21Z
3  poisson-tasks-example 2026-11-06 a  20261106T231318Z-cdf9  complete  local    poisson-tasks-example  8     0                        2026-11-06T23:13:18Z
All output6 lines
ok  3 runs in example-store/runs


#  name                                id                     status    backend  target                 rows  rows without an outcome  created
1  poisson-tasks-example 2026-11-06 c  20261106T231322Z-3382  complete  local    poisson-tasks-example  8     0                        2026-11-06T23:13:22Z
2  poisson-tasks-example 2026-11-06 b  20261106T231321Z-4ba1  complete  local    poisson-tasks-example  8     0                        2026-11-06T23:13:21Z
3  poisson-tasks-example 2026-11-06 a  20261106T231318Z-cdf9  complete  local    poisson-tasks-example  8     0                        2026-11-06T23:13:18Z

6. Compare two arms

Section titled “6. Compare two arms”

The script’s own comparison, run by hand as it suggests: depth 1 against depth 3 on the one outcome the runs hold. Every episode passed in both, so the comparison is no significant change, and the limit it prints says what size of change the eight episodes could have shown.

Terminal window
poisson-prove --store example-store/runs diff "#3" "#2" --event passed
Outputexcerpt, 1 of 16 lines
no significant change  every condition that did not count could have shown a change of 0.990: the largest limit it read is 0.801 (the paired test (McNemar), Holm q 0.05 over 1 tests, 4 times in 5); passed: 0 lower, 0 higher (paired test on shared seeds)
⋯ 15 lines
All output16 lines
no significant change  every condition that did not count could have shown a change of 0.990: the largest limit it read is 0.801 (the paired test (McNemar), Holm q 0.05 over 1 tests, 4 times in 5); passed: 0 lower, 0 higher (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  threshold  0.990 (recorded at sweep time by the runs)
  before  poisson-tasks-example 2026-11-06 a
  after  poisson-tasks-example 2026-11-06 b
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 1 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 1 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 1 tests)
  intervals  on each condition's change: Tango at 95%, simultaneous (Bonferroni) over the 1 tests
  fragility  2 outcomes would change the word: 2 runs of A in generator_hash 49a5f64878314b71ace9bfebeb18050363cefebeedefadf24098778eed895b05, horizon 8.0, slack 4.0, strictness strict that ended in passed made to end otherwise


condition                                                                                                                  event   before  after  change  verdict    rise     fall
generator_hash 49a5f64878314b71ace9bfebeb18050363cefebeedefadf24098778eed895b05 · horizon 8 · slack 4 · strictness strict  passed  1.000   1.000  0.000   unchanged  no room  0.801


  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound


  next: poisson-prove report "poisson-tasks-example 2026-11-06 a" "poisson-tasks-example 2026-11-06 b" --x generator_hash --y horizon

Stored output

Section titled “Stored output”

Every output above is what the command printed, stored in the site repository under src/data/examples/generate-tasks-then-check-with-prove/ with run.py as copied from the Tasks repository at its commit (its sha256 is recorded), the comparison’s diff.json and the store’s run list, as Tasks 0.4.0 and Prove 0.2.1 wrote them. The page cannot show an output, or a file, that is not there.