Changelog

All notable changes to Poisson Tasks.

0.1.0 (2026-10-22)

The guided session

• A bare poisson-tasks on a terminal opens a guided session, with no AI inside. With no tasks.toml in the folder it walks a first run through five steps: set up (which asks first whether to test a model on the built-in tasks or on your own tools), one task, the run with its plan and cost question, the card, and what next. With a tasks.toml it opens on a home screen that reads the folder from disk alone (settings, runs with what each kept and spent, the latest card’s levels, whether the environment is ready) and offers at most five numbered steps for the folder’s state. Enter takes step 1; a short word takes any action: run, again, card, more, replies, task, verify, runs, settings, setup, flow, export, check, home, help and quit.
• Every action is a verb’s command line. Each runs through the same parser and verb as the command, and ends with same as <the command>, so what the session writes is byte for byte what that command writes from a shell, and scripts are one copy away. Nothing calls a model until a yes at the run’s own cost question; the session never passes --yes to run.
• Cards are drawn, never raw Markdown. The card view is the verb’s result; more opens one section of card.md at a time, its tables drawn as tables (Rich 13.4.2 or later, now the floor); replies lists one arm’s episodes with their outcomes and opens one, call by call, with the simulated answers and the model’s last reply.
• Ctrl-C stops the step, never the session. At a question inside a step it stops that step as the command does (nothing written); during a run it is the run’s own interrupt, and the home screen then offers the resume; at the prompt it clears the line, and twice on an empty line it leaves, as quit and Ctrl-D do, with exit 0.
• The load screen is the Poisson distribution at λ = 4 as one row of eighth-block bars in cobalt beside poisson labs, with tasks and the version beneath; monochrome under NO_COLOR, and the one line Poisson Labs tasks <version> with --plain, under TERM=dumb or below 26 columns. It shows once, only in a session.
• Off a terminal there is no session. A bare command in a pipe, under CI, with stdin redirected or with --json keeps its exit 2 and its error line, which now names the session and ends with next: <the folder's first step>. On a terminal with --plain, NO_COLOR or TERM=dumb the session opens in the plain look: numbered menus, a plain prompt, no escape sequence.
• poisson_tasks.folder.read_folder(path) is the library name behind the home screen: a Folder of facts (settings, environment, each run’s kept of total, dollars, card and lock, the newest card’s levels, the tool counts), read from disk with no network and without loading the tokenizer.
• What the session changed in the verbs, so that each is one implementation: init’s first question is what to test, a model on the built-in tasks or on your own tools (own tools then asks for the file before any recipe), its questions are numbered as asked, its note on flows says what a flow is and names --own-flow and [[own.flow]], and its closing screen points at doctor --show-command instead of printing the 18-flag command (--json keeps it); init --force keeps the replaced file as tasks.toml.bak and says so; run holds an advisory lock on <run folder>/run.lock while it runs, so a second run on that folder is refused with nothing spent, and a lock left by a process that died does not count; card’s result names its run (folder, dollars spent, kept of total) and says once how it stands beside the published cards; a card on your own tools no longer suggests card --compare published, and that command reads the written card without --force, prints one verdict line, different ..., writes card/compare.md and prints neither card, on the built-in tasks too; generate --show with no --arm shows the plan’s first arm’s first task and names the other arms instead of refusing; verify with nothing to judge, or on your own tools, ends with a complete next: command; an overwrite refusal’s next: keeps every flag you typed; doctor’s rows say ok, note, refused or not ok, each by what the row is; export’s warning above its commands is one plain line; and in the pretty look no path, URL, command or quoted value is ever split by a line break, and a wrapped row keeps its indent.
• Dependencies. prompt_toolkit>=3.0.53 is declared (it was installed through questionary already; no package is added) and Rich’s floor rises from 11.0.0 to 13.4.2.

The command

• poisson-tasks is the way in. It puts the library behind seven verbs (python -m poisson_tasks is the same command). init asks what to test and what to measure, in plain words, and writes one settings file, tasks.toml (doctor --show-command prints the same setup as one command that asks nothing); it offers to fetch the tokenizer file first if it is missing. generate makes the plan’s tasks with no model and no network and writes them as files, or shows one task in full. run shows the plan, the expected calls and cost and the cap, asks, and runs it against the model, stopping at the cap and resuming after Ctrl-C. card builds a run’s card and says what it means and what it does not claim. verify judges an answer with the task’s own verifier. export writes the configs verifiers’ eval runs. doctor says whether this environment is ready.
• Every verb prints its result first. A problem is one error: line on standard error, saying what to do; the exit codes are 0 done, 1 refused, 2 the command line is wrong, 3 stopped after work started, 4 a check you asked for did not pass, and 130 interrupted; --json prints the result as one document. Options may come before or after the verb, and abbreviated options are refused.
• Two looks, one result. On a terminal the command draws its result with Rich, with arrow-key menus in init. In a pipe, under NO_COLOR or CI, with TERM=dumb, on a standard output that is not UTF-8, or with --json or --plain, it prints plain text with the same facts in the same order. Rich and questionary are ordinary dependencies.
• Five providers, and recipes. init knows openai, anthropic (through its OpenAI-compatible endpoint), ollama, vllm and any other OpenAI-compatible endpoint, and prefills a price only for gpt-5-nano-2025-08-07. A recipe is a named set of arms with a task count and an attempt count (poisson_tasks.arms.RECIPES holds eight: first-look, tool-choice, error-recovery, long-sessions, planning, reading-and-filling, profile and card); [measure] preset takes the ones offered, and a recipe whose knobs are not shown to work yet is refused in one line naming them. The first run’s default is first-look: depth at 1, 3 and 8, 10 tasks each, 30 episodes.
• Refusals before anything is spent. A plan expected to cost more than its cap is refused before the question, with a cap that would let it finish. generate, export and card refuse to overwrite their files without --force. A next: command carries the --config you gave. doctor with no settings file reports on the environment alone. A key variable written the way a shell writes it ($OPENAI_API_KEY) is told to write the name alone.
• What to do after a refusal, and what a printed command costs. generate --show with several arms shows the first arm’s first task and names the others. A card refused because a level’s attempts made no tool call says they answered in text instead, names the files in the run folder that hold the replies, and ends with the command that shows the task they answered. export says, right above the commands it prints, that each calls the model and spends money with no cap.
• Library names added for the command: config.INIT_FLAG_ALIASES, providers.refuse_plain_http_key, poisson_tasks.transcripts (read_transcript and verdict_of, which read an attempt in any of its three forms), poisson_tasks.arms (arms by their id, the generator-version constant, the recipes, knob_status()), poisson_tasks.outcomes (one plain sentence for each outcome label) and poisson_tasks.cards (the published cards by their model’s short name).

Tasks on your own tools

• Bring the tools you already have. An OpenAI tools file (Chat Completions or Responses), an MCP tool list (or an MCP server, started once with your consent only to list its tools, which Tasks never calls) or an OpenAPI 3.0 or 3.1 document is read into one form. A tool Tasks cannot use is refused in one line, and the rest of the set still runs; a whole file is refused when it matches none of the three shapes, when fewer than 2 or more than 256 usable tools remain, or when no tool takes an argument a task could check exactly. Whether a tool reads or writes comes from the file where it says so (MCP’s readOnlyHint, OpenAPI’s methods); OpenAI tools do not say, and count as writes until the settings say otherwise; init and doctor count a tool as you declared it. PyYAML is a declared dependency, for OpenAPI documents written as YAML.
• Tasks are made on those tools, and your backend is never called. A simulator shaped by your schemas answers every call, deterministically, and a verdict is judged by replaying the model’s calls through it. A task is a pure function of your file’s bytes, the arm and the seed. The settings file keeps the tools under [own] (file, format, name, read, write, [own.intents] and [[own.flow]]), each setting has an init option, and generate, run, card, export and doctor work on them. export serves them as the taskset poisson-tasks-own.
• Every dial is offered, within what each set supports. Depth and shape need flows you declare; block goes up to the set’s size; spread needs a read that takes a checkable argument; the decoy-tools dial and control arms are not offered. The recipes lead with the knobs that shipped, and doctor and init show the knob probe’s status for each dial beside it.
• The card on your tools says so first: it measured your tools with simulated responses, is read level against first level within itself, and is never pooled with a published card. card --compare published writes it beside the published card of its model in card/compare.md, each whole, pooling nothing, and prints the one-line verdict different. doctor says when the newest run on the same tool file carries another digest than the file has now.
• verify refuses a tasks file made on your own tools, and --arm in a folder set up for them, in one line. verify does not judge an own-tools task on its own in this release; a run’s card judges every episode by replay.
• A request on your tools leaves nothing to guess. On every dial but arguments it states every value of the call that Tasks can make up from your schemas, required or not, so the model is told which record to change and what to change it to; a free-text value is stated and not checked. generate --show --arm in a folder set up for your tools shows the task generate writes for that arm, on your tools.
• Library names added: poisson_tasks.own and its modules (simulator, generator, verdict, recipes, export and card), the top-level module poisson_tasks_own, Provenance.tools, config.OwnConfig, ProjectConfig.own, Plan.own, runs.own_bundles.

The nine knobs, and what the knob probe found

• Nine knobs have tasks. Beside depth and distractor, similarity (levels 0 to 3: how many digits of the right tool’s ledger number each lookalike tool shares), faults (0, 1, 2 and 4 calls that answer unavailable and must be made again), history (0 to 2400 tokens of earlier tool work in the conversation), block (8 to 64 tools offered), arguments (2 to 8 values each call must carry, copied from the request), shape (1, 2 or 4 records advanced side by side) and spread (the entry code split across 1, 2 or 4 tool answers). Each has its arm ids and turn budget; faults, block, arguments and shape each have a control arm that adds the higher level’s length without its difficulty. A run of any of them has its tool messages replayed against the engine when its records are read, and a disagreement scores nothing.
• The pin ledger has thirty rows, seventeen of them for the new knobs. Every row that can be measured from a bundle is measured on every task.
• Every task id moved, and no task did. The generator version is now "1", and the generator hash covers the three modules that shape a task. The 840 tasks of each published card regenerate with the same prompt, tools, goal and reference solution under a new id. The generator does not change again until the cards are measured again.
• A knob probe measured the seven newer knobs on gpt-5-nano and Qwen3-8B: 60 tasks and 2 attempts an arm, strict pass@1, the lowest level against the highest. faults (bites on Qwen3-8B), history (bites on both) and shape (bites on Qwen3-8B, four wide the harder) shipped, and the recipes that measure them (error-recovery, long-sessions, planning and profile) are offered beside first-look and card. similarity, block, arguments and spread did not ship: they stay in the generator and on the command line, labelled Available · retested in v2.
• Each knob’s measured status is in the package. poisson_tasks.arms.knob_status() reads knob_status.json: for each knob and model, bites, does not bite, top easier or cannot tell, the base and top arms’ strict pass@1, and each contrast’s drop with its 95% interval. No knob is called a non-biter on a model where its base arm passed below 0.15, since a drop of 0.15 could not show there. docs/knobs-and-arms.md has the table, and three things about how the probe ran that are disclosed with it: a check was changed after the first result and read again once, a crash at the start cost one arm a second run, and one model was given more time.
• The decoy-tools dial is experimental. It does not make tasks harder for either published model, so it is not a difficulty setting, and the export’s help for the arm option says so.

The published cards, in the package

• Both published cards ship inside the package, as poisson_tasks/cards/: gpt-5-nano-2025-08-07 (4,200 episodes, about $2.52) and Qwen/Qwen3-8B at revision b968826d (4,200 episodes, served with vLLM on a Colab L4). poisson-tasks card --published rebuilds each from its records and checks it against its stored artifact byte for byte, and the run’s estimate reads them from the package. A test holds both cards to recomputing byte for byte.
• Both say they are not current for this release. They were measured at generator version "0", before the new knobs; the tasks regenerate with the same text and other ids. A card from your own run is not set beside them (card --compare published says so in one line, and the card’s own result carries the same row), and the next release measures the cards again on the current generator.
• A card for any plan. calibration.card_from_run(folder) builds the card of a run folder under the plan’s own design: its tasks per level, attempts, decoding and turn cap. A run that did not finish gives a partial card that says how many attempts it owes; a level never reached is listed as not run, never as zero. card_claims says what a card means and what it does not claim; a model served by its user is recorded as such.
• Every card shows what each level cost: median model calls, median tokens and median episode seconds beside the rates, and three cost tables (calls and tokens, time, and token totals and dollars). A figure that was not recorded reads not recorded, never zero, and time is printed and never compared.
• Depth is read on reaching the goal, with stopping at the goal beside it, because every depth level offers the whole stage ladder and the two published models do not fail the two skills the same way. docs/calibration.md is one page for every published card, rendered from them: where the models’ orderings agree (of 12 adjacent pairs, on none, and they disagree on one), then each card in full.
• A level where more than 10% of attempts made no tool call is not published, and neither is its card; the same refusal is in poisson-tasks card.
• An episode ends after 32 model calls. The taskset pins a turn limit, agent.max_turns = 32, beside the measured harness, so the documented eval command and the calibration run behave the same. An episode that reaches it is scored turn_limit; --env.agent.max-turns N changes it.
• A card rebuilds byte for byte on Python 3.12 and 3.13, not only on 3.11. From 3.12 Python adds a list of decimal numbers with a correction that 3.11 did not make, so on 3.12 and 3.13 a card’s rates moved in their last digit and poisson-tasks card --published (and card --check on a card built on 3.11) said the card no longer rebuilt. A card now adds its figures one at a time, as 3.11 did, on every Python; the published cards and every result on 3.11 are unchanged. A test holds both published cards to rebuilding under the corrected addition.

The run engine

• What a plan will cost is told before anything is spent: runs.estimate reads the published card of the plan’s model (gpt-5-nano’s for a model no card measured), and the plan names which card it read.
• One folder per plan. A run goes in runs/<preset>-<model>/, then -2, -3 for a second run of the same plan or a changed one, with no hash in its name.
• The cap is a bound on what is spent. The spend and token caps are checked before every model call, so the final spend is at most the cap plus the calls in flight when it was reached; the cap holds the folder’s total across runs. spend.jsonl has one line for each episode that ended.
• Resume loses nothing and runs nothing twice. A stopped run keeps every finished episode, and a resume runs exactly the episodes owed; a run in which some episodes failed is resumed too, and its failure stays in the instrument tally. Ctrl-C stops dispatching, gives the episodes in flight ten seconds, writes the ledger and exits 130; a second Ctrl-C stops at once.
• Stops. The provider saying the account is out of credit stops the run. So do five episodes in a row failing before any was scored (the message gives the first error’s status and the provider’s own words), and a run that makes no model call and finishes no episode in 600 seconds. A run in which every episode failed never exits 0.
• What a run changes on your machine is listed: the run folder, ~/.cache/verifiers, ~/.local/bin/uv and uvx, ~/.cache/uv, uv’s receipt in ~/.config/uv, one /tmp/vf-port-* file per episode, and one empty uv lock file in the temporary folder. The harness runs with PIP_REQUIRE_VIRTUALENV=1, so its bootstrap takes uv’s installer every time, and UV_NO_MODIFY_PATH=1, so that installer edits no shell start-up file.
• run and export are one path. runs.verifiers_config is the eval config the run uses for an arm; export --to verifiers writes the same one, with push off and a decoding field declared "not sent" left out of every request.
• A fix to traces written with a project key. The run engine masked any four characters of the key wherever they occurred in a trace line. Every OpenAI project key starts with sk-proj-, and the tasks’ item codes hold -pro, so earlier runs made with a project key could have had nearly every trace line rewritten, leaving lines that were no longer JSON or that failed the replay check when the records were read. Runs made without a key, and the published cards, were not affected. A key’s public prefix (sk-proj-, sk-ant-api03-, gsk_) no longer counts as part of the key; where a provider can put the key, any four of its characters are masked; and no file of a run folder holds twelve characters of the key in a row. Run again any run made with a key before this fix whose records you rely on.

The verifiers export, and the Colab driver

• The verifiers export (poisson_tasks.export): a generated task as a verifiers v1 taskset, scored by the generator’s own verifier, with its tool block served over MCP. The package is the taskset id poisson-tasks, and eval runs it with the null harness in a local subprocess with no flag. ExportTasksetConfig.arm takes an arm’s id or its six fields.
• A second published card needed a Colab driver. poisson_tasks.calibration_run and notebooks/calibration_model_b.ipynb serve Qwen3-8B at a pinned revision with vLLM 0.29.0 on a Colab L4 and run the same card, checkpointing to Google Drive so a dropped runtime resumes; the same driver ran the knob probe’s Qwen3-8B half. Its help describes it without the card’s old name, every option has help text, and each step lists only the options it reads (the notebook hands every step the same four, so a step accepts the rest without listing them).
• Vocabulary v2: the record-id word lists are drawn disjoint from every tool name.
• The name is poisson-tasks (the package poisson_tasks).

Packaging, documents and examples

• The wheel carries the published cards (poisson_tasks/cards/, mapped from the repository’s calibration/ folder), py.typed for poisson_tasks and poisson_tasks_own, the knob probe’s table and the poisson-tasks command; it carries no tokenizer file, no examples and no tests. A test builds the wheel and reads its file list. The module poisson_tasks/cards.py and the data folder poisson_tasks/cards/ share a name; the folder holds no Python file, so the module wins the import, and a test holds it.
• Python 3.11 to 3.13 on macOS and Linux is what the package claims (requires-python = ">=3.11,<3.14", matching verifiers 0.3.1), and hatchling joins the development tools.
• The source archive leaves out .work/, a local working folder: .gitignore names it, since the archive is built from that file.
• The README is for users: install, init, run and card first, then who it is for, what a first run costs and changes on your machine, the nine dials and what the evidence says of each, and the published cards. Contributor material moved to CONTRIBUTING.md. New pages in docs/: commands.md (every command’s help, held equal to what the command prints), config.md, own-tools.md, python.md, prove.md and cli-look.md (the look every command of the lab’s tools keeps); calibration-guide.md became reading-a-card.md. Every command block in them is marked for the check that runs them from a built wheel, and the files a user reads are held to plain words by a test.
• --help and a mistaken argument need no tokenizer file, on the three examples and on the Colab driver’s verify: they read their command line before they load the generator. The examples use generator version "0", the label the published cards carry.
• docs/verifiers-export.md says to give an arm by its id, or as its six fields in one JSON object for a leaky arm, and that the dotted --env.taskset.arm.<field> form is not a way to give one; it also explains export --to verifiers and why the bare eval of a shell is not verifiers’ eval.
• Plain words. The card, the calibration module’s and the Colab driver’s messages and help, the notebook, the engine, the knobs and adapter modules, the tests’ comments and gap-log.md cite no internal decision ids, specification sections, private notes or people.