Tools · in development
Tasks
Tool-use tasks that get harder one knob at a time, each generated with its own verifier.
An agent fails a task and you are left guessing which part of the task did it. This generates the task instead of collecting it: one knob moves between the two sides of a comparison, everything else is held fixed and measured, and the verifier comes out of the same generator as the task. So a change in the pass rate can be pinned to the thing that changed.
Generation makes no model calls and no network calls. It is a deterministic function of a seed, over one mock stateful domain: records that move through stages, and a block of tools that move them. The model, the harness and the statistics are all somewhere else.
It is the fourth of the lab's tools, and the one that feeds the other three. The repository is not public yet, and no study has been run with it. The methods behind the numbers it will produce are one page.
The arms a sweep compares
depth:d3 # 3 dependent calls
depth:d1+h8 # its control: 1 call,
# 7 pairs seeded
distractor:r4@k16 # 4 of 16 tools distract
distractor:r0+len(r8)@k16 # its control: 0 of them,
# the tokens of 8What it is for
What it does
Generates tool-use tasks from a seed, each with the verifier that grades it. A sweep moves one knob, how many dependent calls the task needs or how many look-alike tools sit in the block, and holds the rest fixed and measured. The output is a set of tasks for your agent and a verdict on each episode, ready for a paired comparison.
Who it is for
People who build or evaluate tool-using agents and need to know which property of a task is costing them: comparing models or harnesses, or checking whether a change moved the point where the agent starts to fail. It runs on your own machine, and generating tasks needs no model and no network.
What it replaces
A collected task set with difficulty labels applied afterwards. Between an easy task and a hard one in such a set, the prompt, the tools, the length and the grading all differ at once, so a drop in the pass rate cannot be pinned on any of them. It also replaces a model as the grader: the verifier is code. It does not replace a benchmark's job of ranking models, and does not try to.
How it works
Generate
A seed fixes the scenario: record ids, field values, which class is the target, where every tool sits in the block. An arm is a transform from that scenario and one knob setting to a task. The same seed under every arm of a comparison gives the same scenario, so the two sides pair by construction rather than by sampling.
Pin everything else
Thirteen rows say what each knob's arms hold fixed: the scenario, the system prompt, the tool block, the prompt template and its slots, the overlap between the instruction and the tool names, the observation size. Each row carries how it is measured and what happens when it fails: generation refuses, or the number is reported, or the comparison is void.
Hand it off
Tasks come out two ways: as a verifiers taskset, and as Prove rollout targets, one per arm. It computes nothing. No p-value, no interval, no power, no verdict. Those are Prove's, and this tool never touches them.
The knobs
Workflow depth. A task needs d dependent stage calls on the
target record; each call waits on the token the last one returned. Candidate levels
are 1, 2, 3, 5 and 8. The tool block is 16 tools and is byte-identical at every
level, and the only part of the prompt that moves is the name of the goal stage.
Distractor ratio. At a fixed 16 tools and a fixed depth of 2, r of
the block are functional distractors: the same verb, the same parameters in the same
order, a description from the same template, differing only in which class of record
they act on. Candidate levels are 0, 2, 4, 8 and 14.
The outcomes a task can end in
passed # the only success
wrong_state # stopped in the wrong stage
overshoot # went past the goal stage
collateral_mutation # changed the wrong record
turn_limit # ran out of turns
malformed_call # a call the block cannot take
verifier_error # the verifier refused
api_error # ─┐
wall_timeout # ├ the adapter's, not
replay_mismatch # ─┘ the generator'sTen, closed and named. Seven are the verifier's verdict on an episode; the last three are the adapter's, for a run that never got far enough to have one.
The control arm
Depth moves two things at once: how many calls the task needs, and how much history the agent is reading by the end. So each depth arm has a control at depth 1 carrying the same amount of history, seeded in before the agent's first turn, sized against the calls it stands in for.
It does not match on everything, and the page for a result has to say so: the seeded history is front-loaded, and the agent did not write it. What the control cannot hold fixed is declared in the ledger rather than argued away.
What it is not
Not a benchmark: no leaderboard, no fixed corpus as the product, no model ranking. Not a statistics tool: it never computes a p-value. Not a judge: no model grades any output. Not a harness or an agent framework, and not an environment hub.
It is a generator with a verifier attached, and everything it emits goes to something else to be run and measured.
The reason for the knobs is the shape of the answer. A difficulty axis you can set is an axis you can report: not "the agent scored 61%", but "moving this one thing from here to here changed the pass rate by this much, with everything else held at these values".
What a result will be able to say
The spec fixes the sentence before anything runs, and a result that will not fit it does not get published.
Every slot has to be filled: the generator version, the domain hash, the verifier strictness, the harness and its version, the model and its revision, how it was served and decoded, the knob and the two levels it moved between, the change in the pass rate and its interval, and the N the power gate accepted. Anything a run cannot fill, the write-up cannot claim.
Which also fixes what a result cannot say. Nothing about a production agent, or anyone's product, or real-world tool use. Nothing about another harness, another domain or another sampling setting. No model ranking. The pilot plans two models, and they decode differently, so a sentence putting them side by side is a description and not a test.
Everything in this section is the design, not a finding. No sweep has been run, no result exists, and nothing has been pre-registered.
Where it stands
The generator, the export and the model path are built. No model has been run against it, and nothing has been published.
Seven parts are built and tested: the domain engine, the tokenizer, the knobs with their control arms and the pin ledger, task and verifier generation, the adapter that turns arms into Prove targets on its scripted path (no model in it, which is what proves the pairing), the export to a verifiers taskset, and the standalone model path, which resolves the taskset by id and runs the measured harness against an OpenAI-compatible endpoint. 1,086 tests, each one frozen from a stated invariant before the code under it was written.
Which is the honest version: every part that makes and runs a task exists, and none of it has met a real model; every test drives the model path against a stub. The pinned serving path is not built, no difficulty level has been calibrated against a model, and no pilot has been run, so there is nothing to report. It gets a study when it can carry one, and Sangfroid runs it for you after that.
Ask about it if you have agent tasks you want generated this way. We'll tell you what state it is in that week.