Documentation
Three tools that measure whether a change to an agent is real. Each runs on your own machine and sends nothing home.
| Tool | What it tells you | Start here |
|---|---|---|
| Prove | Whether a change moved the failure rate, in which conditions, and what the run was too small to detect | Prove |
| Trace | At which step runs die, and whether a change made them fail less, with the unchanged version as the control | Trace |
| Tasks | What each level of a difficulty knob is worth to one named model | Tasks |
Prove runs your system and stores every run. Trace reads what Prove stored, or what another evaluation tool already wrote. Tasks generates tasks for either to run on.
None of the three is released yet. Until the first release, install from a clone of the repository; each tool’s page has the lines.
- Knobs, arms and the pin ledger
- One command-line look
- Reading a card
- Running a generated task under verifiers
- Tasks on your own tools
- The commands
- The settings file: `tasks.toml`
- Using it from Python
- What a task is
- What each difficulty level is worth
- Changelog
Integrations
Section titled “Integrations”Bring in results from DeepEval or Ragas and compare two versions with Trace.