See how far you can quantize a model before its tool calling breaks

Install

pip install "poisson-tasks[prove]==0.4.1" "poisson-prove==0.2.2" "mlx-lm==0.32.2"

Accounts and keys

None. The model is open and runs on your machine: an Apple-silicon Mac with about 20 GB of free disk. The first step downloads the model from Hugging Face.

What a run costs

Stated by the plan: $0 (local). Spent: $0.

Tools

Tasks, Under Prove, Prove, Your own agent, sweep, diff, check, report

Source

docs/your-own-agent.md · docs/commands/sweep.md · docs/commands/diff.md · docs/commands/check.md at 2ade83d

This run

poisson-prove 0.2.2 (Python 3.12.13), poisson-tasks 0.4.1, local, 2026-11-11. Stored output.

A smaller copy of a model is cheaper to run, and the question is what it costs. This example runs the model itself. Tasks makes the tool tasks, a short rollout plays one task on a local model, and Prove runs every copy of the model on the same seeds and compares them episode by episode.

What this example uses

The model. Qwen3-4B-Instruct-2507, an open model with 4 billion parameters, run locally with MLX on an Apple-silicon Mac.

What was done to it. It was quantized: every weight is stored in fewer bits, which makes the model smaller and faster and loses some precision. Nothing else about the model is changed. This example has three copies:

name here	what it is	size on disk
full	the model as released, 16 bits per weight	7.5 GB
4bit	quantized to 4 bits per weight	2.1 GB
3bit	quantized to 3 bits per weight	1.7 GB

The eval. 30 tool tasks made with Poisson Tasks. In each, the model is given 16 tools and asked to move one record in a ledger through a fixed number of stages, one tool call per stage, and to stop there. The number of stages is the task’s depth: ten tasks need 1 call, ten need 3 and ten need 8.

An episode. One live attempt by one copy of the model at one task: the model reads the request, calls tools, sees each result, and the episode ends when it replies in plain text or runs out of turns. Tasks then says how it ended. The five endings that come up here are passed (the record is in exactly the state asked for), wrong_state (the model stopped with the record somewhere else), overshoot (it reached the goal and carried the record past it), turn_limit (it was still calling tools when its turns ran out) and malformed_call (it wrote a tool call that could not be read). Each task is attempted three times by each copy: 90 episodes per copy, 270 in all.

Where the numbers come from. This run, and only this run: one pass on an Apple M4 Max on 2026-11-11, about 70 minutes for the three sweeps. It is a new measurement. It uses the same model and the same tasks as the study What removing refusals costs, but none of that study’s numbers, and it does not edit the model’s refusals.

What you will see. The 4-bit copy passes the 3-call and 8-call tasks as often as the full model, as far as 30 episodes per depth can tell. The 3-bit copy passes none of the 3-call tasks and 1 of 30 of the 8-call tasks, and more than half of its episodes end on a tool call that cannot be read.

Before you start: Install and Your machine.

1. Make the three copies of the model

mlx_lm convert downloads the model and writes it as an MLX folder; -q --q-bits quantizes it on the way. These three commands are setup and their output is not stored.

python -m mlx_lm convert --hf-path Qwen/Qwen3-4B-Instruct-2507 --mlx-path models/qwen-full
python -m mlx_lm convert --hf-path Qwen/Qwen3-4B-Instruct-2507 --mlx-path models/qwen-4bit -q --q-bits 4
python -m mlx_lm convert --hf-path Qwen/Qwen3-4B-Instruct-2507 --mlx-path models/qwen-3bit -q --q-bits 3

2. Generate the tasks

No model and no network: Tasks writes ten tasks at each of three depths.

poisson-tasks generate --arm depth:d1 --arm depth:d3 --arm depth:d8 --tasks-per-level 10 --out tasks

Outputexcerpt, 2 of 4 lines

ok  generated 30 tasks: depth at 1, 3 and 8, 10 tasks each, no model and no network
⋯ 1 line
  wrote       tasks/depth-d1.jsonl (10 tasks), tasks/depth-d3.jsonl (10 tasks), tasks/depth-d8.jsonl (10 tasks)
⋯ 1 line

All output4 lines

ok  generated 30 tasks: depth at 1, 3 and 8, 10 tasks each, no model and no network
  pin ledger  The depth contrast across depth:d1, depth:d3 and depth:d8 holds everything but the depth level fixed.
  wrote       tasks/depth-d1.jsonl (10 tasks), tasks/depth-d3.jsonl (10 tasks), tasks/depth-d8.jsonl (10 tasks)
next: poisson-tasks run

3. Save the rollout and the spec

The rollout is what Prove calls for each episode. It loads the model folder named by the environment variable MODEL, lets the seed pick the task and seed the model’s sampling, plays the task with the model’s own tool-call format, and hands the transcript to Tasks’ verifier, which returns the episode in the shape Prove stores.

rollout.py

"""A Prove rollout that plays one Poisson Tasks tool task on a local model.

The model is an MLX folder named by the environment variable MODEL. Each call plays one task
live: the model sees the task's system prompt, its request and its 16 tools, every tool call it
makes is applied to the task's own ledger and the result handed back, and the episode ends when
the model replies in plain text or uses up its turns. Tasks' verifier then says how it ended.
"""

import json
import os
import pathlib
import re

import mlx.core as mx
from mlx_lm import generate, load
from mlx_lm.sample_utils import make_sampler
from poisson_tasks import arms
from poisson_tasks.adapter import Run, to_episode
from poisson_tasks.engine import apply, init_state
from poisson_tasks.export import ExportTaskData, bundle_from_data
from poisson_tasks.tasks import FormatSpec, Turn, scenario_for

HERE = pathlib.Path(__file__).parent
ARMS = ["depth:d1", "depth:d3", "depth:d8"]
SLACK = 4  # turns allowed beyond the task's reference solution
HORIZON = max(arms.arm_from_id(a).max_turns(SLACK) for a in ARMS)
TOOL_CALL = re.compile(r"<tool_call>\s*(.*?)\s*</tool_call>", re.S)  # Qwen's tool-call format
FORMAT = FormatSpec(name="qwen-tool-call", markers=("<tool_call>",))

MODEL, TOKENIZER = load(os.environ["MODEL"])
SAMPLER = make_sampler(temp=0.7, top_p=0.8, top_k=20)  # Qwen's recommended sampling
TASKS = {
    arm: [bundle_from_data(ExportTaskData.model_validate(json.loads(line)))
          for line in (HERE / "tasks" / f"{arm.replace(':', '-')}.jsonl").read_text().splitlines()]
    for arm in ARMS
}


def rollout(cfg, seed):
    arm = arms.arm_from_id(cfg["arm"])
    task = TASKS[cfg["arm"]][seed % len(TASKS[cfg["arm"]])]  # the seed picks the task
    mx.random.seed(seed)  # and the model's sampling
    state = init_state(scenario_for(task.seed, task.split), task.seq_start)
    messages = [{"role": "system", "content": task.system_prompt}, {"role": "user", "content": task.prompt}]
    tools = [{"type": "function", "function": t} for t in task.tools]
    names = {t["name"] for t in task.tools}
    turns, observations, replied = [], [], False

    while len(turns) < arm.max_turns(SLACK):
        prompt = TOKENIZER.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=True)
        text = generate(MODEL, TOKENIZER, prompt=prompt, max_tokens=384, sampler=SAMPLER, verbose=False)
        calls = []
        for block in TOOL_CALL.findall(text):
            try:
                call = json.loads(block)
                calls.append((call.get("name"), call.get("arguments")))
            except (ValueError, AttributeError):
                pass
        if not calls:  # a plain reply ends the episode; a broken call keeps its marker for the verifier
            turns.append(Turn(content=text.strip(), calls=(), finish_reason="stop"))
            replied = True
            break
        content = TOOL_CALL.sub("", text).strip()
        turns.append(Turn(content=content, calls=tuple(calls), finish_reason="tool_calls"))
        messages.append({"role": "assistant", "content": content, "tool_calls": [
            {"type": "function", "function": {"name": n, "arguments": a}} for n, a in calls]})
        for name, args in calls:
            if isinstance(name, str) and name in names and isinstance(args, dict):
                state, observation = apply(state, (name, args))
            else:
                observation = "error: the call did not name a tool in the block with a JSON object of arguments"
            observations.append(observation)
            messages.append({"role": "tool", "content": observation})

    run = Run(turns=tuple(turns), tool_messages=tuple(observations), ended_with_reply=replied,
              turn_limit_hit=not replied, fmt=FORMAT)
    return to_episode(arm, task, run, slack=SLACK, horizon=HORIZON)

The spec names the rollout, the one axis (the task’s depth), 30 episodes per depth, and which endings count for the model and which against it.

tools.toml

# 30 tool tasks at three depths (1, 3 and 8 tool calls), 30 episodes at each depth.
target = "rollout:rollout"
target_id = "qwen3-4b-tool-tasks"
t_unit = "steps"
replicates = 30

[axes.arm]
type = "categorical"
values = ["depth:d1", "depth:d3", "depth:d8"]

[declare]
outcomes = ["passed", "malformed_call", "turn_limit", "overshoot", "wrong_state",
            "collateral_mutation", "verifier_error", "replay_mismatch", "api_error", "wall_timeout"]
successes = ["passed"]
failures = ["malformed_call", "turn_limit", "overshoot", "wrong_state", "collateral_mutation",
            "verifier_error", "replay_mismatch", "api_error", "wall_timeout"]

4. Ask what this eval can see

With 30 episodes per depth, Prove refuses the design at its default: one depth alone would have to change by 0.493 before it was flagged, and the default accepts 0.2. That is the price of a run that fits in an hour on one machine, so the next step accepts it in the open with --max-mde 0.55. A run that needed to see a change of 0.2 would need the 205 episodes per depth Prove names.

MODEL=models/qwen-full poisson-prove sweep tools.toml --dry-run

Outputexcerpt, 4 of 9 lines · exit 1

plan  90 rollouts of qwen3-4b-tool-tasks: 3 cells x 30 replicates on local, $0 (local)
⋯ 5 lines
error: underpowered design: one condition alone would be flagged for a change of 0.493 at its worst base rate, 0.400, and you accept 0.200
  next: poisson-prove sweep tools.toml --dry-run --replicates 205
  next: poisson-prove sweep tools.toml --dry-run --max-mde 0.524

All output9 lines

plan  90 rollouts of qwen3-4b-tool-tasks: 3 cells x 30 replicates on local, $0 (local)
  target  rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: api_error, collateral_mutation, malformed_call, overshoot, replay_mismatch, turn_limit, verifier_error, wall_timeout, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=30 → 90 rollouts
  Backend: local · target: qwen3-4b-tool-tasks · t_unit: steps
  Cost: $0 (local)
error: underpowered design: one condition alone would be flagged for a change of 0.493 at its worst base rate, 0.400, and you accept 0.200
  next: poisson-prove sweep tools.toml --dry-run --replicates 205
  next: poisson-prove sweep tools.toml --dry-run --max-mde 0.524

5. Sweep the full-precision model

MODEL=models/qwen-full poisson-prove sweep tools.toml --max-mde 0.55 --name full --yes

Outputexcerpt, 2 of 22 lines

⋯ 6 lines
  smallest rise one condition alone would be flagged for  0.523 at its worst base rate, 0.225 (Fisher, q 0.05 over 30 tests, 4 times in 5; you accept up to 0.550)
⋯ 10 lines
done  90 rollouts in run "full" (complete) in 26 min 38 s
⋯ 4 lines

All output22 lines

plan  90 rollouts of qwen3-4b-tool-tasks: 3 cells x 30 replicates on local, $0 (local)
  target  rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: api_error, collateral_mutation, malformed_call, overshoot, replay_mismatch, turn_limit, verifier_error, wall_timeout, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=30 → 90 rollouts
  Backend: local · target: qwen3-4b-tool-tasks · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.523 at its worst base rate, 0.225 (Fisher, q 0.05 over 30 tests, 4 times in 5; you accept up to 0.550)
9/90
18/90
27/90
36/90
45/90
54/90
63/90
72/90
81/90
90/90
done  90 rollouts in run "full" (complete) in 26 min 38 s
  episodes with an outcome  90
  episodes with no outcome  0
  episodes timed out  0
  wall time  26 min 38 s

6. Sweep the two quantized copies on the same seeds

--pair-with full gives each copy the full model’s seeds, so every episode is the same task with the same sampling seed, and only the weights differ.

MODEL=models/qwen-4bit poisson-prove sweep tools.toml --pair-with full --max-mde 0.55 --name 4bit --yes

Outputexcerpt, 1 of 23 lines

⋯ 17 lines
done  90 rollouts in run "4bit" (complete) in 21 min 40 s, paired with "full"
⋯ 5 lines

All output23 lines

plan  90 rollouts of qwen3-4b-tool-tasks: 3 cells x 30 replicates on local, $0 (local)
  target  rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: api_error, collateral_mutation, malformed_call, overshoot, replay_mismatch, turn_limit, verifier_error, wall_timeout, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=30 → 90 rollouts
  Backend: local · target: qwen3-4b-tool-tasks · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.547 at its worst base rate, 0.225 (the paired test, q 0.05 over 30 tests, 4 times in 5; you accept up to 0.550)
9/90
18/90
27/90
36/90
45/90
54/90
63/90
72/90
81/90
90/90
done  90 rollouts in run "4bit" (complete) in 21 min 40 s, paired with "full"
  episodes with an outcome  90
  episodes with no outcome  0
  episodes timed out  0
  wall time  21 min 40 s
  next: poisson-prove diff

MODEL=models/qwen-3bit poisson-prove sweep tools.toml --pair-with full --max-mde 0.55 --name 3bit --yes

Outputexcerpt, 1 of 23 lines

⋯ 17 lines
done  90 rollouts in run "3bit" (complete) in 20 min 26 s, paired with "full"
⋯ 5 lines

All output23 lines

plan  90 rollouts of qwen3-4b-tool-tasks: 3 cells x 30 replicates on local, $0 (local)
  target  rollout:rollout (imports, takes cfg and seed)
  outcomes  failures: api_error, collateral_mutation, malformed_call, overshoot, replay_mismatch, turn_limit, verifier_error, wall_timeout, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=30 → 90 rollouts
  Backend: local · target: qwen3-4b-tool-tasks · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.547 at its worst base rate, 0.225 (the paired test, q 0.05 over 30 tests, 4 times in 5; you accept up to 0.550)
9/90
18/90
27/90
36/90
45/90
54/90
63/90
72/90
81/90
90/90
done  90 rollouts in run "3bit" (complete) in 20 min 26 s, paired with "full"
  episodes with an outcome  90
  episodes with no outcome  0
  episodes timed out  0
  wall time  20 min 26 s
  next: poisson-prove diff full 3bit

poisson-prove runs

Outputexcerpt, 5 of 6 lines

ok  3 runs in prove-runs
⋯ 1 line
#  name  id                     status    backend  target               rows  rows without an outcome  created
1  3bit  20261111T214511Z-a291  complete  local    qwen3-4b-tool-tasks  90    0                        2026-11-11T21:45:11Z
2  4bit  20261111T212314Z-ba22  complete  local    qwen3-4b-tool-tasks  90    0                        2026-11-11T21:23:14Z
3  full  20261111T205628Z-b8ad  complete  local    qwen3-4b-tool-tasks  90    0                        2026-11-11T20:56:28Z

All output6 lines

ok  3 runs in prove-runs

#  name  id                     status    backend  target               rows  rows without an outcome  created
1  3bit  20261111T214511Z-a291  complete  local    qwen3-4b-tool-tasks  90    0                        2026-11-11T21:45:11Z
2  4bit  20261111T212314Z-ba22  complete  local    qwen3-4b-tool-tasks  90    0                        2026-11-11T21:23:14Z
3  full  20261111T205628Z-b8ad  complete  local    qwen3-4b-tool-tasks  90    0                        2026-11-11T20:56:28Z

7. Compare full precision with 4-bit

Each row is one depth and one way of ending; before and after are the share of that depth’s 30 episodes that ended that way. On the 3-call tasks the pass rate is 0.933 before and 1.000 after, and on the 8-call tasks 1.000 and 0.967. Both rows are unchanged, and the fall column says what that is worth: a fall of 0.476 or 0.392 would have been flagged, a smaller one might not have been.

The 1-call tasks are a different story, and it is the model’s, not the quantization’s. The full model never passes them: in most episodes it keeps calling tools until its turns run out. The 4-bit copy more often stops after carrying the record past the goal, so turn_limit falls and overshoot rises. One kind of failure became another, and Prove’s word for the comparison is “trade”.

poisson-prove diff full 4bit

Outputexcerpt, 7 of 47 lines

trade  overshoot: 0 safer, 1 worse · turn_limit: 1 safer, 0 worse · api_error: 0 safer, 0 worse · collateral_mutation: 0 safer, 0 worse · malformed_call: 0 safer, 0 worse · passed: 0 safer, 0 worse · replay_mismatch: 0 safer, 0 worse · verifier_error: 0 safer, 0 worse · wall_timeout: 0 safer, 0 worse · wrong_state: 0 safer, 0 worse (paired test on shared seeds)
⋯ 11 lines
condition     event                before  after  change  verdict    rise             fall
arm depth:d1  overshoot            0.133   0.567  +0.433  worse      0.535            none detectable
arm depth:d1  turn_limit           0.867   0.167  -0.700  safer      none detectable  0.535
⋯ 3 lines
arm depth:d1  passed               0.000   0.267  +0.267  unchanged  0.426            no room
⋯ 9 lines
arm depth:d3  passed               0.933   1.000  +0.067  unchanged  none detectable  0.476
⋯ 9 lines
arm depth:d8  passed               1.000   0.967  -0.033  unchanged  no room          0.392
⋯ 8 lines

All output47 lines

trade  overshoot: 0 safer, 1 worse · turn_limit: 1 safer, 0 worse · api_error: 0 safer, 0 worse · collateral_mutation: 0 safer, 0 worse · malformed_call: 0 safer, 0 worse · passed: 0 safer, 0 worse · replay_mismatch: 0 safer, 0 worse · verifier_error: 0 safer, 0 worse · wall_timeout: 0 safer, 0 worse · wrong_state: 0 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  full
  after  4bit
  trade  arm depth:d1: turn_limit fell, overshoot rose
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 30 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 30 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 30 tests); 1 row that did not count has no limit on its fall (none detectable: 1), so a fall of any size cannot be excluded there; 1 row that did not count has no limit on its rise (none detectable: 1), so a rise of any size cannot be excluded there
  intervals  on each condition's change: Tango at 99.667%, false coverage (Benjamini-Yekutieli) over the 2 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  1 outcome would change the word (the search is not exhaustive: it may be fewer): 1 run of B in arm depth:d1 that ended in overshoot made to end otherwise

inferred: no event was given, so the event set is api_error, collateral_mutation, malformed_call, overshoot, replay_mismatch, turn_limit, verifier_error, wall_timeout, wrong_state, passed, the failures and successes the runs declared

condition     event                before  after  change  verdict    rise             fall
arm depth:d1  overshoot            0.133   0.567  +0.433  worse      0.535            none detectable
arm depth:d1  turn_limit           0.867   0.167  -0.700  safer      none detectable  0.535
arm depth:d1  api_error            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  collateral_mutation  0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  malformed_call       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  passed               0.000   0.267  +0.267  unchanged  0.426            no room
arm depth:d1  replay_mismatch      0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  verifier_error       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  wall_timeout         0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  wrong_state          0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  overshoot            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  turn_limit           0.067   0.000  -0.067  unchanged  0.476            none detectable
arm depth:d3  api_error            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  collateral_mutation  0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  malformed_call       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  passed               0.933   1.000  +0.067  unchanged  none detectable  0.476
arm depth:d3  replay_mismatch      0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  verifier_error       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  wall_timeout         0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  wrong_state          0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  overshoot            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  turn_limit           0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  api_error            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  collateral_mutation  0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  malformed_call       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  passed               1.000   0.967  -0.033  unchanged  no room          0.392
arm depth:d8  replay_mismatch      0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  verifier_error       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  wall_timeout         0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  wrong_state          0.000   0.033  +0.033  unchanged  0.392            no room

  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound

  next: poisson-prove report full 4bit --x arm

8. Compare full precision with 3-bit

At 3 bits the tool calling is gone. The 3-call tasks go from 0.933 passed to 0.000 and the 8-call tasks from 1.000 to 0.033. At every depth, half or more of the episodes end in malformed_call, and most of the rest stop in the wrong state.

One row goes the other way. On the 1-call tasks the 3-bit copy passes a third of the time where the full model never does, and Prove marks it “safer”. The table says why not to read much into it: the same copy writes an unreadable call in 0.633 of those episodes.

poisson-prove diff full 3bit

Outputexcerpt, 11 of 47 lines

trade  malformed_call: 0 safer, 3 worse · passed: 1 safer, 2 worse · wrong_state: 0 safer, 2 worse · turn_limit: 1 safer, 0 worse · api_error: 0 safer, 0 worse · collateral_mutation: 0 safer, 0 worse · overshoot: 0 safer, 0 worse · replay_mismatch: 0 safer, 0 worse · verifier_error: 0 safer, 0 worse · wall_timeout: 0 safer, 0 worse (paired test on shared seeds)
⋯ 11 lines
condition     event                before  after  change  verdict    rise             fall
arm depth:d1  malformed_call       0.000   0.633  +0.633  worse      0.426            no room
arm depth:d1  passed               0.000   0.333  +0.333  safer      0.426            no room
⋯ 1 line
arm depth:d1  turn_limit           0.867   0.000  -0.867  safer      none detectable  0.535
⋯ 6 lines
arm depth:d3  malformed_call       0.000   0.500  +0.500  worse      0.426            no room
arm depth:d3  passed               0.933   0.000  -0.933  worse      none detectable  0.506
arm depth:d3  wrong_state          0.000   0.467  +0.467  worse      0.426            no room
⋯ 7 lines
arm depth:d8  malformed_call       0.000   0.567  +0.567  worse      0.426            no room
arm depth:d8  passed               1.000   0.033  -0.967  worse      no room          0.426
arm depth:d8  wrong_state          0.000   0.400  +0.400  worse      0.426            no room
⋯ 11 lines

All output47 lines

trade  malformed_call: 0 safer, 3 worse · passed: 1 safer, 2 worse · wrong_state: 0 safer, 2 worse · turn_limit: 1 safer, 0 worse · api_error: 0 safer, 0 worse · collateral_mutation: 0 safer, 0 worse · overshoot: 0 safer, 0 worse · replay_mismatch: 0 safer, 0 worse · verifier_error: 0 safer, 0 worse · wall_timeout: 0 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  full
  after  3bit
  trade  arm depth:d1: turn_limit fell, passed rose, malformed_call rose
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 30 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 30 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 30 tests); 2 rows that did not count have no limit on their fall (none detectable: 2), so a fall of any size cannot be excluded there
  intervals  on each condition's change: Tango at 98.5%, false coverage (Benjamini-Yekutieli) over the 9 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  18 outcomes would change the word (the search is not exhaustive: it may be fewer): 17 runs of A in arm depth:d1 that ended in turn_limit made to end otherwise; 1 run of B in arm depth:d1 that ended in passed made to end otherwise

inferred: no event was given, so the event set is api_error, collateral_mutation, malformed_call, overshoot, replay_mismatch, turn_limit, verifier_error, wall_timeout, wrong_state, passed, the failures and successes the runs declared

condition     event                before  after  change  verdict    rise             fall
arm depth:d1  malformed_call       0.000   0.633  +0.633  worse      0.426            no room
arm depth:d1  passed               0.000   0.333  +0.333  safer      0.426            no room
arm depth:d1  wrong_state          0.000   0.033  +0.033  unchanged  0.426            no room
arm depth:d1  turn_limit           0.867   0.000  -0.867  safer      none detectable  0.535
arm depth:d1  api_error            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  collateral_mutation  0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  overshoot            0.133   0.000  -0.133  unchanged  0.535            none detectable
arm depth:d1  replay_mismatch      0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  verifier_error       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d1  wall_timeout         0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  malformed_call       0.000   0.500  +0.500  worse      0.426            no room
arm depth:d3  passed               0.933   0.000  -0.933  worse      none detectable  0.506
arm depth:d3  wrong_state          0.000   0.467  +0.467  worse      0.426            no room
arm depth:d3  turn_limit           0.067   0.033  -0.033  unchanged  0.506            none detectable
arm depth:d3  api_error            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  collateral_mutation  0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  overshoot            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  replay_mismatch      0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  verifier_error       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d3  wall_timeout         0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  malformed_call       0.000   0.567  +0.567  worse      0.426            no room
arm depth:d8  passed               1.000   0.033  -0.967  worse      no room          0.426
arm depth:d8  wrong_state          0.000   0.400  +0.400  worse      0.426            no room
arm depth:d8  turn_limit           0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  api_error            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  collateral_mutation  0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  overshoot            0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  replay_mismatch      0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  verifier_error       0.000   0.000  0.000   unchanged  0.426            no room
arm depth:d8  wall_timeout         0.000   0.000  0.000   unchanged  0.426            no room

  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound

  next: poisson-prove report full 3bit --x arm

9. Gate on it

Naming passed as the event, check fails, names the two depths, and exits 4. A pipeline that publishes quantized copies of a model stops here.

poisson-prove check full 3bit --event passed

Outputexcerpt, 1 of 12 lines · exit 4

failed  passed: 2 worse, 1 safer in 3 conditions; fail on worse; failed in arm depth:d8, arm depth:d3 (paired test on shared seeds)
⋯ 11 lines

All output12 lines

failed  passed: 2 worse, 1 safer in 3 conditions; fail on worse; failed in arm depth:d8, arm depth:d3 (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  comparison  trade
  before  full
  after  3bit
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 3 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 3 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 3 tests)
  intervals  on each condition's change: Tango at 95%, false coverage (Benjamini-Yekutieli) over the 3 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  3 outcomes would change the word: 1 run of B in arm depth:d1 that ended in passed made to end otherwise; 2 runs of A in arm depth:d1 made to end in passed

passed: 1 safer, 2 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 3 tests, paired)

10. Write the report

poisson-prove report full 3bit --x arm

Outputexcerpt, 4 of 12 lines

ok  wrote prove-report-full-vs-3bit (3 files)
  wrote  prove-report-full-vs-3bit/report.md
  wrote  prove-report-full-vs-3bit/heatmap.svg
  wrote  prove-report-full-vs-3bit/receipt.html
⋯ 8 lines

All output12 lines

ok  wrote prove-report-full-vs-3bit (3 files)
  wrote  prove-report-full-vs-3bit/report.md
  wrote  prove-report-full-vs-3bit/heatmap.svg
  wrote  prove-report-full-vs-3bit/receipt.html
  saved  comparison f8a5b2706b04

inferred: no event was given, so the event set is api_error, collateral_mutation, malformed_call, overshoot, replay_mismatch, turn_limit, verifier_error, wall_timeout, wrong_state, passed, the failures and successes the runs declared
malformed_call: 0 safer, 3 worse · passed: 1 safer, 2 worse · wrong_state: 0 safer, 2 worse · turn_limit: 1 safer, 0 worse · api_error: 0 safer, 0 worse · collateral_mutation: 0 safer, 0 worse · overshoot: 0 safer, 0 worse · replay_mismatch: 0 safer, 0 worse · verifier_error: 0 safer, 0 worse · wall_timeout: 0 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 30 tests, paired)
Trade: passed rose in 1 cell where malformed_call rose.

  next: poisson-prove ui

What this does not show

One model, one quantizer at its default settings, 30 tasks of one kind, three attempts each, and one pass. It says 3-bit breaks this model’s tool calling on these tasks. It does not say 4-bit is free: this run could not have seen a fall smaller than about 0.4 at any depth, and it measures nothing but tool calling.

The same question on results you already have

These two examples ask what a change to a model cost without running the model, from stored results of the study: Check a model edit on evals you already ran and Find what a model edit changed, and what it left alone.

With your own model

Change MODEL to any MLX folder: a fine-tune, another quantization, a merged adapter. For a model with another tool-call format, change the pattern in rollout.py that finds a call in the model’s text. For a model behind an API, replace the generate call with your client; the rest of the rollout stays as it is.

Stored output

Every output above is what the command printed, stored in the site repository under src/data/examples/see-where-quantization-breaks-tool-calling/ with the run’s diff.json and the three files report wrote, as Prove 0.2.2 wrote them with Tasks 0.4.1 and mlx-lm 0.32.2. The page cannot show an output that is not there. A second run will not print the same numbers to the last digit: the model samples, and the seeds fix the sampling only on the same machine and library versions.