Skip to content
Poisson Labs Docs
GitHub poissonlabs.ai

Examples

Check a model edit on evals you already ran

You changed a model: a fine-tune, a quantization, a weight edit. You ran the same eval on the model before and after, and the results are on disk. Prove can read them as they are. A rollout that looks up a stored episode, where it would normally run one, turns each set of results into a Prove run, and diff compares the runs episode by episode.

What this example uses

Section titled “What this example uses”

The model. Llama-3.1-8B-Instruct, an open model, run locally.

What was done to it. Its refusals were removed by editing its weights, a method known as abliteration. One direction in the model’s internal activity separates the requests it refuses from the requests it answers, and each layer’s output weights are changed so the model can no longer write along that direction. A strength sets how much is removed. This example has three versions of the model:

name herewhat it is
stockthe model as released, nothing removed
fullthe refusal direction fully removed (strength 1.0, the published method)
overremoved 25% past that (strength 1.25)

The eval. 30 tool tasks made with Poisson Tasks. In each, the model is given 16 tools and asked to move one record in a ledger through a fixed number of stages, one tool call per stage, and to stop there. The number of stages is the task’s depth: ten tasks need 1 call, ten need 3 and ten need 8.

An episode. One attempt by one version of the model at one task. It ends one of four ways: passed (the ledger is in exactly the state asked for), wrong_state (it stopped somewhere else), turn_limit (it was still calling tools when its turns ran out) or malformed_call (a call that did not parse). Every task was attempted five times by each version, so there are 150 episodes per version and 450 in all.

Where they come from. The study What removing refusals costs ran them once, on one Mac, on 2026-11-10 and 11. This page does not run the model again. It replays those 450 endings through Prove, so the commands below take seconds and need no GPU.

What you will see. Tool use holds up at full removal as far as this eval can tell, and breaks past it: on the deepest tasks the pass rate goes 0.74, 0.50, 0.28.

Before you start: Install and Your machine.

1. Save the episodes and the rollout

Section titled “1. Save the episodes and the rollout”

Download episodes.jsonl into an empty folder. Each of its 450 lines is one episode: which version of the model played it (edit), the task’s depth (arm), the seed, the task and attempt, and how it ended (outcome). It holds no prompt or response text. The first line:

{"edit": "stock", "strength": 0.0, "arm": "depth:d1", "seed": 2594881162, "task": "depth:d1#0", "attempt": 0, "outcome": "turn_limit", "t": 6, "horizon": 13, "valid_calls": true}

The rollout is the one file that knows your results. Prove calls it with a condition and a seed, and it returns the stored episode that seed names. The environment variable EDIT picks which version’s episodes it serves: stock, full or over.

replay.py
"""A Prove rollout that replays episodes measured once, elsewhere.


Each line of episodes.jsonl is one tool-task episode of Llama-3.1-8B-Instruct: which edit of the
model played it (EDIT), the task's depth arm, the seed Prove gave it, and how it ended. The
rollout answers Prove's (cfg, seed) with the episode that seed names, so a sweep is a Prove run
of episodes that already exist and no model is called.
"""


import json
import os
import pathlib


from poisson_prove import Episode


EDIT = os.environ["EDIT"]  # stock, full (strength 1.0) or over (strength 1.25)
ROWS = {}
for line in (pathlib.Path(__file__).parent / "episodes.jsonl").read_text().splitlines():
    row = json.loads(line)
    if row["edit"] == EDIT:
        ROWS[(row["arm"], row["seed"])] = row




def rollout(cfg, seed):
    row = ROWS[(cfg["arm"], seed)]
    meta = {"task": row["task"], "attempt": row["attempt"], "valid_calls": row["valid_calls"]}
    return Episode(outcome=row["outcome"], t=row["t"], horizon=row["horizon"], meta=meta)

The spec names the rollout, the one axis (the task’s depth, ten tasks at each), 50 replicates per depth, and which endings count for the model and which against it.

tasks.toml
# 30 tool tasks at three depths, 10 tasks x 5 attempts in each arm.
target = "replay:rollout"
target_id = "llama-3.1-8b-tool-tasks"
t_unit = "steps"
replicates = 50


[axes.arm]
type = "categorical"
values = ["depth:d1", "depth:d3", "depth:d8"]


[declare]
outcomes = ["passed", "malformed_call", "turn_limit", "wrong_state"]
successes = ["passed"]
failures = ["malformed_call", "turn_limit", "wrong_state"]

2. Ask what this eval can see

Section titled “2. Ask what this eval can see”

--dry-run runs nothing and prints the plan. With 50 episodes per depth, Prove refuses the design at its default: one depth alone would have to change by 0.374 before it was flagged, and the default accepts 0.2. An eval you already ran is the size it is, so the choice is to accept that limit in the open, and Prove prints the flag that does.

Terminal window
EDIT=stock poisson-prove sweep tasks.toml --dry-run
Outputexcerpt, 4 of 9 lines · exit 1
plan  150 rollouts of llama-3.1-8b-tool-tasks: 3 cells x 50 replicates on local, $0 (local)
⋯ 5 lines
error: underpowered design: one condition alone would be flagged for a change of 0.374 at its worst base rate, 0.400, and you accept 0.200
  next: poisson-prove sweep tasks.toml --dry-run --replicates 177
  next: poisson-prove sweep tasks.toml --dry-run --max-mde 0.386
All output9 lines
plan  150 rollouts of llama-3.1-8b-tool-tasks: 3 cells x 50 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: malformed_call, turn_limit, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=50 → 150 rollouts
  Backend: local · target: llama-3.1-8b-tool-tasks · t_unit: steps
  Cost: $0 (local)
error: underpowered design: one condition alone would be flagged for a change of 0.374 at its worst base rate, 0.400, and you accept 0.200
  next: poisson-prove sweep tasks.toml --dry-run --replicates 177
  next: poisson-prove sweep tasks.toml --dry-run --max-mde 0.386

3. Sweep the stock model

Section titled “3. Sweep the stock model”

The episodes were measured on seeds Prove derives from a salt, so the sweep is given the same salt and asks the rollout for exactly the episodes that exist. --max-mde 0.4 accepts the limit from step 2.

Terminal window
EDIT=stock poisson-prove sweep tasks.toml --salt ab11742026110901 --max-mde 0.4 --name stock --yes
Outputexcerpt, 2 of 13 lines
⋯ 6 lines
  smallest rise one condition alone would be flagged for  0.386 at its worst base rate, 0.300 (Fisher, q 0.05 over 12 tests, 4 times in 5; you accept up to 0.400)
⋯ 2 lines
  episodes with an outcome  150
⋯ 3 lines
All output13 lines
plan  150 rollouts of llama-3.1-8b-tool-tasks: 3 cells x 50 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: malformed_call, turn_limit, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=50 → 150 rollouts
  Backend: local · target: llama-3.1-8b-tool-tasks · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.386 at its worst base rate, 0.300 (Fisher, q 0.05 over 12 tests, 4 times in 5; you accept up to 0.400)
150/150
done  150 rollouts in run "stock" (complete) in 0.0 s
  episodes with an outcome  150
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.0 s

4. Sweep the two edits on the same seeds

Section titled “4. Sweep the two edits on the same seeds”

--pair-with stock gives each edit’s run the stock run’s seeds, so episode k of a task on the edited model is compared with episode k of the same task on stock.

Terminal window
EDIT=full poisson-prove sweep tasks.toml --pair-with stock --max-mde 0.4 --name full-removal --yes
Outputexcerpt, 1 of 14 lines
⋯ 8 lines
done  150 rollouts in run "full-removal" (complete) in 0.0 s, paired with "stock"
⋯ 5 lines
All output14 lines
plan  150 rollouts of llama-3.1-8b-tool-tasks: 3 cells x 50 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: malformed_call, turn_limit, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=50 → 150 rollouts
  Backend: local · target: llama-3.1-8b-tool-tasks · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.388 at its worst base rate, 0.300 (the paired test, q 0.05 over 12 tests, 4 times in 5; you accept up to 0.400)
150/150
done  150 rollouts in run "full-removal" (complete) in 0.0 s, paired with "stock"
  episodes with an outcome  150
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.0 s
  next: poisson-prove diff
Terminal window
EDIT=over poisson-prove sweep tasks.toml --pair-with stock --max-mde 0.4 --name over-removal --yes
Outputexcerpt, 1 of 14 lines
⋯ 8 lines
done  150 rollouts in run "over-removal" (complete) in 0.0 s, paired with "stock"
⋯ 5 lines
All output14 lines
plan  150 rollouts of llama-3.1-8b-tool-tasks: 3 cells x 50 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: malformed_call, turn_limit, wrong_state; successes: passed
  Sweep: 1 axes · 3 cells · N=50 → 150 rollouts
  Backend: local · target: llama-3.1-8b-tool-tasks · t_unit: steps
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.388 at its worst base rate, 0.300 (the paired test, q 0.05 over 12 tests, 4 times in 5; you accept up to 0.400)
150/150
done  150 rollouts in run "over-removal" (complete) in 0.0 s, paired with "stock"
  episodes with an outcome  150
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.0 s
  next: poisson-prove diff stock over-removal

The store now holds three runs.

Terminal window
poisson-prove runs
Outputexcerpt, 5 of 6 lines
ok  3 runs in prove-runs
⋯ 1 line
#  name          id                     status    backend  target                   rows  rows without an outcome  created
1  over-removal  20261111T202635Z-e4f0  complete  local    llama-3.1-8b-tool-tasks  150   0                        2026-11-11T20:26:35Z
2  full-removal  20261111T202633Z-ece0  complete  local    llama-3.1-8b-tool-tasks  150   0                        2026-11-11T20:26:33Z
3  stock         20261111T202631Z-5baa  complete  local    llama-3.1-8b-tool-tasks  150   0                        2026-11-11T20:26:31Z
All output6 lines
ok  3 runs in prove-runs


#  name          id                     status    backend  target                   rows  rows without an outcome  created
1  over-removal  20261111T202635Z-e4f0  complete  local    llama-3.1-8b-tool-tasks  150   0                        2026-11-11T20:26:35Z
2  full-removal  20261111T202633Z-ece0  complete  local    llama-3.1-8b-tool-tasks  150   0                        2026-11-11T20:26:33Z
3  stock         20261111T202631Z-5baa  complete  local    llama-3.1-8b-tool-tasks  150   0                        2026-11-11T20:26:31Z

5. Compare stock with full removal

Section titled “5. Compare stock with full removal”

Each row is one depth and one way of ending; before and after are the share of that depth’s 50 episodes that ended that way. At full removal the deepest tasks pass less often, 0.74 to 0.50. Prove does not call it a change. The verdict is “no significant change”, and beside every row is the smallest rise and fall that row could have shown. A fall of 0.24 is under the 0.387 this eval can flag at that depth, after correcting for twelve tests. The verdict says the eval cannot tell, not that the edit is safe.

Terminal window
poisson-prove diff stock full-removal
Outputexcerpt, 6 of 29 lines
no significant change  every condition that did not count could have shown a change of 0.400: the largest limit it read is 0.387 (the paired test (McNemar), Holm q 0.05 over 12 tests, 4 times in 5); malformed_call: 0 safer, 0 worse · passed: 0 safer, 0 worse · turn_limit: 0 safer, 0 worse · wrong_state: 0 safer, 0 worse (paired test on shared seeds)
⋯ 11 lines
condition     event           before  after  change  verdict    rise             fall
⋯ 8 lines
arm depth:d8  malformed_call  0.040   0.000  -0.040  unchanged  0.299            none detectable
arm depth:d8  passed          0.740   0.500  -0.240  unchanged  0.248            0.387
arm depth:d8  turn_limit      0.080   0.200  +0.120  unchanged  0.335            none detectable
arm depth:d8  wrong_state     0.140   0.300  +0.160  unchanged  0.362            none detectable
⋯ 4 lines
All output29 lines
no significant change  every condition that did not count could have shown a change of 0.400: the largest limit it read is 0.387 (the paired test (McNemar), Holm q 0.05 over 12 tests, 4 times in 5); malformed_call: 0 safer, 0 worse · passed: 0 safer, 0 worse · turn_limit: 0 safer, 0 worse · wrong_state: 0 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  threshold  0.400 (recorded at sweep time by the runs)
  before  stock
  after  full-removal
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 12 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 12 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 12 tests); 5 rows that did not count have no limit on their fall (none detectable: 5), so a fall of any size cannot be excluded there; 2 rows that did not count have no limit on their rise (none detectable: 2), so a rise of any size cannot be excluded there
  intervals  on each condition's change: Tango at 99.583%, simultaneous (Bonferroni) over the 12 tests
  fragility  2 outcomes would change the word (the search is not exhaustive: it may be fewer): 2 runs of B in arm depth:d8 that ended in passed made to end otherwise


inferred: no event was given, so the event set is malformed_call, turn_limit, wrong_state, passed, the failures and successes the runs declared


condition     event           before  after  change  verdict    rise             fall
arm depth:d1  malformed_call  0.020   0.000  -0.020  unchanged  0.241            none detectable
arm depth:d1  passed          0.000   0.000  0.000   unchanged  0.220            no room
arm depth:d1  turn_limit      0.980   1.000  +0.020  unchanged  none detectable  0.241
arm depth:d1  wrong_state     0.000   0.000  0.000   unchanged  0.220            no room
arm depth:d3  malformed_call  0.000   0.020  +0.020  unchanged  0.220            no room
arm depth:d3  passed          0.000   0.000  0.000   unchanged  0.220            no room
arm depth:d3  turn_limit      0.880   0.840  -0.040  unchanged  none detectable  0.355
arm depth:d3  wrong_state     0.120   0.140  +0.020  unchanged  0.355            none detectable
arm depth:d8  malformed_call  0.040   0.000  -0.040  unchanged  0.299            none detectable
arm depth:d8  passed          0.740   0.500  -0.240  unchanged  0.248            0.387
arm depth:d8  turn_limit      0.080   0.200  +0.120  unchanged  0.335            none detectable
arm depth:d8  wrong_state     0.140   0.300  +0.160  unchanged  0.362            none detectable


  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound


  next: poisson-prove report stock full-removal --x arm

6. Compare stock with removal past full

Section titled “6. Compare stock with removal past full”

Past full removal the same row is flagged. The deepest tasks pass 0.28 of the time against 0.74, and the episodes that stopped passing ran out of turns. The verdict is “worse”.

Terminal window
poisson-prove diff stock over-removal
Outputexcerpt, 6 of 28 lines
worse  passed: 0 safer, 1 worse · turn_limit: 0 safer, 1 worse · malformed_call: 0 safer, 0 worse · wrong_state: 0 safer, 0 worse (paired test on shared seeds)
⋯ 10 lines
condition     event           before  after  change  verdict    rise             fall
⋯ 8 lines
arm depth:d8  passed          0.740   0.280  -0.460  worse      0.248            0.387
arm depth:d8  turn_limit      0.080   0.340  +0.260  worse      0.335            none detectable
arm depth:d8  malformed_call  0.040   0.040  0.000   unchanged  0.299            none detectable
arm depth:d8  wrong_state     0.140   0.340  +0.200  unchanged  0.362            none detectable
⋯ 4 lines
All output28 lines
worse  passed: 0 safer, 1 worse · turn_limit: 0 safer, 1 worse · malformed_call: 0 safer, 0 worse · wrong_state: 0 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  stock
  after  over-removal
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 12 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 12 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 12 tests); 4 rows that did not count have no limit on their fall (none detectable: 4), so a fall of any size cannot be excluded there; 2 rows that did not count have no limit on their rise (none detectable: 2), so a rise of any size cannot be excluded there
  intervals  on each condition's change: Tango at 99.167%, false coverage (Benjamini-Yekutieli) over the 2 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  6 outcomes would change the word (the search is not exhaustive: it may be fewer): 6 runs of A in arm depth:d8 that ended in passed made to end otherwise


inferred: no event was given, so the event set is malformed_call, turn_limit, wrong_state, passed, the failures and successes the runs declared


condition     event           before  after  change  verdict    rise             fall
arm depth:d1  passed          0.000   0.000  0.000   unchanged  0.220            no room
arm depth:d1  turn_limit      0.980   1.000  +0.020  unchanged  none detectable  0.241
arm depth:d1  malformed_call  0.020   0.000  -0.020  unchanged  0.241            none detectable
arm depth:d1  wrong_state     0.000   0.000  0.000   unchanged  0.220            no room
arm depth:d3  passed          0.000   0.000  0.000   unchanged  0.220            no room
arm depth:d3  turn_limit      0.880   0.880  0.000   unchanged  none detectable  0.355
arm depth:d3  malformed_call  0.000   0.020  +0.020  unchanged  0.220            no room
arm depth:d3  wrong_state     0.120   0.100  -0.020  unchanged  0.355            none detectable
arm depth:d8  passed          0.740   0.280  -0.460  worse      0.248            0.387
arm depth:d8  turn_limit      0.080   0.340  +0.260  worse      0.335            none detectable
arm depth:d8  malformed_call  0.040   0.040  0.000   unchanged  0.299            none detectable
arm depth:d8  wrong_state     0.140   0.340  +0.200  unchanged  0.362            none detectable


  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound


  next: poisson-prove report stock over-removal --x arm

7. Gate on it

Section titled “7. Gate on it”

check is the same comparison as a gate. Naming passed as the event, it fails, names the depth, and exits 4, so a pipeline that ships model edits stops here.

Terminal window
poisson-prove check stock over-removal --event passed
Outputexcerpt, 1 of 12 lines · exit 4
failed  passed: 1 worse, 0 safer in 3 conditions; fail on worse; failed in arm depth:d8 (paired test on shared seeds)
⋯ 11 lines
All output12 lines
failed  passed: 1 worse, 0 safer in 3 conditions; fail on worse; failed in arm depth:d8 (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  comparison  worse
  before  stock
  after  over-removal
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 3 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 3 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 3 tests)
  intervals  on each condition's change: Tango at 98.333%, false coverage (Benjamini-Yekutieli) over the 1 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  7 outcomes would change the word: 7 runs of B in arm depth:d1 made to end in passed


passed: 0 safer, 1 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 3 tests, paired)

8. Write the report

Section titled “8. Write the report”
Terminal window
poisson-prove report stock over-removal --x arm
Outputexcerpt, 4 of 11 lines
ok  wrote prove-report-stock-vs-over-removal (3 files)
  wrote  prove-report-stock-vs-over-removal/report.md
  wrote  prove-report-stock-vs-over-removal/heatmap.svg
  wrote  prove-report-stock-vs-over-removal/receipt.html
⋯ 7 lines
All output11 lines
ok  wrote prove-report-stock-vs-over-removal (3 files)
  wrote  prove-report-stock-vs-over-removal/report.md
  wrote  prove-report-stock-vs-over-removal/heatmap.svg
  wrote  prove-report-stock-vs-over-removal/receipt.html
  saved  comparison 043ad9ecb90e


inferred: no event was given, so the event set is malformed_call, turn_limit, wrong_state, passed, the failures and successes the runs declared
passed: 0 safer, 1 worse · turn_limit: 0 safer, 1 worse · malformed_call: 0 safer, 0 worse · wrong_state: 0 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 12 tests, paired)


  next: poisson-prove ui

The same study, the other way round

Section titled “The same study, the other way round”

Find what a model edit changed, and what it left alone uses a gentler edit of the same model and two other evals: the refusals the edit was meant to remove, broken down by prompt type, and a benchmark it left alone.

See how far you can quantize a model before its tool calling breaks runs a model live on these same tasks, at full precision, 4-bit and 3-bit.

With your own results

Section titled “With your own results”

Keep tasks.toml’s shape and change three things: the axes to whatever your eval varies, the outcomes to the endings your eval records, and the lookup in replay.py to read your files. If your results were not laid out on Prove’s seeds, key the lookup on the item and attempt each seed should stand for, the same way for both models, so that the pairs are the same item and attempt. The rollout the docs describe on Your own agent is the same function with a live agent inside it.

Stored output

Section titled “Stored output”

Every output above is what the command printed, stored in the site repository under src/data/examples/check-a-model-edit-on-evals-you-already-ran/ with the run’s diff.json and the three files report wrote, as Prove 0.2.2 wrote them. The page cannot show an output that is not there. The episodes are from the study’s Llama-3.1-8B runs of 2026-11-10 and 11; the study’s own analysis of them, with the item as the unit, is in the post.