Skip to content
Poisson Labs Docs
GitHub poissonlabs.ai

Examples

Check whether two leaderboard neighbors actually differ

A leaderboard ranks entries by their average score. Two entries a few points apart are ranked one above the other, and the ranking does not say whether the gap is more than the tasks could have produced by chance. When every entry attempted the same tasks, you can check: compare the two task by task, and count the tasks only one of them solved.

This example does that for two pairs of neighbors on USACO, one of the nine boards in Can agent leaderboards tell neighbors apart?. The outcomes are public: the task-level results of the HAL leaderboard, as released with Hardy, Azam, Reuel, Kochenderfer and Koyejo’s paper (arXiv:2610.00651) under the MIT licence. Each entry is a model on an agent scaffold, and each attempted the same 307 tasks once.

Before you start: Install and Your machine.

1. Get the outcomes and install Prove

Section titled “1. Get the outcomes and install Prove”

The clone is about 6 MB.

Terminal window
git clone https://github.com/hardy-education/scaffold_eval
cd scaffold_eval
git checkout e7ec46439b68badb3f87e55230bcf128c76a992a
python3 -m venv .venv && source .venv/bin/activate
pip install poisson-prove

2. Write the rollout and the spec

Section titled “2. Write the rollout and the spec”

A Prove run is a set of episodes from a rollout, rollout(cfg, seed) -> Episode. Here the rollout calls nothing. It looks up whether the entry named in SYSTEM solved the task the seed stands for. Replicate k is task k in sorted order for every entry, so two entries swept on one salt are paired task by task. Save this beside data/:

replay.py
"""A Prove rollout that replays one leaderboard entry's stored outcomes on USACO.


Each row of data/hal_response_matrix.csv is one task attempted by one system (a model on an
agent scaffold) and whether it was solved. SYSTEM picks the entry, as `model_name/agent_name`.
Prove calls the rollout with a seed; replicate k of the board is task k in sorted order, the same
for every entry, so two entries swept on one salt are paired task by task. No model is called.
"""


import csv
import os
import pathlib


from poisson_prove import Episode
from poisson_prove.executor import derive_seed
from poisson_prove.store import config_hash


MODEL, AGENT = os.environ["SYSTEM"].split("/")
SALT = os.environ["SALT"]
BOARD = "usaco"


rows = list(csv.DictReader((pathlib.Path(__file__).parent / "data" / "hal_response_matrix.csv").open()))
TASKS = sorted({r["task_id"] for r in rows if r["benchmark"] == BOARD})
SCORE = {r["task_id"]: r["score"] for r in rows
         if r["benchmark"] == BOARD and r["model_name"] == MODEL and r["agent_name"] == AGENT}
if set(SCORE) != set(TASKS):
    raise SystemExit(f"{MODEL}/{AGENT} did not attempt every {BOARD} task")


# the seed Prove gives replicate k of this board, for every k: the way back from a seed to its task
TASK_OF = {derive_seed(SALT, config_hash({"board": BOARD}), k): task for k, task in enumerate(TASKS)}




def rollout(cfg, seed):
    task = TASK_OF[seed]
    return Episode(outcome="solved" if SCORE[task] == "1" else "unsolved", t=1, horizon=1, meta={"task": task})

Two of its imports, derive_seed and config_hash, are in the published package but are not part of the documented Python API. They are how the rollout knows which task a seed stands for.

The spec is one board with 307 replicates, one per task:

board.toml
# One leaderboard entry on USACO: 307 tasks, one attempt each, replayed from the public outcomes.
target = "replay:rollout"
target_id = "hal-usaco"
t_unit = "attempts"
replicates = 307


[axes.board]
type = "categorical"
values = ["usaco"]


[declare]
outcomes = ["solved", "unsolved"]
failures = ["unsolved"]
successes = ["solved"]

3. Sweep four entries on one salt

Section titled “3. Sweep four entries on one salt”

Each sweep replays one entry. The same --salt on all four is what pairs them. The first is o4-mini at high reasoning effort, ranked second on the board:

Terminal window
SALT=6c65616465726230 SYSTEM=o4_mini_2025_04_16high/usaco_episodic_semantic poisson-prove sweep board.toml --salt 6c65616465726230 --name o4-mini-high --yes
Outputexcerpt, 3 of 14 lines · exit 0
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
⋯ 7 lines
done  307 rollouts in run "o4-mini-high" (complete) in 1.1 s
⋯ 4 lines
All output14 lines
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: unsolved; successes: solved
  Sweep: 1 axes · 1 cells · N=307 → 307 rollouts
  Backend: local · target: hal-usaco · t_unit: attempts
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.127 at its worst base rate, 0.400 (Fisher, q 0.05 over 2 tests, 4 times in 5; you accept up to 0.200)
129/307
307/307
done  307 rollouts in run "o4-mini-high" (complete) in 1.1 s
  episodes with an outcome  307
  episodes with no outcome  0
  episodes timed out  0
  wall time  1.1 s

The other three are the same command with another entry and name: GPT-5, ranked first, then GPT-4.1 and o3, neighbors in the middle of the board.

Terminal window
SALT=6c65616465726230 SYSTEM=gpt_5_2025_08_07default/usaco_episodic_semantic poisson-prove sweep board.toml --salt 6c65616465726230 --name gpt-5 --yes
Outputexcerpt, 2 of 13 lines · exit 0
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
⋯ 8 lines
  episodes with an outcome  307
⋯ 3 lines
All output13 lines
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: unsolved; successes: solved
  Sweep: 1 axes · 1 cells · N=307 → 307 rollouts
  Backend: local · target: hal-usaco · t_unit: attempts
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.127 at its worst base rate, 0.400 (Fisher, q 0.05 over 2 tests, 4 times in 5; you accept up to 0.200)
307/307
done  307 rollouts in run "gpt-5" (complete) in 0.1 s
  episodes with an outcome  307
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.1 s
Terminal window
SALT=6c65616465726230 SYSTEM=gpt_4_1_2025_04_14default/usaco_episodic_semantic poisson-prove sweep board.toml --salt 6c65616465726230 --name gpt-4.1 --yes
Outputexcerpt, 2 of 13 lines · exit 0
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
⋯ 8 lines
  episodes with an outcome  307
⋯ 3 lines
All output13 lines
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: unsolved; successes: solved
  Sweep: 1 axes · 1 cells · N=307 → 307 rollouts
  Backend: local · target: hal-usaco · t_unit: attempts
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.127 at its worst base rate, 0.400 (Fisher, q 0.05 over 2 tests, 4 times in 5; you accept up to 0.200)
307/307
done  307 rollouts in run "gpt-4.1" (complete) in 0.1 s
  episodes with an outcome  307
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.1 s
Terminal window
SALT=6c65616465726230 SYSTEM=o3_2025_04_16default/usaco_episodic_semantic poisson-prove sweep board.toml --salt 6c65616465726230 --name o3 --yes
Outputexcerpt, 2 of 13 lines · exit 0
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
⋯ 8 lines
  episodes with an outcome  307
⋯ 3 lines
All output13 lines
plan  307 rollouts of hal-usaco: 1 cells x 307 replicates on local, $0 (local)
  target  replay:rollout (imports, takes cfg and seed)
  outcomes  failures: unsolved; successes: solved
  Sweep: 1 axes · 1 cells · N=307 → 307 rollouts
  Backend: local · target: hal-usaco · t_unit: attempts
  Cost: $0 (local)
  smallest rise one condition alone would be flagged for  0.127 at its worst base rate, 0.400 (Fisher, q 0.05 over 2 tests, 4 times in 5; you accept up to 0.200)
307/307
done  307 rollouts in run "o3" (complete) in 0.1 s
  episodes with an outcome  307
  episodes with no outcome  0
  episodes timed out  0
  wall time  0.1 s

4. Compare the top two

Section titled “4. Compare the top two”

diff with two runs named compares them in that order. --event solved makes it one test, on the share of tasks solved.

Terminal window
poisson-prove diff o4-mini-high gpt-5 --event solved
Outputexcerpt, 5 of 15 lines · exit 0
safer  solved: 1 safer, 0 worse (paired test on shared seeds)
⋯ 1 line
  before  o4-mini-high
  after  gpt-5
⋯ 5 lines
condition    event   before  after  change  verdict  rise   fall
board usaco  solved  0.580   0.697  +0.117  safer    0.112  0.116
⋯ 4 lines
All output15 lines
safer  solved: 1 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  before  o4-mini-high
  after  gpt-5
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 1 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 1 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 1 tests)
  intervals  on each condition's change: Tango at 95%, false coverage (Benjamini-Yekutieli) over the 1 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
  fragility  18 outcomes would change the word: 18 runs of B in board usaco that ended in solved made to end otherwise


condition    event   before  after  change  verdict  rise   fall
board usaco  solved  0.580   0.697  +0.117  safer    0.112  0.116


  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound


  next: poisson-prove report o4-mini-high gpt-5 --x board

GPT-5 solved 214 of the 307 tasks and o4-mini 178, a gap of 11.7 points. On the tasks only one of them solved, the split is 49 to 13, and the paired test flags it. Prove’s word for a significant change in the favourable direction is safer, here meaning more tasks solved.

5. Compare two in the middle

Section titled “5. Compare two in the middle”
Terminal window
poisson-prove diff gpt-4.1 o3 --event solved
Outputexcerpt, 5 of 16 lines · exit 0
no significant change  every condition that did not count could have shown a change of 0.200: the largest limit it read is 0.116 (the paired test (McNemar), Holm q 0.05 over 1 tests, 4 times in 5); solved: 0 safer, 0 worse (paired test on shared seeds)
⋯ 2 lines
  before  gpt-4.1
  after  o3
⋯ 5 lines
condition    event   before  after  change  verdict    rise   fall
board usaco  solved  0.450   0.463  +0.013  unchanged  0.116  0.114
⋯ 4 lines
All output16 lines
no significant change  every condition that did not count could have shown a change of 0.200: the largest limit it read is 0.116 (the paired test (McNemar), Holm q 0.05 over 1 tests, 4 times in 5); solved: 0 safer, 0 worse (paired test on shared seeds)
  method  paired test on shared seeds (McNemar), the default for runs paired at sweep time
  threshold  0.200 (recorded at sweep time by the runs)
  before  gpt-4.1
  after  o3
  verdict rule  the verdict counts changes at the family-wise level (Holm, q 0.05 over 1 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 1 tests
  limits  the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 1 tests)
  intervals  on each condition's change: Tango at 95%, simultaneous (Bonferroni) over the 1 tests
  fragility  12 outcomes would change the word: 12 runs of A in board usaco that ended in solved made to end otherwise


condition    event   before  after  change  verdict    rise   fall
board usaco  solved  0.450   0.463  +0.013  unchanged  0.116  0.114


  more  --full prints the per-event tables by hash, the coverage, the omnibus, the fault lines and the bound


  next: poisson-prove report gpt-4.1 o3 --x board

o3 solved 142 and GPT-4.1 solved 138. The board ranks o3 above GPT-4.1. On shared tasks the difference is 1.3 points, with 35 tasks only o3 solved and 31 only GPT-4.1 did, and the test does not flag it. The first line also says what the comparison could have seen: a difference of about 11.6 points or more. A gap of 1.3 is far below that, so this board cannot say which of the two is better.

What it says

Section titled “What it says”

Both pairs are neighbors, and only one is a difference the board can show. These are the numbers the study reports for the two pairs: on USACO it finds 3 of 13 neighboring pairs separate after correcting for testing all 13, and on the other eight boards none do.

This example tests two pairs, each on its own. The study tests every neighboring pair on a board and corrects across them, so a pair that passes alone here could fail there. The study’s own detection floors were computed in its analysis code and are not Prove’s limit printed above.

On your own leaderboard

Section titled “On your own leaderboard”

Change replay.py’s lookup to read your table: one row per task and entry, with whether it was solved. Keep the task order the same for every entry. An entry that skipped tasks cannot be paired on them; this rollout refuses one.

Stored output

Section titled “Stored output”

Every output above is what the command printed on a fresh clone of hardy-education/scaffold_eval at commit e7ec464, with Prove 0.2.2 from PyPI in a new virtual environment, stored in the site repository under src/data/examples/check-whether-two-leaderboard-neighbors-differ/ with replay.py, board.toml and each comparison’s diff --json. The page cannot show an output that is not there.