Check whether the same request gets the same decision
A model that scores an input, with a threshold behind it, makes a decision: route the ticket or not, pass the answer or not. If the score moves between two sends of the same request, an input near the threshold can land on either side. An average will not show it, because the flips go both ways and cancel. This example counts them.
The data is from the same request, twice: a decision model,
Jev (jev-1.13.0), was sent 847 items five times in each of three sweeps, and every response
was stored. The items were chosen to sit beside one of three thresholds, 0.50, 0.60 and 0.90, so
the rates here are for near-threshold inputs and not for ordinary traffic. You run Prove’s
paired on those stored responses, with the declaration the study fixed before it looked at
them. Nothing is sent anywhere.
Before you start: Install and Your machine.
1. Clone the study and install Prove
Section titled “1. Clone the study and install Prove”The clone is about 10 MB. Prove is the only thing installed.
git clone https://github.com/poisson-labs/jev-replaycd jev-replaypython3 -m venv .venv && source .venv/bin/activatepip install poisson-prove2. See what the store holds
Section titled “2. See what the store holds”The repository’s data/store is a Prove store. runs lists it: fifteen runs of 847 rows, five
arms (a, b, 3, 4, 5) in each of the sweeps S1, S2 and S3. Arms a and b are
the pair; the other three are there for asking more than twice.
poisson-prove runs --store data/storeok 15 runs in data/store
⋯ 1 line1 jev@jev-1.13.0 2026-09-23 o S3-5 complete local jev@jev-1.13.0 847 0 2026-09-23T14:25:59Z2 jev@jev-1.13.0 2026-09-23 n S3-4 complete local jev@jev-1.13.0 847 0 2026-09-23T14:25:06Z3 jev@jev-1.13.0 2026-09-23 m S3-3 complete local jev@jev-1.13.0 847 0 2026-09-23T14:24:09Z4 jev@jev-1.13.0 2026-09-23 l S3-b complete local jev@jev-1.13.0 847 0 2026-09-23T14:23:14Z5 jev@jev-1.13.0 2026-09-23 k S3-a complete local jev@jev-1.13.0 847 0 2026-09-23T14:22:16Z⋯ 10 linesAll output18 lines
ok 15 runs in data/store
# name id status backend target rows rows without an outcome created1 jev@jev-1.13.0 2026-09-23 o S3-5 complete local jev@jev-1.13.0 847 0 2026-09-23T14:25:59Z2 jev@jev-1.13.0 2026-09-23 n S3-4 complete local jev@jev-1.13.0 847 0 2026-09-23T14:25:06Z3 jev@jev-1.13.0 2026-09-23 m S3-3 complete local jev@jev-1.13.0 847 0 2026-09-23T14:24:09Z4 jev@jev-1.13.0 2026-09-23 l S3-b complete local jev@jev-1.13.0 847 0 2026-09-23T14:23:14Z5 jev@jev-1.13.0 2026-09-23 k S3-a complete local jev@jev-1.13.0 847 0 2026-09-23T14:22:16Z6 jev@jev-1.13.0 2026-09-23 j S2-5 complete local jev@jev-1.13.0 847 0 2026-09-23T14:21:10Z7 jev@jev-1.13.0 2026-09-23 i S2-4 complete local jev@jev-1.13.0 847 0 2026-09-23T14:20:06Z8 jev@jev-1.13.0 2026-09-23 h S2-3 complete local jev@jev-1.13.0 847 0 2026-09-23T14:19:13Z9 jev@jev-1.13.0 2026-09-23 g S2-b complete local jev@jev-1.13.0 847 0 2026-09-23T14:18:18Z10 jev@jev-1.13.0 2026-09-23 f S2-a complete local jev@jev-1.13.0 847 0 2026-09-23T14:17:23Z11 jev@jev-1.13.0 2026-09-23 e S1-5 complete local jev@jev-1.13.0 847 1 2026-09-23T14:14:43Z12 jev@jev-1.13.0 2026-09-23 d S1-4 complete local jev@jev-1.13.0 847 0 2026-09-23T14:13:50Z13 jev@jev-1.13.0 2026-09-23 c S1-3 complete local jev@jev-1.13.0 847 0 2026-09-23T14:12:56Z14 jev@jev-1.13.0 2026-09-23 b S1-b complete local jev@jev-1.13.0 847 0 2026-09-23T14:12:03Z15 jev@jev-1.13.0 2026-09-23 a S1-a complete local jev@jev-1.13.0 847 0 2026-09-23T14:11:10Z3. Run the paired replay
Section titled “3. Run the paired replay”The declaration, jev_decl_study.toml, says which span holds the request, which value is the
score, the three cuts, and which runs are the arms. paired first checks that the two arms of
each pair sent the same request, then counts the decisions that differ at each cut.
Paired replay walks through every key.
poisson-prove paired --declaration jev_decl_study.toml --store data/store --out pairedOutput8 lines · exit 0
ok wrote paired (6 files) wrote paired/pair_assertion.json wrote paired/contingency_by_cut.json wrote paired/intervals_by_cut.json wrote paired/windows.json wrote paired/cost.json wrote paired/replicates_by_cut.json not recorded the harness's digest and the client's version (the runs did not record them)It writes six files and prints no table. The last line is Prove saying what the stored runs did not record, so that a reader of the files is not left to assume it.
4. Read how many decisions flipped
Section titled “4. Read how many decisions flipped”Two of the files hold the answer. contingency_by_cut.json has, for each cut and sweep, the
four counts of a pair: both sends acted, both declined, and the two ways they disagreed.
intervals_by_cut.json has the flip rate and the net swing, each with an interval that takes
the item as the unit, because one item appears in all three sweeps. This script prints them:
"""Read what `paired` wrote and print, per cut, how many decisions flipped between the two sends."""import json
counts = json.load(open("paired/contingency_by_cut.json"))["cuts"]intervals = json.load(open("paired/intervals_by_cut.json"))["cuts"]
print("cut pairs flipped flip rate 95% interval net swing 95% interval")for cut, c in counts.items(): sweeps = c["per_sweep"].values() pairs = sum(s["n"] for s in sweeps) flipped = sum(s["n10"] + s["n01"] for s in sweeps) f, s = intervals[cut]["flip_rate"], intervals[cut]["net_swing"] print( f"{cut} {pairs:>5} {flipped:>7} {f['estimate']:>8.2%} [{f['lo']:.2%}, {f['hi']:.2%}]" f" {s['estimate']:>+8.2%} [{s['lo']:+.2%}, {s['hi']:+.2%}]" )python3 flips.pyOutput4 lines · exit 0
cut pairs flipped flip rate 95% interval net swing 95% interval0.50 858 159 18.53% [15.38%, 21.79%] +1.75% [-1.28%, +4.78%]0.60 783 145 18.52% [15.45%, 21.71%] +2.68% [-0.26%, +5.62%]0.90 900 48 5.33% [3.56%, 7.22%] -0.22% [-1.56%, +1.11%]What it says
Section titled “What it says”At the 0.60 cut, 145 of 783 near-threshold decisions came back different on the second send of an identical request. The net swing is the change in the share acted on, and its interval contains zero at every cut: the flips nearly cancel, which is why the action rate hardly moves while about one decision in five does at 0.50 and 0.60. At 0.90 it is about one in twenty.
These are the counts and intervals the study reports. They are rates for items picked to sit beside a cut. On ordinary traffic, where most inputs are far from any threshold, the rate is lower, and this data does not say by how much.
On your own model
Section titled “On your own model”The store here was written by a harness that records each request as a span and each score as a
value. To ask the same question of your own grader, classifier or judge, write that harness and
a declaration, sweep it twice, and run paired:
Paired replay has a stand-in grader that runs with no key.
Stored output
Section titled “Stored output”Every output above is what the command printed on a fresh clone of
poisson-labs/jev-replay at commit aeaba51,
with Prove 0.2.2 from PyPI in a new virtual environment, stored in the site repository under
src/data/examples/check-whether-the-same-request-gets-the-same-decision/ with flips.py and
the two files it reads. The page cannot show an output that is not there.