145 of 783

September 24, 2026

Method

847 public items calibrated into narrow bands beside cuts of 0.50, 0.60 and 0.90; three sweeps in separate windows, each sending every item five times with the identical request, the model pinned and no retries; the declared pair compared item by item, with an item-clustered bootstrap interval on the flip rate and McNemar's exact test as the arm-order check.

Result

Jev (jev-1.13.0), sent the identical request twice on items chosen to sit beside a cut: 159 of 858 near-cut decisions flipped at 0.50, 145 of 783 at 0.60 and 48 of 900 at 0.90, while the action rates moved a point or two and the flips nearly cancelled.

Could not detect

Rates for items chosen to sit beside a cut, not for ordinary traffic. One model version in three windows of under five minutes each. A provider-side cache would lower the measured rate, so it is a lower bound on an uncached replay. Nothing about accuracy, other versions, or a week.

Artifacts

report · receipt · code and data

You put a decision model behind a threshold. The model returns a probability, your code compares it to a cut, and something happens or does not: the ticket escalates, or the response ships. Send the same input again a minute later and the probability can come back a tick different. Most of the time that changes nothing. When the probability was sitting next to the cut, it changes the decision.

That is the finding. We sent Jev (jev-1.13.0, from TypeSafe) the identical request twice, on items we had chosen because their probability sat beside a cut, and at two of the three cuts roughly one near-cut decision in five came out different the second time. At the third cut, on the same kind of near-cut items, it was about one in twenty. The overall action rate barely moved, because flips in one direction were matched by flips in the other.

Two things to hold onto before the numbers. These are rates for items picked to sit beside the cut, so they are not rates for ordinary traffic, where most items sit far from any threshold. And “different” means the branch changed, not that the score wobbled. A score that moves from 0.51 to 0.58 at a cut of 0.50 is the same decision twice.

What this means if Jev sits behind a threshold in your system

Read this section first if you run Jev in production. The rest of the note is how we know.

The instability is in the near-cut decisions. An item at 0.02 or 0.98 does not flip. An item at 0.59 or 0.61 with a cut at 0.60 can. How exposed you are depends on how much of your traffic lands near your cut, and this study cannot tell you that number. Your own logs can.

Your aggregate metric will not show it. Across two runs on near-cut items, the action rate at cut 0.50 read 38.8% and then 40.6%, and at 0.90 it read 54.8% and then 54.6%. Underneath, 159 of 858 near-cut decisions at 0.50 and 48 of 900 at 0.90 changed branch. A dashboard that watches the rate sees two numbers a point or two apart and calls it stable. Seeing the flips requires comparing the two runs item by item.

Ties count. Jev returns probabilities at two decimal places, so a returned 0.60 against a cut of 0.60 is exactly on the line. Whether you act on > or >= decides what happens to those items, and it is a choice worth making on purpose. TypeSafe’s documented examples use the strict comparison, so that is the primary rule here, and the weak rule is reported beside it.

Asking more than once helps and does not finish the job. On near-cut items, taking a majority of five calls per decision cut the disagreement between two independent five-call majorities to about two thirds of the single-call figure at 0.50, to about half at 0.60, and to under half at 0.90. It costs five times the tokens and five times the serial latency. The table further down has one row per call count with price and latency, so you can see where the buying stops. It recommends nothing.

This is a measurement of one model version in three windows of a few minutes each. It says nothing about next week, about other versions, or about why.

How many times you have to ask

Once a single call can land on either side of the cut, the operational question is how many calls you need before the decision settles. That is why each item went to Jev five times per sweep rather than twice.

Two measures, both per cut, both on items chosen to sit beside the cut and so not rates for ordinary traffic. agree@j is the share of items where the first j calls all took the same action. It falls as j grows, because more calls give more chances to disagree, and agree@2 is one minus the flip rate, item by item. MS@j is the share of items where two independent majorities of j calls each, taken in different sweeps, reached the same decision. It rises with j, and it is the number that says what a majority vote buys. Both carry item-clustered 95% intervals.

At 0.50 and 0.60, asking five times and getting the same answer every time happened for about six near-cut items in ten. At 0.90 it was about nine in ten. An item that answers at random passes agree@5 one time in sixteen, so the metric bounds instability from below.

The rows below are in the form declared before the numbers existed: for each j, the price, the latency, and “out of 10,000 near-cut decisions, N decided differently” between two independent j-call majorities, with both bounds of the item-clustered interval. These are near-cut decisions rather than ordinary traffic, and no row is a recommendation.

Tokens are the provider’s own recorded input usage, averaged over the 12,705 measurement calls. Price is cited to OpenRouter’s public listing of Jev ($0.042 per million input tokens). Latency is the median recorded call multiplied by j, and the same divided by the four requests we kept in flight.

The five-call row is not a floor. Five calls inside one sweep are minutes apart and share whatever the provider was doing in those minutes. If any caching happened on the provider’s side, agreement here is higher than an uncached replay would show.

What we ran

Jev is a hosted decision model. You send it a state and a yes-or-no proposition and it returns a probability. TypeSafe’s documentation puts a threshold on that probability and acts when it clears. We measured three cuts of the kind people ship: 0.50, 0.60 and 0.90.

The items are public. Banking77 customer-support messages with the question “should this go to this queue”, CLINC150 assistant utterances with the question “is this inside what the assistant handles”, and HelpSteer2 prompt-and-response pairs with the question “is this response ready to send”. All three are CC BY, and all three have been sent to Jev by other people who published their code. We wrote no items.

Flips cannot be measured on items far from a cut, because there are none to find there. So a calibration pass scored a candidate pool once and kept, for each cut, the items whose probability landed in a narrow band beside it: [0.45, 0.55) for 0.50, [0.55, 0.65) for 0.60, [0.85, 0.95] for 0.90. Items were taken in an order fixed by a hash of their text before any score was seen, with no more than 120 from any one source per cut. The bands filled to 286, 261 and 300 items. The two shortfalls are reported as shortfalls, and the corpus was frozen and its digest filed before any measurement call.

Then three sweeps, each in its own declared 20-minute window, each sending every item five times with the identical request, the model pinned to jev-1.13.0 on every call, four requests in flight, zero retries. The first two arms of each sweep are the declared pair, and every headline statistic comes from those two arms alone. The other three arms feed only the agree@j and MS@j figures.

Runs S1-a and S1-b, S2-a and S2-b, S3-a and S3-b, with arms 3 to 5 beside them, through Poisson Prove, pre-release and in testing: report · receipt. Request identity was asserted across all five arms on the whole canonicalised body, with the transport request id the only declared exemption. 2,541 replicate sets, 2,541 matched, none unmatched.

The three sweeps ran in 268, 283 and 275 seconds, inside their windows. The median replay gap between the two arms of a pair was 55.6, 56.3 and 55.2 seconds at the three cuts. Median call latency was about 0.25 s.

Everything below is on the primary rule, act when p > c, applied to the value as returned with no rounding. A returned value exactly on the cut does not act. The weak rule, p ≥ c, was pre-declared as a sensitivity analysis and is reported beside the primary.

The result at each cut

Each cut has its own item set, so the three results are three separate measurements. The design forbids ordering the cuts or reading the three rates as a trend, because a difference between cuts could be the items rather than the cut.

Each cut’s headline is the sentence pre-written for its outcome before any call was made. All three resolved to the same outcome: the interval on the flip rate excludes zero.

Cut 0.50, 286 items × 3 sweeps, 858 pairs:

Two runs of the identical item set scored 38.8% and 40.6% action rates while 159 of 858 actions flipped at the shipped cut — on items picked to sit right beside that cut, so this is not a rate for ordinary traffic — and the flips nearly cancelled.

Flip rate 18.5%, item-clustered 95% interval 15.4% to 21.8%, on near-cut items. Net swing +1.7 points, interval −1.3 to +4.8. Item-level ICC 0.26, design effect 1.52. Ties at exactly 0.50: 91 in the first arm, 72 in the second.

Cut 0.60, 261 items × 3 sweeps, 783 pairs:

Two runs of the identical item set scored 37.9% and 40.6% action rates while 145 of 783 actions flipped at the shipped cut — on items picked to sit right beside that cut, so this is not a rate for ordinary traffic — and the flips nearly cancelled.

Flip rate 18.5%, interval 15.5% to 21.7%, on near-cut items. Net swing +2.7 points, interval −0.3 to +5.6. ICC 0.16, design effect 1.33. Ties at 0.60: 72 and 60.

Cut 0.90, 300 items × 3 sweeps, 900 pairs:

Two runs of the identical item set scored 54.8% and 54.6% action rates while 48 of 900 actions flipped at the shipped cut — on items picked to sit right beside that cut, so this is not a rate for ordinary traffic — and the flips nearly cancelled.

Flip rate 5.3%, interval 3.6% to 7.2%, on near-cut items. Net swing −0.2 points, interval −1.6 to +1.1. ICC 0.30, design effect 1.59. Ties at 0.90: 75 and 73.

The 0.90 result is not a smaller version of the other two and is not to be read against them. It is the one cut where the design could have ruled flips out below one in a hundred, and it did not.

The three per-sweep tables at each cut sit within a few flips of one another. The per-sweep flip rates on near-cut items run 16.1%, 19.2% and 20.3% at 0.50, 18.4%, 17.6% and 19.5% at 0.60, and 5.0%, 5.7% and 5.3% at 0.90. That is spread across three windows on one fixed item set. It is not a variance estimate for anything else.

The first arm always runs before the second, so drift across a window would show up as flips running mostly one way. McNemar’s exact test is the check on that, and it is not a test of whether flips occur: a process that flips a fifth of items evenly in both directions gives it nothing to reject. None of the nine sweep-by-cut tests rejects, and the smallest p is 0.19. At 0.50 and 0.60 the second arm acted a little more often in every sweep, which is why those net swings are positive, and which the test cannot separate from noise at this n.

Under the weak rule, p ≥ c, on the same near-cut items, 164 of 858 flipped at 0.50 (19.1%, 16.2% to 22.1%), 157 of 783 at 0.60 (20.1%, 16.7% to 23.4%), and 46 of 900 at 0.90 (5.1%, 3.4% to 6.9%). Same outcome at every cut. The tie rule moves individual items and does not move the picture.

How the result was checked

The pre-registration, with every cut, band, statistic, cluster unit, floor and outcome sentence, was frozen before any call was made, and everything decided after it is a dated addendum rather than an edit. After the sweeps, an independent recompute, written by a different model family and sharing no code with the tool, recomputed every table, interval and k-metric from the stored response rows. It confirmed every number but one. The one was at cut 0.60: one item’s fifth call had returned an HTTP 529, and our own tool had dropped that item’s declared pair from the analysis along with its five-call metrics, though the pair itself had returned 0.62 and 0.64 and agreed. The pre-registration said to drop it from the five-call metrics only. The tool was fixed, the analysis was regenerated from the stored rows, and the pair was restored: 145 of 783 rather than 145 of 782, the flip rate moving from 18.54% to 18.52%, nothing else changing. The stored rows, the corpus, the harness, the declarations and the recompute are at github.com/poisson-labs/jev-replay, and the recompute runs there without the tool installed.

What this is a measurement of

jev-1.13.0, on 847 items chosen to sit beside three cuts, inside three windows of under five minutes each, on 2026-09-22.

Every rate above is conditional on the selection. The flip rate on ordinary traffic is not measured and is expected to be lower, by an amount that depends on how much of that traffic sits near the cut.

It is a lower bound in one sense. Nothing at the API boundary can detect a provider-side cache hit, and a cache hit makes a replay agree by construction. The measured flip rate is therefore a lower bound on the flip rate of an uncached replay, and the agreement rates are upper bounds.

It is not a trend across cuts, because the three cuts have three different item sets. It is not accuracy or calibration, because there are no ground-truth labels in this design; it says whether the decision reproduces, not whether it is right. And it is not a rate over time. Three windows of a few minutes say nothing about hours, days or a version change.

Items were chosen on a single calibration score, so on average they sit a little further from the cut in the measurement arms than in the pass that chose them. That makes flips less likely, not more.

The version pin is a string. The client sends jev-1.13.0 and the provider echoes it. Every number is dated to the run.

agree@5 cannot see a coin flip. One item in sixteen that answers at random passes it.

There was one errored call in 12,705, the one described above, and no call was retried anywhere.

What is still open

Why an identical request returns a different value is on the other side of the API, and this design does not see there. Routing, batching, sampling and numerical nondeterminism are all candidates, none of them separable from outside, and this note makes no claim among them. The vendor’s documentation does not describe behaviour on repeated identical requests, and its own consistency cookbook notes that its setup cannot separate variation caused by a changed field from variation on identical requests. That gap is what this study measured, from the outside.

Others have measured related things

A public audit, jev-calibration-audit, reported fifteen distinct answers from fifty identical requests. That is a fact about the values. What is added here is a branch-flip rate at cuts people ship, with an interval, on items chosen before any score was seen, under a pre-registration.

Two others, jev-orderby-bench and jev-ood-calibration, reported that returned probabilities are at two decimal places. Our contract check confirmed that before the corpus was built, and it is why ties are counted above. Janus found the best cut moving from 0.67 to 0.37 between two datasets, which is one reason the cuts here were fixed in advance rather than fitted.

What it cost

12,705 measurement calls, 4,876,563 input tokens and 254,080 output tokens by the provider’s own count. At $0.042 per million input tokens, cited to OpenRouter’s public listing of Jev, that is $0.20 for the measurement and $0.28 for the study. The calibration pass and the contract check before it took another 4,516 calls and 1,729,789 input tokens. The whole study stayed under its 7 million token cap.

Three sentences

Two runs of the identical near-cut item sets scored 38.8% and 40.6% at cut 0.50, 37.9% and 40.6% at 0.60, and 54.8% and 54.6% at 0.90, while 159 of 858, 145 of 783 and 48 of 900 of those near-cut decisions flipped, and the flips nearly cancelled.

The flips ran both ways in every one of nine sweeps, and a five-call majority reduced the disagreement without removing it.

The first difference is a returned value on the far side of a cut, and what produces it is not visible from outside the provider.