Ragas

You score two versions of your system with Ragas and compare two means. This page shows how to find out whether the difference is real, or whether the run was too small to tell.

Poisson Trace reads the score table you saved. It runs no metric and calls no model.

Install

pip install poisson-trace
pip install poisson-prove

Nothing here imports Ragas. Tested on tables written by Ragas 0.4.3.

Save your scores

Ragas’s evaluate() writes no result file. It returns a result; save its table, once per version:

result.to_pandas().to_json("base.jsonl", orient="records", lines=True)

The table needs a column that names each case and is the same in both versions: a question id, or the question itself. Use the same cases for both versions. If you can, score each case more than once per version: one run per case is the smallest design there is. Decide the cut before you look: the score under which, or over which, a case counts as missed.

A file written by Ragas’s @experiment is read the same way, by whatever columns your function returned. pandas’s to_json writes each score to at most ten decimal digits, and that is the score the import reads.

One thing about Ragas itself, which this reader never runs. In Ragas 0.4.3 evaluate() posts usage events to Ragas’s own server unless RAGAS_DO_NOT_TRACK=true is set before Ragas is imported. With it set, evaluate() made no network call in our runs, and Ragas still wrote a random usage id to a file, uuid.json, in its folder under the user data directory.

Import and compare

One import is one version. Each import reads one score column and writes one new run into a store in Prove’s format; poisson-trace fix-check then compares the two runs.

poisson-trace-import ragas FILE [FILE ...] --store ROOT --run NAME
        --metric COLUMN
        --cut X ( --below | --above )
        --case-column COLUMN
        [--started ISO-8601] [--json]

Nothing has a default but the start time. You say which column holds the score, where the cut is and which column identifies a case.

FILE is .csv, .jsonl, .json (a list of objects) or .parquet, told by its extension; .gz is read on the first three. Any other extension is refused, and so is a file that is not what its extension says.
--metric is the score column. A column the file lacks is refused, and the refusal lists the file’s columns. A number outside 0 to 1 is kept as it is: the cut decides.
--cut takes a number and exactly one side: --cut 0.5 --below counts a score under 0.5 as missed, --cut 0.3 --above counts a score over 0.3 as missed. A score equal to the cut met it. The outcome is the word met or the word missed. A Ragas table records no threshold, so there is none to fall back on.
A cell that is empty, NaN, null or not a number has no score. Its row has no outcome, the summary counts it as no score, and Trace leaves it out and counts it in the header of its report. A missing score is never read as 0.
--case-column is the column that identifies a case. Rows that share a value are repeats of that case, in the order read, and several files given to one import are read as one version, in the order given. A row with no value there refuses the whole import, and the refusal names the row. In a CSV the value is the text as written: 007 and 7 are two cases.
--started is when that version’s run started, with its zone. Without it the first file’s modification time is used, and the summary says which of the two it was.

The worked example

The two files are in tests/fixtures/ingest/ragas/. They hold the faithfulness, answer correctness and answer relevancy scores of two published result sets, a base model and the same model fine-tuned, on the same 150 rows of 146 questions: one question is asked three times and two are asked twice (huggingface.co/datasets/R0bfried/RAGAS-BASE-llama-3-2-1B-eval2 and …RAGAS-RAFT-llama-3-2-1B-eval2). Those sets carry no licence, so the files hold a hash of each question and the scores, and no text; make.py beside them is the recipe. The Ragas version that scored them is not stated. Over the rows that have a score, mean faithfulness is 0.550 for the base model and 0.566 for the fine-tuned one.

The commands run from the root of a clone and write the store to ./store. The start times are when each set was last changed where it is published.

poisson-trace-import ragas tests/fixtures/ingest/ragas/rag-base.jsonl \
  --store ./store --run rag-base-faithfulness --metric faithfulness --cut 0.5 --below \
  --case-column case --started 2025-03-31T06:56:41Z

# Poisson Trace · import · rag-base-faithfulness

- run written: rag-base-faithfulness · store: ./store
- tool: ragas
- file: rag-base.jsonl · sha256 48e8afe8ab3913ab7a0126124bc4eb67357a626c3e78bd7ad2428bae0cfee19f
- declared: metric faithfulness · cut 0.5 · side below · case_column case
- started: 2025-03-31T06:56:41Z · from: given
- rows 150 · cases 146 · repeats of a case: smallest 1, largest 3
- outcomes: met 95 · missed 52
- rows with no outcome: no score 3

In the command below, OLD and NEW are this run and the other version's run, imported into the same store, the older version first. X is the largest limit of detection you accept, as a share: 0.05 is five points.

next: poisson-trace fix-check OLD NEW --store ./store --event missed --inputs all --max-mde X  # rag-base-faithfulness is OLD or NEW

poisson-trace-import ragas tests/fixtures/ingest/ragas/rag-raft.jsonl \
  --store ./store --run rag-raft-faithfulness --metric faithfulness --cut 0.5 --below \
  --case-column case --started 2025-03-31T07:16:07Z

# Poisson Trace · import · rag-raft-faithfulness

- run written: rag-raft-faithfulness · store: ./store
- tool: ragas
- file: rag-raft.jsonl · sha256 5f5a986fade2bf12531e50e72c3f0532734a5882e18b8bb5bf6304c133975fe7
- declared: metric faithfulness · cut 0.5 · side below · case_column case
- started: 2025-03-31T07:16:07Z · from: given
- rows 150 · cases 146 · repeats of a case: smallest 1, largest 3
- outcomes: met 95 · missed 47
- rows with no outcome: no score 8

poisson-trace fix-check rag-base-faithfulness rag-raft-faithfulness --store ./store \
  --event missed --inputs all --max-mde 0.10 --top 0

# Poisson Trace · fix check · rag-base-faithfulness (old) and rag-raft-faithfulness (new)

## Runs

- old: rag-base-faithfulness · target: import/ragas · status: complete · started: 2025-03-31T06:56:41Z · declared repeats: not recorded
- Prove's tally (old): rows 150 · with outcome 147 · no outcome 3 · timed out 0 · infra 0 · other 3
- new: rag-raft-faithfulness · target: import/ragas · status: complete · started: 2025-03-31T07:16:07Z · declared repeats: not recorded
- Prove's tally (new): rows 150 · with outcome 142 · no outcome 8 · timed out 0 · infra 0 · other 8
- the two runs started 19 min apart
- events: missed
- inputs: every input either run carries · selected 146 · compared 137 · left out 9 (no outcome on the old version 3; no outcome on the new version 6)
- rows used: old 141 (timed out 0, no outcome 0) · new 141 (timed out 0, no outcome 0)
- inputs short of the declared repeats: old 0 · new 0
- inputs with mixed outcomes across their repeats: old 1 · new 0

## The control

The old version, re-run on the same inputs with nothing changed.

| version | runs | failed | failure share | did not fail on re-run |
|---|---|---|---|---|
| old | 141 | 51 | 36.2% | 63.8% |
| new | 141 | 46 | 32.6% | 67.4% |

Re-running these inputs and counting the runs that did not fail would have read 63.8% on the old version, where nothing changed.

## The comparison

exact conditional Cochran-Mantel-Haenszel test, stratified by input over 137 inputs (68 informative), two-sided; α = 0.05.

new failures: observed 46, expected 48.500, variance 17.083; p = 0.629
difference: -3.5 points, new minus old (-15.1 to +8.0; Mantel-Haenszel weights; interval: Sato variance, large-sample, 95%)
by input: p = 0.636 (sign-flip over the 68 inputs that moved, 100,000 draws, seed = 0); not part of the line below

## Consistency

failure share 0.344 · fails again 0.291 · κ -0.080 · within-input variance 0.245 against pooled 0.226
a comparison that ignored the inputs would need 1/(1 − κ) = 0.93 times the runs

## What it could not have detected

proportional change on every input, 80% power, α = 0.05, continuity-corrected normal approximation to the stratified statistic.

- a drop in failure share: could not have detected one below 15.9 points
- a rise in failure share: could not have detected one below 17.7 points

**too small to tell: could not have detected a drop in failure share below 15.9 points, and 10 was asked for**

## Guidance

- repeats: 3 per arm on these 137 inputs would bring the limit to 10 points
- inputs: 356 inputs like these, at the present repeats, would do the same
- ceiling: --max-mde 0.1588 accepts this design as it stands

The report goes on with a note on its guidance, the inputs that changed (no question has enough runs to be flagged) and its block What this cannot say, left out here for length. The command exits 2.

Mean faithfulness rose from 0.550 to 0.566. The run could not have detected a drop in the share of misses smaller than 15.9 points, and 10 was asked for, so it cannot say whether faithfulness moved. It says what would: each question three times per version, or 356 questions.

The Consistency block of this report rests on three questions. Three of the 137 questions compared were asked more than once, and Trace prints a κ, and the line under it, whenever any input has more than one run on a version. The other 134 have one run per version, and for them its figures set the base model’s run beside the fine-tuned model’s. So the κ of -0.080 and the 0.93 say mostly how far the two models miss the same questions, not how consistently either one fails. Do not plan a design from them.

The same two files at the same cut, on the other two scores:

poisson-trace-import ragas tests/fixtures/ingest/ragas/rag-base.jsonl \
  --store ./store --run rag-base-correctness --metric answer_correctness --cut 0.5 --below \
  --case-column case --started 2025-03-31T06:56:41Z

- outcomes: met 43 · missed 107
- rows with no outcome: 0

poisson-trace-import ragas tests/fixtures/ingest/ragas/rag-raft.jsonl \
  --store ./store --run rag-raft-correctness --metric answer_correctness --cut 0.5 --below \
  --case-column case --started 2025-03-31T07:16:07Z

- outcomes: met 63 · missed 87
- rows with no outcome: 0

poisson-trace fix-check rag-base-correctness rag-raft-correctness --store ./store \
  --event missed --inputs all --max-mde 0.10 --top 0

new failures: observed 87, expected 97.000, variance 14.333; p = 0.011
difference: -13.3 points, new minus old (-23.1 to -3.6; Mantel-Haenszel weights; interval: Sato variance, large-sample, 95%)

**different: the new version failed less on these inputs (exact Cochran-Mantel-Haenszel, stratified by input, p = 0.011)**

poisson-trace-import ragas tests/fixtures/ingest/ragas/rag-base.jsonl \
  --store ./store --run rag-base-relevancy --metric answer_relevancy --cut 0.5 --below \
  --case-column case --started 2025-03-31T06:56:41Z

- outcomes: met 101 · missed 49
- rows with no outcome: 0

poisson-trace-import ragas tests/fixtures/ingest/ragas/rag-raft.jsonl \
  --store ./store --run rag-raft-relevancy --metric answer_relevancy --cut 0.5 --below \
  --case-column case --started 2025-03-31T07:16:07Z

- outcomes: met 118 · missed 32
- rows with no outcome: 0

poisson-trace fix-check rag-base-relevancy rag-raft-relevancy --store ./store \
  --event missed --inputs all --max-mde 0.10 --top 0

new failures: observed 32, expected 40.500, variance 9.750; p = 0.009
difference: -11.3 points, new minus old (-19.3 to -3.4; Mantel-Haenszel weights; interval: Sato variance, large-sample, 95%)

**different: the new version failed less on these inputs (exact Cochran-Mantel-Haenszel, stratified by input, p = 0.009)**

On both, the fine-tuned model missed the cut less often: 107 rows of 150, then 87, on answer correctness, and 49 of 150, then 32, on answer relevancy. Three scores at one cut are three tests. The cut, 0.5, was written down before any of them was run.

These are two stored tables. Whether the two models were scored on the same day, with the same retrieval and the same judge, is not in the files, and the report’s closing block says that a difference between two runs is a difference between the runs.

A score Ragas could not give

Ragas gives NaN where a metric cannot be computed, and a saved table holds it as an empty cell, a null or a NaN. Trace leaves those cases out of both versions and counts them at the top of the report.

memory-causal.jsonl.gz and memory-strong.jsonl.gz, in the same folder, are the tables Ragas 0.4.3 gave for a stored memory-retrieval experiment: the same 1,800 lookups on a memory with causal reads and on one with strong reads. Each row holds the ids the memory returned and the ids that answer the question, and Ragas’s two id-based context metrics, which call no model, scored them. Context precision has no score on 1,130 of the 1,800 rows of the causal memory: the rows where nothing was retrieved. The start times given are the two stored runs’ own.

poisson-trace-import ragas tests/fixtures/ingest/ragas/memory-causal.jsonl.gz \
  --store ./store --run memory-causal-precision --metric id_based_context_precision \
  --cut 0.5 --below --case-column user_input --started 2026-09-28T19:37:36Z

- rows 1800 · cases 1800 · repeats of a case: smallest 1, largest 1
- outcomes: met 331 · missed 339
- rows with no outcome: no score 1130

poisson-trace-import ragas tests/fixtures/ingest/ragas/memory-strong.jsonl.gz \
  --store ./store --run memory-strong-precision --metric id_based_context_precision \
  --cut 0.5 --below --case-column user_input --started 2026-09-29T08:26:16Z

- rows 1800 · cases 1800 · repeats of a case: smallest 1, largest 1
- outcomes: met 900 · missed 900
- rows with no outcome: 0

poisson-trace fix-check memory-causal-precision memory-strong-precision --store ./store \
  --event missed --inputs all --max-mde 0.05 --top 0

- inputs: every input either run carries · selected 1800 · compared 670 · left out 1130 (no outcome on the old version 1130)

new failures: observed 338, expected 338.500, variance 0.750; p = 1.000
difference: -0.15 points, new minus old (-0.66 to +0.36; Mantel-Haenszel weights; interval: Sato variance, large-sample, 95%)

**no significant difference (p = 1.000); could not have detected a drop in failure share below 1.7 points**

On the 670 cases left the report reads no significant difference: 339 misses, then 338, -0.15 points, in a design that could have detected a drop of 1.7 points. The header says that 1,130 cases were left out and why, and they are the cases where the causal memory returned nothing. Read that line before the verdict. The two means do not show it: 0.453 over the 670 rows Ragas scored, against 0.458 over 1,800.

What this does not do

It does not run Ragas, a metric or a model. It reads a table.
It reads one score column per import, at one cut. The cut and its side are yours to declare; there is no default.
It does not pair rows by position. Name the column that identifies the case. Rows that share a case are read as repeats of it.
It does not compare means. It compares the share of cases that missed the cut, case by case. The score is kept on every row that has one.
It keeps nothing of any column but the two you name: no answer, context or reference text. The case column’s value is kept as written when it is at most 80 printable characters, and as a SHA-256 digest of it otherwise. If you name the question column as the case, a short question is stored as it is; name an id column to keep questions out.
It does not skip a line it cannot read. A JSON Lines file with a line that is not a JSON object is refused whole, where Ragas’s own loader skips the line.
It does not choose a cut. Trying several cuts or several scores and keeping the one that reads well is not covered by the confidence printed.
With one run per case per version, no single case can be flagged and the consistency of failures cannot be measured. Repeats fix both.
It does not add to a run. A run name the store already holds is refused.
A store the importer wrote is for Trace’s verbs. Prove’s own diff pairs rows by seed, and a score table records none.
Tables from Ragas versions other than 0.4.3 are read by their column names, like any table, and no test runs another version. The published pair above is such a table: the version that scored it is not stated.