Changelog

S0: repository scaffold. S0.1: move lifelines to dev; core is numpy/pandas/scipy/pyarrow. M1: Episode contract and Rollout protocol. M4: Parquet store, run manifest, config_hash, resume support. S3: Go1 v0/v1 golden fixture and published numbers. S3.1: fixture provenance keyed on last manifest-touching commit. M5: Kaplan–Meier, Aalen–Johansen, Wilson, cell survival table. M6: run diff — Fisher/BH, paired McNemar, log-rank, MDE. M2: grid / sobol / scenarios samplers, axis typing, SweepSpec, TOML loader, tool-reliability preset. M3: local executor — seed derivation, plan/pair/resume, pre-flight, batched runs, timeout, caps. M7: report — delta heatmap, worst-cells table, MDE honesty block, markdown/SVG/HTML. M3: modal backend — image spec, parallel map, infra retry, cost guard, parity with local. M10: CLI — sweep, diff, report, check, inspect, runs; keyless mock-agent quickstart. M6: multi-event diff — per-event CIF comparison, single BH family, trade detection, max_mde_unchanged. M7: multi-event report — per-event panels on a shared scale, trade line and marker, max_mde_unchanged honesty, saturation marker, forest plot, single-run survival report. M10: multi-event CLI — inferred events, repeatable —event, check requires explicit events, single-run report, forest flag, readable receipt, local import_path. Fix: competing set is constructed in one place; check refuses unknown events; zero-n cells render as holes. M12: poisson-prove.trace/1 schema with span parentage, wall-clock timing and code location; from_otel and from_litellm converters; record() and @step; inspect and the mock agent read and write the new shape. Fix: inspect emits tab-delimited trace columns, so a span name or error message containing spaces cannot shift them; poisson_prove.record aliases poisson_prove.trace.record. M12: episode stamper — poisson_prove.otel.EpisodeStamper tags every span with poisson_prove.run_id and poisson_prove.episode from a per-context episode() block, behind the [agent] extra. M12: kind ladder extended — fetch_response and execute_task mapped; openinference.span.kind and traceloop.span.kind typed as a rung below gen_ai.operation.name. M3: in-process trace capture — the executor activates the instrumentors named in the sweep file, stamps every span with its episode, and carries the trace into Episode.meta. Fix: max_spans keeps the earliest-started spans and drops the deepest, so a capped trace keeps its root. M4: Store.read(meta=False) and Store.read(keys=) project and filter at the Parquet reader and Store.count reads the footers; diff, report, check and runs never decode meta, and inspect decodes only the rows it prints. M3: a timed-out episode keeps its trace — the child drains a snapshot of its closed and open spans on SIGTERM, poisson-prove.trace/1 gains truncated, errored rows carry meta, retention never strips an errored row, and inspect selects by error as well as outcome. Fix: the drain pipe is created beside the outcome pipe on every timed call, capture or not, and a Modal errored result whose meta is not a mapping is refused like an invalid episode instead of being stored without it. M3: a crashed episode keeps its trace — the error payload takes the dict form on both backends, complete up to the raise with the failing span marked, truncated None. Pilot: golden-suite agent under examples/golden_pilot — HumanEval at 32 seeds through the full loop, chat completions only, own-span paths scrubbed; the first real integration’s span count, dropped and other share recorded. M3/M10: sweep —backend modal no longer imports the target locally; [modal] secrets mounts named Modal secrets on the function, names recorded in the manifest, values never read. M13: poisson-prove serve — a versioned structure for one run pair (poisson-prove.viewpair/1) and a read-only localhost page over it: both arms’ identity from the manifest, an explicit block naming what could not be verified, M7’s stacked SVG embedded verbatim, the honesty block above the fold, and a per-cell drill-down over M6’s own fields. —json prints the structure. Explicit —event only; ephemeral port on 127.0.0.1; writes nothing. S21: error visibility downstream — Store.tally counts with_outcome/no_outcome/timed_out/infra/other from the rows, never a manifest field; multi_diff carries a coverage frame (rows and rows without an outcome per cell in each arm, and why a non-comparable cell is a hole: only_a, only_b, no_outcome_a, no_outcome_b, no_outcome_both); M7 attaches the cause to grid holes with no rendered byte changed; diff’s text output prints the coverage block; poisson-prove.viewpair/2 carries the tally per arm, rows and rows without an outcome per cell, config_text, and a holes list, and states four unverified entries instead of six (three about instrument errors deleted, outcome_labels added); runs prints no_outcome. Every user-facing name says no_outcome, never errored: the count is whether an outcome was stored, not what it was. Fix: sweep’s non-TTY progress branch flushes each decile, so a sweep watched through a pipe or in CI shows its progress while it runs instead of with the end-of-run summary. M6: diff warns when a pairable pair is analysed unpaired — UnpairedAnalysis, a UserWarning subclass raised once per diff when both manifests carry the same run salt and the requested method does not read the pairing; the method is never switched and no rendered byte changes. Pilot: api_error is a first-class outcome — an episode that could not reach the model is no longer labelled crashed, the outcome vocabulary has one definition shared by the agent and numbers.py, and api_error joins the competing set so the crashed CIF stops absorbing provider failures. M3/M10: sweep refuses an underpowered design — the pre-flight MDE now has a threshold (—max-mde, default 0.20) that —yes does not clear, the accepted value is recorded in the run manifest’s environment, and the MDE line no longer calls itself a worst case. S24: an interval on each cell’s delta, at a level adjusted for the size of the grid it was selected from, with the disagreement against the verdict counted rather than reconciled. M14: the dual root — one permutation-calibrated omnibus over the cells x events family, location and dispersion combined by min-p under the sharp null, printed by diff beside the flat family it does not touch. M14/M10: the fault lines — Level 2 of the layered report groups the comparable cells by a resolver that walks a stated order (requested, trace, the sweep’s axes, task family, the whole grid), refuses a grouping whose groups cannot resolve the effect the caller accepts with the same threshold sweep uses, names what it refused and what it tried next, and tests each group’s “no cell in the group changed” under a root that rejected, calibrated by the root’s own permutations and corrected across groups; diff gains —by and —max-group-mde, a fault-lines block and a tree key; nothing else moves. M14/M10: the leaves and the bound — Level 3 of the layered report tests each cell under a rejecting fault line with the grid analysis’s own per-cell test, corrected within that group’s family alone, names the cells it can and counts the ones it cannot, and prints the outer-nodes false discovery rate the tree controls at, computed from its configured depth by one rule (Yekutieli 2008) with its assumption stated; diff gains —depth, a cells block and a bound block; nothing else moves. M7/M10: the layered report on the page — report on a pair renders the omnibus verdict, the fault lines with their grouping and shape, the cells under each rejecting fault line and the outer-nodes bound the tree controls at, into report.md and receipt.html beside the flat family’s figure, every number a field formatted once; report gains the five tree flags; nothing else moves. M13/M10: poisson-prove.viewpair/3 — the structure carries the layered report verbatim with M7’s formatting of every printed float beside it; the page renders the omnibus verdict first, the bound beside the panel honesty, and the fault lines and cells above the heatmap; serve gains the five tree flags; nothing else moves. Fix: the unpaired-analysis warning scopes its correspondence claim to the cells the comparison matched and points at coverage for the rest, instead of promising a counterpart for every episode on a page that can list cells with none; and serve refuses a tree whose (event, config_hash) multiset is not the diff’s, naming a cell each way, where it previously compared only the unit count. A22: pandas 3 port — a stored null reads back as None on pandas 2 and 3 alike; the pandas<3 cap is lifted and the S0 dependency parser reads PEP 508 names. T.4: τ²-bench rollout wrapper under examples/tau2_pilot — airline, τ²’s own reward and termination reasons as the outcome, τ²’s step count as t against its own 200-step horizon; ten episodes measured on Modal at a pinned gpt-4.1-mini, tokens 98.8% of per-episode cost and list price a 2.2x upper bound on it because of prompt caching, 7/10 outcome reproducibility at identical (cfg, seed). No sweep, no diff, nothing in src/poisson_prove. A24: [agent] moves to openai >=2.20,<3 — the instrumentor’s ceiling is openai 3’s httpx2, not openai 2; the golden pilot and P11 move with it. A28/M14: the agreement digest — TreeResult carries a content digest of the rebuild its own construction verified, both renderers compare it instead of trusting an arity, and a second clause refuses a tree grown under another method or q; poisson-prove.viewpair/4. M11a: [chaos] table, keyed fault injection on (logical step, attempt), both httpx hooks, in-tree fault catalogue and applier, the fault carrier on the span, environment[“chaos”]. No new dependency; fault vocabulary from AgentChaos (MIT), see NOTICE. A7: the trace is stored as two columns, envelope and observation bodies, and leaves meta on disk; poisson_prove.trace gains split(), assemble() and OBSERVATION_KEYS; Store.read(trace=) reads either layout; capture on with zero spans is stored as an empty envelope, capture off as null. A8a: poisson-prove.trace/1 gains operation, step_name, outcome, outcome_by, attempt, gate, truncated and bytes_original on the span and alias_map_version on the trace; from_otel lifts the graph.node, openinference, traceloop and poisson-prove.* carriers into them and records which alias map it normalised under; record() takes step_name, attempt and gate and the span handle takes outcome and outcome_by; inspect prints four more columns; infra_attempt and tries replace two colliding uses of attempt. P-1: diff, report, check, serve: paired is seed_identity_match, checked on the stored seeds, beside unpaired_analysis; the headline states the requested method’s pairing; the flat family’s intervals reach report.md, the receipt and poisson-prove.viewpair/4; the permutation floor is labelled wherever a permutation p prints. A8b: an observation value over max_observation_bytes is stored head and tail, together never more than the limit, with its original byte length recorded, per value and whether or not content capture is on; an attribute nobody named is an observation if it is over the limit; the whole-value truncation marker and the per-span byte budget are retired; content = false now also strips OpenInference’s llm.input_messages.* and llm.output_messages.*; the span cap is off by default; and every stored trace passes through finalize() on its way to the store. cli: a missing or empty store says where it looked and what to run next, a missing —x or —event lists the axes and events the runs carry, and an inferred event set is stated on its own line (E1, E3, E4). surfacing: the report, the receipt and the viewer say what was cut — the provenance key truncated, which was the count of episodes re-censored at the common horizon, is episodes censored at horizon; each run’s recorded byte limit, its span count and how many of those spans were stamped as cut follow the run identity, not counted for a run written before the per-value cut; the viewer structure is poisson-prove.viewpair/5; and inspect prints an observation: cut line for a row with a cut span. store: a run stored before A8a reads as the current schema — a trace of exactly the shape A7 stored (six keys, spans of thirteen) is read with the nine fields A8a added set to None (“not recorded”), whether it sits in the trace column or in meta on a part written before A7, so inspect opens the four runs it exited 3 on and prints the spans of the eight it printed nothing for; Trace.from_dict is unchanged, a trace of any other shape is rejected or returned as before, and nothing stored is rewritten. CD: [capture].max_observation_bytes is required and has no default — an int >= 1, or "none" to store every value whole; a [capture] table without it is refused at parse, before the run exists, and a sweep file with no [capture] table declares nothing and has its rows stored uncut. The store cuts at the limit the run declared, so a raised limit is honoured end to end rather than recorded and ignored; truncate_observations and finalize take a required limit: int | None; OBSERVATION_LIMIT_BYTES is OBSERVATION_ROUTING_BYTES, the size at which split routes a value to its own column and never a cap, and OBSERVATION_HEAD_BYTES and OBSERVATION_TAIL_BYTES are retired; the manifest records the declaration as declared and every surface reads it back; the Modal container refuses a capture payload missing a field; and --resume is refused when the declaration differs from what the run recorded, while --pair-with is not. Nothing stored is rewritten. PR-1: poisson_prove.paired — a declaration loaded whole (unknown keys refused at every level, no default B and no default seed, thresholds and prices read from their literal text into Decimal, a sha256 over every field including exemptions), a request canonicaliser that emits number literals verbatim so 0.95 and 0.95000000000000001 are two requests and not one, and assert_pairs, which matches two stored arms on (config_hash, replicate_index) and either calls a pair eligible or excludes it with one of twelve named causes, with counts that reconcile per cut, per arm and per cause. Strata that overlap and an exemption naming a body field are refused; manifests that disagree, a seed that moved, header identity with nothing recorded to assert it on, and a manifest silent about its retry setting are run-level refusals, never per-pair exclusions. executor_modal records infra_retries in the manifest of every fresh Modal run, so a run launched after this can declare retries = "asserted". No rate is computed here; the analysis, the windows and the cost are phases 2 and 3. PR-2: poisson_prove.paired.analyse — one question per declared cut, from a PairAssertion and nothing else: the action is value >= threshold on the Decimal parsed from the stored literal, a value at the cut is action 1 and a tie; the 2×2, the flip rate (n10 + n01) / n and the net swing (n01 − n10) / n are paired quantities per cut and per sweep, every net_swing block labelled direction: "b_minus_a"; the pooled interval is the percentile cluster bootstrap over items the 45/44 note published (seeded from bootstrap_seed alone, clusters stably sorted, whole items resampled), with the declared fallback below five flipping items (a one-sided exact upper bound, the lower bound by observation, the net swing not estimated) and the one-way ANOVA ICC(1) design effect beside it; a per-sweep interval is exact binomial and never called clustered; the pooled n is {items, sweeps} and never their product; McNemar’s exact test sits under arm_order_check only, with can_reject false below six discordant pairs, and nothing downstream reads it; no statistic crosses a cut; the outcome A, B or C fires only against a declared floor, from the pooled flip-rate interval; PR22’s ineligibility rate is carried per cut. contingency_json and intervals_json are the two artifacts. Run against the 45/44 study’s stored arms it reproduces 67 / 23 / 21 / 89, 0.155 to 0.285, p = 0.880, and −0.010 [−0.075, +0.055]. The windows, the cost and the rendering are phase 3. PR-3: poisson-prove paired and report --paired — the windows, the cost, the rendering and the k replicates. A Cut declares its comparison, gt or ge, with no default (retrofit R3), and a sweep may declare more, the arms after a and b, so a replicate set is k = 2 + len(more) calls of one request matched on (config_hash, replicate_index) across every arm and eligible only when all k rows are; the declared pair still carries the whole per-cut analysis, and a declaration with no more writes the bytes it wrote before. windows reads each sweep’s realized span from the request spans’ trace-envelope timestamps, never from wall_time, records held or withdrawn against a declared window without dropping a pair on any timing criterion, and reports the per-cut replay gap and the per-call latency median; cost sums the provider’s own recorded usage per phase, per sweep and per arm with unrecorded spans counted as what they are, prices it only against a declared price echoed as its literal, and --cap-check exits 1 at or above the declared token cap, between sweeps. replicates reports, per cut and under both the declared comparison and the other one, agree@j for j = 2..k (the rate at which the first j calls all took one action; agree@2 is one minus the flip indicator row by row) and MS@j for odd j (the rate at which two sweeps’ j-call majorities agreed), each through the one item-clustered bootstrap with the mirrored exact lower bound below five disagreeing items, the design effect on that metric’s own footing, a paired step between prefixes, and the pre-declared cost framing whole: dollars and latency per decision at each j, and out of 10,000 near-cut decisions the count decided differently with both bounds and the conditioning in one object. Six artifacts at k > 2, five at k = 2; the Paired replay, Replicates, Windows and Cost sections are folded into report.md and receipt.html before the fence, every number as the artifact carries it and no verdict word, with report.py untouched. Fix: a replicate set that broke only past its declared pair keeps the pair. assert_pairs judges arms a and b on their own when a set is not whole, and when they are an eligible pair the set stays on pairs with None for arms 3..k: the pair analysis counts it, pair_assertion.json counts it as an eligible pair, and the set keeps its excluded row with its cause and failing arm and stays out of agree@j and MS@j. Timing columns are built from the rows, so a padded None never turns a nanosecond stamp into a float. A set broken in arm a or b, and every set at k = 2, reads as before. LS: the local serve pass — serve opens with a plain-language Summary (a partial run named first, two complementary outcomes named as one finding, then per outcome the counts, the conditions that got worse with “how sure” on each cell’s q at very sure < 0.001, sure < 0.01, fairly sure < 0.05, and the detection limit with “too small to tell” at 50 pp or more on the panel’s max_mde_unchanged; never a verdict), built once by the new poisson_prove.plain over the payload’s own objects and carried as the eighth top-level key summary of poisson-prove.viewpair/6 (/5 refused by name); each panel gains mde_unchanged as a field and each hole gains where; Run pair moves to sit before Provenance. Three bugs fixed: the figure’s text is inert so a cell’s own label no longer swallows the click; the figure’s title, headline, trade line, legend caption and honesty lines wrap and grow the box instead of leaving it (report’s heatmap.svg and the receipt too; the grid does not move); the Holes block opens with counts by cause and on/off grid, and every off-grid hole names its config on the report’s axes. When two named outcomes are exact complements (MultiDiffResult.complementary_events, an integer identity on the stored counts) the trade line says so on every surface, and the non-complementary trade line no longer says the patch moved failures. The Result block prints one plain line beside “unpaired”. No board id on the page; serve --help marks --x required and describes --method, --theme and --title; refusals name flags, not parameters. New docs/reading-a-serve-page.md. No statistic, no identifier and no stored value changes. A10: a sweep file may declare the steps, outcomes and gates a run expects; the declaration is recorded on the run, a span whose step is a declared gate is stamped gate=“declared” on its way to the store, and report and inspect print never-reached steps, never-reached gates, emitted gates the declaration does not name, and observed outcomes it does not name — against the declared sets, and only for a run that declared; diff refuses two runs whose declared outcome sets differ, and —resume refuses a sweep whose declaration differs from the run’s. Fix: a [chaos] sweep on the Modal backend now arms its fault in the containers. The chaos table rides each map item beside the capture request and reaches worker_batch, where before it was dropped on the host and every episode ran the control arm under a manifest that recorded the fault. run_with_mapper, the orchestration every mapper backend shares, refuses a chaos run whose ok rows come back without backend_meta[“chaos”], naming the run and the record, before it stores them.