# 20260905T185407Z-0313 vs 20260905T195009Z-18ac: failed_assertion

failed_assertion: 0 safer, 1 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 328 tests, paired)

![delta heatmaps](heatmap.svg)

## failed_assertion: 0 safer, 1 worse

| temperature | task_id | n (a/b) | rate_a (pp) | rate_b (pp) | delta (pp) | p | q | verdict | MDE |
|---|---|---|---|---|---|---|---|---|---|
| 0.7 | HumanEval/129 | 32/32 | 37.5 | 100.0 | 62.5 | 1.91e-06 | 6.26e-04 | worse | 34pp |
| 0 | HumanEval/129 | 32/32 | 62.5 | 100.0 | 37.5 | 4.88e-04 | 0.0801 | unchanged | 29pp |
| 0.7 | HumanEval/32 | 32/32 | 62.5 | 84.4 | 21.9 | 0.0923 | 1 | unchanged | 29pp |
| 0 | HumanEval/142 | 32/32 | 34.4 | 53.1 | 18.8 | 0.18 | 1 | unchanged | 34pp |
| 0 | HumanEval/100 | 32/32 | 25.0 | 40.6 | 15.6 | 0.302 | 1 | unchanged | 34pp |
| 0.7 | HumanEval/91 | 32/32 | 87.5 | 100.0 | 12.5 | 0.125 | 1 | unchanged | 12pp |
| 0 | HumanEval/83 | 32/32 | 84.4 | 96.9 | 12.5 | 0.219 | 1 | unchanged | 16pp |
| 0.7 | HumanEval/134 | 32/32 | 68.8 | 81.2 | 12.5 | 0.125 | 1 | unchanged | 26pp |
| 0.7 | HumanEval/100 | 32/32 | 31.2 | 43.8 | 12.5 | 0.219 | 1 | unchanged | 34pp |
| 0.7 | HumanEval/93 | 32/32 | 81.2 | 90.6 | 9.4 | 0.25 | 1 | unchanged | 19pp |
| 0.7 | HumanEval/83 | 32/32 | 62.5 | 71.9 | 9.4 | 0.508 | 1 | unchanged | 29pp |
| 0 | HumanEval/89 | 32/32 | 25.0 | 34.4 | 9.4 | 0.607 | 1 | unchanged | 34pp |
| 0 | HumanEval/5 | 32/32 | 93.8 | 100.0 | 6.2 | 0.5 | 1 | unchanged | 6pp |
| 0 | HumanEval/91 | 32/32 | 90.6 | 96.9 | 6.2 | 0.625 | 1 | unchanged | 9pp |
| 0.7 | HumanEval/10 | 32/32 | 71.9 | 78.1 | 6.2 | 0.625 | 1 | unchanged | 25pp |
| 0.7 | HumanEval/102 | 32/32 | 6.2 | 12.5 | 6.2 | 0.5 | 1 | unchanged | 28pp |
| 0.7 | HumanEval/159 | 32/32 | 9.4 | 15.6 | 6.2 | 0.625 | 1 | unchanged | 29pp |
| 0.7 | HumanEval/119 | 32/32 | 15.6 | 21.9 | 6.2 | 0.727 | 1 | unchanged | 32pp |
| 0.7 | HumanEval/108 | 32/32 | 43.8 | 50.0 | 6.2 | 0.625 | 1 | unchanged | 34pp |
| 0.7 | HumanEval/89 | 32/32 | 37.5 | 43.8 | 6.2 | 0.754 | 1 | unchanged | 34pp |

Worst-powered unchanged cell (temperature=0.7, task_id=HumanEval/110, n=32/32) can only detect a 34pp rise. "1 worse" is an honest claim only above that.
Worst-powered cell overall: 34pp (temperature=0.7, task_id=HumanEval/110).

> This is empirical evidence under a stated test envelope, not formal verification. Cells outside the envelope were not tested; cells inside it were tested at the stated replicate count and can only detect effects at or above the reported MDE.

## Provenance

- events: failed_assertion
- inferred: false
- non event: none
- n tests: 328
- method: mcnemar_bh
- paired: true
- q: 0.05
- t unit: tool_calls
- horizon: 6
- n not comparable: 0
- dropped a: 0
- dropped b: 0
- truncated: 0
- n absent: 0
- proofload version: 0.0.1
- run id a: 20260905T185407Z-0313
- run id b: 20260905T195009Z-18ac
- run salt: f4b697a8e295f01a