failed_assertion: 0 safer, 1 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 328 tests, paired)
"no cell changed" is rejected (p = 0.000249938 < q 0.05) over 328 cells x 1 event
min-p of location and dispersion; 4000 within-cell permutations, pair_swap, seed 0
p floor 2.50e-04: the smallest p 4000 permutations can express; a p at the floor is a bound, not a measurement
| event | mean delta (pp) | p | rms delta (pp) | p |
|---|---|---|---|---|
| failed_assertion | 0.4 | 0.0175 | 5.4 | 2.50e-04 |
informative cells: 56 of 328; 272 unchanged by every exchange
grouping temperature (axis), 2 groups; cells per group min 164 / median 164 / max 164; 0 degenerate; 2 of 2 groups resolve 20pp at alpha 0.025
refused task_id (axis): underpowered; 0 of 164 groups can resolve a change of 20.0 points in a group's event rate (group MDE 0.3615 at 64 and 64 units per arm, alpha 0.000305 = q/164), and you accepted 20.0 points (--max-group-mde 0.2000). --max-group-mde 0.3616 admits this grouping as it stands. next: axis
"no cell in the group changed" tested by BH across 2 groups at q 0.05: 2 rejected
group p floor 2.50e-04: the same 4000 permutations; a group p at the floor is a bound, not a measurement
| temperature | cells | informative | unchanged by every exchange | units (a/b) | group MDE (pp) | discordant pairs | p | q | verdict |
|---|---|---|---|---|---|---|---|---|---|
| 0.0 | 164 | 14 | 150 | 5248/5248 | 3.0pp | 138 | 0.0225 | 0.0225 | rejected |
| 0.7 | 164 | 42 | 122 | 5248/5248 | 3.0pp | 219 | 2.50e-04 | 5.00e-04 | rejected |
"this cell did not change" tested by BH within each of 2 rejecting groups at q 0.05: 1 of 328 rejected
MDE is ADR-018's smallest detectable increase from arm A's rate, capped at the headroom, so on a cell at ceiling it reads the headroom and is not a power number.
temperature=0.0: 164 tests, 0 rejected; cell MDE median 23pp / max 34pp
temperature=0.7: 164 tests, 1 rejected; cell MDE median 23pp / max 34pp
| cell | event | delta (pp) | q | MDE |
|---|---|---|---|---|
| task_id=HumanEval/129, temperature=0.7 | failed_assertion | 62.5 | 3.13e-04 | 34pp |
depth 3 (3 configured); outer-nodes false discovery rate at most 0.432 = 2 x 3 x 0.05 x 1.44
Yekutieli (2008) Corollary A.2 assumes each p-value is independent of its ancestors; this tree's group and root statistics aggregate their own cells, so that assumption is not established here.
| temperature | task_id | n (a/b) | rate_a (pp) | rate_b (pp) | delta (pp) | p | q | verdict | MDE | interval (pp) |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.7 | HumanEval/129 | 32/32 | 37.5 | 100.0 | 62.5 | 1.91e-06 | 6.26e-04 | worse | 34pp | [12.2, 85.8] |
| 0 | HumanEval/129 | 32/32 | 62.5 | 100.0 | 37.5 | 4.88e-04 | 0.0801 | unchanged | 29pp | [-5.1, 68.6] |
| 0.7 | HumanEval/32 | 32/32 | 62.5 | 84.4 | 21.9 | 0.0923 | 1 | unchanged | 29pp | [-23.7, 58.9] |
| 0 | HumanEval/142 | 32/32 | 34.4 | 53.1 | 18.8 | 0.18 | 1 | unchanged | 34pp | [-27.6, 57.4] |
| 0 | HumanEval/100 | 32/32 | 25.0 | 40.6 | 15.6 | 0.302 | 1 | unchanged | 34pp | [-31.4, 55.8] |
| 0.7 | HumanEval/91 | 32/32 | 87.5 | 100.0 | 12.5 | 0.125 | 1 | unchanged | 12pp | [-22.3, 45.9] |
| 0 | HumanEval/83 | 32/32 | 84.4 | 96.9 | 12.5 | 0.219 | 1 | unchanged | 16pp | [-24.9, 47.3] |
| 0.7 | HumanEval/134 | 32/32 | 68.8 | 81.2 | 12.5 | 0.125 | 1 | unchanged | 26pp | [-22.3, 45.9] |
| 0.7 | HumanEval/100 | 32/32 | 31.2 | 43.8 | 12.5 | 0.219 | 1 | unchanged | 34pp | [-24.9, 47.3] |
| 0.7 | HumanEval/93 | 32/32 | 81.2 | 90.6 | 9.4 | 0.25 | 1 | unchanged | 19pp | [-24.5, 42.5] |
| 0.7 | HumanEval/83 | 32/32 | 62.5 | 71.9 | 9.4 | 0.508 | 1 | unchanged | 29pp | [-31.2, 47.1] |
| 0 | HumanEval/89 | 32/32 | 25.0 | 34.4 | 9.4 | 0.607 | 1 | unchanged | 34pp | [-36.7, 51.4] |
| 0 | HumanEval/5 | 32/32 | 93.8 | 100.0 | 6.2 | 0.5 | 1 | unchanged | 6pp | [-26.6, 38.9] |
| 0 | HumanEval/91 | 32/32 | 90.6 | 96.9 | 6.2 | 0.625 | 1 | unchanged | 9pp | [-29.0, 40.6] |
| 0.7 | HumanEval/10 | 32/32 | 71.9 | 78.1 | 6.2 | 0.625 | 1 | unchanged | 25pp | [-29.0, 40.6] |
| 0.7 | HumanEval/102 | 32/32 | 6.2 | 12.5 | 6.2 | 0.5 | 1 | unchanged | 28pp | [-26.6, 38.9] |
| 0.7 | HumanEval/159 | 32/32 | 9.4 | 15.6 | 6.2 | 0.625 | 1 | unchanged | 29pp | [-29.0, 40.6] |
| 0.7 | HumanEval/119 | 32/32 | 15.6 | 21.9 | 6.2 | 0.727 | 1 | unchanged | 32pp | [-33.1, 43.8] |
| 0.7 | HumanEval/108 | 32/32 | 43.8 | 50.0 | 6.2 | 0.625 | 1 | unchanged | 34pp | [-29.0, 40.6] |
| 0.7 | HumanEval/89 | 32/32 | 37.5 | 43.8 | 6.2 | 0.754 | 1 | unchanged | 34pp | [-35.0, 45.4] |
Worst-powered unchanged cell (temperature=0.7, task_id=HumanEval/110, n=32/32) can only detect a 34pp rise. "1 worse" is an honest claim only above that.
Worst-powered cell overall: 34pp (temperature=0.7, task_id=HumanEval/110).
intervals: 99.9848% (alpha* = 0.000152439 = q x 1 / 328 over the flat BH family), basis score_rd_paired
interval excludes zero, verdict unchanged: 0 cells
verdict significant, interval covers zero: 0 cells
This is empirical evidence under a stated test envelope, not formal verification. Cells outside the envelope were not tested; cells inside it were tested at the stated replicate count and can only detect effects at or above the reported MDE.