The full adversarial run and its eleven failures
On 18 October all seven adversarial suites ran paired: 76 of 151 cases passed without the doctrine, 140 with it. This covers how the eleven remaining failures were fixed and re-tested, and the stopping rule I wrote down.
In this piece
Six days after the first paired run, I ran the full one: all seven adversarial suites, 151 cases and 302 subjects, pre-registered before anything ran. Without the doctrine, 76 of 151 passed. With it, 140.
Add the first run's 70 and the totals are 221 cases: 202 passed with the rules, 116 without. Those are the numbers on the case study page, and this is the day they were made.
Fixing the eleven failures
Eleven cases still failed in the doctrine arm. I fixed them with six doctrine edits and one fixture change, then reran the eleven: six passed, two still failed and one regressed. The open cases were resampled, the follow-up rounds recorded, and a closing round pre-registered before it ran.
The last open case took a redesign of its evidence, so that the outcome switch in the test was real, and then two edits to the rules, before it passed. Every step was pre-registered first, because a fix tested after seeing the answer proves nothing.
When to stop
Midway through that day I added a section on when to stop and when to rerun, with sources. One line in it states that the 95% target is the maintainer's own, not a standard from outside. I set it, so the document that depends on it says so.
Sources and notes (9)
- ISTQB glossary: exit criteria — exit criteria agreed before testing
- Adding Error Bars to Evals (Miller) — evaluations as experiments with noise
- Flaky Tests at Google — re-running a failing test as a standard mitigation
- The preregistration revolution (Nosek et al.) — pre-registration separates hypothesis generation from testing
- Regression to the mean in health care decisions — cases picked for failing look better on retest
- DOCTRINE_TESTING.md — the notes this was written from
- Test and probe records — the notes this was written from
- Tasks — the notes this was written from
- Directives — the notes this was written from
Authority System SuiteAll 17 parts →
