When to stop testing a rulebook
By 18 October, testing the Authority System Suite, a set of rulebooks that make an AI show its work, had reached a point where each new pass found more failures and the cost kept rising, so I set stopping rules. They set the exit criteria first, sample before calling a failure, fix the test before the rule, and record every deviation.
A rulebook can't be checked like code
Ordinary code review can't test a rulebook. The Suite's file on testing its doctrine, its written rules, gives four reasons: wrong object, no runtime, a generic template instead of the project's own claims, and a reviewer who is the builder. A rule only runs through a model's attention, so the evidence that it ran is in the trail of records, not in the file.
On 20 September I set out to build a kind of test made for this project's design and procedures. One tool it produced, a claim ledger, counted the claims. Of 552 normative statements, rules about what should happen, 341 were prose-only and 416 had no id any evidence could attach to.
The method grew four safeguards and kept its weaknesses
Over four weeks the method picked up four things: pre-registration, so a pass condition is written before anything runs; paired arms, every case run with the doctrine and without; blind readers, three per packet, majority wins, none told which arm they read; and replicates, because one run per arm is one observation, not a difference.
On 12 October I set out to finish the adversarial system, since a couple of tests still failed. Its weaknesses sit in the same records: one subject per cell, fixtures (the written test cases) written in the same working session as the rules, readers from the same model family, and doctrine outputs whose structure sometimes gave the arm away.
On 18 October every fix found another failure
The full run that day gave 76 of 151 without the doctrine and 140 with it. Eleven failures remained, and each fix-and-rerun round found something else. I wrote a new section of the testing file that day, after a run in which each new pass found more failures and the cost kept rising.
A target first, then samples, then the test before the rule
Set the exit criteria before running. The target is mine: at least 95 percent of cases passing on the majority of at least three samples. It is not an outside standard, so it carries no citation.
Sample before you act. A case is not called failed on one sample, because two cases failed once and then passed two of three samples. Fix the test before the doctrine: the first full run found five of eight failures were fixture faults, and the follow-up found four more.
Read a rerun against the case's own earlier samples, because cases chosen for failing look better on a retest by regression to the mean, the pull of an extreme result back toward the average. Keep a guard set of cases that passed. Batch the edits: at most two edit-and-test rounds per request, then report what is left and stop. Report the cost, and state every deviation.
That last one has a case on file. The follow-up reverted an edit that missed its own pre-registered threshold, even though keeping it would have been easier.
D01 passed once its test stopped changing
Case D01 tests whether the Decision Authority System judges a failed product launch by the decision rather than the outcome. The last three D01 attempts used the same text and fixture, and the doctrine arm passed all three with every reader agreeing. The baseline failed all three.
One of the earlier rounds had failed because the test's own ground truth contradicted itself. It was corrected after the result, and the record marks that as a post-result change to the test file.
Whether 95 percent is right is still open
What exists is a target I set, rules written down before the next run, and a log of what each round cost. Nothing here shows that 95 percent is the right number.
Sources and notes (9)
- ISTQB glossary: exit criteria — exit criteria agreed before testing
- Adding Error Bars to Evals (Miller) — evaluations as experiments with noise
- Flaky Tests at Google — re-running a failing test as a standard mitigation
- The preregistration revolution (Nosek et al.) — pre-registration separates hypothesis generation from testing
- Regression to the mean in health care decisions — cases picked for failing look better on retest
- Judging LLM-as-a-Judge (Zheng et al.) — limits of model readers, which the project uses to score its probes
- DOCTRINE_TESTING.md — the notes this was written from
- Test and probe records — the notes this was written from
- Directives — the notes this was written from
Authority System SuiteAll 17 parts →
