The first paired test of the Authority System Suite
On 12 October all 70 regression-gate cases were run once with the Suite's rules loaded and once without, scored blind and pre-registered. With the rules 62 passed; without them, 40.
In this piece
The Authority System Suite is a set of rulebooks that make an AI show its work. On 12 October I ran its first paired test, built to be hard to fool.
Every regression-gate case across the seven suites, 70 in all, was run twice: once with the doctrine loaded, once without. Each result was scored by three blind readers. The whole run was pre-registered before any subject ran, so I couldn't decide afterwards what counted.
62 of 70 with the rules, 40 without
With the doctrine, 62 of 70 passed. Without it, 40. The check that tests the doctrine's own health went from failing one criterion to passing all five.
Eight cases failed even with the doctrine, and those eight were logged as an open issue. The following days went on tightening the rules those cases exposed and re-running them, with every re-run pre-registered the same way.
More on the Suite on its case study.
Sources and notes (7)
- The preregistration revolution (Nosek et al.) — pre-registration separates hypothesis generation from testing
- Adding Error Bars to Evals (Miller) — evaluations as experiments with noise
- Judging LLM-as-a-Judge (Zheng et al.) — limits of model readers, which the project uses to score its probes
- DOCTRINE_TESTING.md — the notes this was written from
- Test and probe records — the notes this was written from
- Tasks — the notes this was written from
- Directives — the notes this was written from
Authority System SuiteAll 17 parts →
