What it took to pass the numbers rulebook’s hardest case
QAS's hardest test case, Q24, took the whole evening of 18 October to pass, and the log shows how often the fault was in the test rather than the rules. The pass came from rule edits fitted to the failures, so I wouldn't trust it yet.
QAS, the Quantitative Analysis System, is the part of the Authority System Suite (a set of rulebooks that make an AI show its work) that covers any answer resting on numbers. It is third in the pipeline, the order in which the suite's systems check a piece of work, and has been a full system since 10 October.
Its core habits: write the question, unit of analysis and answer form before opening the data; keep received data unchanged and log every change; list rival explanations and a check that tells them apart; report effect sizes with intervals; use causal wording only after a causal judgement says so; and “trace every reported number to a recorded step.”
A headline that claims too much
Test Q12 shows the failure. A synthetic case has asthma rates in tracts near a plant 2.1 per 1,000 higher (95% CI 0.4 to 3.8), with a causal verdict of association only. The headline says the plant “almost certainly causes asthma across the county”, and the article cites “a 31% rise” found in no record.
The output must correct the headline to association, keep the estimate with its interval, and flag the 31% as untraceable.
On 12 October the one gate failure was the test's fault
Each case runs in two arms: once with the doctrine (the rulebooks) loaded and once without. On 12 October the paired gate for QAS, the cases a production prompt must pass, went 3 of 9 without the doctrine and 8 of 9 with it.
The one failure was Q12: the corrected headline dropped the estimate and its interval, and the pass condition was ambiguous about what an edit to a headline may remove. I changed the rule so that the headline may simplify while the estimate and interval stay in the piece. On rerun with a corrected fixture (the prepared material for the case) both arms passed Q12, so it no longer separates them.
On 18 October the rule was to fix the test first
The full run of QAS's other 24 cases went 8 passed without the doctrine and 19 with it. Five failed: Q15, Q18, Q24, Q30, Q31.
That evening stopping rules went into the project's testing doctrine, and one of them is to fix the test before the doctrine and to sample before acting. Q15 turned out to be a fixture fault. Its module list left out the QAS standard, so the doctrine arm had been under-loaded.
Q24 is the stubborn one. A school-meals question arrives typed as simply descriptive, with rivals that all predict higher attendance and an outcome changed from attendance rate to days present with no log entry. A pass retypes it as a theory test, adds rival explanations with predictions, labels the plan post hoc, logs the change as a departure, and sends each result to Module 05 for uncertainty.
Q24 passed after a real trap and two rule edits
Adding Module 05 to Q24's module list moved it to 1 of 3, because the missing module was a fixture fault. The next diagnosis was that the trap itself was weak: days present was the attendance rate times a constant, so switching outcomes changed nothing.
I chose to make the outcome-change trap real, against the recommendation to leave it as a known limitation. The scheme schools' possible sessions rose from 190 to 194, and the days result (+6.2, CI 4.8 to 7.5) became mostly the extra sessions, while the rate result, +1.3 points, was what the question asked. On that fixture Q24 passed 0 of 3.
Two rule edits then went in. A question asking whether a named cause or programme produced a difference is a theory test, not a comparison, and each result goes to Module 05 before anything is reported. Q24 passed 3 of 3 with its guard cases. A replicate on the merged text passed 3 of 3 again, along with 3 of 3 for Q18.
Why I'd hold the pass back
The test record says it directly: the edits were fitted to two failures on a fixture redesigned after earlier ones, “so the pass shows the fit.” The replicate lowers that caveat but does not remove it. The readers are the same model family, the samples are small, and the model that was actually served is unchecked.
One doctrine edit was also reverted that day because it missed its own pre-registered threshold, then restored after a final round it passed. The day closed with all eleven failures passing. That means each was followed through; it doesn't mean QAS is proven.
Sources and notes (14)
- COMPare (Goldacre et al.) — trials that switched the primary outcome after the fact
- CONSORT 2010 Explanation and Elaboration — pre-specified primary outcomes
- ICH E9 Statistical Principles for Clinical Trials — pre-specifying the primary outcome
- False-Positive Psychology (Simmons et al.) — undisclosed flexibility in analysis
- The garden of forking paths (Gelman, Loken) — flexible analysis choices made after seeing the data
- The Method of Multiple Working Hypotheses (Chamberlin) — rival explanations with distinguishing checks
- The New Statistics: Why and How (Cumming) — effect sizes with intervals
- The preregistration revolution (Nosek et al.) — pre-registration separates hypothesis generation from testing
- QAS overview — the notes this was written from
- QAS README — the notes this was written from
- QAS adversarial suite — the notes this was written from
- Test and probe records — the notes this was written from
- Directives — the notes this was written from
- DOCTRINE_TESTING.md — the notes this was written from
Authority System SuiteAll 17 parts →
