Most of the suite’s 552 rules had nothing enforcing them
On 20 September I counted the rules in the Authority System Suite for the first time. Of 552 normative statements, 341 (62%) had nothing named beside them that could enforce them. A rule is only as good as whatever makes the model follow it, so that number shaped how the suite is built.
The suite is a set of rulebooks that make an AI show its work. That day there were four: TAS (the Topic Authority System) for evidence, RAS (the Research Authority System) for research, LAS (the Logic Authority System) for logic and DAS (the Decision Authority System) for decisions. TAS's changelog already had entries from 16 September, and the other three repositories were scaffolded on 19 September. Three more systems arrived on 10 October and have their own pieces.
The router, the short file that decides which rulebooks a question needs, gives the purpose of all of it in one sentence: “Output that looks like it met the standard when it did not is the failure this entire system exists to prevent.”
The first audit found problems that fail silently
The first audit across all four systems, written the same day, found problems that fail silently. The word “tier” meant three different things in three systems (TAS tiers A to F, RAS tiers 1 to 8, DAS tiers A to F), so a value from one could validate in another and mean something else. And in all four systems the user was the only source with no tier.
Ordinary review tools hadn't found any of this. So on 20 September I set out to build a kind of test made for this project's systems design and procedures, and to work out why the built-in tools couldn't see the gaps. The project's testing doctrine records that direction.
Most of the rules had nothing behind them
The tool that came out of it is a claim ledger: a script that extracts every normative sentence (a modal, an imperative or a table row, anything that says what must happen) and gives it an id. Then it asks two questions of each. Is anything named beside it that could make it fire, such as a field, a check, a script or a test id? And has the project's record ever seen it act?
The first run, at about 07:20 UTC on 20 September, found 552 claims. Of those, 341 were PROSE-ONLY: 62%, with nothing named that enforces them. And 416 were UNTRACEABLE: 75%, meaning the claim names no record id, so no evidence could ever attach to it.
It also caught something on its own. A rule written an hour earlier said that every system named in the response's “loaded:” line (where the AI announces which rulebooks it loaded) must have a matching read step in the trace (its log of steps). That rule had no check behind it. The ledger flagged it, and a check was written.
The count is a heuristic, and it has moved
The ledger is a heuristic and its file says so: a claim's class is “a prompt to look, not a verdict.” Some rules can't carry a mechanism because what they govern is a reply, and replies aren't kept. On 21 September I decided that those rules, once each is marked with its reason, come out of both the total and the prose-only count.
By 27 September the ledger read 828 claims with 269 PROSE-ONLY (32%), and on 12 October 1,041 claims with 313 (30%). The file list grew between runs, so those rows are not like for like.
Whether any of it helps is still open
The suite's README keeps the claim small. Its “Honest limits” section says: “Most of this is not enforced.” It says the suite doesn't prevent hallucination or sycophancy but makes both “catchable”, and that whether any of it improves output “is empirical”. The other pieces in this series describe the tests that try to answer that.
Sources and notes (8)
- AGENTS.md specification — the instruction-file convention the router follows
- README — the notes this was written from
- AGENTS.md router — the notes this was written from
- DOCTRINE_TESTING.md — the notes this was written from
- Project log — the notes this was written from
- Claim ledger summaries — the notes this was written from
- Umbrella overview — the notes this was written from
- TAS changelog — the notes this was written from
Authority System SuiteAll 17 parts →
