AI sounds sure.Make it show
its work.
The Authority System Suite is a set of plain-text rulebooks, tests and record-keeping tools that make an AI model's work checkable by someone who wasn't there. It runs on any model, in a coding agent, a project or a plain chat, with nothing to install.
- My role
- Creator and director: the questions, hypotheses, design and every call
- When
- Sept–Oct 2026, 20 working sessions
- Scale
- 7 systems · 267 adversarial tests · 68 published sources
- Status
- Open licence. Results self-reported, not yet independently confirmed
adversarial cases passed by the same model with the rules, against 116 without: 91% against 52%.Paired, pre-registered and blind-scored. Self-reported, not yet independently confirmed.
How it was tested →Fluent is not the sameas checked.
An AI model writes with the same confidence whether the work behind it was done or not. A paragraph built from three cherry-picked sources reads exactly like one built from a fair search. The reader can't tell, so mistakes pass as answers until something expensive breaks.
Lawyers have filed invented case law, consultancies have refunded reports with made-up citations, and a model update had to be rolled back for agreeing with everyone. The failures come in four shapes:
Process, not model size,is the lever.
- Process beats capability.Investigators, auditors and courts already work in stages that each leave something checkable, and refuse to treat a claim as true until it has a source. Write that discipline down and an AI can follow it every time, without being reminded.
- The human is half the problem.The rules also have to hold against the person using them: ego, faulty memory, a leading question. Nothing tiered the user's own assertions, so now something does.
- A rule nobody can see firing isn't a rule.The main quality measure became the share of rules with no mechanism behind them.
- Externalise memory.Anything an agent is expected to remember will eventually be forgotten; anything it is expected to re-read will not be. So the state lives in files, not in the context window.
- Cheaper models, closer to premium ones.Written discipline should close part of the gap on procedure-heavy work. A goal, not yet measured.
Many features of AI that are locked behind paywalls are things that could be stored in what I am building, to allow everyone to have access to them for free.
Ony, on why it should be free
Honest limit
It doesn't prevent hallucination or agreement. It makes both catchable. That is a weaker promise than “no errors” and a more useful one.
A building codeplus a site logbook.
The seven systems are the code for seven kinds of construction. The umbrella is the part of the code that applies to every building. The Paper Trail is the logbook an inspector reads after the crew has gone home.
Five systems run as a pipeline in the order the work happens: source → fact → quantity → inference → decision. Lower number wins a conflict, so a source research rules unusable can't support a fact, and a claim still alleged can't carry a recommendation. Two support systems sit beside the pipeline and issue no verdicts. A router sends work by what the AI is about to produce, not by keywords in the question.
A design decision
Three systems each had a field called “tier”, meaning three different things, so a value could pass in one system and mean something else in the next. The fix was to namespace the values rather than map them: TAS:A, RAS:3, DAS:B.
source → fact → quantity → inference → decision
Two support systems beside the pipeline. They issue no verdicts.
RAS
Research
“Can this source be relied on?”
Guards againstConfidence a fair search wouldn't support
Documentation peoplecan actually follow.
I wanted documentation a reader can follow without losing attention: visuals, section headers, double columns, metaphors, flow charts, a glossary and an index, said more simply without diluting the meaning.
Every section has two layers: a plain-language guide on top and the verified source underneath, unchanged. Each part ends with the same check: what a tool or test actually enforces, against what is still only a written rule.
- 319-page guide, 591-page complete edition, a web edition with search
- 12 parts, 116 sections, a 182-term glossary, 49 data charts
- One writer agent and a separate checker agent for every part
- About 1,360 statements checked against the sources; 142 corrected
Tested like an experiment,not a demo.
adversarial cases passed by the same model with the rules, against 116 without: 91% against 52%.
Without the rulesWith the rules
Cases passed per system, both rounds combined (regression gates on 12 Oct, full suites on 18 Oct 2026). First-pass results.
- Paired: every case run with and without the rules, on the same model
- Pre-registered: what counts as a pass was written before the run
- Blind: three readers score each answer without knowing which arm it came from
- Stopping rules taken from the testing literature, written before the reruns
A hard call
One fix passed its case but missed the threshold I'd set for it in advance. I reverted it. A rule you bend when it costs you a result isn't a rule.
What these numbers are not
The answers and the readers came from one model family, the results are self-reported, and early failures were fixed against the failing answers before re-running. Independent reviewers and other models are next.
The record ofhow it got here.
The first system, TAS, starts life as rules inside one ChatGPT project, then is rewritten so any model can run it.
The founding goal is written down: the best possible searching, sourcing and reasoning, stored as portable text any AI can be given.
False start: the first session spends four turns insisting three of the systems don't exist, because a lookup failed.
A failed lookup is not evidence of absence. Every session now keeps a list of what it has not verified.
I notice I'm the one catching every mistake: the AI's built-in checks aren't. That becomes the project's testing discipline.
Turning point. I ask what has actually been fixed. The honest answer: the last nine cycles fixed almost nothing and withdrew six earlier claims.
Stop generating findings; fix the metric. Work is redirected the same evening.
The share of rules with nothing enforcing them falls from 55% to 25% in a day, mostly through checks that each bind a whole table.
First paired probe: 13 tests, same model with and without the rules. 5 of 13 pass without, 13 of 13 with.
The handbook and the Rigor Manager, a local app that shows the rules firing live and lets a person confirm or dispute the AI's work.
I ask whether the system's starting point is research-backed. It isn't.
A published source now stands behind every module rule, and no web search starts without a research plan.
Three more systems: numbers (QAS), evidence gathering (EGS), ethics and harm (EHS).
Every system's regression gate run paired, pre-registered and blind-scored: 62 of 70 with the rules, 40 of 70 without.
The remaining 151 cases: 140 with, 76 without. Stopping rules from the testing literature are written before the reruns, and one fix that missed its own threshold is reverted.
A record showing only outputs would misrepresent this work as the model's. Most of what shaped it came from here.
From the project's record of who decided what
Running insidea real product.
The suite is built into the Journalism Workbench, my private investigation app. The AI's instructions there are the suite's own modules, its run controls are the suite's settings in plain words, and its answers must pass the suite's output checks before a person even sees them.
It runs both ways. The app maps every value it stores to a term in the suite, and when the app needs something the suite doesn't define, that's reported as a gap in the suite rather than patched over in the app. The product is a test harness for the method.
See the Workbench →What isn't on this page
The rule text, the prompts and the test cases stay out on purpose. Publishing a test lets anyone train to it, and the wording is what makes the method work. Happy to walk through any of it in conversation.
Being wrong is ordinary and recoverable. Being wrong invisibly is not.
The design constraint behind all of it
More case studies
All work and career →


Get in touch.
I'm open to creative technical project management, founding teams, and agentic AI engineering and research. I read everything that comes in, and I answer.
onysnowstudios@gmail.comLinkedIn →
