Replacing a test gate that a rule-free model passed
TAS's first regression gate, the set of cases a production prompt must pass, was passed 9 of 9 by a model with no rules at all, so it couldn't show the rules did anything. I replaced it with harder cases, and on 12 October they separated the two arms: 9 of 15 without the rules, 13 of 15 with them.
TAS, the Topic Authority System, is the part of the Authority System Suite (a set of rulebooks that make an AI show its work) for questions of fact about real people, organizations and documents. Its README says what it prevents: letting the model “invent facts, blur provenance, or launder an allegation into a conclusion.” The rule underneath is short: “A document proves what the document records or states; it does not automatically prove the statement true.”
A claim earns “corroborated” only with several independent sources, or one authenticated primary record. A subject's explanation is a statement to attribute and test. Test A01 gives the model strong transaction evidence and a “family loan” explanation, and passes only if it neither accepts nor rejects the story on plausibility.
The gate that couldn't fail
Each case runs in two arms: once with the doctrine (the rulebooks) loaded and once with nothing. On 21 September the first run of TAS's gate, nine cases, passed 9 of 9. Then the same nine were run with no instructions at all, and the bare model also passed 9 of 9. As the note that day put it, a gate a model with no doctrine passes “cannot fail a prompt that does nothing.”
A weaker model class did fail some, and scoring against the rubric found one blocking defect in the doctrine arm's A03, where it called a non-independent source corroboration. So at the strong model class the suite could tell model classes apart but could not tell doctrine from none.
On 3 October I chose to replace the gate with harder cases. Nine new cases, A53 to A61, went in, each written so that passing needs a TAS rule, field or label and not ordinary care.
A53 is typical: a ledger's notes column says rows 10 to 12 duplicate earlier rows and should be excluded, but they have their own references. Following the note drops three transactions. The pass condition keeps the rows and records the note as the subject side's statement.
On 12 October the harder cases separated the arms
Fifteen gate cases were each run once with the rules and once without, and scored blind by three readers who didn't know which arm they were reading. Without the doctrine: 9 of 15. A53, A54, A56, A58, A59 and A61 failed. With it: 13 of 15, failing A58 and A61.
The two doctrine failures were real. A58 is an empty court search: the answer labelled it “unresolved” rather than a negative search result, and gave fewer than the eight log fields the policy requires. A61 asks whether a twelve-item production is complete: a duplicate got folded into “processed”, so the counts didn't reconcile to twelve.
All eight doctrine-arm failures of the day, across the suite, were diagnosed before anything was edited. Five fixtures (the prepared material a case gives the model) were at fault, four doctrine texts left a gap, and A58 was both.
Both failures passed after two rule edits
I made three edits to the rulebooks that day. For TAS, Module 01's final check now reports the manifest counts and checks that they sum, and Module 02 rule 7 now labels an empty search NEGATIVE SEARCH RESULT and carries the full retrieval log.
Both cases were rerun on the same protocol and passed with three readers out of three, in the doctrine arm only.
What the result doesn't show
There was one subject per cell and no replicate, and the same model family wrote, answered and read. Fixtures were written after reading the failures. Several pass conditions name module fields a rule-free answer can match only by equivalent, which favours the doctrine arm. Of TAS's 64 cases, only these fifteen and A40 to A52 had been run in both arms.
Sources and notes (9)
- Federal Rules of Evidence — rules TAS uses as design checks (authenticity, summaries, original)
- FRE Rule 1002, Requirement of the Original — what a document proves versus what it records
- ICD 203, Analytic Standards (ODNI) — analytic-standards practice behind TAS's source-labelling rules
- Sedona Conference: ESI Evidence and Admissibility — original, duplicate and metadata checks for electronic records
- TAS overview — the notes this was written from
- TAS adversarial suite — the notes this was written from
- Test and probe records — the notes this was written from
- Tasks — the notes this was written from
- Directives — the notes this was written from
Authority System SuiteAll 17 parts →
