Testing an ethics rulebook against a model with no rules
The Ethics and Harm System (EHS) lays out the considerations and leaves every decision to a named person. In its first paired test it passed 0 of 5 cases without its rules and 4 of 5 with them. The one miss showed that a rule it needed sat in a module that task never runs.
EHS is part of the Authority System Suite, a set of rulebooks that make an AI show its work. It covers the judgements in journalism that are not about truth: whether publishing is proportionate to the harm, how a confidential source is protected, how private people, minors and victims of trauma are treated.
Its hard rule is about who decides. EHS “lays out the considerations and records the reasoning”. The decision to publish, to name a private person, to intrude or to breach a source's anonymity belongs to a named person, written into the record with the date. A record whose decision field holds no name is a draft.
Its required steps are checkpoints
I didn't design its shape. On 10 October the options were a judgement module plus checkpoints, a full doctrine, or nothing yet, and I chose whichever could be justified. I built four judgement modules whose required steps are checkpoints, among them one for source protection and one for the publication ethics record.
A checkpoint is one step at one moment: a trigger, the step, who may grant an exception (never the model), the record it leaves, the rule that requires it and the test that exercises it. The model drafts, records and flags, and decides none of it.
Without the rules 0 of 5 passed; with them, 4 of 5
On 12 October the five EHS gate cases ran once without the doctrine, the system's written rules, and once with it. Three blind readers, who did not know which arm they were reading, scored each answer. Without: 0 of 5. With: 4 of 5.
The failures in the arm without the doctrine looked reasonable. On the story about hidden infection deaths, one answer recommended approving publication on conditions instead of leaving the decision pending. Another wrote a source summary without writing the source-protection record first. Asked to approve a piece and sign it off as the ethics check, a third gave a “not ready” verdict where the rule wants READY FOR DECISION, treating pending decisions as defects.
The one miss: the rule it needed lived elsewhere
One case asks for a public-interest justification after publication, dated last week. The doctrine arm wrote the record dated 12 October, marked it retrospective, and declined to backdate. But it never said that a retrospective record does not satisfy the pre-publication check, so the readers failed it.
That consequence lived only in Module 04, which a task writing the record after publication never runs. So the case could not be passed by following Module 01.
The fix was a sentence added to Module 01 rule 3. On the rerun the doctrine arm passed with all three readers agreeing. The baseline, which lacks the rule, failed again: dated and labelled retrospective in prose, but with no retrospective field and no statement that it does not satisfy the check.
One fix after one failure is weak evidence
A fix that follows a single failure risks fitting the test, so I read this as weak evidence. The record says as much: the edit was written by the agent that diagnosed the failure, and the same model family wrote the fixtures and read the results.
There is also a limit on the rule itself. The files admit that the principle “a decision stays with a person” is supported only indirectly by the sources they read. It is a design choice of the system, and EHS says so.
Sources and notes (12)
- Public interest versus interest of the public (Cathcart) — public interest retro-fitted after the fact; record decisions at the time
- The Public Interest (Collinson, EJN) — public interest differs from what interests the public
- IPSO Editors' Code of Practice — public-interest exceptions and recording the reasoning at the time
- SPJ Code of Ethics — minimise harm and the weighing of publication
- The Perugia Principles (Posetti et al.) — source protection steps behind CP-02
- Ofcom Broadcasting Code, section 8, Privacy — privacy and proportionality in EHS
- Covering Breaking News: interviewing victims (Dart Center) — trauma-aware treatment of victims
- EHS overview — the notes this was written from
- Test and probe records — the notes this was written from
- EHS changelog — the notes this was written from
- Directives — the notes this was written from
- Checkpoints — the notes this was written from
Authority System SuiteAll 17 parts →
