Checking the media suite plan against its primary sources
When the media suite's mockup was checked against primary sources on 18 October, several of its claims did not hold. Four of its ten model slots ship file formats it does not list, one model was marked non-commercial without the licence saying so, and three detector figures could not be matched to the papers cited. Nothing of the suite is built, and twelve questions to me are still open.
The media suite is my plan for going through photos, video and audio in the Workbench: cleaning them, isolating sounds, and recognising sounds, places, context, accents and languages. The research, written on 11 October, begins with what the app did then. It read the text in images (OCR), stored a photo as dated picture evidence whose date is typed by hand, and read video and audio only through OpenAleph, the platform it reads its files with, with no analysis.
Every machine output is treated as a clue
The research reviews how verification teams work, from the Berkeley Protocol, Bellingcat and the Verification Handbook, and draws four design rules from them. Keep originals untouched and hash them on arrival, recording a fingerprint of their exact contents. Treat every machine output as a clue rather than a fact, stored with the tool and settings that made it, and confirmed by a person. Combine several weak signals, and show the workings.
Its build order has four phases and 24 items. Phase one is cheap and needed by everything: hashing, a metadata reader, duplicates, transcription with word timestamps, and keyframes.
What it leaves out on purpose
It leaves out a “fake or real” verdict from any detector, face recognition against outside databases, and naming people from faces or estimating emotion, age or ethnicity. It also leaves out voice-print matching of a named person without an expert, and using enhanced images to read plates or identify people.
Non-commercial models and AGPL components are left out unless I decide the app will never be commercial or redistributed. The research records its own limit: accuracy figures are publishers' own benchmark numbers, and real material will usually do worse.
Several claims did not match the sources
By 18 October there was a media suite mockup of 60 boards. A round of source verification set its claims beside the primary sources: live Hugging Face model cards (the documentation published with each model), licence texts, W3C and IETF documents, US statutes, arXiv papers, NIST and C2PA documents. The mockup was left unedited and the differences were reported.
A model slot is a place in the design where one AI model plugs in. Four of the ten slots ship formats the mockup does not list: Whisper small as a CTranslate2 file, Omnilingual ASR as a .pt checkpoint, diarization (working out who speaks when) as .bin, and translation as a 5.48 GB pytorch_model.bin. The mockup says safetensors, a format that cannot carry runnable code, for all four, and its own rule refuses pickle formats, which can, unless a reason is written.
The sound-separation model was marked non-commercial; the SAM licence text has no such clause, though it has trade-control and end-use limits. The detector figures of 79%, 61% and 38% could not be matched to any table in the papers cited. One claim about SynthID's retention is contradicted by Google's privacy notice.
Some claims held: the NLLB non-commercial label, the Media Fragments syntax, the cited accessibility criteria, and the position of giving no fake-or-real verdict.
Twelve questions are still open
The research ends with twelve open questions for me. Some are technical, such as what GPU the main machine has. Some are about principle: whether the app will ever be sold or redistributed, which decides whether non-commercial models are allowed; whether face clustering within one case is acceptable; whether accent identification, which today covers only 16 English accents, is worth including.
One is a name: the research suggests “The Darkroom”, as a suggestion only.
No answers are recorded in the notes I read, and nothing of the suite is built. The next step in the project's start-here note is mockups for approval, then a study of the open-source suites at github.com/storytold, whose licences were checked on 11 October.
Sources and notes (12)
- Berkeley Protocol on Digital Open Source Investigations (OHCHR) — one of the verification practices the four design rules draw on
- Bellingcat Online Investigations Toolkit — verification practice the research reviewed
- Verification Handbook, 3rd edition — verification practice the research reviewed
- Model card: Systran/faster-whisper-small — checked on 18 Oct: the transcription slot ships a CTranslate2 file
- Model card: facebook/nllb-200-1.3B — checked on 18 Oct: the non-commercial label holds
- SAM-Audio licence text — checked on 18 Oct: no non-commercial clause for the sound-separation slot
- SynthID — checked on 18 Oct: its privacy notice contradicts the retention claim
- Media Fragments URI 1.0 (W3C) — checked on 18 Oct: the fragment syntax holds
- storytold on GitHub — the open-source suites Ony asked to study next
- Media suite research — the notes this was written from
- Source verification, round 2 — the notes this was written from
- Media suite UX research map — the notes this was written from
Journalism WorkbenchAll 12 parts →
