Compile
The corpus is a fixed set of files plus one manifest row per document.
Every document is registered with its id, type, date, publisher, source URL, and file fingerprint. Pages are extracted and stored as written, because a quote is checked against a page, not against a paraphrase of one.
Text is split into passages of a few hundred tokens that never cross a table, and each passage stays tied to the page it sits on. A copy of every file is hosted so a citation can point at a page anchor rather than at a download.
- Idempotent
- Ingest is keyed by file hash. Running it twice over the same corpus inserts nothing the second time; a manifest with one new document processes only that document.
- Extraction is auditable
- Every row records how its text was extracted, so a bad extractor can be traced after the fact. Optical character recognition is a fallback for scanned pages only and is never run over a real text layer.
- Nothing we wrote is a document
- A transcription, an extract, or an analysis is never registered as a source. Only a public body's own published record is.