Foundation
Data.
The files beside a collection.
Which DuckDB files sit next to a collection, which layer writes them, and which reads them.
Chapter 1
The files beside a collection
A PDAI collection is a folder of documents and pages. Beside them stand DuckDB files, each written by one layer and read by others:
- the Stanza analysis — sentences, tokens, dependencies, features and entities, one row per token;
- the fact sidecar — letters, pages, sentences, persons, topics and footnotes as structured facts, together with the evidence space;
- the sentence record — sentences, clauses, keys and the normaliser’s footnotes with their grounds;
- the corpus dictionary of LWgrammar;
- the global register of individuals, shared by all collections.
A sentence is addressed by its document, its page and its number on the page; the number restarts on every page, so an address without the page is wrong. The same address appears in a footnote, in an explication, in a citation and in a deep link of the reader.
The certified document infrastructure that produces the pages and their analysis is described in PDAI (21 pages).
Chapter 2
One register of individuals
Collections spell names in many ways. The global register gives each person, place and institution one identifier — at present 13,081 persons, 2,892 places and 6,357 institutions. Collection builders write only the spellings they found; whether two spellings name one individual is decided by an agent following a written instruction, with a second reader, and an owner’s correction wins.
The normaliser reads the register and never writes to it. When a book names a person the register knows only from letters — Newton in Euler’s E57 — the binding is recorded as a candidate with its reasons, not entered as a decision.