--- title: "PDAI: A Certified Document Infrastructure for Computational Historical Scholarship" subtitle: "From PDF to a Queryable, Audited, and Learning-Enabled Corpus" running-title: "PDAI Document Infrastructure" book: false toc: true author: - name: Gerd Graßhoff affiliations: - name: Humboldt-Universität zu Berlin department: Philosophisches Institut address: Unter den Linden 6 city: Berlin country: Germany postal-code: 10099 - name: Deutsches Museum department: Forschungsinstitut address: Museumsinsel 1 city: München country: Germany postal-code: 80538 orcid: 0000-0002-4450-357X email: gerd.grasshoff@hu-berlin.de abstract: | Computational epistemology — the systematic study of how knowledge claims in historical science can be evaluated computationally — requires primary sources that are queryable down to the page and sentence. This paper presents PDAI (Portable Document AI), a certified document infrastructure that transforms scanned scholarly editions into structured, queryable corpora with full production audit and learning-enabled quality assurance. The infrastructure has three layers: the PDAI format (a flat, predictable XML directory structure validated against a declared DTD grammar), the production pipeline (a Partitur-governed sequence from ingest through schema-first OCR, pilot-gated batch processing, and a mandatory certify-and-learn stage), and the contextual data stores (corpus file with lemma postings and BM25, famulus file, gazetteer, LWgrammar lexicon, register of individuals) that make a certified page answerable to structured scholarly queries. A reference layer, the Famulus Reference, gives every document, page and segment a stable `fr:` address, a citation drawn only from recorded bibliographic data, and provenance and validation records, kept in one DuckLake and served by one MCP server. The paper describes the nine structural categories of the PDAI page grammar and shows through five case studies how they capture the genuine typographic diversity of early modern scholarly editions — epistolary apparatus, astronomical ephemerides, cuneiform transliteration, manuscript catalogue entries, and philosophical correspondence. A page enters the corpus only after the audit gate: every defect is attributed to one of four root-cause layers, registered as a versioned learning case, and closed through an A/B-gated instruction improvement loop. The result is a certified chain from PDF to the ai-viewer endpoint where pages serve as Grounding Evidence for computational epistemological inquiry — citable, provenance-tracked, and regression-anchored. keywords: - digital humanities - document processing - optical character recognition - computational epistemology - PDAI format - schema-first OCR - scholarly infrastructure - large language models - grounding evidence bibliography: ai-scholar-bibliography.bib format: arxivai-typst: default --- # Introduction: Documents as Grounding Evidence The history of science is a record of claims. Kepler claims that planetary motion obeys an area law; Euler derives the shape of tidal forcing from a Newtonian potential; the Babylonian astronomical diaries record Jupiter's daily position in degrees and seconds. These claims are embedded in primary sources — letters, treatises, observation records — most of which reached the digital age as scanned PDFs: visually faithful, computationally opaque. ```{=typst} #arxivai.sidebar(title: [Grounding Evidence])[ *Grounding Evidence* is the direct textual or observational basis on which a historical knowledge claim rests — as distinct from secondary literature that interprets or summarises it. A page of Kepler's correspondence is Grounding Evidence for claims about his reasoning; a monograph about Kepler is Sekundärevidenz. The distinction is operational: PDAI pages are structured to serve as Grounding Evidence, which means they must be citable at the sentence level, provenance-tracked, and answerable to structured queries without paraphrase. ] ``` A scanned PDF preserves what a document looks like. It cannot answer the question: *on how many pages of the Kepler correspondence does Kepler mention the law of refraction?* To answer this question computationally, a system needs not images but structured transcriptions — transcriptions that record not just characters but the elements of scholarly apparatus: headers, sections, paragraphs, marginal notes, footnotes, figures, tables, mathematical expressions, and the epistolary skeleton of a letter. It needs those transcriptions validated against a declared grammar so that queries can be written against a stable schema. And it needs provenance: which model produced this transcription, under which instruction, on which date. The Portable Document AI (PDAI) infrastructure addresses these requirements. This paper describes its format, its production pipeline, and the certify-and-learn system that makes a PDAI page a trustworthy unit of Grounding Evidence. Three companion papers carry related material: the orchestration architecture behind the query pipeline is presented in [@grasshoff2026aipartitur]; the linguistic index layer (lemma tables, dependency parses, DuckDB query engine) in [@grasshoff2026aistanza]; and the research platform that reads PDAI pages as evidence for scholarly operations in [@grasshoff2026aierasmus]. The reference layer is documented with its package (`pdai/famulus-reference/README.md`); its addresses are consumed by the justification library famulus-proof (`development/famulus-proof`). # The PDAI Format ## The problem with PDF The PDF standard [@pdf2020] solves a real problem: document appearance is preserved across platforms. This made it the canonical format for digitised historical collections. But the solution creates a different problem for computational scholarship. A scanned PDF encodes position on a page, not structure. A footnote marker in the text and its referent at the page foot share no programmatic connection; a column in a table of astronomical data has no type. Even PDFs with embedded text layers from conventional OCR contain errors, lose structure, and cannot distinguish a marginal annotation from the main text. The consequence is that the scholarly question — what does this document say about this topic — can be asked of a PDF only through full-text string search, which neither understands structure nor tolerates the orthographic variation of historical texts. The TEI guidelines [@tei2023] offer rich semantic encoding, but at a cost: TEI markup is written by human editors and is not automatically produced from scans. For large historical corpora — tens of thousands of pages across dozens of editions — TEI encoding is not feasible. What is needed is automatic production of structured XML from page images, at sufficient quality to serve as Grounding Evidence. ## The PDAI directory structure A PDAI document is a directory on the filesystem. The directory structure separates concerns while maintaining clear relationships between layers: ``` /Volumes/PDAI-1/pdai-collections// status.json readiness manifest (pdai-complete) _corpus.duckdb sentence corpus, lemma table, BM25 (corpus/2) _famulus.duckdb derived tables: FQL, evidence, sentence records, dossier, famulus reference, letters, register gazetteer_*.parquet person, place and domain gazetteer _lake/ the collection's Parquet files in the DuckLake / .pdf original source PDF biblio.xml bibliographic identity (biblio.dtd) pdai-config.json harvest and processing record pages/ page_NNN/ page_NNN.png page image at production resolution page_NNN.txt raw OCR text (fastimage, used as signal only) page_NNN.xml structured PDAI XML (the primary output) segments/ letters/ materialised letter records structure.xml document-level assembly ~/ai-mac-studio/pdai/collections//quality/ _defects.json structural audit results / tag_hierarchy_protocol.jsonl structural fingerprints for regression learning_fault_dossier.json fault cases for the learning loop audit_report_.pdf LettreAI Typst certification report ``` The separation is principled. Source images occupy `pages/`; OCR outputs are co-located with the page. `segments/` holds assembled records that span multiple pages. `structure.xml` is the document-level view. The audit and learning layer `quality/` lives with the collection's code, not with its data: it accumulates without touching production files. This structure embodies two design principles: separation of concerns (no processing step overwrites source data) and accumulative growth (the quality folder grows as auditing deepens, without invalidating transcriptions). ## The canonical page DTD Every PDAI page is validated against `page.dtd` v4.0 (or a collection wrapper of it), maintained at `pdai/pipeline/ai-scholar/src/ai_scholar/schemas/page.dtd`. The DTD is the grammar contract: any query written against the PDAI schema can rely on the elements in @tbl-dtd-categories being present and structurally correct. The page language, together with the collection wrappers described below, covers nine structural categories. Seven are declared by the canonical grammar itself; the epistolary apparatus comes from the letter wrapper `page_letter.dtd`, and the `` element from the wrappers that need it: | Category | Elements | Role in scholarly apparatus | |------------|------------------------------------|--------------------------| | Page frame | ``, ``, `
`, `