PDAI XML Schema Reference

DTD definitions for the PDAI document processing pipeline

Overview

AI-Scholar uses DTD (Document Type Definition) schemas to define and validate XML at each stage of the processing pipeline. The schemas form a hierarchy: a base page.dtd for OCR output, extended page schemas for enriched structures, and segment-level schemas for assembled logical units.

All DTD files reside in ai-scholar/src/ai_scholar/schemas/.

Schema Hierarchy

The PDAI schema system has two layers: page schemas (one XML file per physical page) and segment schemas (one XML file per logical unit assembled from multiple pages).

graph TD
    subgraph pageSchemas [Page Schemas — Stage 1 and 2]
        page[page.dtd] --> page_translated[page_translated.dtd]
        page --> page_annotated[page_annotated.dtd]
        page --> page_letter[page_letter.dtd]
        page --> page_section[page_section.dtd]
        page --> page_normalized[page_normalized.dtd]
    end

    subgraph segmentSchemas [Segment Schemas — Assembled Units]
        letter_dtd[letter.dtd]
        section_dtd[section.dtd]
        parno_dtd[parno.dtd]
    end

    page_letter -.->|assembly| letter_dtd
    page_section -.->|assembly| section_dtd
    page -.->|extraction| parno_dtd

PDAI schema hierarchy — page schemas (top) and segment schemas (bottom)

Page Schemas

page.dtd — Base Page (Stage 1)

The foundation schema for all page XML. Produced by the vision-LLM OCR step: one page image yields one XML file.

Exemplar documents: E57, Swerdlow1973, Maestlin1973 (all in pdai-reference)

Root Structure

<page pidx="042">
  <meta model="gemini-3.8-flash"
        instruction_template="full_page_scholarly"
        lang="la" processed_date="2026-02-15"
        page_index="042" />
  <header>CAPUT TERTIUM.</header>
  <main>
    <p><parno>21.</parno> Cum igitur ...</p>
    <figure ref="page_042_fig_1">
      <caption>Fig. 3</caption>
    </figure>
    <marginal side="left" type="line-number">210</marginal>
  </main>
  <footnotes>
    <footnote>Reference citation.</footnote>
  </footnotes>
  <footer>42</footer>
</page>

Elements

Element	Content Model	Description
`page`	`(meta, header?, main, footnotes?, footer?)`	Root: a single physical page
`meta`	`EMPTY`	Processing metadata (self-closing)
`header`	`#PCDATA`	Running header text
`main`	`(p \\| figure \\| marginal)*`	Primary content container
`p`	`(#PCDATA \\| fn \\| parno \\| hi \\| br \\| figure)*`	Paragraph with mixed content
`fn`	`#PCDATA`	Inline footnote marker (content = number)
`parno`	`#PCDATA`	Paragraph number in scholarly editions
`hi`	`#PCDATA`	Inline formatting; `rend` attribute
`br`	`EMPTY`	Line break (verse/lists only, never in prose)
`figure`	`(caption?, description?)`	Figure reference; `ref` links to image file
`caption`	`(#PCDATA \\| hi)*`	Figure caption
`description`	`(#PCDATA \\| hi)*`	Figure description
`marginal`	`#PCDATA`	Marginal annotation
`footnotes`	`(footnote)*`	Container for page footnotes
`footnote`	`(#PCDATA \\| fn \\| p \\| hi)*`	Single footnote entry
`footer`	`#PCDATA`	Printed page number or running footer

Key Attributes

Element	Attribute	Values	Required
`page`	`pidx`	Zero-padded page index	Yes
`meta`	`model`	AI model name	Yes
`meta`	`lang`	Language code	Yes
`meta`	`output_schema`	Schema name	No
`hi`	`rend`	`italic \\| bold \\| spaced \\| superscript \\| subscript`	No
`figure`	`ref`	Image filename	Yes
`marginal`	`side`	`left \\| right`	No
`marginal`	`type`	`line-number \\| folio \\| date \\| reference \\| topic`	No

Notes

LaTeX mathematics appears as plain text: $...$ (inline) and $$...$$ (display). There is no <equation> element.
<fn> contains only the footnote number as text content. There is no data-fn attribute (removed in v3.0).
<marginal> replaces the older <marginalia> element (renamed in v3.0).

page_letter.dtd — Letter Page (Stage 2)

Extends page.dtd for correspondence documents by adding <letter-head> and <letter-closing> to the page content model.

Exemplar documents: kgw-13, Goldbach, kirch-06 (all in pdai-reference)

Instruction template: ai-instructions/structure/letter_tagging.j2

Additional Elements

Element	Content Model	Description
`letter-head`	`(reference?, sender?, addressee?, place?, date?, archive?, incipit?)`	Letter metadata from the edition
`letter-closing`	`(greeting?, place-date?, salutation?, signature?, postscriptum?, address?, marginal?, p)*`	Letter closing section
`reference`	`#PCDATA`	Edition reference number
`sender`	`#PCDATA`	Sender name
`addressee`	`#PCDATA`	Recipient name
`incipit`	`#PCDATA`	Opening words
`greeting`	`#PCDATA`	Closing greeting
`signature`	`#PCDATA`	Signature text
`postscriptum`	`(#PCDATA \\| p)*`	Postscript content
`address`	`(#PCDATA \\| p)*`	Address block

Example (KGW-13, page 38)

<page pidx="038">
  <meta model="gemini-3.8-flash"
        instruction_template="letter_ocr"
        lang="de" processed_date="2026-02-10"
        page_index="038" output_schema="page_letter"
        document_type="correspondence" />

  <letter-head>
    <reference>19. KONRAD VISCHSES AN KEPLER</reference>
    <sender>Konrad Vischses</sender>
    <addressee>Johannes Kepler</addressee>
    <place>Tübingen</place>
    <date>14./24. Mai 1595</date>
    <archive>Pulkowo, Sternwarte</archive>
  </letter-head>

  <main>
    <p>Letter body content...</p>
  </main>

  <letter-closing>
    <greeting>Vale</greeting>
    <signature>Tuus M C Vischses</signature>
  </letter-closing>
</page>

page_section.dtd — Section Page (Stage 2)

Extends page.dtd for monographs and articles by adding hierarchical <section> elements to <main>.

Exemplar documents: E57 (chapters), Swerdlow1973 (article sections), Maestlin1973 (secondary article)

Instruction template: ai-instructions/structure/section_tagging.j2

Additional Elements

Element	Content Model	Description
`section`	`(heading?, (p \\| section \\| figure \\| marginal)*)`	Hierarchical section; may nest
`heading`	`(#PCDATA \\| hi)*`	Section heading text

Section Attributes

Attribute	Description
`level`	Numeric depth (1 = chapter, 2 = section, etc.)
`l`	Legacy level string
`id`	Section identifier (e.g. `chapter_3`)

page_translated.dtd — Translated Page (Stage 3)

Extends page.dtd with translation-specific metadata attributes. The structural elements are identical to page.dtd; only <meta> gains additional attributes.

Exemplar documents: E57 (page_NNN_translation_english.xml), kgw-13 (page_NNN_translation_english.xml)

Instruction templates: ai-instructions/translation/scholarly.j2, ai-instructions/translation/correspondence.j2

Additional Meta Attributes

Attribute	Description
`source_lang`	Original language of the source text
`target_lang`	Translation target language code
`translator_agent`	Agent that performed translation
`validation_status`	`valid` or `invalid`

Example

<page pidx="001">
  <meta model="gemini-3.8-flash"
        instruction_template="scholarly_translation"
        lang="en" source_lang="la" target_lang="en"
        translator_agent="translation-english"
        output_schema="page_translated" />
  <main>
    <p><parno>1.</parno> Every change that occurs in any body...</p>
  </main>
</page>

page_annotated.dtd — Annotated Page (Stage 6)

Extends page.dtd with semantic annotation elements for named entity recognition.

Exemplar documents: No pdai-reference exemplar yet (planned).

Instruction templates: ai-instructions/entities/ner.j2, ai-instructions/entities/person.j2

Additional Elements

Element	Content Model	Description
`entity`	`#PCDATA`	Generic named entity
`person`	`#PCDATA`	Person name
`place`	`#PCDATA`	Geographic location
`date-ref`	`#PCDATA`	Date reference
`citation`	`#PCDATA`	Scholarly citation or reference

page_normalized.dtd — Normalised Page

Simplified schema for cleaned plain text output. This does not extend page.dtd; it has an independent structure optimised for downstream text processing.

Exemplar documents: No pdai-reference exemplar yet (planned).

Segment Schemas

Segment schemas define the structure of assembled logical units — letters, chapters, paragraphs — that are derived from multiple page XML files. Page boundaries are preserved within segment bodies as <page-break pidx="NNN"/> elements.

letter.dtd — Assembled Letter

Schema for a complete letter assembled from one or more page_letter.dtd pages.

Exemplar: kgw-13 segments/letters/letter_001/letter_001.xml

Instruction template: ai-instructions/structure/letter_assembly.j2

Root Structure

<letter id="letter_001" number="1"
        pages="15">
  <letter-meta sender="BÜRGERMEISTER UND RAT VON WEIL"
               addressee="UNIVERSITÄT TÜBINGEN"
               date_original="22. Mai 1590 (a. St.)"
               place="Weil" language="de"
               source_pages="15"
               assembled_date="2026-02-10" />

  <letter-head>
    <reference>1. BÜRGERMEISTER UND RAT VON WEIL AN DIE UNIVERSITÄT TÜBINGEN</reference>
    <sender>Bürgermeister und Rat von Weil</sender>
    <addressee>Universität Tübingen</addressee>
  </letter-head>

  <body>
    <page-break pidx="015"/>
    <p>Letter body content spanning multiple pages...</p>
  </body>

  <letter-closing>
    <greeting>Vale</greeting>
    <signature>Bürgermeister und Rat</signature>
  </letter-closing>

  <editorial>
    <footnotes>
      <footnote source-page="015">Editorial note.</footnote>
    </footnotes>
  </editorial>
</letter>

Key Design

<page-break pidx="NNN"/> preserves provenance: every passage in the segment body can be traced to its source page
<editorial> collects footnotes from all pages, with source-page attribute for citation
The pages attribute on <letter> lists the page range for quick reference

section.dtd — Assembled Section/Chapter

Schema for a chapter or section assembled from page_section.dtd pages.

Exemplar: Planned from E57 chapters.

Instruction template: ai-instructions/structure/section_assembly.j2

Key Elements

Element	Content	Description
`section`	`(section-meta, heading, body)`	Root: assembled chapter
`section-meta`	`EMPTY`	Metadata: title, page range
`body`	`(page-break \\| p \\| section \\| figure \\| marginal \\| footnotes)*`	Content with page provenance

parno.dtd — Assembled Paragraph

Schema for a single numbered paragraph extracted from pages.

Exemplar: E57 segments/paragraphs/parno_001/parno_001.xml

Key Elements

Element	Content	Description
`paragraph`	`(paragraph-meta, body)`	Root: one numbered paragraph
`paragraph-meta`	`EMPTY`	Page range and section cross-reference
`section-ref`	`EMPTY`	Which chapter this paragraph belongs to

famulus-reference/1 tables

Next to the DTDs, which define the XML of pages and segments, every collection has a relational form in schema famulus-reference/1 (pdai/famulus-reference/src/famulus_reference/schema/core.sql). It is stored in <coll>/<coll>_famulus.duckdb and in the collection schema of the DuckLake. The DTDs themselves are stored in fr_dtd.

Table	Content	Relation to the XML
`fr_collection`	title, description, licence, rights holder, DTD id, contract, landing, languages, period	collection descriptor
`fr_source`	kind, holding institution, shelfmark, origin URL, IIIF manifest, licence, sha256, path	source files
`fr_document`	title, creator, date, language, pages, entry type, biblio sha	document folder, `biblio.xml`
`fr_page`	image path, sha256, size, IIIF canvas	`pages/page_NNN/`
`fr_segment`	seg_type (figure, print, instrument, letter, record, inscription …), parent, page range, bbox, DTD element, attrs	DTD object elements, `segments/`
`fr_text`	text layers `ocr`, `diplomatic`, `normalised`, `translation` with sha256	`page_NNN.xml`, translations, `index.duckdb` pages
`fr_entity`, `fr_mention`	persons and places (register `_register`, GND) with evidence grade	annotated entities
`fr_citation`	BibTeX, CSL-JSON, text citation, locator, gaps	`biblio.xml` only
`fr_provenance`	step, tool, model, instruction, input/output sha256, file, run	pipeline logs
`fr_validation`	check, grade `DOCUMENTED`/`INFERRED`/`HYPOTHESIS`/`GAP`, certificate sha256	audit, famulus-proof certificates
`fr_identifier`	scheme, value, resolver (URN, ARK, DOI, GND, shelfmark, IIIF)	—
`fr_dtd`	DTD text, version, sha256, status `draft`/`deployed`/`approved`	the DTD files of this page
`fr_harvest`	datestamp, set, deleted flag	OAI-PMH

A collection DTD drafted by the intake skill (pdai/collections/<coll>/config/page.dtd, wrapping page.dtd v4.0) keeps fr_dtd.status = draft until the owner approves it at the review gate.

Schema Registry

The registry at src/ai_scholar/schemas/registry.yml defines all schemas with their inheritance relationships, enabling automated pipeline validation.

schemas:
  page:
    file: page.dtd
    description: "Base page structure from OCR processing"
    extends: []

  page_letter:
    file: page_letter.dtd
    description: "Letter/correspondence page structure"
    extends: [page]

  page_section:
    file: page_section.dtd
    description: "Section/chapter enriched page structure"
    extends: [page]

  page_translated:
    file: page_translated.dtd
    description: "Translated page with language metadata"
    extends: [page]

  page_annotated:
    file: page_annotated.dtd
    description: "Page with semantic entity annotations"
    extends: [page]

  page_normalized:
    file: page_normalized.dtd
    description: "Normalised text without footnotes"
    extends: []

  letter:
    file: letter.dtd
    description: "Assembled letter from page_letter pages"
    extends: []

  section:
    file: section.dtd
    description: "Assembled chapter from page_section pages"
    extends: []

  parno:
    file: parno.dtd
    description: "Assembled numbered paragraph"
    extends: []

Schema-to-Exemplar Mapping

Every schema should be exercised by at least one document in pdai-reference:

Schema	Exemplar Document	Subcollection	Notes
`page.dtd`	E57, Swerdlow1973, Maestlin1973	enestroem, kepler-prim, kepler-sec, examensarbeiten	All documents have base OCR pages
`page_letter.dtd`	`kgw-13`, Goldbach	kepler-gw, eulerOO	Correspondence documents
`page_section.dtd`	E57, Swerdlow1973, `kgw-03`	enestroem, kepler-prim, kepler-gw	Chapter/section structure
`page_translated.dtd`	E57, `kgw-13`	enestroem, kepler-gw	English translations
`page_annotated.dtd`	—	—	Planned
`page_normalized.dtd`	—	—	Planned
`letter.dtd`	`kgw-13`	kepler-gw	Materialised letters
`section.dtd`	—	—	Planned from E57 chapters
`parno.dtd`	E57	enestroem	130 numbered paragraphs

Pipeline Validation

When chaining agents, the orchestrator validates schema compatibility between pipeline steps:

sequenceDiagram
    participant O as Orchestrator
    participant R as SchemaRegistry
    participant A1 as OCR_Agent
    participant A2 as Translation_Agent

    O->>R: Get A1 output_schema
    R-->>O: page
    O->>R: Get A2 input_schema
    R-->>O: page
    O->>R: is_compatible?
    R-->>O: true
    O->>A1: Execute
    A1-->>O: page XML
    O->>R: Validate against page.dtd
    R-->>O: valid
    O->>A2: Execute
    A2-->>O: page_translated XML

Schema validation during pipeline execution

Compatibility Rules

Same schema: page -> page — compatible
Extended schema: page_translated -> page — compatible (superset)
Incompatible: page_normalized -> page — incompatible (different structure)

CLI Commands

# List all registered schemas
ai-scholar schemas list

# Show schema details
ai-scholar schemas info --schema page_letter

# Validate an XML file against a schema
ai-scholar schemas validate --schema page_letter \
  --file ../pdai-reference/kepler-gw/kgw-13/pages/page_038/page_038.xml

# Validate an entire pipeline
ai-scholar validate-pipeline letter-pipeline