PDAI Document Structure

Page-centric architecture for AI-driven scholarly document processing

Overview

PDAI (Page-Document AI) organises processed scholarly documents in a standardised page-centric structure. Every document follows the same canonical layout: each page of the source PDF (or scanned image collection) becomes a subfolder containing all derivatives of that page — OCR XML, rendered HTML, translations, extracted figures, and agent outputs. On top of this page layer, a segment layer assembles cross-page logical units (letters, chapters, paragraphs) that can be processed and cited independently.

This document describes the canonical PDAI 2.0 structure. For the formal DTD reference, see XML Schemas. For the collection-level organisation, see PDAI Collections.

Design Principles

Page-centric: All derivatives of a page — image, XML, HTML, translations, figures — share the same stem and live together in pages/page_NNN/. The page is the atomic unit of the archive.
Consistent naming: page_NNN[_agent].ext with underscores only (no hyphens in filenames).
Three-digit padding: page_000, page_001, …, page_999.
Segments as derivatives: Segments (letters, chapters, paragraphs) are derived from pages, never the reverse. Every segment references back to its source pages.
Schema-validated: XML at every stage conforms to a DTD from the schema hierarchy in ai-scholar/src/ai_scholar/schemas/.

Canonical Directory Tree

collection/DocumentName/
├── DocumentName.pdf              # Original PDF (or absent for scan sources)
├── manifest.json                 # Document metadata + segment mappings
├── ai-orchestrator.yml           # Agent pipeline configuration (optional)
│
├── css/                          # Display stylesheets
│   └── default.css
│
├── templates/                    # Document-specific Jinja2 overrides
│   └── segmenter.j2
│
├── pages/                        # PAGE LAYER — one subfolder per page
│   ├── page_000/
│   │   ├── page_000.png          # Source image (from PDF or scan)
│   │   ├── page_000.xml          # Base OCR output (page.dtd)
│   │   ├── page_000.html         # Rendered HTML (from XML)
│   │   ├── page_000_fig_1.png    # Extracted figure
│   │   ├── page_000_translation_english.xml   # Translation agent
│   │   ├── page_000_translation_german.xml    # Translation agent
│   │   └── page_000_annotated.xml             # Annotation agent
│   ├── page_001/
│   │   └── ...
│   └── page_NNN/
│
└── segments/                     # SEGMENT LAYER — derived logical units
    ├── letters/                  # Segment type: correspondence
    │   ├── letter_001/
    │   │   ├── letter_001.xml    # Assembled letter (letter.dtd)
    │   │   ├── letter_001.html
    │   │   └── letter_001_translation_english.xml
    │   └── letter_NNN/
    │
    ├── chapters/                 # Segment type: book structure
    │   ├── chapter_001/
    │   │   └── chapter_001.xml   # Assembled chapter (section.dtd)
    │   └── ...
    │
    └── paragraphs/               # Segment type: numbered paragraphs
        ├── parno_001/
        │   └── parno_001.xml     # Assembled paragraph (parno.dtd)
        └── ...

Collection folder

A collection folder on PDAI-1 holds the document folders plus two famulus reference entries:

/Volumes/PDAI-1/pdai-collections/<coll>/
├── <DocumentName>/                 # document folders as above
├── <coll>_famulus.duckdb           # famulus DB, schema famulus-reference/1 (rebuildable)
└── _lake/                          # DuckLake Parquet files of collection schema <coll>
    └── legacy/                     # Parquet files of legacy schema legacy_<coll>

_lake/ is written only by the DuckLake writer on mac-studio; do not edit or move its files. See DuckLake.

Pages: The Primary Layer

Every PDF page becomes a subfolder under pages/. The page is the fundamental unit of the PDAI archive. All processing begins with pages and all derived artefacts (translations, annotations, segments) trace back to specific pages.

File Types within a Page Folder

File Pattern	Schema	Description
`page_NNN.png`	—	Source page image (300–480 DPI)
`page_NNN.xml`	`page.dtd`	Base OCR output
`page_NNN.html`	—	Rendered HTML for viewer display
`page_NNN_fig_M.png`	—	Extracted figure M from this page
`page_NNN_translation_LANG.xml`	`page_translated.dtd`	Translation to target language
`page_NNN_annotated.xml`	`page_annotated.dtd`	Entity annotations

Page Processing Stages

flowchart LR
    IMG["page_042.png"] --> OCR["page_042.xml\n(page.dtd)"]
    OCR --> HTML["page_042.html"]
    OCR --> TRANS["page_042_translation_english.xml\n(page_translated.dtd)"]
    OCR --> TAG["page_042 enriched\n(page_letter.dtd or page_section.dtd)"]
    OCR --> ANNOT["page_042_annotated.xml\n(page_annotated.dtd)"]
    IMG --> FIG["page_042_fig_1.png"]

Processing stages for a single page

Segments: The Derived Layer

Segments are logical units of a document — letters, chapters, numbered paragraphs — that span one or more physical pages. They are derived from the page layer through a two-step process:

Segmentation: The segmenter agent analyses the full document XML and produces manifest.json, which maps pages to logical segments.
Materialisation: The materialise command reads the manifest, extracts content from the relevant page XMLs, and writes assembled segment XML to segments/{type}/{type}_NNN/.

Why Segments Matter

Segments enable AI clients to process complete logical units rather than arbitrary page fragments. A letter that spans pages 15–17 of a printed edition can be processed as a single XML document. The segment XML preserves page provenance through <page-break pidx="NNN"/> markers, so every passage in the segment can be cited by its publication page number.

Segment Types

Type	Schema	Typical Source	Example
`letters`	`letter.dtd`	Correspondence editions	`kgw-13`, Goldbach
`chapters`	`section.dtd`	Monographs, treatises	E57
`paragraphs`	`parno.dtd`	Numbered scholarly texts	E57 (130 paragraphs)
`documents`	—	Diplomatic acts	ActaTominicana1523
`sections`	`section.dtd`	Journal articles	Swerdlow_1973

Page Provenance in Segments

Every segment body contains <page-break/> markers that record which page each passage originates from:

<letter id="letter_002" number="2" pages="16-17">
  <body>
    <page-break pidx="016"/>
    <p>Content from page 16...</p>
    <page-break pidx="017"/>
    <p>Content continuing on page 17...</p>
  </body>
  <editorial>
    <footnotes>
      <footnote source-page="016">Note from page 16.</footnote>
      <footnote source-page="017">Note from page 17.</footnote>
    </footnotes>
  </editorial>
</letter>

This design enables precise citation: “Letter 2, found on pp. 16–17” — referencing the original publication pages, not arbitrary file indices.

Segment Translations

Translations can exist at both the page level and the segment level:

Page-level translations (pages/page_NNN/page_NNN_translation_english.xml) translate each page independently. They are produced first, during page-level agent processing.
Segment-level translations (segments/letters/letter_001/letter_001_translation_english.xml) translate the complete logical unit. They can be produced either by re-translating the assembled segment or by assembling page-level translations using the manifest page mappings.

The relationship between these two levels:

flowchart TD
    subgraph pageLevel [Page Level]
        P16["page_016_translation_english.xml"]
        P17["page_017_translation_english.xml"]
    end

    subgraph segLevel [Segment Level]
        L2["letter_002_translation_english.xml"]
    end

    P16 -->|assemble via manifest| L2
    P17 -->|assemble via manifest| L2

Relationship between page-level and segment-level translations

manifest.json

The manifest file maps pages to segments and contains document metadata. It is generated by the segmenter agent and validated by the assess command.

Structure

{
  "document": {
    "name": "kgw-13",
    "title": "Johannes Kepler Gesammelte Werke, Band XIII",
    "language": "de",
    "page_count": 446
  },
  "segments": {
    "parts": {
      "type": "edition",
      "items": [
        {"id": "briefe", "title": "Briefe 1590-1599",
         "type": "main_content", "pages": [14, 15, ..., 384]}
      ]
    },
    "letters": {
      "type": "correspondence",
      "schema": "letter_collection",
      "numbering": "arabic",
      "items": [
        {"id": "letter_001", "number": 1,
         "sender": "BÜRGERMEISTER UND RAT VON WEIL",
         "addressee": "UNIVERSITÄT TÜBINGEN",
         "date": "22. Mai 1590 (a. St.)",
         "pages": [15]}
      ]
    }
  }
}

Segment Item Fields

Each segment item must include:

id — unique identifier within the segment type
pages — list of page indices (0-based) this segment spans
number — sequential number for validation (if the segment type has numbering)

Additional fields depend on the segment type (e.g. sender, addressee, date for letters).

Naming Conventions

Content Type	Pattern	Example
Page image	`page_NNN.png`	`page_042.png`
Page OCR	`page_NNN.xml`	`page_042.xml`
Page HTML	`page_NNN.html`	`page_042.html`
Agent output	`page_NNN_agentname.xml`	`page_042_translation_english.xml`
Figure	`page_NNN_fig_M.png`	`page_042_fig_1.png`
Segment	`segtype_NNN.xml`	`letter_001.xml`
Segment agent	`segtype_NNN_agent.xml`	`letter_001_translation_english.xml`

Rules:

Underscores only — no hyphens in filenames (agent names normalised: translation-english becomes translation_english)
Three-digit zero-padded page numbers (000–999)
Figures numbered per page: fig_1, fig_2, etc.

Key Locations

Resource	Path
DTD schemas	`ai-scholar/src/ai_scholar/schemas/`
Instruction templates	`../ai-instructions/` (centralised)
OCR templates	`../ai-instructions/ocr/`
Structure templates	`../ai-instructions/structure/`
Translation templates	`../ai-instructions/translation/`
Reference collection	`../pdai-reference/`
Path resolution API	`ai_scholar.pdai_structure.PDAIStructure`
Famulus DB	`<coll>/<coll>_famulus.duckdb`
Lake files	`<coll>/_lake/` (catalog `pdai-collections/_lake/famulus.ducklake`)

Processing Workflow

flowchart TD
    subgraph ingestion [1. Ingestion]
        PDF[PDF or Scans]
        PAGES["pages/page_NNN/page_NNN.png"]
    end

    subgraph ocr [2. OCR Processing]
        XML["pages/page_NNN/page_NNN.xml"]
    end

    subgraph segmentation [3. Segmentation]
        MANIFEST[manifest.json]
    end

    subgraph pageAgents [4a. Page-Level Agents]
        AI_PAGE["pages/page_NNN/page_NNN_agent.xml"]
    end

    subgraph materialise [4b. Materialisation]
        SEGMENTS["segments/type/type_NNN/"]
    end

    subgraph segAgents [5. Segment-Level Agents]
        AI_SEG["segments/type/type_NNN/type_NNN_agent.xml"]
    end

    subgraph export [6. Export]
        HTML_OUT["pages/page_NNN/page_NNN.html"]
    end

    PDF --> PAGES
    PAGES --> XML
    XML --> MANIFEST
    XML --> AI_PAGE
    MANIFEST --> SEGMENTS
    XML --> SEGMENTS
    SEGMENTS --> AI_SEG
    XML --> HTML_OUT

Complete PDAI processing workflow

CLI Commands

# Process PDF into PDAI structure
ai-scholar process ../pdai-reference/enestroem/E57

# Generate manifest with segmentation
ai-scholar segment ../pdai-reference/enestroem/E57

# Assess manifest for sequence validity
ai-scholar assess ../pdai-reference/enestroem/E57

# Materialise segments
ai-scholar materialise ../pdai-reference/eulerOO/Goldbach --segment-type letters

# Page-level agent
ai-scholar agent ../pdai-reference/enestroem/E57 translation-english --pages 0-50

# Segment-level agent
ai-scholar agent ../pdai-reference/eulerOO/Goldbach summarize-letter --segment letters

# Migrate legacy V1 structure
ai-scholar migrate ../pdai-reference/enestroem/E57 --dry-run

Viewer Integration

The AI-Viewer reads documents from the page-centric structure:

cd /path/to/2026project
python -m http.server 4100
open http://localhost:4100/ai-viewer/viewer.html?doc=pdai-reference/enestroem/E57

The viewer uses manifest.json for segment navigation, pages/page_NNN/page_NNN.html for content display, and css/ for display format switching.