PDAI Collections & Repositories

Machine-portable repository registry for scholarly documents

Overview

PDAI (PDF AI) documents are organized into collections (physical directories on disk) and repositories (logical groupings by scholarly project). The configuration is centralized in ai-platform/config/repositories.yaml and machine-specific volume paths are defined in ai-platform/machines/*.json.

All paths are resolved at runtime by lib/env.py, making the system fully machine-portable.

Collections

Collections are directories containing PDAI document folders. Each machine can have:

Primary collection: {PROJECT_BASE}/pdai/ (Dropbox-synced, available on all machines)
External collections: Machine-specific volumes (e.g., external drives), defined in machines/*.json under storage.volumes.external_pdai

from lib.env import platform_env

# All accessible collection roots
for coll in platform_env.pdai_collections:
    print(f"  {coll} (exists={coll.exists()})")

Repositories

Repositories group documents by scholarly project. Each repository has a name, optional port/domain for web serving, and a list of documents or a prefix pattern for auto-assignment.

Active Repositories

Repository	Name	Port	Domain	Documents	Prefix
`eulerOO`	Euler Opera Omnia	5103	euler-opera.lettre.ai	E57, Goldbach	`E`
`keplerGW`	Kepler Gesammelte Werke	5104	kepler-gw.lettre.ai	KGW-03, KGW-13	`KGW-`
`kirch`	Kirch Astronomical Correspondence	5109	kirch.lettre.ai	(all KIRCH-*)	`KIRCH-`
`copernicusGW`	Copernicus Studies	5105	copernicus.lettre.ai	Swerdlow_1973	—
`humanism`	Humanism and Renaissance	—	—	ActaTominicana1523	—

Planned Repositories

Repository	Name	Port	Domain	Status
`waefler`	Waefler-Hamidi Archive	5106	waefler.lettre.ai	planned
`karman`	Von Karman Archive	5725	karman.lettre.ai	planned
`sundials`	Ancient Sundials (BSDP)	5720	sundials.lettre.ai	planned
`babylonianAstronomy`	Babylonian Astronomy	—	—	planned
`medievalDiagram`	Medieval Diagrams	—	—	planned

Document Assignment

Documents are assigned to repositories by:

Explicit listing: Document ID appears in the repository’s documents list
Prefix matching: Document ID starts with the repository’s prefix string
Fallback: Unmatched documents are assigned to open

from lib.env import platform_env

# Resolve a document's repository
repo = platform_env.resolve_document_repository("E57")      # → "eulerOO"
repo = platform_env.resolve_document_repository("KIRCH-05")  # → "kirch"
repo = platform_env.resolve_document_repository("unknown")   # → "open"

PDAI Document Structure

Each document folder follows the V2 page-centric layout:

{document_id}/
  {document_id}.pdf              # Original PDF
  pdai-config.json               # Format version, metadata, content flags
  manifest.json                  # Segmentation: chapters, paragraphs, page_count
  pages/
    page_000/
      page_000.png               # Page image (full resolution)
      page_000.xml               # OCR XML (conforms to page.dtd v3.0)
      page_000.html              # Rendered HTML
      page_000_translation_{lang}.xml
      page_000_fig_{n}.png       # Extracted figures
  segments/
    {segment_type}/
      {segment_id}/
        {segment_id}.xml         # Segment XML

For the complete specification, see:

Document Structure — Full V2 page-centric layout, naming conventions, manifest schema, processing workflow (Seite noch nicht erstellt)
DTD Schema Reference — XML schemas (page.dtd, page_letter.dtd, etc.), element tables, validation rules

Machine-Specific Volumes

PDF harvesting sources and targets are machine-specific:

# In machines/mac-studio.json
"volumes": {
  "pdf_sources": ["/Volumes/2025Berlin", "/Volumes/Repository"],
  "pdf_target": "/Volumes/2025Berlin/pdf-Sammlungen",
  "external_pdai": []
}

from lib.env import platform_env

# Available PDF sources (filtered to existing)
for src in platform_env.pdf_sources:
    print(f"  {src}")

# PDF sorting target
print(f"Target: {platform_env.pdf_target}")

Famulus reference

Every collection is also a famulus reference: each item has an fr: address, a citation from recorded bibliographic data, provenance and validation records. Details: Famulus reference, DuckLake.

Item	Location / form
`fr:` address	`fr:<collection>/<doc>[/<pidx:04d>[/<segment>]]`, e.g. `fr:hartmann/bsb00028171/0137/print1`; used as famulus-proof `Clause.address`
Famulus DB	`/Volumes/PDAI-1/pdai-collections/<coll>/<coll>_famulus.duckdb` (schema `famulus-reference/1`; rebuildable with `famulus-reference build --collection <coll>`)
Lake folder	`/Volumes/PDAI-1/pdai-collections/<coll>/_lake/` — Parquet files of the collection schema; `_lake/legacy/` for the legacy schema `legacy_<coll>`
Collection code	`pdai/collections/<coll>/config/` — `famulus.yaml`, `page.dtd`, `segments.yaml`
New collections	skill `pdai-collection-intake` (`pdai/skills/pipeline/pdai-collection-intake/`): import folder → inventory → profile → DTD draft → segment plan → review gate → build → famulus DB → lake export. See Skills
Lake writes	mac-studio only (`famulus-reference export`); clients queue job `famulus-lake-export` in `~/syncthink/queue/`

Configuration Files

File	Purpose
`ai-platform/config/repositories.yaml`	Repository definitions, collection paths, harvester settings
`ai-platform/machines/*.json`	Machine-specific volume paths
`ai-platform/lib/env.py`	Runtime path resolution and machine detection
`ai-scholar/src/ai_scholar/schemas/*.dtd`	XML validation schemas