KarmanManager

Comprehensive AI Edition package for converting the digitized Collection of Theodor von Karman Nachlass scans to structured XML format.

Overview

KarmanManager processes scanned images from the KarmanArchive and converts them into structured XML with automatic recognition of letter metadata (sender, recipient, date, place). The package uses google-genai for the optional Gemini backend and local Ollama / LM Studio clients for the primary 2026 OCR workflow.

Features

• Archive Scanning: Automatically identifies and processes folders containing JPG images
• OCR Processing: Supports Ollama (including cloud models), Gemini API, and LM Studio for high-quality text extraction
• Letter Recognition: Automatically extracts letter metadata (sender, recipient, date, place)
• Structured XML Output: Generates well-formed XML following ai-ocr format standards
• Batch Processing: Handles entire archive or selected folders
• Karman2026 Mode: Specialized processor for the simplified img/ and xml/ folder structure
• Adaptive Rate Limiting: Intelligent worker pool management to optimize cloud API usage and avoid rate limits
• Real-time Monitoring: Progress bars and speed statistics during processing

Installation

cd /Users/friedrichstr/ai-mac-air/scholarly/projects/ai-karman-manager
uv sync

On Mac Air, KarmanConfig.from_env() auto-detects the Syncthing mirror (/Volumes/golden4T/karman-archive-studio) if the environment variables are not set. On other hosts, export them explicitly.

# Mac Air
export KARMAN_ARCHIVE_PATH="/Volumes/golden4T/karman-archive-studio"
export KARMAN_OUTPUT_PATH="/Volumes/golden4T/pdai-collections/karman-nachlass"
export KARMAN_OCR_TEMP="/Volumes/golden4T/tmp/ai-ocr-karman"

# Mac Studio
export KARMAN_ARCHIVE_PATH="/Volumes/pdai-1/KarmanArchive"
export KARMAN_OUTPUT_PATH="/Volumes/pdai-1/pdai-collections/karman-nachlass"
export KARMAN_OCR_TEMP="/Volumes/pdai-1/tmp/ai-ocr-karman"

Syncthing mirror setup

The Dropbox copy of KarmanArchive on Mac Air is cloud-only and cannot be read locally. The real source (/Volumes/pdai-1/KarmanArchive on Mac Studio) is mirrored to /Volumes/golden4T/karman-archive-studio via Syncthing, managed by the pdai-sync CLI integrated in this workspace (src/pdai_sync/).

1. Apply the sync configuration (drives Mac Studio remotely over Tailscale): bash cd /Users/friedrichstr/ai-mac-air/scholarly/projects/ai-karman-manager uv run pdai-sync apply
2. On Mac Air, wait for sync: bash ./check_karman_archive_sync.sh
3. Ingest into PDAI: bash uv run karman-ingest

Set the Gemini API key if you use the Gemini backend:

export GEMINI_API_KEY="your-api-key"
# or
export GOOGLE_API_KEY="your-api-key"

Usage

Process entire archive (Legacy Format)

karman-manager process \
  --archive-path "$KARMAN_ARCHIVE_PATH" \
  --output-path "$KARMAN_OUTPUT_PATH" \
  --backend ollama \
  --ollama-model qwen3.5:cloud

Process specific folder (Legacy Format)

karman-manager process \
  --archive-path "$KARMAN_ARCHIVE_PATH" \
  --output-path "$KARMAN_OUTPUT_PATH" \
  --backend ollama \
  --ollama-model qwen3.5:cloud \
  --folder "1-20_ANGUS_ROBERT_W_1928-1942"

Process with Karman2026 Structure

For the new simplified structure (img/ and xml/ folders) and the canonical PDAI layout, use the monitored script:

cd /Users/friedrichstr/ai-mac-air/scholarly/projects/ai-karman-manager
export KARMAN_ARCHIVE_PATH="/Volumes/golden4T/karman-archive-studio"
export KARMAN_OUTPUT_PATH="/Volumes/golden4T/pdai-collections/karman-nachlass"
export KARMAN_OCR_TEMP="/Volumes/golden4T/tmp/ai-ocr-karman"
uv run python run_sample_2026_monitored.py

If the Syncthing mirror is not yet complete, generate a structural preview first:

uv run karman-ingest-preview --folder 1-20_ANGUS_ROBERT_W_1928-1942

See docs/KARMAN_PREVIEW_EXAMPLE.md for the expected preview XML output.

See docs/KARMAN_2026.md for the Mac-Air-specific workflow, docs/KARMAN_PDAI_STRUCTURE.md for the full PDAI collection contract, docs/SYNCTHING_KARMAN_ARCHIVE_SETUP.md for the mirror setup, and docs/SYNCTHING_REFERENCE.md for the complete Syncthing architecture reference.

Output Structure

Legacy Format

KarmanArchive_XML/
├── 1-1_ABRAHAM_MAX_1921-1922/
│   ├── 1-1_ABRAHAM_MAX_1921-1922_0001.xml
│   ├── 1-1_ABRAHAM_MAX_1921-1922_0002.xml
│   └── ...
├── 1-2_ACKERET_JACOB_1935-1958/
│   └── ...
└── collection_metadata.xml

Karman2026 Format

karman2026/
├── 1-1_ABRAHAM_MAX_1921-1922/
│   ├── img/
│   │   ├── 1-1_ABRAHAM_MAX_1921-1922_0001.jpg
│   │   └── ...
│   └── xml/
│       ├── 1-1_ABRAHAM_MAX_1921-1922_0001.xml
│       └── ...
└── ...

XML Format

The output XML follows the ai-ocr format:

<page pidx="N">
  <meta model="qwen3.5:cloud" instruction_template="karman_letter_ocr" lang="orig" processed_date="..." />
  <header>...</header>
  <main>
    <letter number="N">
      <from>Sender Name</from>
      <to>Recipient Name</to>
      <place>Location</place>
      <date>Date</date>
    </letter>
    <p>Letter content...</p>
  </main>
  <footnotes>...</footnotes>
</page>

Additional scripts

• setup_karman_archive_sync_mac_studio.py — Deprecated. Use uv run pdai-sync apply instead (drives Mac Studio remotely over Tailscale; no need to run anything on Mac Studio).
• pdai-sync (integrated CLI, src/pdai_sync/) — Syncthing control plane: uv run pdai-sync apply, probe, status --all, set-key, device-id. Config in config.toml.
• check_karman_archive_sync.sh — Checks the Syncthing mirror status on Mac Air.
• wait_for_sync.sh — Polls the Syncthing mirror until it is idle and complete.
• karman-ingest — Ingests the mirrored source archive into the PDAI collection layout.
• karman-ingest-preview — Generates PDAI-conform preview XMLs before the mirror/OCR data is available.
• run_karman_2026_pipeline.sh — End-to-end pipeline: sync-check → ingest → OCR sample.
• restart_ocr_after_sync.sh — Clean previews, wait for sync, ingest, and run OCR (useful after Mac Studio source updates).
• check_pdai_status.sh — Quick status overview of the karman-nachlass PDAI collection.
• clean_previews.sh — Removes preview XMLs and resets registry status before a real OCR run.
• run_pdai_ocr_batch.py — Batch Stage-A OCR over many PDAI documents with adaptive concurrency (--from-file <doc_ids.txt> or --doc-ids a,b,c; defaults to every doc in the collection).

Dependencies

• google-genai: For the optional Gemini backend (never google-generativeai)
• lxml: For XML building
• Pillow: For image processing
• jinja2: For template rendering
• tqdm: For progress bars and monitoring
• requests: For API communication

License

Apache-2.0