PDF Harvest

Fast, massively parallel PDF filename harvester with SQLite caching, built for Python 3.14.

Features

• Massively parallel using Python 3.14's free-threading and subinterpreters
• Intelligent caching with SQLite branch-level tracking
• Flexible patterns with wildcard and alternative matching (no regex overhead)
• Fast string operations for pattern matching
• Multi-root scanning across volumes and directories

Requirements

Python 3.14 or later

Installation

Install Python 3.14 with uv

uv python install 3.14

Create virtual environment and install

cd /path/to/CopernicusPDF
uv venv --python 3.14 .venv
source .venv/bin/activate
uv pip install -e .

Quick Start

Search all volumes for Copernicus PDFs

pdfharvest --root /Volumes --file-name "*Copernic*"

Search with multiple alternatives

pdfharvest --root ~ --file-name "*Copernic*|*Gesamtausgabe*|*Ptolemy*"

Use subinterpreter backend for maximum parallelism

pdfharvest --root /Volumes --file-name "*Copernic*" --backend interpreters

Python 3.14 Features

Free-Threading (No-GIL)

The default thread-based backend leverages Python 3.14's free-threaded builds for true multi-core parallelism. All worker threads execute simultaneously across CPU cores without GIL contention.

Subinterpreters

The --backend interpreters option uses concurrent.futures.InterpreterPoolExecutor to distribute work across isolated subinterpreters, each with its own GIL. This provides true parallelism even on standard Python builds.

Usage

pdfharvest --root /Volumes --file-name "*Copernic*|*Gesamtausgabe*"

Common Options

• --root PATH - Root directory to scan (repeatable)
• --file-name PATTERN - Wildcard pattern for filename stem (use | for alternatives)
• --file-ext EXT - File extension (default: pdf)
• --dir-name PATTERN - Directory name pattern to constrain search
• --case-sensitive - Enable case-sensitive matching
• --workers N - Number of parallel workers (default: CPU count × 4)
• --backend {threads,interpreters} - Execution backend (default: threads)
• --force-rescan - Ignore cache and rescan everything
• --json - Output results as JSON lines

Pattern Syntax

Patterns use * as wildcard:

• *Copernic* - contains "Copernic" anywhere
• Gesamtausgabe* - starts with "Gesamtausgabe"
• *scan - ends with "scan"

Use | for alternatives:

• *Copernic*|*Ptolemy*|*Kepler* - matches any of the three

Caching

pdfharvest maintains an SQLite cache (default: ~/.pdfharvest/cache.db) that records which directory branches have been scanned for which patterns. Subsequent runs with the same pattern skip cached branches, making repeated searches extremely fast.

Force a complete rescan:

pdfharvest --root /Volumes --file-name "*Copernic*" --force-rescan

Documentation

Comprehensive documentation is available in docs/pdfharvest-guide.qmd.

To render as HTML with Quarto:

quarto render docs/pdfharvest-guide.qmd

Architecture

• pdfharvest/cli.py - Command-line interface
• pdfharvest/patterns.py - Fast string-based pattern matching
• pdfharvest/db.py - SQLite caching and persistence
• pdfharvest/walker.py - Parallel directory traversal with thread and interpreter backends

Performance

On Python 3.14 free-threaded builds, pdfharvest achieves near-linear scaling with CPU cores for filesystem scanning. The intelligent branch cache ensures repeated searches complete in milliseconds rather than minutes.

Tested on trees with millions of files across multiple volumes.

Examples

Find all Copernicus manuscripts

pdfharvest \
  --root /Volumes \
  --file-name "*Copernic*|*Copernicus*|*De_Revolutionibus*" \
  --json > copernicus_files.jsonl

Scan only research directories

pdfharvest \
  --root /Volumes \
  --file-name "*" \
  --dir-name "*research*|*papers*"

Maximum parallelism with subinterpreters

pdfharvest \
  --root /Volumes \
  --file-name "*Copernic*" \
  --backend interpreters \
  --workers 64

License

MIT