ai-dataframe

Fast indexing and filtering for PDAI document collections.

Built on DuckDB (ART-indexed entity lookups, Full-Text Search) with Polars for zero-copy data interchange.

Install

pip install -e .

Quick Start

Python API

from ai_dataframe import PdaiCorpus

# Ingest PDAI documents
corpus = PdaiCorpus("research.duckdb")
corpus.ingest("/path/to/pdai/")

# Filter paragraphs by entities
results = corpus.filter(
    persons=["Kepler"],
    date_range=("1590-01-01", "1600-12-31"),
    text_search="Stipendium",
    limit=50,
)
print(results)  # Polars DataFrame

# Explore entities
corpus.entities.list_entities("person")
corpus.info()

CLI

# Ingest
ai-dataframe ingest /path/to/pdai/ --db research.duckdb

# Statistics
ai-dataframe info --db research.duckdb

# Query with filters
ai-dataframe query --db research.duckdb --person Kepler --date-range 1590-01-01 1600-12-31

# List entities
ai-dataframe entities --db research.duckdb --type person

Architecture

• Parser: Extracts paragraphs from PDAI XML files (pages, letters, paragraph segments)
• DuckDB backend: Three-table schema with ART indexes for sub-millisecond entity lookups
• Entity Registry: Normalized key:value → integer ID system for persons, places, dates
• FTS: DuckDB Full-Text Search with BM25 scoring on paragraph text