---
title: "The Famulus data layer"
subtitle: "One corpus file and one derived file per collection, measured"
format:
  arxivai-typst:
    keep-typ: true
    book: false
author:
  - name: Gerd Graßhoff
    affiliations:
      - name: Philosophisches Institut, Humboldt-Universität zu Berlin
        city: Berlin
        country: Germany
      - name: Forschungsinstitut des Deutschen Museums
        city: München
        country: Germany
    orcid: 0000-0002-4450-357X
    email: gerd.grasshoff@hu-berlin.de
abstract: |
  The PDAI collections of Famulus were served from 2,655 DuckDB files (63.5 GB), a ChromaDB
  vector store per collection (17.6 GB) and per-document intermediates. Most of it repeated the
  same data: every token was stored three to four times, the paragraphs of a correspondence
  three times, the dossier facts three times, and 29.6 GB of per-document databases fed a corpus
  that one parse of the page documents yields directly. This paper describes the layer that
  replaced it: per collection one corpus file, rebuilt from the Stanza page documents and
  carrying a BM25 index, and one file for everything derived from it; two small stores that
  services write at query time stay apart. On 28 collections the new corpus file reproduces
  every table of the old one, row for row, or the difference was traced to a stale old file
  (among them 454 pages of the Euler–Goldbach correspondence that the old incremental build
  never added). The large corpus files are 5–9 times smaller and build in seconds to minutes; of the
  76 recorded SQL statements of the FQL bench 72 return the same rows, and the other four differ for
  stated reasons. The vector store was removed.
  On 200 paraphrase probes BM25 with a character 4-gram index for German and French finds the
  source sentence in the top 50 for 0.830 of them, the vector store did for 0.895, and in the
  top 10 for 0.725 against 0.890; that loss is stated, not hidden.
bibliography: bibliography.bib
---

# The starting point, measured

Famulus answers questions to scholarly collections from evidence it can address: a page, a
sentence, a letter, a figure [@grasshoff2026famulusreasoning; @grasshoff2026aischolar]. Each
collection on the PDAI volume had grown a database per purpose. A measured inventory of the
volume on 20 October 2026 (@tbl-inventar, left columns; the script `dblayer_inventar.py` walks the
volume, nothing is estimated) counted 2,655 DuckDB files [@raasveldt2019duckdb] with 63.5 GB,
a ChromaDB store per searchable collection with 17.6 GB, 145,656 Stanza page documents
[@qi2020stanza] with 19.0 GB and as many CoNLL-U exports with 3.2 GB.

A read-only survey of the code that writes and reads these files found three kinds of
overhead. *Duplication*: a token was a row of the corpus table `stz_tokens`, again a row of
`stz_deps` (one row per token), again in `lemma_index.parquet`, again in the document's own
`index.duckdb`, and its morphological features once more as one row per feature in `stz_feats`;
in karman-nachlass the dossier facts stood in the dossier projection, in the famulus-reference
file and in the lake. *Dead structures*: tables written by a builder and read by no code
(`fq_segments`, `fq_tag_topic`, `fx_formen`, `sr_concept_up`, `do_nennung`, `do_querverweis`, the
postings of an early aggregation step). *Hazards*: a step that wrote a document's page text
deleted the whole per-document file and with it the parsed tables; the evidence-space tables were
written into the published FQL file, which the next FQL publication replaced; and the
multiprocessing pool of the Stanza batch waited forever when a worker process died.

# Principles

1. **One spine, derived elsewhere.** The linguistic spine of a collection — documents, pages,
   page text, sentences, tokens with their dependency and features, named entities, the lemma
   index and the full-text index — is one file, `<c>_corpus.duckdb`. Everything built from it
   or from the page XML for answering, proving and displaying is one other file,
   `<c>_famulus.duckdb`.
2. **Rebuildable from files.** The corpus file is built from the Stanza page documents and the
   page XML alone; no database is its input. A full build replaces it; a delta replaces the
   changed documents.
3. **Publication by swap.** Every writer works on a copy (an APFS clone, which costs nothing until
   blocks change) and replaces the published file atomically after its own gates passed. A reader
   never sees a half-written file, and a failed build leaves the published file untouched.
4. **One lock per collection.** Corpus runs take an exclusive lock on the collection, writers of
   single pages (OCR) a shared one; two Stanza runs on one collection can no longer interleave.
5. **Explicit gaps.** A missing index or store is an error with a name, never an empty result.
6. **No model text in the evidence.** The page text of the corpus is the printed text of
   `<main>`; descriptions written by a model (`description`, `figDesc`, `meta`) do not enter it.
7. **Short-lived connections.** Every reader opens read-only for one operation.

Two stores do not fit principle 3 and stay files of their own: the LWgrammar usage store, whose
search counts the search service increments while answering, and the concept cache of ai-query,
which it fills at query time. A file that is only ever replaced whole cannot take such writes.

# The corpus file

`<c>_corpus.duckdb` (corpus/2) has one token table. Dependency head, its lemma and part of speech,
the feature string, character offsets and the NER tag are columns of `stz_tokens`; the former
tables `stz_deps` and `stz_feats` are views over it, so every reader keeps its SQL. The lemma
index is a table `stz_lemma` with the six columns of the former parquet file, physically ordered by
lemma: a lookup reads only the row groups whose minimum and maximum enclose the lemma, the same
pruning the parquet file gave, without a second file that can fall out of step. `stz_page_text`
holds the printed text of every page, which the famulus-reference build had read from the
per-document databases. No ART index is built; the tables are written in locus order and
DuckDB's zone maps prune.

Full-text recall is a BM25 index [@robertson2009bm25] over the sentence text and the lemmatised
sentence (`lemma_text`), built with DuckDB's full-text extension. Its default tokenizer keeps only
the letters a–z; the index is built with a pattern that keeps every letter and digit, so Greek and
accented Latin are indexed. Compound words defeat word-level BM25: a paraphrase says "Theorie der
Grenzschicht" where the source says "Grenzschichttheorie". For German and French a second index
over character 4-grams of the words is kept, and its ranking is fused with the word ranking by
reciprocal rank fusion.

The build reads the page documents of a collection in parallel, appends them through one writer,
sorts, builds the lemma table and the two indexes, and checks gates before the swap: documents,
pages, sentences and tokens equal the counts of the page documents; `stz_lemma` has one row per
token with a lemma; each index has one row per indexed sentence; no sentence key repeats. A delta
copies the published file forward, replaces the changed documents and rebuilds the lemma table
and the indexes; above about a tenth of the documents a full build is cheaper and leaves no free
blocks behind.

Two defects of the old pipeline were found on the way and fixed at their source. A worker that
died left `Pool.imap_unordered` waiting for its task forever; the batch now runs on a process pool
that reports a dead worker, re-runs the jobs that were in flight one by one, and records a page
that kills a worker twice as an error with its name. The page that killed workers on
20 October was a subject index whose text had no sentence punctuation: Stanza received a
"sentence" of 5,525 words, whose dependency parse needs memory quadratic in its length.
Sentence segmentation now cuts any unit above 200 words at word boundaries into near-equal
pieces.

# The famulus file

`<c>_famulus.duckdb` holds the tables of the FQL sidecar (`fq_*`), the evidence space (`fx_*`), the
sentence records and their normal form (`sr_*`, `fnf_*`), the dossier projection (`do_*`), the
famulus reference (`fr_*`) and the correspondence tables of the collection's main database
(`letters_resolved`, the `ent_*` projection of the register). Its builders stay what they were;
each keeps its private build state where it needs one (the FQL build keeps the cache of its
language-model answers in a work file that never leaves the machine) and hands its tables over:
the tables of the handed-over file replace the same-named tables of the famulus file, the other
builders' tables stay untouched, scalar macros come along and the BM25 indexes of the handed-over
tables are rebuilt. The defect of the evidence tables disappears with this: they are no longer
written into a file that another builder replaces.

The merge of the existing stores into the famulus file checks every copied table's row count
against its source and leaves behind only tables no code reads (@tbl-zusammenfuehrung). A table
that existed in two sources would have been recorded as a conflict; none did.

Which file a role of a collection resolves to is decided in one place, `ai_lemma_index.stores`,
from a recorded state per collection (`legacy` or `migrated`, separately for the corpus and the
famulus file), never from which files happen to exist. The readers were moved to it in nine code
bases (ai-lemma-index, famulus-proof, ai-query, ai-viewer, famulus-reference, karman-manager,
LWgrammar, famulus-logic and the orchestrator); a ratchet test fails when a store name is written
literally again.

# Evaluation

**Disk.** @tbl-inventar compares the volume before and after the switch of all 28 collections with a
Stanza corpus. DuckDB files fell from 2,655 with 63.5 GB to 298 with 10.8 GB; the vector stores (17.6 GB)
and the CoNLL-U exports are gone. One per-document database whose write-ahead log DuckDB cannot replay
(pirckheimer-sec, `jahrbuch-2012`) was kept and is named in the cleanup report. Every removed file was first written to a tar archive outside the
synchronised tree, read back and compared by checksum.

**Parity.** @tbl-umstellung lists the switch. For 23 collections every table of the old corpus
file is equal in the new one — the row count and an order-independent checksum of all columns,
computed in one process — and so is the lemma index. Four collections differed, and in each case
the old file was the stale one: in eulerOO one page had been re-parsed after the old file was
built; in causality the lemma parquet dated from August; and in eulerOO-IVA the old corpus held
only 802 of the 1,256 pages of the Euler–Goldbach volume, because its incremental build added
missing documents but never missing pages of a document already present; and waefler-prim's old
file still held a document whose folder had been removed from the collection. The new file contains
the 454 missing pages, 8,416 sentences that no search could find before. karman-context had no
complete old corpus to compare.

**Speed.** A full build of the largest corpus, karman-nachlass (1,066 documents, 8.7 million
tokens), took 65 s without and 93.5 s with the German 4-gram index on a loaded machine; a delta
of one document 14–17 s. @tbl-latenz gives lookup latencies measured on both files before the old
one was removed: the lemma table answers in the time of the parquet file (p95 3.0 against 5.3 ms
on karman-nachlass) and the dependency view faster than the old table (23.7 against 49.4 ms).
Feature filters through the `stz_feats` view unnest every token and are three to five times slower
than the old table; filtering the token column `feats` directly runs at the old table's speed
(30–45 ms against 45 ms on karman-nachlass); the sentence-record build and the usage reading of the
evidence space read the column, two measurement scripts of the notebook still use the view. The shipped BM25 stage answers with a p95 of
25 ms (Latin, enestroem), 33 ms (French, hevelius), 38 ms (English) and 60 ms (German,
karman-nachlass, with the 4-gram fusion).

**Answers.** The 76 SQL statements recorded by a run of the FQL bench on the old stores were
replayed with their recorded bindings on the new ones (@tbl-fql). 72 return the same rows. Two
differ between two runs on the old stores as well (an unordered limit); the statement that lists
the collection's FQL tables returns fewer tables, because the dead ones were left behind.

**Recall without embeddings.** The vector store was the recall path for paraphrases. Before it
was removed, 200 sentences (50 each in Latin, German, English and French from four collections)
were paraphrased by gemini-3.8-flash, and each engine was asked to find the source sentence from
its paraphrase (@tbl-paraphrase). The bge-m3 store [@chen2024bgem3] found it in the top 50 for
0.895 of the probes; word-level BM25 for 0.795, with German at 0.62; adding the 4-gram index for
German and French raises BM25 to 0.830 overall and German to 0.74. In the top 10 the loss is
larger: 0.725 against 0.890. The second embedding store, EmbeddingGemma 2, had been built only for
a few collections and recalled almost nothing. The vector store is not kept for the remaining
difference: the 4-gram index closes half of it, and the paraphrase recall that matters for the
proofs of @grasshoff2026famulusreasoning runs over lemma and register bindings, not over
similarity.

# Limits

The measurements of speed were taken while other processes parsed and indexed collections on the
same machine; they are upper bounds. The paraphrase probe measures known-item retrieval of one
sentence, not the recall of all sentences that express a proposition. The store of the shared concept ontology is still
copied into each collection's sentence records. The answer check of ai-query delivered 11 of 14
golden questions on the new layer; for the three that failed the store queries return the same
rows on the old and the new files, so the failures lie in the routing of the questions, but
there is no run of these questions on the old layer from the same day to compare.

# Tables

{{< include _tabellen/inventar.md >}}

{{< include _tabellen/umstellung.md >}}

{{< include _tabellen/zusammenfuehrung.md >}}

{{< include _tabellen/latenz.md >}}

{{< include _tabellen/paraphrase.md >}}

{{< include _tabellen/fql.md >}}