famulus-logic

Benchmark for classifiers that judge whether a probing expression P stands in a logical relation to any sentence of a list drawn from the PDAI collections — and in which relation. Consequence is not restricted to material implication: conceptual rules (Junggeselle → unverheirateter Mann), synonymy, hyponymy, converse relations, calendar arithmetic and geographic inclusion all count, provided the rule is a rule of meaning and not an unstated fact.

Every sentence and every probe must be an Aussage (true or false).

Pipeline

PDAI page XML / fresh index.duckdb
   │  harvest      split, Aussage pre-filter, provenance (path, OCR model, sha256, mtime)
   ▼
data/sentences.json
   │  probe        LLM: confirm Aussage · probes for each target relation · near misses
   ▼
data/benchmark.json        sentences with `attributes` = gold labels per probe
   │  bench        probe × [source sentence + distractors] → classifier
   ▼
runs/bench.duckdb          `runs` (one row per run: error matrix, wall time, classifier properties)
                           `predictions` (one row per pair) · view `leaderboard`
runs/<run_id>.json         full run record

Files

Labels: relation of S (corpus sentence) to P (probe)

Each label is defined by the truth-value combinations of (S, P) that are impossible; classifiers judge the four cells and the label follows.

Families for the coarse matrix: follows (first three), opposes (next three), none.

Near misses

Probes built to look like a relation (mimics) while their gold label is another one: converse direction, overlap without rule, false synonym, role swap, quantifier shift, modal shift, temporal shift, entity substitution, negation scope, contrary-as-contradictory, attitude drop, missing premise. The trap rate is the share of near misses a classifier labels as the relation they mimic.

Usage

uv sync --extra dev
uv run famulus-logic relations

# 1) sentences (default root $PDAI_ROOT or /Volumes/golden4T/pdai-collections, canonical layout)
uv run famulus-logic harvest --collections kepler-prim karman-nachlass --per-doc 200
#    local snapshots while the drive is offline:
uv run famulus-logic harvest --root ~/ai-mac-air/pdai/collections \
  --glob 'kirch/quality/dual-read/*/page_*.xml' 'hevelius/pilot/*/page_*.xml' --per-doc 150

# 2) probes (cached per sentence; resumes after interruption)
uv run famulus-logic probe --model deepseek-v4.1-flash:cloud --limit 500 --workers 8

# 3) runs
uv run famulus-logic bench --classifier lexical
uv run famulus-logic bench --classifier fast --split 0.7
uv run famulus-logic bench --classifier llm:kimi-k3:cloud --mode list --decide cells
uv run famulus-logic report                    # leaderboard
uv run famulus-logic report --run <run_id>     # error matrix of one run
uv run famulus-logic audit --run <run_id>      # disagreements → runs/audit.jsonl for gold review

LLM backend: any OpenAI-compatible endpoint; default local Ollama (FAMULUS_LOGIC_LLM_BASE=http://localhost:11434/v1, FAMULUS_LOGIC_LLM_KEY).

The fast classifier (TF-IDF + logistic regression, sentences held in an in-memory DuckDB table) needs the optional fast extra: uv sync --extra fast.

What a run records

runs row: classifier name, version, model, mode (pair/list), decision (cells/label), all properties (temperature, seed, prompt-template sha256, relations sha256), benchmark sha256, task parameters, environment (host, python, git commit + dirty flag) — and the results: accuracy, macro/weighted F1, Cohen's κ, the fine 7×7 and coarse 3×3 error matrices, related-vs-independent F1, accuracy on positives vs. near misses, trap rate, distractor false-related rate, list-level detection F1 / localization / exact-list accuracy, wall time, pairs/s, latency p50/p95, tokens in/out, error counts.

Documentation

docs/famulus-logic.qmd (German; HTML + PDF, lettreai theme) describes the catalog, the pipeline, the metrics and the pilot results. Tables come from scripts/docs_generate.py; refresh after every run:

uv run python scripts/docs_generate.py
cd docs && quarto render famulus-logic.qmd --to lettreai-html
QUARTO_TYPST=$(which typst) quarto render famulus-logic.qmd --to lettreai-typst

Argument reconstruction and the three-sentence test

A second layer on top of the pairwise relations: inferential contribution. Does the premise pair (A, B) support a conclusion C, and how? The judge marks the eight truth-value cells of (A, B, C). The label (linked, convergent, a_suffices, restates_a, a_necessary, linked_against, … 17 in all) is computed from them as prime implicants, with impossible premise combinations as don't-cares. For n premises, n + 2 cells decide minimal sufficiency.

On top of that test, argue reconstructs the argument DAG of a section the way a minimal theory is built in the regularity theory of causation:

• nodes are sentences (split into units where a sentence carries its own inference);
• edges are minimally sufficient premise sets;
• the thesis is the sink that organises most of the section;
• sentences outside every minimal set are excluded, with a reason.

The full account, with Euler's E57 §§ 9–10 as the worked example, is in docs/argument-reconstruction.md.

uv run famulus-logic support
uv run famulus-logic triad --judge llm:kimi-k3:cloud                       # tests/fixtures/triads.json
uv run famulus-logic triad --items examples/e57/triads.json --judge llm:kimi-k3:cloud
uv run famulus-logic argue --section examples/e57/section.json --reading Q --distractors   # oracle
uv run famulus-logic argue --section examples/e57/section.json --reading Q --distractors \
    --proposer llm:deepseek-v4.1-flash:cloud --judge llm:deepseek-v4.1-flash:cloud

Caveats

• Gold is generated. Probe labels come from an LLM; gold_status is generated when its four-cell judgement matches its label, otherwise inconsistent_cells (excluded from runs). Use audit and set gold_status to verified/human after review. Do not benchmark the generator model against its own gold without saying so.
• Distractor gold is assumed (independent). Same-document distractors (assumed:same_doc) are harder and occasionally truly related; they are reported separately in metrics.distractors.
• PDAI integrity. harvest uses index.duckdb only if it is not older than the page OCR XML of its document; otherwise it reads the XML and counts index_stale. Sentence text is never altered beyond whitespace.