--- title: "Golden Queries in AI Erasmus" subtitle: "A testbed for scholarly inquiry: query forms, grounding, and the monitor log" running-title: "Golden Queries in AI Erasmus" book: false toc: true author: - name: Gerd Graßhoff email: gerd.grasshoff@hu-berlin.de orcid: 0000-0002-4450-357X affiliations: - name: Humboldt-Universität zu Berlin department: Philosophisches Institut address: Unter den Linden 6 city: Berlin country: Germany postal-code: 10099 - name: Deutsches Museum department: Forschungsinstitut address: Museumsinsel 1 city: München country: Germany postal-code: 80538 abstract: | A research platform is only as trustworthy as the questions it can be held to. This paper presents the *golden queries* of AI Erasmus — a canonical set of thirteen scholarly questions, each with a declared form of inquiry, a grounding document, and an oracle — and the *monitor log* that makes every answer inspectable. A golden query is a research question whose expected answer is declared in advance, so the pipeline is audited against a reference rather than against a model's fluency. Two kinds of oracle are in force: a *count oracle*, a verified number, or a narrow band around one, that the pipeline must match, and a *floor oracle*, a register-grounded minimum a healthy run must clear. We walk the thirteen canonical goldens across four corpora — Euler, Kepler, Pirckheimer, and von Kármán — showing seven forms of inquiry: lemma co-occurrence, attestation, person mention, correspondence roster, letter subject, place-and-person lookup, and concept statements. For each we show how the question is grounded in a document layer, and we reproduce the monitor log — segmented section by section, each paragraph explaining the goal solution with its strengths and limits. We close with the reliability metrics the golden set makes possible: recall of the general filter wherever a sentence-level must-include set exists, the independent DuckDB audit of the ground tier, and the record of the gates that have held the set — the audits of an engine since retired, and the routing gate and delivery gate of the live chat path, measured again on 20 October 2026. We then show the *learning loop* that leads from a monitor-log deficit to a versioned patch — worked through a real case, a misresolved person name, in which the A/B gate could not decide and the patch was promoted by hand, on the record. The set is designed to grow: new queries are introduced, tested, and promoted as the platform's scope widens, and existing goldens are revisited when the corpus changes beneath them. keywords: - digital humanities - golden queries - query grounding - monitor log - PDAI - lemma search - correspondence networks - reliability metrics bibliography: bibliography.bib format: arxivai-typst: default --- # Introduction: The Question of Trust A scholar's instrument earns trust the way a workshop does: by being held to account. In Erasmus's printing house the amanuenses copied, collated, fetched, and counted, and the judgement about what a variant meant never left the desk. The modern instrument that AI Erasmus builds [@grasshoff2026aipartitur; @orchestrator2026] staffs those repeatable moves with specialists — but a specialist that answers fluently is not the same as a specialist that answers *right* [@grasshoff2026aierasmus]. The difference is the subject of this paper. Meaning, in the lemma queries that follow, is use under a citable rule [@grasshoff2026lwgrammar; @wittgenstein1953]: a co-occurrence of *lex* and *natura* is a fact about pages, not a claim that Kepler wrote of a *lex naturae*. The problem is the asymmetry of failure. A lemma search that returns nothing is honest: "no letters from this sender in this range" is scholarship's ordinary weather. A generated answer is never empty; pressed for something it does not know, a model produces fluent material with no page behind it, and the failure is invisible precisely where it is worst. The platform's answer to this asymmetry is *provenance* — every result carries the page addresses it rests on, the method that produced it, and the scope it was computed over. But provenance alone is not enough. A method trace tells you *how* an answer was produced; it does not tell you whether the answer was *right*. ```{=typst} #arxivai.concept-box(title: [The golden-query stance])[ A golden query is a research question together with a form of inquiry, a grounding document, and an oracle. An oracle is the declared expected answer against which a run is checked — not the human label later assigned to the run as a whole. Two kinds are in force: a count oracle (a verified number) and a floor oracle (a register-grounded minimum). The golden set is the platform's objective: the learning loop maximises the fraction of golden queries whose oracle is satisfied. Without goldens, an answer is judged by fluency; with them, it is judged by accountability. ] ``` This paper is the full statement of that testbed. It makes five contributions: 1. **The basic approach.** What a golden query is, how its two kinds of oracle are declared, and why the golden set is the objective of the learning loop (Chapter 2). 2. **The query forms.** The seven forms of inquiry the canonical set exercises — lemma co-occurrence, attestation, person mention, correspondence roster, letter subject, place-and-person lookup, and concept statements — each with its evidence path (Chapters 3–4). 3. **Grounding.** How each question is grounded in a document layer, and what the grounding documents are (Chapter 5). 4. **The monitor log.** The mapping function that turns a run into a readable, sectioned protocol, reproduced and explained paragraph by paragraph, with the strengths and limits of each section (Chapter 6), and the reliability metrics the set makes possible (Chapter 7). 5. **The learning loop.** How an annotated deficit in the monitor log becomes a versioned, A/B-gated patch to the pipeline — the mechanism by which the fraction of goldens whose oracle is satisfied is to be raised, and the record of what it has done so far (Chapter 8). The protocol for extending the set is Chapter 9. The set is designed to grow. New queries are introduced, tested against their oracles, and promoted into the canonical set as the platform's scope widens. The paper closes on what the golden set changes in the craft of building a scholarly instrument. # The Basic Approach: Golden Queries as the Objective ## What a golden query is A golden query is a research question together with three things that distinguish it from an ordinary request to the server: - a *form of inquiry* — the kind of scholarly act the question asks for (attesting a lemma, counting correspondents, extracting statements about a concept); - a *grounding document* — the corpus layer that can answer it (lemma index, letter register, gazetteer, sentence index); - an *oracle* — the declared expected answer against which a run is checked. A *query family* is the grouping of goldens that share one form of inquiry and one evidence path. The form is the act; the family is the set of goldens that exercise it. The oracle is the load-bearing part: it is what turns a query run from a demonstration into a test. Two kinds of oracle must not be collapsed. A **count oracle** is a verified number, derived from the register, that the pipeline must match — `n_letters: 11` for ground-pirckheimer-2 — or a narrow band around such a number: the register count for ground-pirckheimer-1 is 75 correspondents, and its oracle accepts 70 to 85. A **floor oracle** is a register-grounded minimum, recorded in the golden's `truth` note, that a healthy run must clear and a broken route must fail — `n_sentences.min: 30` for R1, whose register union on 12 August 2026 is 86 sentences. A third thing is not an oracle at all: the **run verdict** (`correct`, `partial`, `incorrect`) that a human labeller assigns to the run as a whole after reading a sample of top sentences. The learning loop maximises the fraction of runs whose oracle is satisfied; the labeller's verdict is the human check on whether that satisfaction was the right scholarly answer. The canonical set of thirteen goldens is indexed in a manifest [@grasshoff2026aiquery]. Each entry names its tier, its corpus, its query, and its role: ```{=typst} #arxivai.score-box( "version: 2 name: euler-kepler-pirckheimer-karman corpora: [enestroem, eulerOO, kepler-prim, kepler-sec, pirckheimer, karman-nachlass] goldens: - id: R1 tier: catalog corpus: [enestroem, eulerOO] query: \"Sentences in Euler Enestroem with lemma lex natura\" role: lemma cooccurrence (lex + natura) - id: ground-pirckheimer-1 tier: ground corpus: [pirckheimer] query: \"How many correspondents did Pirckheimer have between 1520 and 1530?\" oracle: {n_unique_correspondents: {min: 70, max: 85}} role: verified correspondence-count ground", id: "goldens/manifest.yaml", path: "canonical set (13)", marks: ( (line: 3, note: [Six corpora, four centuries, three languages — one set of operations.]), (line: 7, note: [The catalog tier: lemma co-occurrence over Euler's Eneström corpus #cite().]), (line: 15, note: [The ground tier: a band around the verified count of 75 correspondents.]), ), plays-on: "ai-query golden audit", ) ``` The tiers matter, and they must not be read as "shape versus number". As of August 2026 every canonical golden carries an oracle of one of the two kinds above. The **catalog** tier (R1, R2, R4, R5) holds lemma and person queries whose oracles are floors on sentence counts, grounded in a direct probe of the lemma index. The **ground** tier (ground-pirckheimer-1, -2) holds queries whose oracles are verified facts, derived from the Pirckheimer letter register [@pirckheimerBriefwechsel]. The **karman** tier (K1–K5) holds queries over the von Kármán archive [@karmanFindingAid], with floor oracles on correspondent or letter counts. The **concept** tier (T4-kepler-motrix, T4-euler-aether) holds statement-extraction queries whose oracles are floors on the number of grounded statements. A catalog golden tests whether the pipeline understands the question *and* clears a register floor; a ground golden tests whether it matches a register-derived number. The manifest holds a fourteenth entry, `euler-goldbach`, outside the canonical thirteen: a routing-only golden whose oracle is a grounding verdict, not an answer (Chapter 9 returns to it). ## Why the golden set is the objective The learning loop of AI Erasmus maximises the aggregate golden score — the fraction of golden queries answered correctly or partially. This is a deliberate choice of objective. It means the platform is not optimised to sound scholarly; it is optimised to answer the questions a scholar actually asks, as judged against a reference. The choice has a consequence that is worth stating plainly. A golden query that the pipeline cannot answer is not a failure to be hidden; it is a finding. When the monitor shows a golden returning the wrong subtask, or a specialist returning an empty set where the oracle expects a count, that is the learning loop's raw material. The golden set turns the pipeline's mistakes into a ledger that can be read, attributed to a subagent, and patched. ```{=typst} #arxivai.method-box(title: [The verdict is assigned to the run, not the item])[ A golden item is an assessment *query run* that a human has evaluated and labelled. The run verdict — correct, partial, or incorrect — is always assigned to the run as a whole, never to individual sentences or documents in the result set, and it is not the oracle. A query returning 500 sentences is still a single run with a single verdict. The labeller reads 5–15 top sentences to form a judgement of the run, not all 500. ] ``` ## The pipeline the goldens exercise Every golden is run through the same path an ordinary request to the server takes: the Partitur `ai-query-phase1`, played by the Conductor's harness in the function behind the chat endpoint [@grasshoff2026aipartitur]. The first stages read the question: one instructed model decode turns it into entity spans and a structured filter record, the kind of question follows from that structure by declared rules, and persons, places, dates, and the collection scope are resolved against the registers. Later steps depend on what these stages decided: a correspondence question routes to a register specialist, a lemma question to the index, a person question to the gazetteer. Until late August 2026 the goldens were also audited on an older engine, an AND-OR planner (`orchestrate()`) that has since been retired; Chapter 7 keeps the measurements of the two paths apart. The golden set is deliberately spread across these paths. Some goldens exercise the fast lemma trail; others exercise the register specialists; still others exercise the entity gazetteer. The spread is the point: a golden set that only tested one path would certify only one path. # The Query Families and Their Forms of Inquiry A scholarly question, phrased in ordinary language, is routed by *decomposition* — the classification of the question into a family with a declared evidence path, read by declared rules from the question's decoded structure. The golden set exercises the families that matter most for historical and conceptual research. | Family | Form of inquiry | Example golden | Evidence path | |--------|-----------------|----------------|---------------| | Lemma co-occurrence | Sentences where two lemmata appear together | R1, R2, K5 | Lemma index (boolean AND) | | Attestation | Whether a single lemma appears at all | R4 | Lemma index (single lemma) | | Person mention | Passages where a named person is typed as a person | R5, K3 | Person gazetteer + letter bodies | | Correspondence roster | Who wrote to whom, or how many letters, in a year range | K1, K2, ground-pirckheimer-1 | Resolved letter register | | Letter subject | What a set of letters is about | ground-pirckheimer-2 | Letter register + body text | | Place-and-person lookup | Letters bound to a place and a person together | K4 | Place gazetteer + letter register | | Concept statements | Grounded statements about a concept, with implications | T4-kepler-motrix, T4-euler-aether | Sentence index + concept-statements specialist | : The query families exercised by the golden set: each family is the set of goldens that share one form of inquiry and one evidence path. {#tbl-families} @tbl-families is a working map, not a closed ontology. Its point is the third column: every family names the layer it reads, so every answer arrives knowing where it came from. The golden set is a sample of these families — a sample chosen to cover the paths the platform must not get wrong. # The Golden Set: Thirteen Queries, Four Corpora {#sec-golden-set} This chapter walks the thirteen canonical goldens. For each we state the query, its form of inquiry, its grounding, and its oracle — a count where the oracle is a verified fact, a floor where it is a register-grounded minimum. The monitor log of one of them is reproduced in full in Chapter 6. ## Lemma co-occurrence: R1, R2, K5 The lemma co-occurrence query asks for sentences where two lemmata appear together. It is the workhorse of conceptual research: "where do *lex* and *natura* co-occur?" is the question behind a claim about how a concept is used. ```{=typst} #arxivai.operation-box( title: [Lemma co-occurrence], question: [Sentences in Euler Enestroem with lemma lex natura], steps: ( (agent: "fasttext-classify", does: [classifies the question as a co-occurrence query — intent=cooccurrence, scope=sentence]), (agent: "collection-select", does: [fixes the scope to the Eneström corpus — the score's iron rule keeps scope terms out of any content filter]), (agent: "stanza-filter", does: [attests the lemmata lex and natura in the lemma index and retrieves every co-occurring sentence with its page]), (agent: "assemble", does: [groups the evidence into an answerable shape, every item with a locus]), ), answer: [a set of sentences where lex and natura co-occur, each with its page address — the raw material of a claim about usage], ) ``` The grounding is the lemma index, built by Stanza under Universal Dependencies [@qi2020stanza; @nivre2020ud] and stored in DuckDB for interactive query [@raasveldt2019duckdb]: formerly the file `lemma_index.parquet`, since 20 October 2026 the table `stz_lemma` of the collection's corpus file `_corpus.duckdb` [@grasshoff2026famulusdatalayer]. The oracle for R1 is a floor, not a single exact count: the pipeline must produce `lemma_ncf = [[lex],[natura]]` with the right collections and at least 30 co-occurring sentences. The register truth of 12 August 2026 is 86 sentences (enestroem 70 in 34 documents, eulerOO 16 in 7). R2 is the same concept scoped to Kepler [@keplerGW], testing fan-out across `kepler-prim` and `kepler-sec`, with a floor of 15 against a register truth of 45 in kepler-prim and 0 in kepler-sec. K5 is the same shape over the von Kármán archive, exercising the general filter on a corpus without a `letter_pages` table, with a floor of 100 against a register truth of 1,426 sentences. The form's specific difficulty is the *iron rule*: a collection name must never enter the lemma filter. "Lex natura in Kepler's writing" must produce `lemma_ncf = [[lex],[natura]]` with `collections=[kepler-prim]`, never `lemma_ncf = [[lex],[natura],[kepler]]`. The catalog golden R2b, kept outside the canonical thirteen, is the regression test for exactly this disambiguation — the "Lex natura in Kepler's writing" bug that once treated the corpus scope as a co-occurring lemma. ## Attestation: R4 The attestation query asks whether a term appears at all. It is the negative control of scholarship: "does vacuum appear in Euler's writings?" is a question whose honest answer may be "no". ```{=typst} #arxivai.operation-box( title: [Attestation], question: [Does vacuum appear in Euler's writings?], steps: ( (agent: "fasttext-classify", does: [classifies the question as an attestation query — intent=attestation, scope=sentence]), (agent: "stanza-filter", does: [probes the lemma vacuum against the lemma index of the Eneström corpus]), (agent: "assemble", does: [returns the verdict — found or absent — with the evidence]), ), answer: [a verdict on whether vacuum is attested, with the sentences that attest it — or an honest empty answer], ) ``` The form's specific virtue is that it tests the pipeline's honesty. An attestation query whose answer is "absent" must return an empty result with its trace, not a plausible guess — the amanuensis stance applied to the negative case. R4 itself is a positive case: the register truth of 12 August 2026 is 55 sentences with *vacuum* (enestroem 46, eulerOO 9), and its oracle is a floor of ten sentences, so a pipeline that answered "absent" would fail it. A canonical golden for the negative case, a term the corpus does not attest, is still missing from the set. ## Person mention: R5, K3 The person-mention query asks where a person is named in a text. It is the first step of prosopography: "where is Tycho mentioned as a person in Kepler?" is the question behind a reading circle or a reception study. ```{=typst} #arxivai.operation-box( title: [Person mention], question: [Where is Tycho mentioned as a person in Kepler?], steps: ( (agent: "fasttext-classify", does: [classifies the question as an entity query — intent=entity, scope=sentence]), (agent: "person-classify", does: [resolves the name Tycho against the person gazetteer and types it as a person]), (agent: "gazetteer-entity", does: [maps the surface form to the registry entity and probes the corpus]), (agent: "lemma-execute", does: [retrieves the sentences where the person is named, with their pages]), ), answer: [the passages where Tycho is named as a person, each with its page — the raw material of a prosopographical claim], ) ``` The form's specific difficulty is *typing*. "Tycho" must be resolved as a person, not treated as a bare lemma. The golden R5 tests that the gazetteer-entity subagent appears in the trail and that the entity is typed correctly. K3 is the same shape over the von Kármán archive: "who mentions Einstein in his letters to Karman?" routes to the `mention-in-letters` specialist, which reads the letter register and the letter bodies. ## Correspondence roster: K1, ground-pirckheimer-1 The correspondence-roster query asks who wrote to whom, within a range of years. It is the heart of network research: "who corresponded with Karman between 1930 and 1940?" is the question behind a correspondence network. ```{=typst} #arxivai.operation-box( title: [Correspondence roster], question: [How many correspondents did Pirckheimer have between 1520 and 1530?], steps: ( (agent: "fasttext-classify", does: [classifies the question as a correspondence-count query — intent=roster]), (agent: "person-classify", does: [resolves Pirckheimer against the person gazetteer and sets wanted_role=correspondent]), (agent: "date-classify", does: [extracts the year range 1520–1530]), (agent: "correspondence-roster", does: [reads the resolved letter register, filters by year, and counts distinct correspondents]), ), answer: [75 distinct correspondents between 1520 and 1530 — the register count around which the oracle's band of 70 to 85 is set], ) ``` The form's specific virtue is that it has a *count oracle*. The register count for ground-pirckheimer-1 is 75 distinct correspondents, and the oracle accepts a band of 70 to 85 around it. An independent DuckDB query of `letters_resolved` — `COUNT(DISTINCT other_id)`, excluding the main person and unknown senders, years 1520–1530 — returns the same 75, and still did when it was repeated on 20 October 2026. That query is independent of the orchestrator and of any model call; it is not independent of the register schema, and the audit notes record that it matches the specialist's year-window predicate. What it certifies is that the number 75 is a fact about the table, not an artefact of the pipeline's routing. K1 is the same form of inquiry over a different corpus and a floor oracle. The query is *"Who corresponded with Karman between 1930 and 1940?"* The register truth of 12 August 2026 is 494 raw names / 493 canonical ids in that window; the floor is 200, set so that canonicalization merges never fail a healthy run and a broken route cannot clear it. The orchestrator audit of 24 August 2026, on the retired engine, returned 491 distinct correspondents and an oracle verdict PASS; Chapter 7 reports what the live path returns. ## German letter-count: K2 K2 is the correspondence-roster form asked in German, as a letter count rather than a who-list: *"Wie viele Briefe schrieb Millikan an Karman?"* It is the golden the previous walkthrough omitted, and it is in the canonical thirteen. ```{=typst} #arxivai.operation-box( title: [German letter-count], question: [Wie viele Briefe schrieb Millikan an Karman?], steps: ( (agent: "fasttext-classify", does: [classifies the German question as a correspondence-count query — intent=roster]), (agent: "person-classify", does: [resolves Millikan and Kármán against the person gazetteer]), (agent: "correspondence-roster", does: [reads the resolved letter register and counts letters from Millikan to Kármán]), ), answer: [n_letters in the band 3–25 — the floor-and-ceiling oracle that rejects both a zero-hit failure and a corpus-wide inflation], ) ``` The register truth of 12 August 2026 is 4 letters under a strict sender-Millikan / recipient-Kármán filter, 16 Millikan-sent letters overall, many with an empty recipient, and about 22 with Robert A. Millikan in either role. The oracle band 3–25 accepts strict, recipient-lenient, and either-role counting, and rejects the 42 letters of a corpus-wide count. The audit run of 24 August returned `n_letters = 10` and an oracle verdict PASS. The form's specific difficulty is the German cue path together with a directed count: the pipeline must not treat the question as an English who-list and must not inflate the count to the whole archive. ## Letter subject: ground-pirckheimer-2 The letter-subject query asks what a set of letters is about. It is the synthesis question: "what is the subject of the letters from Schöner between 1520 and 1527?" is the question behind a summary of a correspondence. ```{=typst} #arxivai.operation-box( title: [Letter subject], question: [What is the subject of the letters from Schöner between 1520 and 1527?], steps: ( (agent: "fasttext-classify", does: [classifies the question as a letter-subject query — intent=roster]), (agent: "person-classify", does: [resolves Schöner and sets wanted_role=correspondent]), (agent: "date-classify", does: [extracts the year range 1520–1527]), (agent: "letter-subject", does: [reads the register, retrieves the letters, and summarises their subject]), ), answer: [a summary of the letters' subject — the oracle requires the topic saphea (the astrolabe) to appear], ) ``` The form's specific difficulty is that the oracle is a *loose substring* check, not an exact number. The expected answer requires `n_letters: 11` and a `subject_summary` containing "saphea" — the astrolabe topic that dominates the Schöner correspondence. The loose check is deliberate: prose synthesis is not expected to be byte-identical between runs, but the load-bearing fact (the topic) must appear. ## Place-and-person lookup: K4 The place-and-person query combines a place and a person: "letters sent from Pasadena by Millikan" is the question behind a study of where a correspondent wrote from. ```{=typst} #arxivai.operation-box( title: [Place-and-person lookup], question: [Letters sent from Pasadena by Millikan], steps: ( (agent: "fasttext-classify", does: [classifies the question as a correspondence query]), (agent: "person-classify", does: [resolves Millikan and sets wanted_role=correspondent]), (agent: "place-classify", does: [resolves Pasadena against the place gazetteer]), (agent: "correspondence-roster", does: [reads the register, filters by place and person, and returns the letters]), ), answer: [the letters sent from Pasadena by Millikan, each with its page], ) ``` The form's specific difficulty is the *joint resolution* of place and person. The golden K4 tests that Pasadena resolves as a place and Millikan as a person, and that the two bind together in the register filter. The partitur benchmark pins the expected binding: `place_binding: pasadena` and `letter_ent_name: millikan`. The register truth of 12 August 2026 is 13 letters under a sender-Millikan / place-Pasadena filter, 19 if Millikan is allowed in either role; the oracle band is 5–25. The known limitation, recorded in the manifest, is that "sent BY Millikan" direction is not enforced. K4 is also the golden that the routing gate of 20 October 2026 got wrong: it named the subtask `mention-in-letters` where the manifest expects `correspondence-roster` (Chapter 7). ## Concept statements: T4-kepler-motrix, T4-euler-aether The concept-statements query asks what an author *says* about a concept, and what those statements imply. It is a different form of inquiry from lemma co-occurrence: R1 returns sentences where *lex* and *natura* co-occur; T4-kepler-motrix asks for grounded statements about the *vis motrix* and their implications. The retrieval unit is the statement, not the sentence, and the specialist is `concept-statements`. ```{=typst} #arxivai.operation-box( title: [Concept statements], question: [What statements does Kepler make about the vis motrix, and what do they imply?], steps: ( (agent: "fasttext-classify", does: [classifies the question as a concept-statements query]), (agent: "collection-select", does: [fixes the scope to kepler-prim — Kepler is corpus scope, not a co-occurring lemma]), (agent: "concept-statements", does: [segments grounded statements about vis motrix from the sentence index and records their implications]), ), answer: [at least five grounded statements from at least three documents — the floor oracle against a register truth of 145 motrix-anchored sentences in 15 kepler-prim documents], ) ``` T4-euler-aether is the same form in German, over Euler: *"Welche Aussagen macht Euler über den Äther?"* It exercises the German cue path ("Aussagen über X") and Latin-corpus seed resolution from a German concept name. The register truth of 12 August 2026 is 269 `aether%` sentences across 15 enestroem documents. For both T4 goldens the floor is five statements from three documents, resting on at least twenty sentences. The audit of 24 August 2026 returned oracle PASS for both (T4-kepler-motrix: 7 statements, 26 documents; T4-euler-aether: 43 statements, 5 documents). One limit belongs in the same paragraph as the result. The floor oracle checks counts, not the prose of the synthesis. In the same audit run the T4-kepler-motrix synthesis reported a single matched document for a query it rendered as "in kepler-prim, kepler-sec", with no lemma in it, while the specialist returned seven statements from 26 documents. That mismatch is a finding for the learning loop, not a silent pass. # Grounding: Documents and the Grounding Locus Every golden query is grounded in a document layer. The grounding document is the layer that answers the question — the lemma index for a co-occurrence query, the letter register for a roster query, the gazetteer for a person query. The grounding *locus* is the specific address — collection, doc_id, page index — that a claim rests on. ## The grounding layers The platform's substrate is the PDAI page: one raster image and one structured transcription under a single address [@grasshoff2026aischolar]. Above the pages, derived layers accumulate: - **The lemma index** (`stz_lemma` in `_corpus.duckdb`, formerly `lemma_index.parquet`) — one row per token, built with Stanza, stored in DuckDB [@grasshoff2026aiindex; @grasshoff2026famulusdatalayer]. Grounds co-occurrence and attestation queries over the Eneström, Kepler, and Euler corpora [@enestroem1910; @keplerGW; @eulerOOIV]. LWgrammar writes OED-form usage entries from that index [@oed; @grasshoff2026lwgrammar]. - **The letter register** (`letters_resolved`) — the edition's metadata of sender, recipient, and date per letter. Grounds correspondence roster and letter-subject queries [@pirckheimerBriefwechsel; @karmanFindingAid]. - **The person and place gazetteers** — canonical entities with their variant forms. Ground person-mention and place-and-person queries. - **The stanza index** (`stz_sentences`) — the sentence layer, with dependency parses [@grasshoff2026aistanza]. Grounds the sentence-level evidence and the concept-statements family. Each golden names its grounding layer in its definition. R1 grounds in the lemma index; ground-pirckheimer-1 grounds in the letter register; R5 grounds in the person gazetteer. The grounding is what makes the oracle checkable: the independent audit re-runs the query directly against the layer, bypassing the pipeline, and compares. ```{=typst} #arxivai.method-box(title: [The grounding contract])[ Every answer carries its grounding locus: the collection, the doc_id, and the page index it rests on. The chain from answer back to page is never broken. This is what allows the golden set to print numbers and mean them — and what allows the independent audit to check them. ] ``` ## The independent audit The ground-tier check the golden set offers is an *independent audit* of the register, not a second run of the pipeline. For ground-pirckheimer-1 the audit runs `COUNT(DISTINCT other_id)` over `letters_resolved` and confirms the count oracle of 75. For ground-pirckheimer-2 it confirms 11 letters and the saphea topic. The audit does not call the orchestrator and does not call a model. It does use the same table, and for the year window it uses the same predicate as the specialist — the audit notes say so. What it answers is whether the oracle is a fact about the source layer. What it does not answer is whether the specialist's code is free of that predicate. ground-pirckheimer-3 is in that audit and not in the canonical thirteen. It is the candidate used as the worked example of the learning loop in Chapter 8: four correspondents mention Georg Hartmann in letters to Pirckheimer, against thirteen index-cited letters. The gates of Chapter 7 therefore run two canonical ground goldens; @tbl-independent reports three rows because the candidate was audited alongside them. # The Monitor Log and Its Mapping Function {#sec-monitor-log} The monitor log is the readable record of a golden run. It is produced by a single script, `query_monitor.py`, which plays the Phase-0 Partitur once, selects the answering Partitur from the register, hands the understanding to the orchestrator, and writes the shared trail as numbered sections §1–§11. The mapping function is the heart of it: it turns the raw event stream and the goal tree into a sectioned protocol that a reader can follow. ```{=typst} #arxivai.method-box(title: [The mapping function])[ The monitor is not a separate diagnostic system and not a second pipeline. It is the readable imprint of the one execution path. The runner plays Phase 0 once, selects the answering Partitur, and writes the shared trail as numbered sections. The iron rule: never invent or paraphrase a monitor log. The log is the deliverable. ] ``` The sections are the anatomy of one answer. §1 shows the Phase-0 Partitur being played — the classification, the person and place resolution, the collection selection. §2 shows the Partitur selection from the register. §3 prints the full goal text of the chosen Partitur. §4 shows the signals — persons, years, collections. §5 is the execution stream, the chronological protocol of every event. §6 is the annotated goal tree, joining the plan to the run. §7 lists unbound execution. §8 shows the LLM calls. §9 is the answer and goal check. §10 is the verdict. §11, added after the run reproduced below, gathers the findings of the whole run into one report. ## The monitor log, section by section We reproduce the monitor log of a golden run of 6 August 2026 (run `22f1a524055b4f72`) — the mention-in-letters query over Pirckheimer — and explain each section, with its pros and cons. The run used `gemini-3.5-flash-lite`, then the default model; since 20 October 2026 the default is `gemini-3.8-flash`. The query is: ``` Which correspondent mentions Georg Hartmann in his letters to Pirckheimer? ``` ### §1 — UNDERSTAND: the Phase-0 Partitur, played The first section is the richest and the longest, deliberately, because Phase 0 carries the entire routing decision. It shows the fixed Partitur `ai-query-understand` (registered today under the identifier `query-understanding`) being played — not as a Python call sequence, but as a score execution. The section shows the setting (the context the query was asked in), the goal text verbatim from the register, the stages played with their guards and budgets, the full rendered instruction of the classifier, the attribute glossary, and the routing hints. The stages played tell the story of the classification: ``` ✓ fasttext-classify pdai:aiq_classify 1502.6ms 2500 ok ✓ person-classify pdai:aiq_person_classify 0.1ms 80 ok ✓ place-classify pdai:aiq_place_classify 0.0ms 50 ok ○ date-classify pdai:aiq_date_classify 0.0ms 200 skipped ✓ collection-select pdai:aiq_collection_select 4.9ms 40 ok ``` The classifier returns `subtask_type=mention-in-letters` with confidence 0.95. The person classifier resolves nine persons. The date classifier is skipped — there is no temporal token in the query. The collection selector fixes the scope to `pirckheimer`. **Pro.** The section makes the routing decision fully inspectable. A reader can see which classifier answered, with what confidence, and which stages were skipped and why. The SHA check confirms the disk YAML matches the register. **Contra.** The section is long, and it repeats textbook material on every run — the specialist catalogue and the full instruction. For repeated runs this is redundant. The specialist definition is also slightly inaccurate for some subtasks: it says a specialist answers "from a structured register, not from lemma retrieval", which is not true of `terminology-find`, which attests lemmata over the lemma index. ### §2 — PARTITUR SELECTION The second section shows the register choice. Candidates come only from `partitur_register.duckdb.partiturs`, filtered by goal-path prefix and term score. Skills are not searched. ``` • goal_path prefix: query/historical/ • excluded kind: understanding • selected ai-query-phase1 @ query/historical/stage-one (term_score=1.767) candidates: → ai-query-phase1 score=1.767 query/historical/stage-one ``` **Pro.** The demarcation is stated every time: candidates only from the register, Phase-0 scores never answer. The candidate list shows the alternatives that were *not* chosen. **Contra.** The section shows the score but not the reasoning. A reader sees that `ai-query-phase1` won, but not *why* the root Partitur beat a dedicated specialist Partitur. The decision should be commented, not just scored. ### §3 — PARTITUR TEXT The third section prints the full goal text of the chosen Partitur from the register — not truncated, not paraphrased. ``` Beantworte eine historische Fachfrage aus belegter Evidenz: Artikuliere zunächst, was die Frage verlangt und welche Dokumente überhaupt als Antwort infrage kommen (Grounding-Primärquellen vs. Sekundärrepositorien). Filtere diese Evidenzmenge dann — immer billigst und selektivst zuerst — auf die minimale Menge relevanter Passagen. Erst dieses kleine Resultat wird in die volle LLM-Anfrage injiziert. Jede Behauptung im Ergebnis trägt einen Grounding-Locus (collection, doc_id, pidx). ``` **Pro.** Completeness here is correct. Without §3 the selection is an ID; with it, the goal of the entire execution is readable: specify the evidence set, filter cheaply, inject only the remainder into the LLM, ground every claim. **Contra.** None material. Full goal text is the right default for this section. ### §4 — SIGNALS The fourth section shows the condensed Phase-0 output that all later stages read as a common board — without re-parsing the query. ``` persons = ['Pirckheimer, Willibald', 'Pirckheimer, Klara', ..., 'Sauermann, Georg'] years = {} collections = ['pirckheimer'] ``` **Pro.** The three signals — persons, years, collections — are the shared board. The section is the right headline. **Contra.** The section is too flat. The persons list shows only the canonical name strings, not the provenance — the surface term, the entity id, the role match. This is the most common review objection to §4. In this run, the token "Georg" collides with "Sauermann, Georg", so "Georg Hartmann" is not resolved as a bigram — an entity-resolution gap that the flat list hides. ### §5 — EXECUTION STREAM The fifth section is the chronological protocol of every event in the AO* run: `node_open`, `node_solved`, `subagent_start`, `subagent_end`, `decision`, `route`. It is the timeline. ``` [ 71] step= 7 info node_open route 10.6ms open LEAF:execute mention-in-letters [ 72] step= 8 info subagent_start mention-in-letters 0.0ms mention-in-letters starting [ 73] step= 8 info decision mention-in-letters 221.7ms mention-in-letters returned n_out=14 [ 74] step= 8 info subagent_end mention-in-letters 221.7ms mention-in-letters done in 221.7ms [ 76] step= 8 info node_open mention-in-letters 224.2ms open LEAF:assemble final answer [ 77] step= 9 info subagent_start assemble 0.0ms assemble starting [ 78] step= 9 info decision assemble 51.9ms verdict=found groups=14 ``` **Pro.** The stream is the process trace with milliseconds and intermediate decisions. Without it, cache hits, skip reasons, and order would be invisible. Structurally identical runs are collapsed to a single line, which keeps the log readable. **Contra.** The event data JSON is often truncated to 220 characters — half readable, half redundant with §6. And when the metapartitur cache disagrees with the classifier, the stream shows the contradiction honestly but does not explain it. ### §6 — ANNOTATED GOAL TREE The sixth section is the join of the register goal tree (PLAN) and the actual run (RUN). It is the argumentative form — *this* goal was pursued, *that* alternative was rejected, *this* leaf is missing. ``` ▸ [AND] ai-query-phase1#root PLAN promising RUN ok ├─ ✓ analyse PLAN promising RUN ok 27ms in=1 out=1 ├─ ▸ [OR] ai-query-phase1#answer-from-register PLAN promising RUN ok │ ├─ ○ correspondence-roster PLAN promising(unused OR-alt) RUN not-taken │ ├─ ✓ mention-in-letters PLAN promising RUN ok 222ms in=1 out=14 │ ├─ ○ source-finder PLAN promising(unused OR-alt) RUN not-taken ├─ ✓ goal-check PLAN promising RUN ok 43ms in=1 out=1 ``` **Pro.** This section shows the OR alternatives that were not taken, the leaves that ran, and the plan-run agreement at the OR node. The `!! PLANNED-NOT-RUN` and `!! CONTRACT-MISS` markers make gaps visible. **Contra.** The `PLANNED-NOT-RUN` markers mix genuine gaps with substituted paths — a leaf that was skipped because a specialist substituted for it reads the same as a leaf that was genuinely forgotten. The monitor should distinguish "really forgotten" from "substituted by a specialist". ### §7 — UNBOUND EXECUTION The seventh section lists steps that ran but have no register goal node under the answering Partitur — plus the Phase-0 steps bound to the Understand score. ``` bound to the Phase-0 score (ai-query-understand, played once in §1): • understand status=ok ms=1704.41 • classify status=ok ms=1507.52 • metapartitur status=ok ms=2256.84 in=11 out=1 • partitur status=ok ms=348.22 in=1 out=11 ``` **Pro.** The section makes the seam visible: Phase 0 and the metapartitur are execution reality but not leaves of `ai-query-phase1`. Without §7 they would appear in §6 as ghosts or vanish. **Contra.** The list should carry a binding recommendation for each unbound step — "belongs to Phase-0", "engine-internal control node", or "genuine register gap". ### §8 — LLM CALLS The eighth section lists every model invocation with its phase tag, model, stage, milliseconds, error, and prompt head. ``` n_calls = 2 (phase-0=1 execute=1) active_model=gemini-3.5-flash-lite [00] phase=phase-0 model=gemini-3.5-flash-lite backend=gemini stage=fasttext-classify ms=1464.2 err=None [01] phase=execute model=gemini-3.5-flash-lite backend=gemini stage=metapartitur ms=2207.9 err=None ``` **Pro.** Cost, model choice, and prompt provenance are all visible. The prompt head proves that no inline prompts were used — the SKILL.md path is in the text. **Contra.** For audits the response head is often missing — the monitor shows the prompt but not the first characters of the model's answer. ### §9 — ANSWER / GOAL CHECK The ninth section condenses the answer side: specialist fields, counts, goal-check score, and violated stage contracts. ``` specialist = mention-in-letters total_count = 4 assemble_verdict = found goal_satisfied = True goal_score = 0.7 stage contracts = 5/6 stages met - correspondence-roster: FAILED roster_has_correspondents: contract check failed top_entries: - Schöner, Johann (count=8) - Savorgnanus, Petrus (count=3) ``` **Pro.** The goal gate answers "was what was asked delivered?" The contracts from the Partitur register check the blackboard keys. **Contra.** The section must show the answer itself more strongly — the term list with its grounding loci and viewer URLs — not just the metrics. And the contract failures are often schema noise: contracts belonging to the *correspondence-roster* specialist fail on a *mention-in-letters* run, because the goal check evaluates all register contracts, not just those of the active specialist. ### §10 — VERDICT The tenth section is one line of truth. ``` resolution = CLOSED (partitur=ai-query-phase1, leaves=30, missing=0) execution = 31 steps, 96 events, 2 llm calls, 3953 ms join = ran=24 planned-not-run=4 ran-not-planned=6 phase-0=2 answer = count=4 goal_satisfied=True goal_score=0.7 ``` **Pro.** The closing judgement without scrolling. `CLOSED` means goal resolution had no missing leaves — independent of whether the answer satisfied the goal. **Contra.** The distinction between resolution-closed and goal-satisfied is correct and important, but it should be clearer in the badge — `[closed]` versus `[goal-fail]`. ## What the monitor does well, and where it falls short The monitor is a very good *observation* instrument and not yet a complete *explanation* instrument. It does well at: one source of truth (script output, not ad-hoc dump); Partitur before execution (text, choice, signals, then stream); the PLAN × RUN argumentative form; the specialist path without fake validation; and making term learning visible. It falls short at: §4 persons without provenance (two "Schöner" entries unexplained); no answer evidence with `collection/doc_id/pidx`; the metapartitur-cache-versus-classify contradiction shown but not commented; and contract failures of the wrong specialist cluttering the goal check. These are the next sharpenings, in priority order: - §4 persons as structured rows with surface term, entity id, and role; - a cache-conflict line when metapartitur and classifier disagree; - a §9 evidence block with grounding loci; - a contract filter restricted to the active specialist; - splitting `PLANNED-NOT-RUN` into `substituted` versus `missing`. # Reliability and Metrics The golden set makes reliability measurable. Three measurements matter most: the recall of the general filter, the independent DuckDB audit, and the record of the gates through which the whole set has been run. ## Recall of the general filter The general filter is the recall-first stage of the funnel: it gathers sentence and page candidates from the selected collections with *no false negatives* (false positives are acceptable, because the next stage interprets). The false-negative audit checks that every sentence the oracle says must be included is in fact included. | Golden | Verdict | Filter n | Must-include | FN | Recall | |--------|---------|----------|--------------|----|--------| | R1 | PASS | 5032 | 86 | 0 | 1.0 | | R4 | PASS | 2 | 2 | 0 | 1.0 | | R2 | PASS | 2407 | 45 | 0 | 1.0 | | R5 | PASS | 1656 | 1228 | 0 | 1.0 | | ground-pirckheimer-1 | SKIP | 362 | — | — | no sentence oracle | | ground-pirckheimer-2 | PASS | 82 | 29 | 0 | 1.0 | | K1 | SKIP | 1500 | — | — | unknown | | K2 | SKIP | 2181 | — | — | unknown | | K3 | SKIP | 1543 | — | — | unknown | | K4 | SKIP | 3599 | — | — | unknown | | K5 | SKIP | 3171 | — | — | unknown | : The general-filter false-negative audit of 8 August 2026, covering the then-canonical eleven goldens. Recall is 1.0 on every golden for which a sentence-level must-include set was stated. T4 is not in this audit. {#tbl-fn} @tbl-fn reports a scoped result, not a suite-wide one. On 8 August 2026 the recall-first filter dropped none of the sentences the must-include sets required, for the five goldens that have such a set (R1, R4, R2, R5, and ground-pirckheimer-2's saphea pages). The must-include cardinalities are the register probes recorded in the manifest `truth` notes: R1 union 86, R2 kepler-prim 45, R5 the sampled floor against a 5,146-sentence tycho% truth. The six SKIP rows are goldens for which "which sentences must the filter not drop?" does not apply: ground-pirckheimer-1 is a correspondent count against `letters_resolved`; K1–K5 retrieve letters or persons, not sentences. The two T4 goldens, promoted after this audit, have not yet been run through the sentence-level false-negative script. ## The independent DuckDB audit The independent audit queries the source layer without calling the orchestrator. @tbl-independent is the audit of 2026-08 against `letters_resolved` / `letter_pages` / `stz_sentences`. | Golden | Status | Verdict | Summary | |--------|--------|---------|---------| | ground-pirckheimer-1 | canonical | PASS | oracle=75 duckdb=75 letters=354 | | ground-pirckheimer-2 | canonical | PASS | letters=11/11 saphea attested | | ground-pirckheimer-3 | candidate | PASS | correspondents=4, index_letters=13 | : The independent DuckDB audit of the two canonical ground goldens and the Chapter 8 candidate. {#tbl-independent} The two canonical ground goldens match their count oracles: 75 correspondents, inside the band of 70 to 85, and 11 letters with the saphea topic. The third row is the candidate used in Chapter 8, not a third canonical ground golden. The audit shares the register schema and, for the year window, the specialist's predicate; it does not share the orchestrator, the Partitur, or any model call. ## The gates and their record The canonical goldens have been run through three gates, and their results must not be pooled, because each exercises something different. The *orchestrator audit* drove the older AND-OR engine, which has since been retired; it graded each run against its oracle. The *routing gate* (`route_goldens.py`) runs a golden only through the analysis — comprehension, grounding, and the determinacy rules — and checks the route the manifest expects: subtask, grounding verdict, and top collection. The *delivery gate* (`chat_golden_loop.py`) runs a golden through the function behind the chat endpoint, the path a scholar's question takes, and checks that an answer is delivered and that it satisfies the oracle. | Date | Gate | Model | Result on the canonical thirteen | |:------------|:---------------------|:-----------------------|:----------------------------------------------| | 2026-08-12 | orchestrator audit (retired engine) | — | 13 oracle PASS; wall time 53,370 ms | | 2026-08-24 | orchestrator audit (retired engine) | gemini-3.5-flash-lite | 12 PASS; K5 FAIL (an Ollama call timed out) | | 2026-09-29 | routing gate | gemini-3.5-flash-lite | 9 of 14 routes match; R1, R4, R5, K1, K5 do not | | 2026-09-29 | delivery gate | gemini-3.5-flash-lite | 7 delivered and correct; 6 wrong: R1, R5, ground-pirckheimer-2, K1, K4, K5 | | 2026-10-20 | routing gate | gemini-3.8-flash | 13 of 14 routes match; K4 does not | | 2026-10-20 | delivery gate | gemini-3.8-flash | 9 delivered and correct; 3 wrong: R4, R5, K4; 1 error: K3 (time-out after 240 s) | : The gates through which the canonical goldens have been run. The routing gate also covers the routing-only entry `euler-goldbach` (14 routes). Records: `partitur/goldens/README.md` (12 Aug), `orchestrator_audit.json` (24 Aug), `reports/gates/` in ai-query (29 Sep, 20 Oct). {#tbl-gates} @tbl-gates says three things. First, a pass on the retired engine was not a pass on the live path: the thirteen goldens that all passed in August were, on 29 September, delivered correctly in seven cases. Second, the live path has moved. Between 29 September and 20 October R1, K1, K5, and ground-pirckheimer-2 recovered; R4 now returns four sentences where its floor requires ten; K3 ran out of time; R5 returns two sentences where fifty are required; and K4 returns 499 letters where the band allows at most 25 — the corpus-wide inflation the band was written to reject. The routing gate names the cause for K4: the subtask came out as `mention-in-letters` instead of `correspondence-roster`. Third, the two runs of October and September differ in the model as well as in the pipeline (the default changed to `gemini-3.8-flash` on 20 October), so the differences cannot be laid at the pipeline's door alone. The delivery gate of 20 October ran thirteen goldens two at a time in about nineteen minutes; its per-golden times include waiting and are not latencies, which is why the operation boxes of Chapter 4 print none. # The Learning Loop: From Monitor to Promoted Patch The monitor log and the reliability metrics are diagnostic: they show *that* a golden failed and *where*. They do not, by themselves, fix anything. The learning loop is the mechanism that closes that circle — it turns an annotated deficit in a monitor log into a versioned, A/B-gated patch to the pipeline itself, and it is built never to trust a patch until the patch has re-earned the golden set's verdict. Where the gate cannot decide, the record must say so; the worked example below is such a case. ## Two entry points, two gates The loop has two paths into a promotion decision, each with its own gate. ```{=typst} #arxivai.method-box(title: [Path A — metric-driven tunable loop])[ *Trigger*: automatic, or `python -m ai_query.learn.loop run`. *Target*: a subagent's tunable YAML (never a `SKILL.md`, never `partitur.yaml`). *Selection*: the worst-scoring subagent with `support_n > 0`; a subagent without a tunable file is skipped for the next-worst. *Gate*: a fast re-run of the failing support set must raise the subagent's score; then the full suite must strictly raise the aggregate, the mean of the subagents' metric scores. ] ``` ```{=typst} #arxivai.method-box(title: [Path B — annotation-driven instruction/partitur loop])[ *Trigger*: a user annotates a specific stage of a specific run, then `POST /api/learn/from-annotations`. *Target*: determined by the annotation's `target` field — a `SKILL.md` instruction, or a subagent's `partitur.yaml`. *Objective*: for a `SKILL.md` patch, the golden pass rate, measured on the unpatched instruction first; for a `partitur.yaml` patch, the aggregate score across all subagents, because a partitur governs a whole agent, not one stage. ] ``` Path B keeps a patch only if its measure improves and has not fallen by more than one percentage point below the baseline; Path A demands a strict gain of the aggregate. The `SKILL.md` branch is judged by the golden pass rate rather than by the subagent's metric because the metric scorecard writes no rows for several subagents, whose score therefore stands at 1.0 and cannot show an improvement. Every patch is snapshotted before it is applied — to `assessment/versions/` for tunables, to `assessment/instruction_versions/` for `SKILL.md` files, and to `assessment/partitur_versions//` for partitur YAMLs — so a rejected or later-superseded patch can be restored exactly. ```python # loop.py — the partitur A/B gate improved = new_agg > baseline_agg no_regression = new_agg >= baseline_agg - 0.01 kept = improved and no_regression ``` The record of what the loop has done is shorter than its design. Up to 9 October 2026 its version history held 68 instruction versions — 46 active, 18 rejected, 4 superseded — and no Partitur version at all: the partitur branch has never kept a patch, and neither `assessment/versions/` nor `assessment/partitur_versions/` holds a snapshot. ## Worked example: ground-pirckheimer-3, Georg Hartmann This is the same query family as K3 in @sec-golden-set — mention lookup inside a correspondence — but a different golden: `ground-pirckheimer-3` asks *"Which correspondent mentions Georg Hartmann in his letters to Pirckheimer?"* in the Pirckheimer corpus rather than K3's Einstein-in-Karman query. It is a candidate, not one of the canonical thirteen; @tbl-independent already records the register fact the pipeline was missing: four correspondents and thirteen index-cited letters. Georg Hartmann is no correspondent of Pirckheimer: he has no entry in the collection's register of correspondents and is listed only in the edition's person index. The register does hold another Georg, the correspondent Georg Sauermann. **The deficit.** The monitor log of run `de7b615a0c5040b5` (6 August 2026) shows what happened: ``` §4 SIGNALS persons = ['Pirckheimer, Willibald', …, 'Sauermann, Georg'] §5 EXECUTION STREAM … "letter_ent_ids": {"pirckheimer": ["ent_sauermann_georg"]} … §10 VERDICT answer = count=4 goal_satisfied=True goal_score=0.6 ``` The token *Georg* was bound to `Sauermann, Georg`, the phrase *Georg Hartmann* was not resolved, and the letter filter was built on the wrong person. The goal check nevertheless passed; earlier runs of the same question, asked in German on 1 August, had been flagged automatically when their goal check failed. **The annotation.** An annotation of 7 August located the failure at the planning stage: the entity filter produced no letter identifiers for *Georg Hartmann*, although the edition's person index lists him, and the roster specialist returned no correspondents, although Schöner and Savorgnanus demonstrably mention him. The plan, it said, must take a person-mention path: filter on *Georg Hartmann*, group by correspondent, restrict to the Pirckheimer collection. **The patch.** The loop turned this annotation, together with one on the Schöner letter-subject golden (ground-pirckheimer-2), into a new version of the planning instruction, `plan v1.1`. Against its predecessor, kept as `v1.0`, it adds a single rule: when a query targets a specific person, or a roster of senders and years, the stage record must carry explicit filters (`sender`, `time_window`, `letter_ent_ids`) instead of falling back on an empty general filter. **The gate.** The A/B gate could not decide. The planning stage is scored by intent match, which stood at 1.0 before and after the patch; a measure that cannot rise cannot show an improvement. The version row says so in its own words: `plan v1.1` was "manually promoted — A/B gate bypassed: plan metric is intent_match=1.0 in assessment DB, not goal_check", with `score_before = 1.0` and `score_after = 1.0`. **The result.** Both annotations are still open. The record shows a patch promoted by judgement where the gate had no discriminating measure, and it shows that the deficit was not closed by that patch alone: in the fifteen monitored runs of 9 August the persons signal carries the surface string `Georg Hartmann`, bound to no register entry, and the answer stage returned no canonical entries — while the goal check passed every one of these runs, nine of them with a score of 1.0. A passing verdict on an empty answer is itself a finding for the loop, and the sharpest one this case produced. ## Discussion: what the loop buys, and what it risks **In favour.** The loop never proposes a patch without a logged deficit to justify it — no speculative tuning. Every change is versioned and reversible: `plan v1.0` is kept beside `v1.1`, and the baseline of every `SKILL.md` patch is snapshotted before the patch is applied. The annotation record and the version row stay linked, so a later reader can trace live behaviour back to the observation that motivated it — and, in the Hartmann case, to the reason why the gate was bypassed. **Against.** A gate is only as discriminating as its measure. Where the measure is saturated, as the planning stage's intent match was, the gate cannot tell a good patch from a bad one, and promotion falls back on judgement. The LLM's patch proposal is bounded by a minimal-change instruction, but "minimal" is judged by the model, not verified structurally — the stored text of `plan v1.1` is wrapped in a stray Markdown code fence that the model added and no check removed. And the loop refuses to retry a validation failure automatically: a human must sharpen the annotation and restart it, which is a deliberate brake against runaway iteration, but it does mean a bad annotation can stall the loop rather than self-correct. # Extending the Set The golden set is designed to grow. New queries are introduced, tested against their oracles, and promoted into the canonical set as the platform's scope widens. The extension protocol is deliberately simple: 1. **Introduce.** A new query is added as a *candidate* — a golden with an expected shape but no verified oracle. 2. **Test.** The candidate is run through the pipeline and the monitor. The trail is inspected for the right intent, the right trail, and a plausible result set. 3. **Verify.** Where the query has a countable answer, the independent DuckDB audit confirms it. Where it has a subject, the oracle is a loose substring check. 4. **Promote.** A candidate whose oracle is verified and whose run is clean is promoted to `status: golden` and joins the canonical set. The set already shows the pattern. The concept tier — T4-kepler-motrix and T4-euler-aether — was promoted into the canonical thirteen after floor oracles were written from the 12 August 2026 register probes. The karman tier grew from the partitur benchmark's proven shapes, including the German letter-count K2. Each addition widened the coverage of the query families without breaking the existing goldens. The extension protocol is the platform's answer to a real risk: that the golden set becomes a closed exam the pipeline is tuned to pass. The protocol keeps the set open; the independent audit keeps the count oracles as facts about the source layer. A golden can also go stale while the pipeline stands still, because the corpus changes beneath it. The routing-only entry `euler-goldbach` asks how many letters the Euler–Goldbach correspondence holds and expects the grounding verdict GAP: when it was written, the volume of the Euler correspondence edition that contains these letters (Series IVA, volume 4) was absent from the indexed collection. Since its ingest on 1 October 2026 the letter table of that collection holds 214 letters with Goldbach as sender or recipient, 196 of them in that volume. The routing gate of 20 October still matched the expected GAP — a match against a truth that no longer holds. A golden's truth note is dated for this reason, and a change to a collection obliges a review of the goldens grounded in it. ## Candidates still under test: terminology-find The seven query families of @sec-golden-set are not the ceiling. Two candidates still probe a different shape: *"Where does Euler discuss force in E57?"* and *"How does Euler formulate Newton's law of motion in E57?"* — both scoped to Enestroem document E57 (*Theoria motus corporum solidorum seu rigidorum*, 1765). Unlike the lemma and attestation goldens, these ask the pipeline to locate a *concept*, not a single lemma: "force" spans `vis`/`vires` (Latin), `Kraft`/`Kräfte` (German), and `force`/`forces` (French); "Newton's law of motion" spans `lex motus` and its vernacular equivalents [@zaratiana2024gliner]. These candidates are still candidates. Their `expected_answer` is a lower bound on sentence count (`n_sentences.min: 10` and `.min: 5` respectively), not a verified oracle, and their notes say so explicitly: *"Promote after GLiNER2 control run confirms oracle."* The intended grounding method was a sentence-level gold control set built by GLiNER2 labelling of E57's own sentences, against which precision and recall could be measured directly. No accepted control set was ever produced, and on 8 October 2026 GLiNER was taken out of the platform's pipeline, and the probe with it. The candidates therefore stay candidates until a new verification method has been declared and has actually run — the extension protocol applied: a candidate's status is not upgraded on the strength of a method that was planned. # Conclusion A research platform is only as trustworthy as the questions it can be held to. The golden queries of AI Erasmus are that holding: a canonical set of thirteen scholarly questions, each with a declared form of inquiry, a grounding document, and an oracle — a count oracle where a number was derived from the register, a floor oracle where a register probe supplies a minimum. The set's design is its argument. The thirteen goldens span four corpora, three languages, and seven query families — lemma co-occurrence, attestation, person mention, correspondence roster, letter subject, place-and-person lookup, and concept statements. The ground tier matches verified numbers; the catalog, karman, and concept tiers clear register floors. On 8 August 2026 the general filter achieved recall 1.0 on the five goldens that have a sentence-level must-include set. On 12 August 2026 the thirteen canonical goldens all received oracle PASS in the orchestrator audit — on an engine since retired. On the live chat path, on 20 October 2026, nine of the thirteen were delivered and correct, and the routing gate matched thirteen of fourteen expected routes. The independent DuckDB audit confirms the two canonical ground goldens as facts about `letters_resolved`. The monitor log makes a run inspectable, section by section. The learning loop turns an annotated deficit in that log into a versioned patch behind an A/B gate, and the Georg Hartmann case records what the gate could and could not decide: a patch promoted by hand where the measure was saturated, and a goal check that passed an empty answer. The extension protocol keeps the exam open: T4 joined the canonical thirteen; the E57 terminology-find queries remain candidates, now without the control method they were waiting for; and the routing-only Goldbach entry shows that a golden must be reviewed when its collection changes. The Famulus Handlungsmodell (described in @grasshoff2026aierasmus) extends the scope of what the goldens audit: they now test not only the filter stage but every Partitur in the iterative expansion loop — the correspondence-roster specialist, the terminology finder, the entity roster. The gate remains the same: no Partitur enters the repertoire without a PASS on the relevant canon members, measured from a baseline taken *before* the draft; so far no generated Partitur has been registered. The golden set is the mirror in which every new repertoire entry must see itself. An oracle tests whether an answer is right in its count or its topic; it does not show why the answer holds. That is the task of the Famulus justification layer, in which the answer to a question put to a correspondence is the conclusion of an argument over addressed records, checked by critical questions and stored with the notebook that asked it [@grasshoff2026famulusreasoning]. The two complement each other: an oracle that passes on an unsound argument, and an argument whose conclusion misses its oracle, are findings for both. The goldens can also be read and run as a notebook. The generator of Famulus Notebook writes one block per golden, with its collections and oracle in the preamble and its role, ground, and expected route in the paragraph before it; a run of that notebook is a run of the golden check whose result stays as a document [@grasshoff2026famulusnotebook]. The workshop in Basel is staffed; the golden set is how we know the staff is doing its job. # Acknowledgements Thanks are due to the archives and editorial projects whose editions underpin the golden queries, and to collaborators in the AI Platform project for the index, viewer, query, and orchestrator infrastructure.