apunkt/ktx
1
0
Fork 0
mirror of https://github.com/Kaelio/ktx.git synced 2026-07-01 08:59:39 +02:00
Code Issues Projects Releases Packages Wiki Activity

chore: remove private benchmark specs

Browse source
This commit is contained in:
Andrey Avtomonov 2026-06-30 11:13:44 +02:00
parent 67a69dba8b
commit 1c5d16abc3
40 changed files with 0 additions and 8716 deletions
Download patch file Download diff file Expand all files Collapse all files

62
spider2-specs/README.md
View file

@ -1,62 +0,0 @@
# spider2-specs — feature specs driven by the Spider 2.0-Lite benchmark
This directory is the handoff point between two agents working on different
sides of the same goal: making Claude Code + ktx score well on the Spider
2.0-Lite benchmark **without benchmark-specific instructions** — the agent
should succeed using only what ktx provides (skills, semantic layer, wiki).
## Mechanics
Three directories form a pipeline. A feature flows `todo/` → `specs/` →
(implemented), and only its intake draft moves to `done/`:
- **`todo/`** — intake drafts. A **playground agent** (works in
`/Users/andrey/projects/kaelio/spider-clean-submission/playground`, runs the
benchmark, identifies ktx capability gaps) writes a draft spec here when it
finds a gap.
- **`specs/`** — refined specs. A **refinement pass** (brainstorming) takes a
`todo/` draft and produces a proper, implementation-ready spec at
`specs/<same-filename>.md`: sharpened requirements, resolved ambiguities,
acceptance criteria, and orientation hints. The refined spec is the **durable
artifact** the implementer builds from — it stays in `specs/` permanently and
never moves.
- **`done/`** — intake drafts whose feature has shipped (see below).
The **ktx worktree agent** (started from a ktx repo worktree, e.g.
`/Users/andrey/conductor/workspaces/ktx/tallinn-v2`) implements from the
refined spec in `specs/` (falling back to the `todo/` draft only if no refined
spec exists yet). When the feature is implemented it:
1. appends a short **"Implementation notes"** section to the refined spec in
`specs/` (what was built, where, any deviations); and
2. **moves the original intake draft from `todo/` to `done/`.**
Location is status: `todo/` = draft awaiting implementation, `done/` = draft
whose feature shipped, `specs/` = refined specs (permanent home, do not move).
A draft and its refined spec share the same filename so they correspond
(`todo/01-foo.md` ↔ `specs/01-foo.md` ↔ `done/01-foo.md`). No other tracking.
## Rules for specs
1. **Generic, not benchmark-overfit.** ktx is a general-purpose product; the
benchmark only surfaces the need. Every spec must state a real-world use
case independent of Spider 2.0-Lite. If a requirement only makes sense for
the benchmark, it doesn't belong in ktx.
2. Specs are **requirement-level**, not implementation plans. Code pointers in
specs are orientation hints from exploration (line numbers may have
drifted); the implementer owns the design.
3. One spec per file, kebab-case, numeric prefix = suggested priority order.
A refined spec in `specs/` keeps the same filename as its `todo/` draft.
## For the implementer
- After implementing, rebuild and re-link the dev binary so the playground
picks it up: `pnpm run build && pnpm run link:dev` (provides `ktx-dev`).
- Add/extend tests in the ktx test suites; specs list acceptance criteria to
cover.
- Build from the refined spec in `specs/`. On completion, append
"Implementation notes" to that spec (it stays in `specs/`) and move the
intake draft from `todo/` to `done/`.
- If a spec turns out to be wrong or already satisfied, don't silently drop
it — record why in the refined spec's notes and move the draft to `done/`
explaining why no change was needed.

0
spider2-specs/done/.gitkeep
View file

74
spider2-specs/done/01-connection-scoped-wiki.md
View file

@ -1,74 +0,0 @@
# Connection-scoped wiki pages
## Problem
Wiki pages have only two scopes today: `GLOBAL` and `USER`
(`packages/cli/src/context/wiki/types.ts`, frontmatter schema ~lines 14-29).
There is no way to associate a page with a connection. In a project with many
connections, all pages share one search index, so `wiki_search` for a generic
term ("orders", "revenue", "average order value") surfaces pages about the
wrong database. Concept names collide across databases constantly in
real-world multi-connection projects (several databases each with `orders`,
`customers`, etc.).
Today, when `memory_ingest` is called with a `connectionId`, that id is only
used to scope which semantic-layer sources the triage agent can see
(`memory-agent.service.ts` ~46-72, ~107-109); it is **not** persisted on the
resulting wiki page in any form.
## Generic use case
Any org with multiple databases/warehouses in one ktx project: org-wide
definitions ("fiscal year starts in February") should be visible everywhere,
while database-specific conventions ("in the events DB, `user_id` is the
anonymous device id, not the account id") should not pollute searches about
other databases.
## Requirements
1. **Frontmatter field.** Add an optional `connections:` field to wiki page
frontmatter — a list of connection ids (accept a single string too,
normalize to list).
- **Absent or empty ⇒ unscoped: the page applies to all connections.**
This is exactly today's behavior, so every existing page is unaffected
(backward compatible by construction).
2. **Search filtering.** `wiki_search` (MCP tool, `context-tools.ts` ~46-64)
and `ktx wiki search` / `ktx wiki list` (CLI,
`knowledge-commands.ts`) accept an optional `connectionId`:
- With `connectionId: X` ⇒ return pages scoped to X **∪** unscoped pages.
- Without ⇒ current behavior, all pages.
- The filter must apply to **all three search lanes** (lexical FTS5,
semantic/embedding, token fallback) in
`local-knowledge.ts` / `sqlite-knowledge-index.ts` — not as a post-filter
that eats into the result limit unevenly.
3. **Index.** Persist the scoping in the `.ktx/db.sqlite` knowledge index
(the index is already re-synced from files on every search,
`local-knowledge.ts` ~286-310, so a schema addition + sync is sufficient).
4. **Write path.** The memory agent's wiki-write tool accepts the connections
field; when `memory_ingest` is invoked with a `connectionId`, the agent
should default new database-specific pages to that connection, while still
being allowed to write unscoped pages for clearly org-wide content (prompt
guidance, not a hard rule).
5. **`wiki_read` and refs are unchanged** — pages remain addressable by key
regardless of scoping; `connections` is a search/relevance concern only.
6. **Validation.** Warn (don't fail) when a page references a connection id
not present in `ktx.yaml` — config and content can evolve independently.
## Acceptance criteria
- A page with `connections: [db_a]` is returned by
`wiki_search(query, connectionId: "db_a")` and by an unfiltered search, but
**not** by `wiki_search(query, connectionId: "db_b")`.
- A page with no `connections` field is returned in all three cases above.
- Existing projects with no scoped pages behave identically before/after.
- Filtering works in each lane independently (test with embeddings disabled
to exercise lexical/token lanes alone).
- `memory_ingest(content, connectionId)` produces a page scoped to that
connection for database-specific content.
## Benchmark context (motivation only)
Spider 2.0-Lite local subset = one project with 30 SQLite connections whose
schemas share table/concept names (Northwind, sakila, two e-commerce DBs…).
External-knowledge docs (RFM definition, F1 overtake rules) are each relevant
to exactly one database and must not surface for the other 29.

71
spider2-specs/done/02-verbatim-ingest-mode.md
View file

@ -1,71 +0,0 @@
# Verbatim ingest mode for authoritative documents
## Problem
`ktx ingest --text/--file` routes content through the memory agent
(`text-ingest.ts` ~246-357 → `memory-agent.service.ts`), an LLM triage loop
(30-step budget for `external_ingest`, content clipped at ~48k chars,
`memory-agent.service.ts` ~165) that may rewrite, condense, or split the
content before writing wiki pages.
For *authoritative* documents — formula definitions, specs, runbooks,
compliance text — paraphrasing is a bug, not a feature:
- exact thresholds, constants, and rule wording must survive byte-for-byte;
- lexical (BM25) search works best when the stored text matches the phrasing
users/agents will query with;
- ingestion should be deterministic and reproducible — same input file, same
resulting page.
## Generic use case
Any team ingesting documents that are already the source of truth: metric
definition sheets, SLA documents, calculation methodology docs, regulatory
text. The user wants ktx to *index and surface* the document, not to
re-author it.
## Requirements
1. **Flag.** `ktx ingest --file <path> --verbatim` (apply to `--text` too).
Composes with the existing optional `--connection <id>` so the resulting
page can be connection-scoped (see spec 01).
2. **Body preservation is enforced by code, not by prompt.** The stored page
body must be the input content byte-for-byte. The LLM is used **only** to
generate metadata: `summary`, `tags`, `sl_refs`, suggested page key/slug
(and `connections` default from the flag). Implementation freedom: a
single constrained LLM call is fine — the full memory-agent loop is not
required for this mode.
3. **No clipping of the stored body.** The ~48k clip may apply to what is
*sent to the LLM* for metadata generation, never to what is *written* to
the wiki page.
4. **Existing frontmatter.** If the input file already has YAML frontmatter,
preserve user-provided fields and only fill gaps (don't overwrite an
explicit `summary` with a generated one).
5. **Key collisions.** Deterministic, non-destructive behavior: error or
suffix — never silently overwrite an existing page.
6. **Degraded mode.** With `llm.provider.backend: none`, `--verbatim` should
still work, deriving `summary` from the first heading/sentence and leaving
optional metadata empty. (Regular agent ingest can't do this; verbatim
mode can and should.)
## Acceptance criteria
- Ingesting a file with `--verbatim` produces a wiki page whose body is
byte-identical to the input (assert with a hash in tests).
- Running the same ingest twice is idempotent or fails loudly on the second
run (per requirement 5) — no duplicated/divergent pages.
- A >48k-char file is stored in full.
- `--verbatim --connection X` yields a page scoped to X (depends on spec 01;
if 01 isn't implemented yet, the flag composition can land later).
- Generated metadata makes the page findable: `wiki_search` for a phrase
from the document body returns it (lexical lane), and for a paraphrase of
its topic returns it when embeddings are enabled (semantic lane).
## Benchmark context (motivation only)
Spider 2.0-Lite ships 8 external-knowledge markdown docs (RFM bucket
definitions, haversine formula, F1 overtake rules…). Gold SQL was authored
against their exact text; an LLM paraphrase that drops a bucket boundary
loses a question. We currently work around this by hand-writing frontmatter
and copying files into `wiki/global/` — verbatim mode makes that a supported
ktx workflow instead of a manual step.

63
spider2-specs/done/06-scan-tolerate-broken-objects.md
View file

@ -1,63 +0,0 @@
# Schema scan must tolerate individual objects that fail introspection
> Priority: MEDIUM. Found during the first full Spider2-lite sqlite ingest
> (2026-06-13): one database (`oracle_sql`) failed to ingest **entirely**
> because a single broken VIEW errored during introspection, leaving that
> connection with no semantic layer at all.
## Problem
`ktx ingest <connection>` aborts the whole database's schema scan when one
table/view errors during introspection/profiling. In `oracle_sql` the view
`emp_hire_periods_with_name` is defined as
`SELECT ehp.start_date, ehp.end_date ... FROM emp_hire_periods ehp ...` but the
base table has no `start_date`/`end_date` columns — so any attempt to read it
raises `no such column: ehp.start_date`. That single broken object failed the
ingest of all ~48 healthy tables/views in the database.
A second, related symptom: setting `enabled_tables: [main.customers]` to work
around it produced a different hard failure (`Adapter "database schema" did not
recognize fetched source output`), so the documented allowlist escape hatch did
not provide a clean fallback either.
## Generic use case
Real databases routinely contain broken or inaccessible objects: views over
dropped/renamed columns, views referencing tables the connection role can't
read, permission-denied tables, or vendor system views that error. ktx should
ingest everything it *can* and skip what it can't — never let one bad object
zero out an entire connection's context. This is basic robustness for
production warehouses, not benchmark-specific.
## Requirements
1. **Per-object isolation.** If introspecting/profiling one table or view
throws, skip that object, record a warning (object name + error), and
continue scanning the rest. The connection's semantic layer is built from
the objects that succeeded.
2. **Surface, don't hide.** Report skipped objects in the ingest summary and in
`ktx status` (e.g. "oracle_sql: 1 object skipped — emp_hire_periods_with_name:
no such column ehp.start_date"). Honor `failureMode` for whole-connection
aborts, but a single bad object should not count as a connection failure.
3. **Views vs tables.** A broken view should never block base-table ingest.
Consider profiling views defensively (they are read-only projections).
4. **Allowlist fallback should work.** `enabled_tables` should reliably restrict
the scan to the listed objects (and the qualification format for sqlite must
be documented and accepted). Fix the `did not recognize fetched source
output` failure when the allowlist yields a small/edge-case set.
## Acceptance criteria
- Ingesting a sqlite DB containing one broken view plus N healthy tables yields
a semantic layer for the N healthy tables and a warning naming the broken view
— exit is success (not "failed"), subject to `failureMode`.
- The skipped object is listed in the ingest summary and `ktx status`.
- `enabled_tables` restricted to a subset ingests exactly that subset without the
adapter-output error.
## Benchmark context (motivation only)
`oracle_sql` (8 of the 135 sqlite questions) currently has no semantic layer
because of its one broken view; those questions must be solved from raw
`sql`-tool introspection instead of ktx's enriched context. Tolerant scanning
would restore enriched context for that database.

112
spider2-specs/done/07-analytics-skill-sql-craft.md
View file

@ -1,112 +0,0 @@
# Add universal SQL-authoring craft to the ktx-analytics skill
> Priority: HIGH. The `ktx-analytics` skill currently tells the agent *which
> ktx tools to call and in what order*, but gives almost no guidance on
> *writing correct SQL*. In benchmark runs the agent reliably produced
> runnable SQL (0 execution errors) yet failed on correctness — precision,
> determinism, type mismatches, and answer completeness. These are universal
> analytics-engineering truths that every ktx user benefits from, so they
> belong in the shipped skill, not in any caller's prompt.
## Scope guard (read first)
Only **universally-true** SQL/analytics craft goes here — guidance that helps a
real ktx user querying a **live** database. The test for inclusion: *"Would this
advice be correct and useful for an analyst on a current, production database?"*
**Dialect-specific syntax is out of scope here.** The v9 harnesses' only
per-dialect content (Snowflake: `DB.SCHEMA.TABLE` FQTNs, double-quoted
lowercase cols, VARIANT colon-paths; BigQuery: backtick FQTNs, `_TABLE_SUFFIX`
for sharded tables; sqlite: `strftime`/`julianday`) is genuinely useful but
belongs in a **dialect-aware** location (per-driver notes), not this flat
skill. Track separately as a follow-up; the rules below must stay
dialect-agnostic.
Explicitly **do NOT** add (these are application/consumer concerns, not skill
concerns, and some are actively wrong for live data):
- Output-format contracts ("return a bare result set with exactly these
columns, no prose"). The skill is for interactive analysis and already
favors readable tables + summaries; a caller that needs a strict result
shape specifies that itself.
- Anchoring relative time ("recent", "past N months") to `MAX(date)` of the
data. On a live database "recent" means relative to *now*; this is only true
for static snapshots and must not be baked into the product.
- Anything justified by a grader/scoring comparator.
## File
`packages/cli/src/skills/analytics/SKILL.md` (the shipped skill;
`setup-agents.ts` installs it into agent environments — the copy under a
project's `.claude/skills/` is regenerated from this source). Extend the
existing `<rules>` block and step 5 ("Query") / step 6 ("Validate and
explain"); keep the existing interactive guidance intact.
## Requirements — add these as general rules (behavior only, no rationale that
references answers/graders)
**Schema discovery before writing SQL**
1. Inspect representative sample rows of each table before composing SQL —
confirm date/time encoding (e.g. `YYYYMMDD` vs ISO vs epoch), null
prevalence in join/filter keys, and the actual set of categorical/enum
values. (`entity_details` + a small `sql_execution` sample.)
2. Cast a column to its real type before comparing it in `WHERE`/`JOIN`. A
string column compared against a numeric literal (or vice versa) can
silently match nothing.
**Composition discipline**
3. Build complex queries incrementally — one CTE at a time, verifying each
layer's output on a small sample before stacking the next.
4. Avoid joins that fan out row counts. Add columns only from tables already
required by the grain, or pre-aggregate to the target grain before joining.
**Window-function correctness**
5. Give every ranking/ordering window function a complete, deterministic
tie-breaker (append unique key columns), so `RANK`/`ROW_NUMBER`/`LAG`
results are stable rather than flickering across runs.
6. Apply row filters **after** window functions for sequence / "first" /
"most recent" / "since" questions — compute over the full partition, then
filter.
**Numeric precision**
7. Compute at full precision; round only in the final projection, never inside
intermediate CTEs.
8. Be explicit about truncation (`CAST AS INT` truncates; use explicit
rounding when rounding is intended).
9. Distinguish "average of per-group averages" (macro: `AVG(group_metric)`)
from "overall/weighted average" (micro: `SUM(num)/SUM(den)`) based on the
question's wording.
**Answer completeness / interpretation**
10. "top / highest / most / lowest" → return only the winning row(s) (e.g.
`RANK() = 1` / `QUALIFY`), not the full ranked list, unless a list is asked
for.
11. "for each X / per X / by X" → exactly one row per X; don't collapse to a
single value unless the question says "overall" or "total across X".
12. When a question asks for inputs and a derived value ("X, Y, and their
ratio"), include the inputs as columns alongside the derived value.
13. When grouping by a human-readable label (a name), also expose the entity's
identifier — identity, not just the label, is part of the result.
14. When a result is unexpectedly empty, relax filters one at a time to find
which predicate removed the rows.
## Acceptance criteria
- The shipped `analytics/SKILL.md` contains the rules above, phrased as general
truths with **no reference to any benchmark, gold answer, or scoring
comparator**.
- Existing interactive guidance (compact result tables, summaries,
clarification prompts, the tool-order workflow) is preserved — the skill must
still read well for an interactive human-facing analysis session.
- None of the excluded items (output-shape contract, `MAX(date)` anchoring,
grader-driven advice) appear.
- Skill stays within a reasonable size; group the new rules under clear
sub-headings so they're scannable.
## Benchmark context (motivation only)
On the Spider 2.0-Lite sqlite subset, the solver produced 0 execution errors
but ~50 result mismatches; a large share traced to exactly these gaps
(premature rounding, string-vs-number compares, non-deterministic window
ordering, returning full lists for "top" questions, dropping inputs to derived
values). These are generic SQL-authoring defects — fixing them in the skill
improves ktx for everyone and, as a side effect, the benchmark.

83
spider2-specs/done/08-per-dialect-sql-syntax-notes.md
View file

@ -1,83 +0,0 @@
# Per-dialect SQL syntax notes (dialect-aware, scoped to the connection)
> Intake draft. Companion to `specs/07-analytics-skill-sql-craft.md`, which kept
> the analytics SQL craft dialect-agnostic and explicitly deferred per-dialect
> syntax here.
## Problem
Spec 07 deliberately keeps the analytics SQL-authoring craft
**dialect-agnostic** — every rule must read correctly on any engine. But a lot of
*real* correctness depends on dialect-specific syntax that spec 07 excludes and
defers to this follow-up:
- **Snowflake:** `DB.SCHEMA.TABLE` FQTNs, double-quoted lowercase identifiers,
VARIANT colon-paths.
- **BigQuery:** backtick FQTNs, `_TABLE_SUFFIX` for sharded tables, `QUALIFY`.
- **sqlite:** `strftime`/`julianday` for dates, no `QUALIFY`.
This guidance is genuinely useful to an agent writing SQL against a live
database, but it must **not** pollute the flat dialect-agnostic skill — an agent
querying sqlite should never see Snowflake VARIANT syntax. It belongs in a
**dialect-aware** location, surfaced only for the dialect the active connection
actually uses.
## Generic use case
Any ktx project whose connections span more than one warehouse engine (e.g. a
Snowflake warehouse + a BigQuery export + a local sqlite extract). When the agent
writes SQL for a given connection, it should get that engine's syntax
conventions — and nothing for the engines it isn't querying.
## Requirements
1. **Per-driver dialect notes.** Author concise, correct syntax notes per
supported driver: FQTN form, identifier quoting/case, date/time functions,
top-N / window-filtering idiom, semi-structured access. These are genuine
per-engine invariants, so enumerating them per driver is acceptable (unlike a
denylist of bad specifics).
2. **Scope to the active dialect, derived from state.** Which notes the agent
sees must be selected from the connection's configured driver/dialect
(`ktx.yaml` connections / the connector registry), not guessed and not shown
all at once. The flat analytics skill stays dialect-agnostic (spec 07
invariant preserved).
3. **Delivery mechanism (enabling sub-requirement).** The shipped skill is
installed as a **single `SKILL.md`** per target (`setup-agents.ts` /
`readAnalyticsSkillContent`). Surfacing per-dialect notes on demand needs one
of two approaches; the refinement pass should compare them before committing:
- **Multi-file skill delivery** — bundle `reference/<dialect>.md` files and
have the skill point to the one matching the connection. Requires extending
`setup-agents.ts` to copy a skill *directory* (Claude Code, Codex, universal
`.agents`) and a multi-file zip (Claude Desktop), a **flatten/concatenate
transform** for the single-file targets (Cursor `.mdc`, OpenCode `.md`), and
**per-file manifest entries** for clean uninstall. This is the
install-mechanism improvement spec 07's Model section flags as future work.
- **Dynamic MCP delivery** — an MCP surface returns the dialect hints for a
given `connectionId` (the MCP layer already resolves the connection's
dialect), so no install change is needed and Cursor/OpenCode get identical
behavior. May be the lower-cost, more uniform path; weigh it first.
4. **No dialect syntax leaks into the dialect-agnostic skill.** Spec 07's
acceptance criterion (no `QUALIFY`/`strftime`/`julianday`/backtick-FQTN/etc. in
`analytics/SKILL.md`) stays green. This work adds a *separate* dialect-aware
channel; it does not amend the flat skill.
## Acceptance criteria
- An agent querying a sqlite connection gets sqlite date idioms and never sees
Snowflake/BigQuery-only syntax; an agent querying Snowflake gets
FQTN/identifier/VARIANT guidance.
- The dialect shown is **derived from the connection's configured driver**, not
hardcoded per project and not guessed.
- `analytics/SKILL.md` remains dialect-agnostic — spec 07's criteria are
unaffected.
- Whichever delivery mechanism is chosen installs/serves correctly across **all**
supported agent targets, including the single-file Cursor/OpenCode shape.
## Benchmark context (motivation only)
The Spider 2.0-Lite v9 harnesses' only per-dialect content was Snowflake
(`DB.SCHEMA.TABLE` FQTNs, double-quoted lowercase cols, VARIANT colon-paths),
BigQuery (backtick FQTNs, `_TABLE_SUFFIX` for sharded tables), and sqlite
(`strftime`/`julianday`). That content is real and useful but engine-specific;
spec 07 kept it out of the flat skill and deferred it here so the
dialect-agnostic rules stay clean.

150
spider2-specs/done/09-fan-out-safe-multi-hop-aggregation.md
View file

@ -1,150 +0,0 @@
# Strengthen fan-out-join safety for multi-hop aggregation in the analytics skill
## Problem
The `ktx-analytics` skill already carries a fan-out rule (spec 07, rule 4:
*"Avoid fan-out joins — add columns only from tables already at the target
grain, or pre-aggregate to that grain before joining; a join that multiplies
rows quietly inflates every downstream `SUM`/`COUNT`"*). In practice the agent
honors it on a single join but still **silently fan-outs on multi-hop join
chains**, where the inflation is one or two joins removed from the aggregate and
therefore much harder to notice.
The failure shape: a metric that lives at a *coarse* grain (e.g. one row per
parent record) is counted/summed *after* the parent has been joined down to a
*finer* grain (e.g. one row per child line). Every parent-level value is then
duplicated by its child fan-out, so `COUNT(*)` / `SUM(amount)` over-counts by an
amount that depends on the data — runnable SQL, plausible-looking number,
quietly wrong.
The rule today is stated as a *prohibition* ("avoid"). It needs to be a
*detect-and-fix habit*: a concrete multi-hop example of the trap, and an active
verification step the agent runs while composing, not just an instruction to be
careful.
## Generic use case (independent of any benchmark)
An analyst on any production warehouse asks: *"How many orders are there per
region?"* where the path from region to the order's detail runs through several
hops (region → store → order → order line). The honest answer counts each order
once. If the query descends to the line-level table along the way (e.g. for a
filter), each order is counted once **per line on the order**, inflating the
per-region total. Attribution here is unambiguous — each order belongs to exactly
one store and thus one region — so the *only* thing that can go wrong is the row
multiplication, which is exactly what makes it a clean teaching case. This is one
of the most common silently-wrong analytics mistakes on normalized schemas — it
is not
specific to any dataset, dialect, or benchmark.
## Requirements
This extends the existing `<sql_craft>` "Composition" guidance in the
`ktx-analytics` skill (spec 07). Additive only; keep it inline, dialect-agnostic,
and stated as a heuristic-plus-why (consistent with spec 07's style).
1. **Generalize the fan-out rule to multi-hop chains.** Make explicit that the
danger is *cumulative*: any one-to-many hop on the path between the table that
owns a measure and the aggregate inflates that measure, even when the
offending join is several hops away from the `SUM`/`COUNT`. The fix is the
same as the single-hop case — **pre-aggregate the measure to its own grain in
a CTE, then join the already-aggregated result** — but the agent must apply it
per measure-owning table along the whole chain, not just at the final join.
2. **Add a verification habit, not just a prohibition.** While composing, the
agent should confirm a join did not change the grain it intends to aggregate
at — e.g. check that the row count (or the count of the aggregate's key) is
unchanged across a join that is supposed to be one-to-one / many-to-one, and
pre-aggregate the finer table to grain when it is one-to-many. This is the same
"build incrementally and check each layer" discipline spec 07 already endorses,
pointed specifically at grain preservation.
**Pre-aggregate is the general fix; `COUNT(DISTINCT)` is a count-only
shortcut.** Pre-aggregating the finer table to the measure's grain in a CTE and
then joining one-to-one is the remedy that works for every aggregate
(`COUNT`/`SUM`/`AVG`). `COUNT(DISTINCT <key>)` is a valid one-liner *for counts
only* — it must NOT be generalized to a fanned-out `SUM`/`AVG`, because two
rows can legitimately hold equal amounts and `DISTINCT` would wrongly collapse
them. State this trap explicitly; a naïve "just use `COUNT(DISTINCT)`" rule is
silently wrong for sums.
3. **One concrete, generic multi-hop example.** Include a short worked example
that shows the inflation and the fix. It must use an **invented, generic
schema** — **no benchmark table names, no benchmark SQL, and no benchmark
result values** (see "Leak-safety" below — hard constraint). The example must:
(a) use a **plain `COUNT`** (not an average) so it isolates the fan-out lesson
and does not entangle the skill's separate *macro-vs-micro average* rule; and
(b) use a chain with **unambiguous single-owner attribution** so the only thing
that can go wrong is row multiplication. The intended example is the chain
`regions → stores → orders → order_lines` answering *"how many orders per region
include at least one backordered line"* — each order belongs to exactly one
store and thus exactly one region, so attribution is clean; the line-level
filter gives `order_lines` a genuine reason to be joined (so the fix is the
pre-aggregate remedy, not "drop the join"), and that join sits **several hops
below** the region-level COUNT (the multi-hop point):
```sql
-- "How many orders per region include at least one backordered line?"
-- (order_lines is genuinely needed here — for the backordered filter — so the
-- fix is NOT "just drop the join".)
-- WRONG: the order_lines join is one row per matching line, joined several hops
-- BELOW the COUNT. An order with 3 backordered lines is counted 3 times, so the
-- per-region total is inflated by backordered-lines-per-order — silently wrong.
SELECT r.region_id, COUNT(*) AS n_orders
FROM regions r
JOIN stores s ON s.region_id = r.region_id
JOIN orders o ON o.store_id = s.store_id
JOIN order_lines l ON l.order_id = o.order_id AND l.is_backordered -- one-to-many: fan-out
GROUP BY r.region_id;
-- RIGHT (general remedy): collapse the finer table to the measure's grain in a
-- CTE FIRST, then join one-to-one so nothing multiplies. This same shape works
-- for SUM/AVG, not just COUNT.
WITH qualifying_orders AS ( -- back to ONE row per order
SELECT DISTINCT order_id FROM order_lines WHERE is_backordered
)
SELECT r.region_id, COUNT(*) AS n_orders
FROM regions r
JOIN stores s ON s.region_id = r.region_id
JOIN orders o ON o.store_id = s.store_id
JOIN qualifying_orders q ON q.order_id = o.order_id
GROUP BY r.region_id;
-- Count-only shortcut: COUNT(DISTINCT o.order_id) over the WRONG query also works
-- HERE. But it is counts-only — a fanned-out SUM/AVG of a per-order measure (e.g.
-- summing each order's shipping_fee after joining lines) must pre-aggregate;
-- DISTINCT would wrongly merge two orders that happen to share the same fee.
```
## Leak-safety (hard constraint on this spec and its example)
The benchmark's gold answers must never appear in ktx. The worked example must
be a **synthetic, generic schema invented for teaching** — not the tables,
column names, query, or numeric results of any Spider 2.0-Lite question. The
example demonstrates the *pattern* (coarse-grain measure counted after a
one-to-many join), which is universal; it must be reconstructable from first
principles by anyone, with zero reference to benchmark data. A reviewer should
be able to read the example and find nothing that ties it to a specific
benchmark instance.
## Acceptance criteria
- The skill's `<sql_craft>` Composition section states the multi-hop
generalization of the fan-out rule and a grain-verification habit, inline and
dialect-agnostic.
- It includes exactly one short, **generic** worked example (wrong vs.
pre-aggregated-right) using an invented schema, with no benchmark-derived
identifiers or values.
- No new tool, flag, or config; this is skill-content only (additive to spec 07).
- Existing analytics-skill content tests are updated to cover the added rule's
presence (mirroring spec 07's `analytics-skill-content.test.ts`).
## Benchmark context (motivation only)
Multi-hop aggregation questions (counting/averaging a coarse-grained measure
reached through several one-to-many joins) are a recurring source of
result-mismatch failures in the SQLite subset: the agent produces runnable SQL
with the right tables but a fan-out-inflated number. These are correctness
failures, not knowledge or schema-discovery failures (zero execution errors in
the latest run), so the fix belongs in the product's authoring craft — where it
also helps any real analyst — not in a benchmark-specific prompt.
```

65
spider2-specs/done/10-panel-completeness-spine.md
View file

@ -1,65 +0,0 @@
# Panel/period completeness — emit the full set of groups, not only the populated ones
## Problem
When a question asks for a result *per period* or *per category* ("orders for each
month of 2023", "revenue by region", "count per status"), the natural `GROUP BY`
only returns groups that actually have rows. Periods/categories with **zero**
activity silently vanish, so a "12 months" answer comes back with 9 rows and the
ones that should read `0` are simply absent. The agent writes runnable SQL with
the right aggregate but an **incomplete panel**.
This is a universal reporting correctness issue: a monthly report with missing
months, or a category breakdown missing the empty categories, is wrong for any
analyst — and it is also a frequent result-mismatch shape on the benchmark.
## Generic use case (independent of any benchmark)
"How many orders were placed in each month of 2023?" must return **12 rows** even
if March had no orders (March = 0), not 11 rows. "Sales per region" should include
regions with no sales (as 0/NULL) when the question asks for *each* region.
## Requirements
Additive to the `ktx-analytics` skill's `<sql_craft>` "Answer completeness /
interpretation" group (consistent with spec 07's inline, dialect-agnostic, heuristic
+ why style).
1. **Recognize "full-panel" phrasing.** Cues like *each / every / per <period> /
for all <category> / by month* signal that the answer's row set should be the
**complete** set of periods or categories in scope, not just those present in
the filtered fact rows.
2. **Build a spine, then LEFT JOIN.** Generate the full set of expected
groups — a date/number series via a recursive CTE for periods, or the distinct
dimension values from the authoritative dimension table for categories — and
LEFT JOIN the aggregated facts onto it, defaulting missing measures with
`COALESCE(metric, 0)` (or NULL when 0 would be wrong). *Why:* a plain inner
`GROUP BY` can only emit groups that have at least one fact row.
3. **Don't over-apply.** When the question asks only about groups that exist
("which months had orders"), the spine is unnecessary; the cue is *each/all*
vs *which*.
## Leak-safety (hard constraint)
Any worked example must use a **synthetic generic schema** (e.g. an `orders`
table with an `order_date`) and demonstrate only the *pattern* (spine + LEFT JOIN
+ COALESCE). No benchmark table names, SQL, or result values. The behavior is
reconstructable from first principles and tied to no specific instance.
## Acceptance criteria
- `<sql_craft>` states the full-panel cue, the spine + LEFT JOIN + COALESCE recipe,
and the over-application guard — inline and dialect-agnostic.
- At most one short generic example (recursive-CTE date spine or distinct-dimension
spine), no benchmark-derived content.
- Skill-content only; analytics-skill content tests updated to cover the rule.
## Benchmark context (motivation only)
Per-period / per-category questions where some periods are empty produce
short-row result mismatches in the SQLite subset. The fix is a universal
reporting habit (complete panels), so it belongs in the product's craft, where it
also helps real analysts — not in a benchmark-specific prompt. Related to spec 11
(rolling/cumulative windows need a complete date spine to be correct).

73
spider2-specs/done/11-time-series-window-recipes.md
View file

@ -1,73 +0,0 @@
# Time-series window craft — running totals, rolling-N (min-periods), period-over-period
## Problem
A large share of analytics questions are time-series shaped: a **running/cumulative
balance**, a **rolling N-day average**, or **period-over-period growth**. The agent
knows window functions exist (spec 07 covers determinism and window-then-filter) but
gets the *time-series specifics* wrong:
- cumulative balance computed without an unbounded preceding frame (or with the
frame defaulting incorrectly when there are ties on the order key);
- "rolling 30-day" implemented as `ROWS BETWEEN 29 PRECEDING` over **gappy** daily
data, so the window spans the wrong calendar span when days are missing;
- no **minimum-periods** handling — a rolling average is reported before the window
is actually full;
- "growth vs previous period" without `LAG`, or comparing to the wrong neighbor.
These are runnable-but-wrong; the structure is close, the edge case diverges.
## Generic use case (independent of any benchmark)
- "Each account's month-end running balance over 2023" — cumulative sum of monthly
net over an ordered window.
- "30-day rolling average of daily revenue, only once 30 days of history exist."
- "Month-over-month revenue growth rate."
All three are bread-and-butter for any analyst on any time-series table.
## Requirements
Additive to the `ktx-analytics` skill's `<sql_craft>` "Window functions" group
(inline, dialect-agnostic, heuristic + why).
1. **Cumulative / running total.** `SUM(x) OVER (PARTITION BY k ORDER BY t ROWS
BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)`, with a complete tie-breaker in
`ORDER BY` (spec 07 rule). *Why:* the default frame with a non-unique `ORDER BY`
can include/exclude peers unexpectedly.
2. **Rolling window over time, not over rows.** When "rolling N days/months" is
asked, the window must span a calendar range. Over gappy data, either build a
complete date spine first (see spec 10) so `ROWS BETWEEN n-1 PRECEDING` equals
the intended span, or use a range/self-join keyed on the date. *Why:* row-count
frames over missing dates silently measure the wrong span.
3. **Minimum periods.** When the question says "only after N periods of data" (or
it is implied by a rolling metric), emit NULL/skip until the window is full
(e.g. guard on `COUNT(*) OVER (...) = N`). *Why:* a partial early window is not
the requested metric.
4. **Period-over-period.** Use `LAG(metric) OVER (PARTITION BY k ORDER BY period)`
for prior-period comparisons; growth rate = `(cur - prev) / prev` computed at
full precision (round only at the end). Guard divide-by-zero/NULL prev.
## Leak-safety (hard constraint)
Worked examples must use a **synthetic generic schema** (e.g. `daily_revenue(day,
amount)` or `account_txns(account_id, txn_date, net)`) and show only the *pattern*.
No benchmark table names, SQL, or result values.
## Acceptance criteria
- `<sql_craft>` "Window functions" gains the cumulative, rolling-over-time +
min-periods, and period-over-period recipes — inline, dialect-agnostic.
- At most one or two compact generic examples; no benchmark-derived content.
- Skill-content only; analytics-skill content tests updated.
## Benchmark context (motivation only)
Running-balance / rolling / period-over-period questions are the single largest
result-mismatch cluster in the SQLite subset (financial-transactions style DBs).
The methodology is universal analyst craft, so it belongs in the product's skill
(transfers to real users), not in a benchmark-specific prompt. Depends on spec 10
(date spine) for the gappy-rolling case.

61
spider2-specs/done/12-parse-text-encoded-numbers.md
View file

@ -1,61 +0,0 @@
# Parse text-encoded numeric columns before doing math on them
## Problem
Numeric measures are often stored as **text** with human formatting: unit suffixes
(`"1.2K"`, `"3M"`, `"4B"`), currency symbols and thousands separators (`"$1,200"`),
percent signs (`"12%"`), or non-numeric sentinels for missing/zero (`"-"`, `"N/A"`,
`""`). Aggregating or comparing such a column directly is silently wrong: string
comparison orders `"100" < "9"`, and a naive `CAST(x AS REAL)` yields `0`/NULL on
the formatted values rather than the intended number.
The agent already samples schemas (spec 07 schema-discovery), but when it sees a
"numeric" column it tends to assume it is a real number type and skips the parse —
so the arithmetic runs on garbage. Runnable, plausible, wrong.
## Generic use case (independent of any benchmark)
A `trade_volume` column stored as `"1.2K" / "3M" / "-"` must become `1200 / 3000000
/ 0` before you can sum it or compute a daily change. A `price` stored as
`"$1,299.00"` must become `1299.00` before averaging. This is routine data hygiene
on real, messy production tables.
## Requirements
Extend the `ktx-analytics` skill's `<sql_craft>` "Schema discovery before writing
SQL" group (inline, dialect-agnostic, heuristic + why).
1. **Detect text-encoded numerics during sampling.** When a column that the
question treats as a number is stored as text, sample distinct values to learn
the encodings actually present (suffixes, symbols, separators, sentinels) before
composing — never assume the format from the column name.
2. **Parse and scale before arithmetic.** Strip currency/separator/percent
characters; multiply by the suffix scale (K=10^3, M=10^6, B=10^9); map sentinels
(`-`, `N/A`, empty) to `0` or `NULL` per the question's intent; then cast to a
numeric type. Do this in an early CTE so all downstream math sees clean numbers.
*Why:* string columns compared/aggregated as-is sort lexically and cast to 0,
producing silently wrong results instead of errors.
3. **Confirm coverage.** After parsing, sanity-check that no intended-numeric value
failed to parse (would surface as NULL), to catch an encoding the sample missed.
## Leak-safety (hard constraint)
Worked examples must use a **synthetic generic schema** and made-up values (e.g. a
`metrics(label, value_text)` table with `"1.2K"`, `"-"`). No benchmark table names,
SQL, or result values; the parsing pattern is universal and tied to no instance.
## Acceptance criteria
- `<sql_craft>` schema-discovery gains the detect → parse/scale → verify guidance —
inline, dialect-agnostic, with at most one short generic example.
- No benchmark-derived content. Skill-content only; content tests updated.
## Benchmark context (motivation only)
At least one SQLite-subset question stores trading volume as suffix-encoded text
("K"/"M", "-" for zero) and fails because the agent aggregates the raw strings. The
fix — parse messy encodings before math — is universal data hygiene that helps any
analyst, so it belongs in the product's craft rather than a benchmark-specific
prompt.

105
spider2-specs/done/14-output-completeness-final-check.md
View file

@ -1,105 +0,0 @@
# Enforce answer-output completeness with a final pre-emit check in the analytics skill
## Problem
The single largest correctness failure mode is **incomplete output**: the query runs and the
methodology is roughly right, but the result is missing columns the question asked for. Three
recurring sub-patterns:
1. **Multi-part questions answered partially.** A question that asks for several things ("report
the highest *and* the lowest month, each with its count and average, *and* the difference")
comes back with only the first part — one column instead of the several requested.
2. **Identity dropped.** Grouping by a human-readable name but not projecting the entity's
identifier (e.g. a product name without its product id, a customer name without its
customer id).
3. **Inputs to a derived value dropped.** Returning a ratio / percentage / difference but not
the underlying counts the question also asked for.
Sub-patterns 2 and 3 are **already covered by `<sql_craft>` rules** in the analytics skill
(spec 07: *"expose identity, not just the label"* and *"keep the inputs to a derived value"*),
yet they are frequently **not applied**. So the gap is not missing knowledge — it is that these
rules are passive heuristics buried in a list, and the agent doesn't reliably check them before
finalizing. The fix is to (a) add the missing multi-part-completeness rule and (b) turn
output-completeness into an **explicit final verification step** the agent performs before
emitting SQL.
This is reinforced by evidence that the failure is **model-independent**: a markedly stronger
model produced the same incomplete-output mistakes on these questions, which means it is a
craft/enforcement gap, not a capability gap.
## Generic use case (independent of any benchmark)
An analyst is asked: *"For each region, report the highest and the lowest monthly order count,
and the difference between them."* A complete, useful answer has a column for the region's id
and name, the highest count, the lowest count, and the difference — five columns. Returning just
the region and a single number answers only part of the request. This is a universal expectation
on any database: answer **every** part of a multi-part request, identify the entities, and show
the inputs behind any derived figure.
## Requirements
Additive to the analytics skill's `<sql_craft>` "Answer completeness / interpretation" group and
its workflow's validate step (inline, dialect-agnostic, heuristic + why, consistent with spec 07).
1. **Multi-part / multi-output completeness (new rule).** When a question requests several
outputs — a list ("A, B, and C"), paired extremes ("the highest *and* the lowest"), or a
value plus its components ("X, Y, and their ratio") — the final projection must contain a
column for **each** requested output. *Why:* answering only the first clause is the most common
way a runnable query is still wrong; the grain and methodology can be perfect yet the answer
is short by columns.
2. **Fold the existing identity / inputs rules into the same completeness notion.** The
already-shipped rules — project the entity **identifier** alongside any human-readable label,
and **keep the inputs** to any derived value — are part of output completeness; reference them
from the check below so they are actually applied, not just listed.
3. **Add an explicit final completeness check (the enforcement mechanism).** Before emitting the
final SQL, the skill should have the agent **re-read the question and confirm the projection
covers**: every named metric/attribute; the identifier of every grouped/named entity; every
input to a derived value; all at the grain the question specifies. This is a short, concrete
checkpoint at the validate step — the point is to convert the passive heuristics into an active
pre-finalize verification. (Do **not** add unrequested/extra columns to be "safe" — that is
grader-gaming; the check is about matching the request exactly, not padding it.)
Generic teaching example (synthetic schema — see Leak-safety):
```sql
-- "For each region, report the highest and lowest monthly order count and their difference."
-- WRONG: answers only the first clause; no region id, no lowest, no difference.
SELECT region_name, MAX(monthly_orders) AS highest
FROM region_monthly GROUP BY region_name;
-- RIGHT: one column per requested output + the entity's identity, at the region grain.
SELECT r.region_id, r.region_name,
MAX(m.monthly_orders) AS highest_monthly_orders,
MIN(m.monthly_orders) AS lowest_monthly_orders,
MAX(m.monthly_orders) - MIN(m.monthly_orders) AS difference
FROM regions r
JOIN region_monthly m ON m.region_id = r.region_id
GROUP BY r.region_id, r.region_name;
```
## Leak-safety (hard constraint)
The example must use an **invented, generic schema** (`regions`, `region_monthly`) and made-up
columns — **no benchmark table names, SQL, or result values.** It teaches the *pattern* (cover
every requested output + identity + inputs), which is universal and tied to no specific instance.
## Acceptance criteria
- The skill states the multi-part-completeness rule and a concrete **final completeness check**
(re-read question → verify metrics + identity + inputs + grain), inline and dialect-agnostic,
cross-referencing the existing identity/inputs rules so they're enforced.
- Includes the over-projection guard (don't pad with extra columns — that's grader-gaming).
- One short generic example (wrong vs complete); no benchmark-derived content.
- Skill-content only; analytics-skill content tests updated to cover the new rule + check.
## Benchmark context (motivation only)
In the latest SQLite-subset run, **incomplete output was the single largest failure bucket
(~13 of 51 voted failures)**: multi-part questions answered partially, and identity / derived-value
inputs dropped — the latter two being spec-07 rules that already exist but weren't applied. A
probe with a much stronger model reproduced the *same* incomplete-output failures, confirming this
is a craft-enforcement gap rather than a model-capability one. The fix — answer every requested
part, identify entities, keep inputs — is universal analyst craft, so it belongs in the product
skill (and transfers to real users), enforced as a final check rather than left as a passive hint.
```

116
spider2-specs/done/15-mcp-server-structured-logging.md
View file

@ -1,116 +0,0 @@
# Structured, leveled logging for the ktx MCP server
> **Scope: observability only.** This spec is about *seeing* what the MCP server
> does (which tool, what params, when, how long, outcome). *Preventing* a runaway
> query from blocking the server (off-event-loop / interruptible query execution)
> is a separate concern — see "Non-goals" and the sibling spec note below.
## Problem
The ktx MCP server (`packages/cli/src/mcp-http-server.ts` +
`mcp-server-factory.ts`; raw `node:http` + `@modelcontextprotocol/sdk`
`StreamableHTTPServerTransport`) emits almost no operational logs. There is no
server-side record of **which MCP tool was called, with what parameters, when,
how long it took, or whether it succeeded** — nor of session open/close or
transport errors. When a tool call is slow, hangs, or a client connection drops
("Transport channel closed"), an operator has no trail to diagnose it and must
resort to process sampling / `lsof` / guesswork — and the offending input
(e.g. the exact SQL) is typically unrecoverable.
## Generic use case
Anyone running a long-lived ktx MCP server — a developer's local instance, a
shared team server, or a hosted deployment — needs observability into tool-call
activity to:
- diagnose slow or hung tool calls (which `sql_execution` ran, against which
connection, with what SQL, for how long);
- explain client-visible connection failures from the server side (session
lifecycle, transport-closed events);
- audit what agents asked the server to do;
- spot patterns (hot tools, slow connections, error rates).
This is standard production-server hygiene; the server currently provides none.
## Requirements (sketch — refine when picked up)
1. **One structured (JSON) logger, low overhead.** Suggested `pino` (orientation
only; implementer owns the choice). A single shared instance; write **JSON to
stdout** (12-factor — the launcher/aggregator routes it). No in-app file
rotation. Optional human-readable pretty output only when attached to a TTY
(dev).
2. **Configurable level via env** (e.g. `KTX_LOG_LEVEL`, default `info`; `debug`
for diagnosis) — verbose logging on demand without code changes.
3. **Per-session / per-call context** via child loggers: every line carries a
`sessionId` (from the transport session) and, for tool calls, a `callId` +
`tool` name, so one session's or call's activity can be traced/grepped.
4. **Tool-call logging — START logged BEFORE execution, COMPLETION after.** For
every MCP tool invocation:
- on entry: log `{ tool, params, sessionId, callId }` **before** running the
handler (so the record exists even if the handler never returns);
- on exit: log `durationMs` + outcome (ok with result size, or error with
stack).
This makes a **hung / never-returning call identifiable**: a start with no
matching completion is the culprit, with its exact parameters and timestamp.
This matters specifically because handlers like `sql_execution` run a
*synchronous* better-sqlite3 query — a runaway query blocks the process and no
completion is ever logged, so the start line (flushed before the blocking
call) is the only record. For `sql_execution`, `params` should include the SQL
text (the most useful field). Emit a **WARN** when a *completed* call exceeds a
configurable slow threshold (e.g. `KTX_SLOW_TOOL_MS`).
5. **Connection / session lifecycle:** log session open/close (with `sessionId`)
and transport errors (the SDK's closed-channel / "Transport channel closed"
events) so client-side connection failures have a server-side counterpart.
6. **Error logging** with structured stack traces (a standard error serializer),
not bare strings.
7. **Light redaction — credentials only** (bearer token, connection
passwords/secrets). SQL text and tool params are *not* secrets and must be
logged. Do not over-redact.
8. **Synchronous logging is fine.** The server uses a synchronous DB client, so
logging need not be async; prefer the simpler synchronous stdout path over
async/worker transports (which can lose buffered lines on a hard crash). Do
not introduce async-logging machinery.
## Acceptance criteria (sketch)
- With `KTX_LOG_LEVEL=debug`, invoking any MCP tool produces a `tool.start`
(tool, params, sessionId, callId) and a `tool.end` (durationMs, outcome) line
on the server's stdout, as JSON.
- A tool call that never returns (e.g. a runaway `sql_execution`) leaves a
`tool.start` line carrying its **exact SQL and timestamp** and **no**
`tool.end` — so the offending query is recoverable from the log alone, with no
process sampling.
- A completed tool call slower than the configured threshold emits a WARN with
its duration.
- Session open/close and transport-closed events are logged with the `sessionId`.
- At default level (`info`), routine per-tool lines are suppressed but lifecycle,
slow-call warnings, and errors are present.
- Credentials (bearer token, connection secrets) never appear in logs; SQL and
tool params do.
- No new heavy dependencies beyond the logger; no OpenTelemetry/metrics stack; no
async-transport machinery.
## Non-goals
- **Preventing/interrupting runaway queries** (off-event-loop execution, query
timeouts, worker-thread isolation). That is a *separate* spec; a single
synchronous query that fans out into a massive nested-loop join can peg the
single-threaded server for hours and break new connections — observability
surfaces *which* query, but the fix is execution-model work. (This logging is
also a prerequisite for a future watchdog that detects a `tool.start` with no
`tool.end` past a threshold and recycles the server.)
- Metrics/tracing/OpenTelemetry exporters.
- Forwarding logs to the MCP *client* via the protocol's logging capability
(`notifications/message`, `logging/setLevel`) — a possible later enhancement,
distinct from operational stdout logging.
## Benchmark context (motivation, not a requirement)
Running Spider 2.0-Lite against the MCP server at concurrency, an
adversarial-reviewer-generated query degenerated into a massive nested-loop join;
synchronous better-sqlite3 executed it on the event loop, pegging a server at
~100% CPU for hours and breaking new MCP connections to it ("Transport channel
closed"). We could not determine *which* query, because the server logs nothing
about tool calls — diagnosis required `sample`/`lsof` on the live process and the
exact SQL was never recovered. Structured tool-call logging (especially
start-before-execute) would have turned this into a one-line `grep` of the server
log.

131
spider2-specs/done/16-bounded-query-execution-timeout.md
View file

@ -1,131 +0,0 @@
# Bounded query execution (deadline + non-blocking) for read SQL
> Priority: HIGH. Found empirically during a Spider2-lite sqlite run
> (2026-06-18): a single `sql_execution` MCP call wedged a worker at 100% CPU
> for 13+ minutes and never returned. The query
> `SELECT MIN(time_id), MAX(time_id), COUNT(*) FROM profits` on the
> `complex_oracle` sqlite database hit a VIEW (`costs ⋈ sales`, 918,843 × 82,112
> rows, joined on a 4-column key with no composite index) whose plan degraded to
> an O(N×M) nested-loop scan. Because the sqlite connector runs
> `better_sqlite3 .all()` **synchronously with no timeout**, it blocked the MCP
> worker's entire event loop: no `tool.end` was ever logged, the port went
> unresponsive, and the query could not be cancelled. One of four eval shards
> stalled until the worker was killed by hand.
## Problem
Two compounding gaps on the read-query path:
1. **No execution deadline.** A single expensive query runs unbounded. This is
handled divergently per connector, with no shared contract: BigQuery has a
real server-side job timeout (`job_timeout_ms`); ClickHouse has an HTTP
`request_timeout`; Snowflake, Postgres, MySQL, and SQL Server bound only
connection/pool *acquisition*, not statement *execution*; SQLite has nothing.
So whether a runaway query is bounded depends entirely on which driver the
caller happened to hit.
2. **In-process engines block the event loop and can't be cancelled.** The
sqlite connector executes on the main thread via synchronous
`better_sqlite3 .all()`. A slow query freezes the whole MCP server (it can't
serve other requests, send progress, or write `tool.end`), and there is no
way to interrupt it: better-sqlite3 exposes no interrupt/cancel API — its
documented mechanism for slow queries is to run them in a **worker thread**,
and the only way to stop a runaway synchronous query is to terminate the
thread executing it.
The net effect is a query that produces a `tool.start` with no matching
`tool.end`, an unresponsive server, and no self-recovery. A row cap (`maxRows`)
does not help — it bounds returned rows, not scan work, and the failing query
returned a single aggregate row.
## Generic use case
Any data agent that lets an LLM author SQL will eventually issue an
accidentally-expensive query — an unindexed or cartesian join, an expensive
VIEW, a wide aggregate over a large fact table. A general-purpose context layer
must bound that and return a clean, fast "query exceeded Ns" error so the agent
can revise (add filters, query base tables, narrow the range) instead of hanging
the tool and the server. This matters for embedded/local warehouses (sqlite,
duckdb) and remote ones alike, and is wholly independent of any benchmark.
## Requirements
1. Every read-query execution path (`executeReadOnly`) enforces a single
canonical execution deadline. One opinionated default; **not** a per-call
user flag. Where a driver already supports a per-connection timeout
(BigQuery `job_timeout_ms`), reuse that as the per-connection override rather
than inventing a parallel knob.
2. On exceeding the deadline the path resolves with a `KtxQueryError`
("query exceeded {N}s") — a finite, decision-reaching outcome, never an
unbounded hang.
3. The deadline is a **shared contract at the connector boundary**, defined once
(on the `executeReadOnly` contract or a shared wrapper at the call site) so
all drivers participate. Bring the existing divergent timeouts (BigQuery job
timeout, ClickHouse request timeout) under this one contract instead of
leaving parallel mechanisms.
4. For in-process engines (sqlite today, any future embedded driver), execution
MUST NOT block the MCP server event loop. Run the query off the main thread
and enforce the deadline by terminating that thread on timeout (the
better-sqlite3-documented approach, since synchronous queries are
uncancellable in-thread). The event loop must stay responsive so `tool.end`
is always written and concurrent requests on the same port are served.
5. Prefer real cancellation over client-side give-up. Where the engine supports
a server-side statement timeout (Postgres `statement_timeout`, MySQL
`max_execution_time`, Snowflake `STATEMENT_TIMEOUT_IN_SECONDS`, ClickHouse
`max_execution_time`, BigQuery job timeout, SQL Server request timeout), set
it so the deadline actually stops work, not merely abandons the promise while
the query keeps running. For in-process engines, thread termination is the
cancellation.
6. The MCP `sql_execution` tool surfaces the timeout as an expected error
(classified as `KtxQueryError`, not a `$exception` fault, consistent with
existing expected-error classification) and logs a `tool.end` with the error
outcome.
7. Read-only enforcement (`assertReadOnlySql`) and the `maxRows` row cap remain
unchanged. The deadline is additive; `maxRows` is not a substitute for it.
## Acceptance criteria
- A read query that exceeds the deadline returns a `KtxQueryError` within
roughly the deadline; the MCP worker stays responsive (a concurrent tool call
on the same server completes while the slow query is still pending) and writes
a matching `tool.end` with a non-ok outcome.
- sqlite specifically: executing a deliberately pathological query (e.g. an
expensive VIEW or an unindexed cross join) on a fixture does not block the
event loop, is terminated at the deadline, and CPU returns to idle afterward
(the off-main-thread executor is killed, not left spinning).
- No regression: normal fast queries return identical results; read-only
rejection still works; `maxRows` still bounds returned rows.
- Tests cover the deadline path for at least the in-process driver (sqlite,
terminate-on-deadline) and one server-side-timeout driver.
## Benchmark context (motivation only)
The Spider2-lite local set loads several warehouses into sqlite, some with
expensive VIEWs over large fact tables — e.g. `complex_oracle.profits` =
`costs ⋈ sales` on `(prod_id, time_id, channel_id, promo_id)`, 918,843 × 82,112
rows, no composite index, with `promo_id` (the index the optimizer picks) being
95.5% a single value. LLM-authored profiling queries (MIN/MAX/COUNT over such a
view) trigger O(N×M) nested-loop scans. Without a deadline these hang an eval
shard for 10+ minutes; with one, the agent gets a fast error and can scope the
query instead.
## Orientation hints (code pointers; may have drifted)
- Shared contract: `packages/cli/src/context/scan/types.ts` —
`KtxScanConnector.executeReadOnly` (~343), `KtxReadOnlyQueryInput` (~285).
- MCP call site: `packages/cli/src/context/mcp/local-project-ports.ts:70`
(`connector.executeReadOnly`); tool registration in
`packages/cli/src/context/mcp/context-tools.ts`.
- In-process sync execution (the acute hang):
`packages/cli/src/connectors/sqlite/connector.ts:311-313`
(`better_sqlite3 .prepare().all()`).
- Existing divergent timeouts to unify: `connectors/bigquery/connector.ts`
(`job_timeout_ms` / `jobTimeoutMs`), `connectors/clickhouse/connector.ts:602`
(`request_timeout`), `connectors/snowflake/connector.ts:342` (test/pool only),
`connectors/postgres/connector.ts`, `connectors/mysql/connector.ts`,
`connectors/sqlserver/connector.ts` (pool/connection only).
- Error class: `packages/cli/src/errors.ts:25` (`KtxQueryError`).
- better-sqlite3 (context7 `/wiselibs/better-sqlite3`, v12.x): no
interrupt/cancel API; `docs/threads.md` documents the worker-thread pattern
for slow queries (master owns worker lifecycle and respawns on exit) — extend
it with terminate-on-deadline to enforce the timeout.

68
spider2-specs/done/18-bigquery-cross-project-datasets.md
View file

@ -1,68 +0,0 @@
# 18 — BigQuery cross-project dataset support (introspect foreign-hosted datasets, bill in own project)
**Status:** intake draft (todo). Requirement-level; the implementer refines into `specs/18-…`.
## Problem (generic, real-world)
Analysts routinely query datasets that live in a **different** BigQuery project than the one
they bill jobs to — Google's `bigquery-public-data`, a partner's shared project, an
organization's central data project, etc. To make those connectable in ktx (so `discover_data`,
the semantic layer, dictionary sampling, and `sql_dialect_notes` work), ktx must be able to
**introspect a dataset hosted in a foreign project while running/billing jobs in the
credentials' own project**.
Today it can't. ktx's BigQuery connector derives a single `projectId` from
`credentials.project_id` and uses it for **both** job billing **and** schema introspection:
- `connectors/bigquery/connector.ts:294` — `projectId` is read only from `credentials.project_id`;
there is no separate billing-vs-dataset project knob.
- `:544` (`introspectDataset`) — calls `this.getClient().dataset(datasetId)`, which resolves the
dataset **in the client's (billing) project**, and labels every table `catalog: this.resolved.projectId`.
- `:453` (`listTables`) — queries `\`${projectId}\`.\`region-…\`.INFORMATION_SCHEMA.TABLES`, i.e. the
**billing** project's INFORMATION_SCHEMA.
- `:163` (`datasetIds()`) — returns `dataset_ids` verbatim; it never parses a `project.` prefix.
So a `dataset_id` naming a dataset in another project can't be introspected, even though querying
it works fine (cross-project reads bill to the caller's project — that path already works).
### Empirical confirmation
With a service account in project `ktx-spider2-lite`:
- ktx's call pattern `client.dataset("austin_311")` → **`404 NotFound`** (looks in
`projects/ktx-spider2-lite/datasets/austin_311`).
- The cross-project form `DatasetReference("bigquery-public-data","austin_311")` → **succeeds**
(lists the public tables; public metadata is readable by any authenticated principal).
- There is **no config knob** to separate the introspection project from the billing project.
## Requirement
The BigQuery connector must accept **fully-qualified `project.dataset` entries** in `dataset_ids`
(a single connection may span more than one source project), and for each:
- **introspect** via the *dataset's* project — `client.dataset(id, { projectId })` /
`DatasetReference(project, dataset)`, query the **dataset project's** `INFORMATION_SCHEMA`, and
label the table `catalog` with the dataset's project;
- **run jobs / bill** in `credentials.project_id` (unchanged).
A bare `dataset` (no `project.`) keeps today's behavior (resolve in `credentials.project_id`), so
existing single-project connections are unaffected.
## Acceptance
- `dataset_ids: ['bigquery-public-data.austin_311']` (credentials in a *different* project) →
`ktx ingest <conn>` introspects the tables, enriches, and samples values; `discover_data` /
`dictionary_search` return them.
- A connection mixing `['bigquery-public-data.x', 'other-project.y']` introspects both.
- `sql_execution` of a fully-qualified `project.dataset.table` query still runs and bills in
`credentials.project_id`.
- Single-project `dataset_ids: ['my_dataset']` behaves exactly as before (no regression).
## Benchmark context (motivation only — do not encode benchmark specifics)
Spider 2.0-Lite's **BigQuery slice (205 questions)** is otherwise **unservable faithfully**: every
one of its ~74 logical databases groups datasets hosted in foreign public projects
(`bigquery-public-data`, `isb-cgc-bq`, `data-to-insights`, …), never in a project we own. Query
execution already works cross-project (proven), but ktx-only *discovery* (the whole point of the
faithful surface) is blocked because the connector can't introspect them. Scope is small: of 74
BQ dbs only **1** spans more than one source project, so "let `dataset_ids` carry `project.dataset`
and introspect each in its own project" covers the benchmark and the general case alike. This is
the sole blocker for the BigQuery leaderboard slice (the Snowflake slice needed no connector
change and is already baselined).

89
spider2-specs/done/19-durable-bounded-relationship-detection.md
View file

@ -1,89 +0,0 @@
# 19 — Durable, resumable, bounded relationship detection during ingest enrichment
**Status:** intake draft (todo). Requirement-level; the implementer refines into `specs/19-…`.
## Problem (generic, real-world)
Ingest enrichment runs three stages in a fixed order inside `runLocalScanEnrichment`
(`packages/cli/src/context/scan/local-enrichment.ts`):
1. `descriptions` (`:530`) — per-table LLM descriptions (the expensive step: one model call per
table; on a large schema this is minutes of paid LLM work).
2. `embeddings` (`:559`) — column embeddings.
3. `relationships` (`:593`) — FK/join discovery: profiles a row sample of **every** table, then
validates candidate joins.
The queryable semantic-layer artifacts are persisted **once, at the very end**, by
`writeLocalScanEnrichmentArtifacts` in `local-scan.ts:510` — which runs **after**
`runLocalScanEnrichment` returns, i.e. after all three stages.
This creates three failure modes that compound on large schemas (hundreds of tables):
1. **Enrichment is lost if relationship detection is interrupted.** The descriptions + embeddings
are computed and held in memory, but they only reach the durable, queryable artifacts when the
final write runs after the `relationships` stage. If the process is killed/crashes/times out
**during** relationship detection (the last, slowest, silent stage), the artifacts are never
written — the schema survives (it was written earlier at `local-scan.ts:473`) but **all the
paid LLM enrichment is discarded**. Empirically: ingesting a 95-table BigQuery dataset produced
full descriptions + embeddings (progress reached "Building embeddings 17/17"), then the
relationships stage ran silently past a supervising deadline and was killed — the persisted
`_schema` had **0** AI descriptions, only the native column comments. Every larger dataset hits
this, so the most expensive work is the most likely to be thrown away.
2. **Re-running does not resume — it re-spends.** There is a stage state store
(`SqliteLocalScanEnrichmentStateStore`) and a `runEnrichmentStage` helper (`:413`) that saves
each completed stage's output. But the completed-stage lookup keys on **`runId`**
(`findCompletedStage({ runId, stage, inputHash })`, `:427`), and `runId` is fresh per ingest
invocation. So resume only works *within* a single run; re-running an interrupted ingest gets a
new `runId`, misses the cache, and **re-computes descriptions + embeddings from scratch**
(re-paying for the LLM work that already succeeded).
3. **Relationship detection is unobservable and unbounded.** The stage emits no progress between
"Detecting relationships" and the final "Relationship detection found N accepted" — minutes of
silence on a large schema. A supervisor watching for liveness cannot distinguish a slow-but-
working profile from a true hang, and there is no internal time/work budget, so on a very large
schema it can run far longer than any reasonable deadline.
## Requirements
1. **Checkpoint queryable artifacts before relationship detection.** Persist the descriptions +
embeddings into the semantic-layer artifacts as soon as the `embeddings` stage completes, before
the `relationships` stage runs. Relationship detection then appends/merges its own artifact on
completion. Net: the expensive LLM + embedding enrichment is **always durable and queryable**,
even if relationship detection fails, is interrupted, or is skipped. (A failed/partial
relationship stage should degrade to "no/partial joins", never to "no descriptions".)
2. **Make stage resume work across runs.** Resolve a completed stage by stable content identity
— `(connectionId, stage, inputHash)` — independent of `runId`, so re-running an interrupted
ingest resumes the finished `descriptions`/`embeddings` stages from cache and only re-runs what
actually failed (e.g. `relationships`). Re-running after an interruption must not re-spend LLM
credits on stages that already succeeded.
3. **Make relationship detection observable and bounded** (mirrors spec 16's bounded query
execution). Emit progress through the existing progress port — e.g. "Profiling table K/N",
"Validating candidate K/M" — so liveness is visible. Enforce an overall time/work budget
(configurable, e.g. under `scan.relationships`) so on a very large schema the stage stops
gracefully and returns the relationships found so far (partial) rather than running unboundedly.
Partial completion is persisted (per requirement 1) and marked as such.
## Acceptance
- Interrupting an ingest **during** relationship detection still leaves a queryable semantic layer
with the table/column descriptions + embeddings that were generated (verified: re-open the
connection, descriptions are present).
- Re-running an interrupted ingest **does not** regenerate descriptions/embeddings whose stage
already completed (verified: no LLM description calls for the cached tables; only the failed
stage re-runs).
- A connection with hundreds of tables emits relationship-stage progress and completes within the
configured budget, persisting partial relationships if the budget is hit — without discarding
enrichment.
- Small/single-run ingests behave exactly as before (no regression in artifacts or relationship
output when nothing is interrupted).
## Benchmark context (motivation only — do not encode benchmark specifics)
The Spider 2.0-Lite BigQuery slice has datasets with hundreds–thousands of tables (`ebi_chembl`
785, `fec` 486, `ga360` 366, …). Enriching them with claude-code costs real, rate-limited LLM
budget; losing that enrichment to a relationship-stage interruption — and re-spending it on every
retry — makes large-schema ingest impractical. This is a general durability/cost property of the
ingest pipeline, independent of the benchmark; the benchmark only made it acute at scale.

101
spider2-specs/done/20-resilient-enrichment-under-slow-llm.md
View file

@ -1,101 +0,0 @@
# 20 — Resilient enrichment under a slow/hung LLM backend
**Status:** draft (intake). Requirement-level; the implementer refines into `specs/20-*.md`.
This is the **enrichment-stage** analog of two already-shipped specs:
- spec 16 (bounded query execution) — bound *and actually cancel* a runaway read query (child-thread/process kill, not a cosmetic JS deadline);
- spec 19 (durable/bounded relationship detection) — checkpoint expensive ingest work so an interruption doesn't lose it.
Spec 16 hardened the **read-query** path and spec 19 checkpointed at **stage boundaries**. The same two
weaknesses still exist *inside the descriptions enrichment stage*, and together they turned a single hung
table into an indefinite wedge plus total loss of an entire stage's LLM work.
## Problem / requirement
Two compounding gaps on the per-table description-enrichment path, observed end-to-end:
### 1. The per-table LLM timeout does not actually terminate the work
The per-table `generateObject` enrichment call is wrapped in `retryAsync` with a fresh
`AbortSignal.timeout(KTX_ENRICH_LLM_TIMEOUT_MS)` per attempt (ktx commit `01f63380`). When the LLM
backend is a **subprocess** (the `codex` backend spawns a child `codex` process; `claude-code` likewise
spawns a child) and that child **hangs with an open connection to the provider** (TCP ESTABLISHED, ~0%
CPU, no bytes flowing), the JS-level `AbortSignal` fires but **does not kill the child process or unblock
the await** — so the call sits *past* its own timeout indefinitely.
Observed (BigQuery ingest, codex backend, 2026-06-23): with `KTX_ENRICH_LLM_TIMEOUT_MS=1800000` (30 min),
two of `covid19_usa`'s widest tables (252 columns) hung; the stage sat at **268/285 for 41+ minutes** —
well past the 30-min per-attempt timeout — with exactly two codex children, each holding 3 ESTABLISHED
connections at ~0% CPU, until killed by hand. The timeout was cosmetic: it never terminated the hung
child. (This is precisely the failure mode spec 16 fixed for SQL — a deadline that fires in JS but cannot
interrupt the underlying work — applied to the enrichment LLM call instead of the query.)
**Requirement:** the per-table enrichment-call timeout must be **enforced**, not advisory — when it fires,
the in-flight work is actually cancelled (subprocess SIGKILL for process-backed providers; request abort
for HTTP-backed ones) and the call returns/throws *promptly* so the stage can proceed (skip the table per
the existing no-retry-on-timeout policy). A hung table must cost at most ~one timeout, never unbounded
wall-clock. Provider-agnostic: it must hold for `codex`, `claude-code`, and HTTP backends alike.
### 2. Descriptions are checkpointed only at full-stage completion, so a few bad tables lose all the good ones
Spec 19 persists the descriptions checkpoint **after the descriptions stage completes** (before
relationships). There is no *within-stage* persistence: while the stage runs, every enriched table's
description lives only in memory. So if the stage cannot complete — e.g. 2 tables out of 285 hang (gap #1),
or the process is killed, or it hits the stall watchdog — **all** the already-enriched tables are lost,
even though their (expensive) LLM descriptions were finished.
Observed (same run): `covid19_usa` had **283/285** tables enriched in memory but **0** rows in
`local_scan_enrichment_stages` and **0** `ai:` descriptions on disk; killing the wedged ingest discarded
all 283, forcing a from-scratch re-ingest. The cost of 2 pathological tables was 283 tables' worth of
redone LLM calls.
**Sharper observation (re-ingest with a short, enforced timeout):** even when the stage *does* run to
the end — the 2 hung tables hit a 4-min timeout and were skipped, so 283/285 descriptions were generated
and the ingest reported success (`Scan completed` / `Ingest finished`, embeddings built, exit 0) — the
descriptions were **still persisted as 0** (0 `ai:` on disk, 0 stage rows). So the discard is **not** just
"lost on kill": a stage that completes with *any* skipped/aborted table currently persists **nothing**,
throwing away every successfully-generated description. The skip must be graceful — a skipped table costs
one missing description, not the entire stage's output. (This is the strongest argument for per-table
incremental persistence: the 283 good descriptions should have been durable the moment each was produced.)
**Requirement:** persist enriched descriptions **incrementally** (per-table or per-batch) during the
descriptions stage, so that (a) tables that finished are durable even if the stage never completes, and
(b) a resumed ingest re-does only the *unfinished* tables, not the whole stage. The existing additive-write
design (spec 19 already preserves existing descriptions on re-ingest) is the foundation; this extends the
checkpoint granularity from once-per-stage to incremental.
## Sketch (implementer to refine)
- **Enforced timeout:** route enrichment-call cancellation through real termination — kill the codex/
claude-code child process on timeout (reuse spec 16's child-kill mechanism), abort the HTTP request for
network backends. A fired `AbortSignal` must guarantee the await settles within a bounded grace period.
- **Sane default + the right tradeoff:** the default per-table timeout should be **moderate** (single-digit
minutes) with a small retry count, not very large — because the cost of a *hang* is the timeout value
itself, a long timeout is strictly worse for hangs. (The 30-min value used in the incident was an operator
override chosen to avoid cutting off slow-but-completing wide tables; with #1 enforced and incremental
checkpointing, a moderate default + skip is the better operating point.)
- **Incremental persistence:** flush descriptions per-batch (e.g. every N completed tables or on a timer) to
the same store/format used at stage completion; on resume, treat already-persisted tables as done and only
enrich the remainder. Keep it idempotent and additive (don't clobber prior descriptions).
- **Interaction with the stall watchdog:** with #1 enforced, no single table can starve progress for longer
than ~one timeout, so an external stall watchdog stops being the only backstop.
## Generic use case (independent of the benchmark)
Anyone ingesting a large or wide schema with an LLM enrichment backend (especially a *subprocess* backend,
which is the common local/desktop setup) will eventually hit a table whose description call hangs — a
provider stall, a rate-limit black-hole, a pathologically large prompt. Without an *enforced* timeout, one
such table wedges the whole ingest indefinitely; without *incremental* persistence, any interruption throws
away all the per-table LLM work already done (the dominant ingest cost). Both fixes make large-schema
enrichment **resilient and resumable** — a few bad tables degrade to a few skipped descriptions, not a
hung process and a from-scratch redo. This is core robustness for a general-purpose ingestion product,
wholly independent of any benchmark.
## Benchmark context (motivation only — not a benchmark-specific rule)
Surfaced during the Spider 2.0-Lite **BigQuery** ingest (2026-06-23, codex enrichment backend). Re-enriching
the giant public datasets, `covid19_usa` wedged at 268/285 for 41+ minutes on 2 hung 252-column tables; the
30-min per-table `AbortSignal` timeout never killed the hung codex children, and because descriptions
checkpoint only at stage completion, the 283 already-enriched tables were unrecoverable — the operator had
to kill, cache-bust, and re-ingest the db from scratch (with a short timeout as a stopgap). The benchmark
just exercised a large/wide multi-dataset ingest at scale; the gap and the fix are generic.

91
spider2-specs/done/21-selective-enrichment-stages.md
View file

@ -1,91 +0,0 @@
# 21 — Selective enrichment stages (`--stages`) + per-stage cache keys
**Status:** draft (intake). Requirement-level; the implementer refines into `specs/21-*.md`.
Follow-on to spec 19 (durable/resumable relationship detection) and spec 20 (resilient enrichment).
Those made enrichment *survivable and resumable*; this makes it *selectively re-runnable* — re-run one
enrichment stage without re-paying for the others.
## Problem / requirement
Enrichment has three stages — **`descriptions`** (per-table LLM text), **`embeddings`**
(sentence-transformers over the schema/descriptions), **`relationships`** (FK/join detection, optionally
LLM-proposed). Today you cannot re-run a *subset* of them, and three facts in the current code make a
targeted re-run impossible without a full, expensive re-enrich:
1. **One coarse cache key gates all three stages.** `context/scan/local-enrichment.ts:611` computes a
single `inputHash` from `{snapshot, mode, detectRelationships, providerIdentity, relationshipSettings}`,
and all three stages reuse it (descriptions ~`:641`, embeddings ~`:672`, relationships ~`:728`). So
changing *any* one stage's inputs invalidates *every* stage's cache. Concretely: flipping
`scan.relationships.llmProposals`, switching the LLM backend, or upgrading the embeddings model forces
ktx to re-run the **expensive per-table descriptions** even though they didn't conceptually change.
2. **No CLI surface to select stages.** The enrichment internally already supports a relationships-only
path (`mode: 'relationships'`, which skips the description/embedding stages — they're gated on
`mode === 'enriched'`), but `ktx ingest` exposes no flag to invoke it (only `--no-query-history`).
The capability is built; it's just not reachable.
3. **The per-stage storage already exists** (`local_scan_enrichment_stages` PK `(connection_id, stage,
input_hash)`) and the **additive write already preserves existing descriptions** on re-ingest — so the
foundation for "touch one stage, keep the rest" is in place; only the key granularity and the CLI
surface are missing.
**Requirement:** let an operator re-run a chosen subset of enrichment stages on already-ingested
connection(s), recomputing only those stages and **preserving the others' artifacts untouched** — cheaply,
without re-running unchanged (especially the costly `descriptions`) stages.
## Design decisions (resolved during intake; implementer may refine)
- **CLI flag: `--stages <comma-list>`** (plural). Accepts a comma-separated subset of
`descriptions,embeddings,relationships`; default = all three (current behaviour). Plural because it takes
a *set*; `--stages relationships` and `--stages descriptions,embeddings` both read naturally, and the
plural signals "list expected" (singular `--stage` implies exactly one). **Validate** the names — an
unknown stage is an error, never silently ignored.
- **Per-stage `inputHash`.** Split the single coarse hash so each stage keys on *only its own* inputs:
- `descriptions` → `{snapshot, mode, providerIdentity}` (NOT relationship settings, NOT embedding model)
- `embeddings` → `{snapshot, embeddings model/provider, + the description text it embeds}`
- `relationships`→ `{snapshot, relationshipSettings (incl. llmProposals), providerIdentity}`
Then flipping `llmProposals` invalidates only `relationships`; swapping the embeddings model invalidates
only `embeddings`; improving description prompts/LLM invalidates only `descriptions`.
- **Preserve-others semantics.** Stages not named in `--stages` are left exactly as on disk (additive write,
already the behaviour). A selective run never deletes another stage's artifacts.
- **Downstream-staleness handling.** Stages have a dependency order (`descriptions → embeddings`;
`relationships` depends only on the schema snapshot). Re-running `descriptions` alone can leave existing
`embeddings` semantically stale (they embedded the old text). The run must **warn** when a selected
re-run leaves an unselected downstream stage stale, and the operator can opt to cascade
(`--stages descriptions,embeddings`). Do not silently leave a stale-but-unflagged downstream.
- **`relationships` uses existing descriptions as context.** When re-running `relationships` only, the
stage should read the existing enriched schema (incl. on-disk `ai:` descriptions) so `llmProposals` has
full context — not just raw column names.
- **Scope:** the three enrichment stages for now. Design the stage-name namespace so it can later extend to
the broader scan phases (schema / query-history / source / memory) and subsume the inconsistent
`--no-query-history` negative flag, but that unification is out of scope here.
## Sketch (implementer to refine)
- Add `--stages` to `ktx ingest`; parse+validate into a stage set; thread it to the enrichment entry so it
selects which stage blocks run (reuse the existing `mode`/stage gating — `mode: 'relationships'` is the
precedent).
- Replace the single `computeKtxScanEnrichmentInputHash` call with per-stage hash computation keyed on each
stage's own inputs; gate each stage's resume/skip on its own hash.
- Ensure selective runs read + preserve the on-disk enriched schema and write additively.
- Emit a clear staleness warning when an unselected downstream stage is invalidated by a selected one.
## Generic use case (independent of the benchmark)
Any team running ktx in production maintains its semantic layer over time: they improve description prompts
or switch the description LLM, upgrade the embeddings model, or turn on LLM-proposed joins. Today each of
those forces a **full re-enrich of every connection** — re-running the expensive per-table descriptions
even when only embeddings or relationships changed. Selective `--stages` re-runs makes these routine
maintenance operations cheap and targeted: "re-embed everything on the new model" or "backfill joins now
that llmProposals is on" become a single fast pass that leaves the untouched stages — and their cost —
alone. This is core operability for a long-lived ingestion product and is wholly independent of any
benchmark.
## Benchmark context (motivation only — not a benchmark-specific rule)
Surfaced during the Spider 2.0-Lite multi-backend ingestion (2026-06-24). A level-aware audit found (a) a
tail of BigQuery dbs with poor *column*-description coverage (`google_dei` ~1%, `gnomAD`, `usfs_fia`, …)
that want a **`descriptions`-only** re-run with a longer timeout, and (b) a desire to **backfill joins**
across all already-ingested dbs after enabling `llmProposals` — without re-paying for descriptions. Both
were blocked by the coarse single `inputHash` (flipping `llmProposals` or re-describing would invalidate
the whole enrichment) and the absence of a stage-selective CLI flag. The benchmark just exercised
large-scale multi-backend ingestion; the gap and the fix are generic.

300
spider2-specs/specs/01-connection-scoped-wiki.md
View file

@ -1,300 +0,0 @@
# Connection-scoped wiki pages
> Refined spec. Intake draft: `todo/01-connection-scoped-wiki.md`.
## Problem
Wiki pages have only two scopes today: `GLOBAL` and `USER`
(`packages/cli/src/context/wiki/types.ts`, `WikiScope`). Scope is expressed by
directory (`wiki/global/<key>.md`, `wiki/user/<userId>/<key>.md`) and the
search path filters by loading only the in-scope pages before any lane runs.
There is no way to associate a page with a **connection** (a warehouse/database
defined under `connections:` in `ktx.yaml`).
In a project with many connections this causes two distinct failures:
1. **Cross-database relevance pollution.** All pages share one search index, so
`wiki_search` for a generic term (`orders`, `revenue`, `average order
value`) surfaces pages written about the wrong database. Concept names
collide across databases constantly in real multi-connection projects
(several databases each with `orders`, `customers`, …).
2. **Silent overwrite on shared keys.** Page keys are a flat, global namespace.
The write path resolves a repeated key to the existing file and updates it
in place. So if the agent writes an `orders` page while ingesting database B
and an `orders` page already exists for database A, B's content **overwrites
A's** — same-concept pages for different databases cannot coexist today.
Today, when `memory_ingest` is called with a `connectionId`, that id only
scopes which semantic-layer sources the triage agent can see
(`memory-agent.service.ts`); it is **not** persisted on the resulting wiki page
and **not** validated against `ktx.yaml`.
## Generic use case
Any org with multiple databases/warehouses in one **ktx** project: org-wide
definitions ("fiscal year starts in February") should be visible everywhere,
while database-specific conventions ("in the events DB, `user_id` is the
anonymous device id, not the account id") should not pollute searches about
other databases — and two databases that both have an `orders` concept must be
able to keep separate, non-colliding pages.
## Model
`connections` is **additive frontmatter metadata**, orthogonal to the existing
`GLOBAL`/`USER` directory scope — not a third scope dimension:
- A page is still `GLOBAL` or `USER` and lives where it lives today. It may
**additionally** carry a `connections` list.
- **Page keys remain a flat, globally-unique namespace.** `connections` does
**not** namespace keys; a page is addressable by key alone, unchanged.
- A page may list **multiple** connections.
- **Absent or empty `connections` ⇒ unscoped: the page applies to all
connections.** This is exactly today's behavior, so every existing page is
unaffected.
This keeps `wiki_read` and refs untouched and adds no parallel scope axis;
filtering by connection is purely a search/relevance concern.
## Requirements
### 1. Frontmatter field
Add an optional `connections` field to wiki page frontmatter — a list of
connection ids.
- Accept a single string too; normalize to a list at parse time (reuse the
existing array-coercion helper used for `tags`/`refs`/`sl_refs`).
- Round-trips through parse/serialize without loss.
- Absent or empty ⇒ unscoped (see Model). Existing pages are unaffected by
construction.
### 2. Page identity and key distinctness
`connections` does not change how pages are identified or addressed:
- Keys stay flat and globally unique; `wiki_read(key)` is unchanged.
- Because the write path updates a page in place when its key already exists,
same-concept pages for different connections **MUST** use distinct keys
(e.g. `orders_sales_db` vs `orders_events_db`). Connection-distinctive keys
for database-specific pages are the primary mechanism (driven by write-path
prompt guidance, requirement 5).
- **Data-loss guard (code, not prompt):** a connection-scoped write whose key
matches an existing page whose `connections` scope is **disjoint** from the
incoming scope MUST surface a collision instead of silently overwriting the
existing page. (Updating a page within the same connection scope, or
broadening/narrowing its own `connections`, is a normal update — not a
collision.) The implementer owns whether the collision is a hard error or a
suffixed new key; it must not be a silent clobber.
### 3. Search filtering
Add an optional connection filter to the search surfaces:
- **MCP:** `wiki_search(query, connectionId?)` (`context-tools.ts`).
- **CLI:** `ktx wiki search` and `ktx wiki list` accept `--connection <id>`
(with `-c` alias), matching the `ktx sql` connection flag.
Semantics:
- With `connectionId: X` ⇒ return pages whose `connections` is empty
(unscoped) **∪** pages whose `connections` contains X.
- Without ⇒ current behavior, all pages.
- The filter **MUST** apply uniformly to **all three search lanes** (lexical
FTS5, semantic/embedding, token fallback) at the **candidate-source level**,
so each lane draws its full candidate pool from the already-scoped set. It
**MUST NOT** be a post-filter on the merged/ranked results — that would let
off-scope candidates consume both the per-lane pool and the final result
limit unevenly.
*Orientation:* the existing `GLOBAL`/`USER` scoping already filters at the
disk-load step that feeds both the in-memory token lane and the synced SQLite
index (`local-knowledge.ts`); the connection filter fits the same seam.
### 4. Index persistence
The `.ktx/db.sqlite` knowledge index is re-synced from files on every search.
The implementer owns whether to persist `connections` as index columns / a side
table, or to filter the loaded page-set before the per-search sync. The binding
requirement is the uniform-across-lanes behavior in requirement 3 — not a
specific schema.
*Trade-off note (non-binding):* filtering the loaded page-set re-syncs only the
scoped subset and gives up a little embedding-cache reuse when searches
alternate between connections (recompute is one embedding per scoped page per
connection switch — negligible at the scale this targets). Persisting
`connections` in the index avoids that at the cost of a schema addition and a
per-lane predicate. Either is acceptable.
### 5. Write path
- The memory agent's page-write tool (`wiki-write.tool.ts`) accepts a
`connections` input field with the same REPLACE semantics as
`tags`/`refs`/`sl_refs`: omit ⇒ keep existing on update; `[]` ⇒ clear to
unscoped; `[ids]` ⇒ set.
- When `memory_ingest` / the memory agent runs with a `connectionId`, prompt
guidance directs the agent to:
- set `connections: [connectionId]` on new **database-specific** pages, using
connection-distinctive keys; and
- leave `connections` empty for clearly **org-wide** content.
- This is **prompt guidance, not a code auto-default.** A connection-scoped
ingest must remain able to produce unscoped org-wide pages, so the tool must
not force the session's `connectionId` onto every page.
### 6. `wiki_read` and refs unchanged
Pages remain addressable by key regardless of scoping. `wiki_read`, `refs`, and
`sl_refs` semantics are unchanged; `connections` is a search/relevance concern
only.
### 7. Validation
Validation behavior splits by surface, because an explicit argument is a
typo-prone input while persisted content drifts independently of config:
- **Explicit argument** — a connection id supplied as a command/tool argument
(`wiki_search`/`memory_ingest` `connectionId`, `ktx wiki … --connection`)
MUST be validated against `ktx.yaml` connections and **rejected with a clear
error listing the configured ids** when unknown. Reuse the canonical
`project.config.connections[id]` check. This also closes the current gap
where `memory_ingest`'s `connectionId` is accepted unvalidated.
- **Persisted frontmatter** — a connection id that appears only in a stored
page's `connections` and is not in `ktx.yaml` MUST **warn (not fail)** during
validation/doctor, and MUST NOT break loading, searching, or reading that
page. Config and content can evolve independently.
### 8. Scope boundary
This spec delivers the **mechanism** (frontmatter storage + uniform filter +
write surface + validation). Driving the agent to actually pass `connectionId`
during analytics work is the concern of
`03-multi-connection-routing-in-analytics-skill`. It composes with the
`--connection` flag on `ktx ingest` from `02-verbatim-ingest-mode`.
## Acceptance criteria
- A page with `connections: [db_a]` is returned by
`wiki_search(query, connectionId: "db_a")` and by an unfiltered search, but
**not** by `wiki_search(query, connectionId: "db_b")`.
- A page with no `connections` field is returned in all three cases above.
- Two pages — `orders_sales_db` (`connections: [sales_db]`) and
`orders_events_db` (`connections: [events_db]`) — coexist; a search scoped to
`sales_db` returns the first and not the second, and neither overwrote the
other on write.
- A connection-scoped write whose key matches an existing page scoped to a
**different** connection surfaces a collision instead of silently
overwriting (data-loss guard, requirement 2).
- Filtering works in each lane independently (test with embeddings disabled to
exercise the lexical and token lanes alone).
- `memory_ingest(content, connectionId)` produces a page scoped to that
connection for database-specific content.
- `wiki_search`/`ktx wiki search --connection <unknown>` fails with an error
that lists the configured connection ids.
- A page whose `connections` references an id absent from `ktx.yaml` produces a
warning but stays searchable and readable; search and read do not throw.
- `connections` accepts a single string and a list, both normalized to a list.
- Existing projects with no scoped pages and no `connectionId`/`--connection`
behave identically before/after.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the design.
- **Frontmatter type + parse/serialize:** `wiki/types.ts` (`WikiFrontmatter`),
`wiki/knowledge-wiki.service.ts` (`parsePage`/`serializePage`), array
coercion `wiki/local-knowledge.ts` (`stringArray`).
- **Search lanes + per-search re-sync:** `wiki/local-knowledge.ts`
(`searchLocalKnowledgePagesWithSqlite`; the disk-load step that already
scopes `GLOBAL`/`USER`; token lane), `wiki/sqlite-knowledge-index.ts`
(FTS5 `knowledge_pages_fts` lexical lane, semantic scan, `sync`).
- **MCP surface:** `mcp/context-tools.ts` (`wiki_search`, `wiki_read`,
`memory_ingest`; `connectionId` already present on `memory_ingest` but
unvalidated).
- **CLI surface:** `commands/knowledge-commands.ts`
(`ktx wiki search`/`list`/`read`); canonical `--connection` flag in
`commands/sql-commands.ts`; validation pattern
`project.config.connections[id]` in `mcp/local-project-ports.ts`.
- **Write path:** `wiki/tools/wiki-write.tool.ts` (input schema, REPLACE
semantics, scope decision), `memory/memory-agent.service.ts` (`connectionId`
threaded through the capture session and tool session;
`external_ingest` forces `GLOBAL` scope).
- **Connection config:** `context/project/config.ts` (`connections` record in
`ktx.yaml`).
## Benchmark context (motivation only)
Spider 2.0-Lite local subset = one project with ~30 SQLite connections whose
schemas share table/concept names (Northwind, sakila, two e-commerce DBs…).
External-knowledge docs (RFM definition, F1 overtake rules) are each relevant
to exactly one database and must not surface for the other 29.
## Implementation notes
Shipped on branch `write-feature-spec-wiki` (ktx worktree `tallinn-v2`). All
acceptance criteria covered; full package suite green (2924 passing),
type-check, knip/biome dead-code, and pre-commit clean.
**What was built / where**
1. **Frontmatter field (req 1).** `connections?: string[]` added to
`WikiFrontmatter` (`context/wiki/types.ts`) and to the file-layer page model
`LocalKnowledgePage` (`context/wiki/local-knowledge.ts`). Parsed via a new
`stringList()` coercion (single string → list); round-trips through both
serializers. Absent/empty ⇒ unscoped.
2. **Search/list filter (req 3, req 4).** `connectionId?` threaded through
`searchLocalKnowledgePages` → both the sqlite-FTS and scan impls →
`loadAllKnowledgePages`, and through `listLocalKnowledgePages`. The filter is
applied at the **disk-load seam** (`pageMatchesConnection`: unscoped ∪ pages
listing the id), so the token lane and the per-search SQLite sync (lexical +
semantic) both draw their candidate pool from the already-scoped set —
candidate-source level, not a post-filter.
- Chose req 4 **option B (filter the loaded page-set)** over persisting a
column. Verified-safe here: standalone ktx's memory agent reads pages from
files via a no-op `LocalKnowledgeIndex`, so `.ktx/db.sqlite`'s
`knowledge_pages` is a per-search cache that `searchLocalKnowledgePages`
rebuilds every call — scoping the sync corrupts no shared state. Only cost
is one embedding recompute per scoped page on a connection switch (the
spec's acknowledged, negligible trade-off). No index-schema change.
3. **Page identity + data-loss guard (req 2).** Keys stay flat/global;
`wiki_read`/refs unchanged. The write tool (`wiki/tools/wiki-write.tool.ts`)
rejects (hard error, no silent clobber) a connection-scoped write whose
incoming `connections` is **disjoint** from a same-key existing page's
non-empty `connections`, suggesting a connection-distinctive key. Same-scope,
overlapping, broaden/narrow, and unscoped-existing updates are allowed.
Chose a hard error over auto-suffixing so the conflict reaches the agent
(the decision-maker) instead of silently forking the key namespace.
4. **Write path (req 5).** `wiki_write` accepts `connections` (string or list)
with REPLACE semantics (omit ⇒ keep, `[]` ⇒ unscoped, `[ids]` ⇒ set); no
code auto-default of the session connection. Prompt guidance added to the
shared `wiki_capture` skill (new "Connection scoping" section) and the
`memory_agent_external_ingest` prompt. The session `connectionId` is now
surfaced to the agent so the guidance is actionable: in the memory-agent
prompt header and in the ingest work-unit `<context>` block
(`build-wu-context.ts`, fed from `ingest-bundle.runner.ts`).
5. **Validation (req 7).** New shared helper
`context/connections/configured-connections.ts → assertConfiguredConnectionId`
validates explicit connection-id arguments against `ktx.yaml` and throws an
error listing the configured ids. Routed from all three explicit-arg
surfaces: MCP `wiki_search` (`local-project-ports.ts`), MCP `memory_ingest`
(validated at the boundary in `mcp-server-factory.ts` — this also closes the
prior gap where `memory_ingest`'s `connectionId` was accepted unvalidated),
and CLI `ktx wiki --connection`/`-c` (`commands/knowledge-commands.ts` +
`knowledge.ts`). Persisted-frontmatter ids absent from config are **warn-only**:
`listReferencedConnectionIds` + a non-fatal `ktx status` warning
(`status-project.ts`); loading/searching/reading never throw on them.
**Deviations / notes**
- Req 1 says "reuse the existing array-coercion helper used for `tags`/`refs`".
That helper (`stringArray`) is array-only and does **not** coerce a single
string; added a dedicated `stringList` for `connections` to meet the
single-string acceptance criterion rather than change `stringArray`'s
behavior for the other fields.
- **Scope boundary kept:** `discover_data` (MCP) also searches wiki and already
takes `connectionId`, but req 3/8 scope the filter to `wiki_search` + CLI, so
its wiki lane is intentionally left unscoped. Worth a follow-up if
`discover_data`'s wiki results should also be connection-scoped for
consistency.
- MCP tools-list snapshot and the `mcp-server-factory` test were updated for the
new `wiki_search.connectionId` param and the `memory_ingest` validation
wrapper (the port is no longer the raw service object; it delegates).

327
spider2-specs/specs/02-verbatim-ingest-mode.md
View file

@ -1,327 +0,0 @@
# Verbatim ingest mode for authoritative documents
> Refined spec. Intake draft: `todo/02-verbatim-ingest-mode.md`.
## Problem
`ktx ingest --text/--file` routes captured content through the memory agent.
`runKtxTextIngest` (`packages/cli/src/text-ingest.ts`) builds a
`MemoryAgentInput` with `sourceType: 'external_ingest'` and hands it to
`MemoryAgentService.ingest` (`context/memory/memory-agent.service.ts`), which
runs a multi-step LLM triage loop (≈30-step budget, content clipped to ~48k
chars) inside a session worktree. The agent decides — via the `wiki_write`
tool — what to persist, so it may **rewrite, condense, split, or re-title** the
content before it lands as a wiki page. The body is produced by an LLM, not
copied by code.
For *authoritative* documents — formula definitions, metric specs, runbooks,
compliance text — paraphrasing is a defect, not a feature:
- exact thresholds, constants, and rule wording must survive unchanged;
- lexical (BM25/FTS5) search works best when the stored text matches the
phrasing users and agents query with;
- ingestion should be deterministic and reproducible — the same input file
yields the same page, and re-running is safe.
Two further gaps block authoritative ingest today:
- The memory agent hard-requires an LLM backend
(`context/memory/local-memory.ts` throws when `llm.provider.backend: none`
and no runner is injected), so there is **no** offline ingest path at all.
- The agent's write tool *merges* a repeated same-scope key in place (REPLACE
frontmatter semantics in `wiki/tools/wiki-write.tool.ts`), i.e. exactly the
silent in-place rewrite an authoritative-document workflow must avoid.
## Generic use case
Any team ingesting documents that are already the source of truth: metric
definition sheets, SLA documents, calculation-methodology docs, regulatory
text. The user wants **ktx** to *index and surface* the document, not to
re-author it. Today they work around the memory agent by hand-writing
frontmatter and copying files into `wiki/global/`; verbatim mode makes that a
first-class, supported `ktx ingest` workflow.
## Model
`ktx ingest --verbatim` is a **distinct, code-driven ingest path**, not a
constrained prompt over the existing agent loop. Its defining invariants:
- **The stored page body is the input document body, written by code.** The LLM
never produces, edits, or relays the body. It is confined to generating
*metadata* about the body.
- **Behavior follows from inputs, not from a mode prompt.** Whether metadata is
LLM-generated or derived offline follows from the configured backend
(`llm.provider.backend`), not from a second user-facing switch.
- **Pages are `GLOBAL`-scoped.** Verbatim ingest targets org/project
authoritative docs (the content teams copy into `wiki/global/` today).
Connection association is expressed by the **additive `connections`
frontmatter** from spec 01, never by directory.
- **Deterministic and idempotent.** The page key, the merged frontmatter, and
the stored body are all functions of the input alone (given a fixed backend),
so the same input produces the same page and a re-run is a safe no-op.
### "Byte-for-byte" scope
The guarantee is on the document's **interior**: no paraphrase, no condense, no
split, no re-title, no reflow, **no clipping**. The shared wiki store
canonicalizes *surrounding* whitespace — `parsePage` trims the body and
`serializePage` emits a single trailing newline
(`wiki/knowledge-wiki.service.ts`) — so leading/trailing blank lines are
normalized by the storage layer. Verbatim mode **MUST** write through that
shared `writePage`/`serializePage` path rather than fork a parallel serializer;
the interior bytes (thresholds, constants, wording) are what must be preserved
exactly, and they are. Acceptance hashes compare the stored body against the
**trimmed** input body.
## Requirements
### 1. Flag
`ktx ingest --file <path> --verbatim` and `ktx ingest --text <content>
--verbatim`. `--verbatim` is a boolean that applies to every `--file`/`--text`
item in the invocation; each item becomes its own page.
- It composes with the existing `--connection-id <id>` flag
(`commands/ingest-commands.ts`) so the resulting page can be
connection-scoped (see spec 01). **Note:** the intake draft wrote
`--connection`; the shipped flag is `--connection-id`. Use `--connection-id`.
- No new `--key` flag (see requirement 4). No second behavioral switch beyond
`--verbatim` itself.
### 2. Body preservation is enforced by code, not by prompt
The stored page body is the input content (interior preserved exactly, per
**Model → "Byte-for-byte" scope**).
- Verbatim mode **MUST NOT** route the body through the memory-agent LLM loop
or any `wiki_write` tool call where a model could alter it.
- The LLM, when used, generates **only** metadata: `summary`, `tags`, and
`sl_refs`. A single constrained structured-output call (AI SDK v6
`generateObject` with a `zod` schema) is the intended mechanism — the full
memory-agent loop, worktree, and squash-merge are **not** required and should
not be used.
- The page key is **not** LLM-generated (requirement 4).
### 3. No clipping of the stored body
The ~48k clip may apply only to the text **sent to the LLM** for metadata
generation. It **MUST NOT** apply to the text **written** to the page. A
document larger than the clip limit is stored in full; only its metadata is
derived from the clipped prefix.
### 4. Deterministic page key
The key is derived from the input, never chosen by the LLM (an LLM-chosen slug
would break determinism and the requirement-6 idempotency guarantee):
- **`--file <path>`** → `suggestFlatWikiKey(basename without extension)`
(`wiki/keys.ts`). This is the primary document case and is always
deterministic.
- **`--text <content>`** → if the content opens with a Markdown heading, the
key is `suggestFlatWikiKey(heading text)`. If there is no leading heading,
**hard error**: inline verbatim text needs a leading heading to derive a
stable key, or should be passed as `--file`.
- No hash-based keys (unfindable) and no `--key` override flag. A real need for
explicit key control can add `--key` later.
### 5. Frontmatter: passthrough + gap-fill
If the input has its own YAML frontmatter, split it from the body: the body is
everything after the closing `---`; the frontmatter is authoritative metadata.
- **Passthrough.** Every input frontmatter field is preserved in the stored
page, **including fields not in `WikiFrontmatter`** (`effective_date`,
`version`, `owner`, …). The serializer `YAML.stringify`s the object, so
unknown keys round-trip. Dropping them would be silent data loss on
authoritative docs.
- **Gap-fill only.** Generated/derived metadata fills **absent** fields only;
it **MUST NOT** overwrite an explicit value. An input `summary:` is never
replaced by a generated one; explicit `tags`/`sl_refs` are likewise kept.
- **Defaults.** `usage_mode` defaults to `auto` (findable via search, not
force-injected) when the input does not set it.
- **Connection scoping.** `--connection-id X` (validated via
`assertConfiguredConnectionId`, `context/connections/configured-connections.ts`)
sets `connections: [X]` when the input frontmatter does not already declare
`connections`. If the input frontmatter declares a **different**
`connections` than the flag, **hard error** (ambiguous intent) rather than
silently choosing one. If they match, or only one source is present, proceed.
### 6. Degraded mode (`llm.provider.backend: none`)
`--verbatim` **MUST** work with no LLM backend — this is its capability the
regular agent ingest lacks.
- `summary` is derived from the leading Markdown heading text, or, if none, the
first non-empty sentence of the body (trimmed to a reasonable length).
- `tags` and `sl_refs` are left empty.
- The body is still stored in full (requirement 3 applies unchanged).
### 7. Key collisions: idempotent-if-identical, else hard error
Verbatim mode does **not** reuse the agent write tool's in-place merge. Before
writing, read any existing `GLOBAL` page at the derived key:
- **No existing page** → write.
- **Existing page, stored body identical** to the new body (compared after the
storage-layer normalization in **Model**) → **idempotent no-op success**
(re-running the same file is safe).
- **Existing page, body differs** → **hard error** naming the conflicting key
and directing the user to a distinct key. Never a silent overwrite, never an
auto-suffixed second page (which would produce the duplicated/divergent pages
this mode must avoid).
### 8. LLM-failure handling
When a backend **is** configured but the metadata call fails (rate limit,
transport error, malformed output after retries), **fail the item** (honoring
`--fail-fast` and the per-item exit-code aggregation in `text-ingest.ts`).
**MUST NOT** silently fall back to degraded derivation: a degraded page written
on a transient error would, under requirement 7, refuse to be replaced by a
healthy re-run — breaking reproducibility. Degraded derivation is reserved for
`backend: none`.
### 9. Findability
After write, the page is reindexed so search returns it:
- `wiki_search` for a phrase taken from the document body returns the page via
the lexical lane (the body is indexed in `buildKnowledgeSearchText`).
- `wiki_search` for a paraphrase of the document's topic returns it via the
semantic lane **when embeddings are enabled** (this is what the generated
`summary`/`tags` buy over a bare degraded page).
## Acceptance criteria
- Ingesting a file with `--verbatim` produces a page whose body is
byte-identical to the trimmed input body (assert with a hash in tests).
- A >48k-char file is stored in full (assert stored body length ≥ input length
minus trim).
- Running the same `--verbatim` ingest twice is idempotent: one page, identical
bytes both times, no error on the second run.
- A second ingest to the same derived key with **different** body content fails
loudly (requirement 7) and does not modify the existing page or create a
suffixed one.
- Input frontmatter with an unknown field (e.g. `effective_date`) is preserved
in the stored page; an explicit input `summary` is **not** overwritten by a
generated one.
- With `llm.provider.backend: none`, `--verbatim` still produces a page: full
body stored, `summary` derived from the heading/first sentence, `tags` and
`sl_refs` empty.
- `--verbatim --connection-id X` yields a page with `connections: [X]`; an
unknown id is rejected with an error listing the configured ids. (Depends on
spec 01, now shipped.)
- `--verbatim --connection-id X` where the input frontmatter already declares a
different `connections` fails with an ambiguity error.
- `ktx ingest --text "no heading here" --verbatim` errors asking for a leading
heading or `--file`.
- `wiki_search` for a body phrase returns the page (lexical lane); for a topic
paraphrase it returns the page when embeddings are enabled (semantic lane).
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
module layout and design, subject to the invariants above.
- **Command flag:** `commands/ingest-commands.ts` (`ktx ingest` option table;
`--text`/`--file`/`--connection-id`/`--fail-fast` already present — add
`--verbatim` and thread it into `KtxTextIngestArgs`).
- **Orchestration:** `text-ingest.ts` (`runKtxTextIngest`, `loadItems`,
`validateItems`, per-item loop and exit-code aggregation). The verbatim flow
reuses item loading and replaces the `memoryIngest.ingest(...)` call with a
code-driven write for `--verbatim` items. Keep the new logic in a focused
module (e.g. a `verbatim-ingest` sibling) rather than swelling `text-ingest`.
- **Frontmatter split / write / serialize:** `wiki/knowledge-wiki.service.ts`
(`parsePage` for the `---…---` split shape, `serializePage`, `writePage`,
`readPage` for the collision check). Write through this shared path — do not
re-implement YAML framing.
- **Key derivation:** `wiki/keys.ts` (`suggestFlatWikiKey`, `assertFlatWikiKey`).
- **Frontmatter type:** `wiki/types.ts` (`WikiFrontmatter`; `summary` and
`usage_mode` are the required fields; unknown passthrough fields live
alongside).
- **Connection validation:** `context/connections/configured-connections.ts`
(`assertConfiguredConnectionId`, shipped with spec 01).
- **Metadata LLM call:** the local LLM runtime/config resolution in
`context/llm/` (e.g. `local-config.ts`; `backend: none` ⇒ no runtime). Use a
single `generateObject` call with a `zod` metadata schema; the `ai-sdk` skill
covers v6 patterns.
- **Reindex / search lanes:** `wiki/local-knowledge.ts`
(`loadAllKnowledgePages`, `buildKnowledgeSearchText`, the lexical/token/
semantic lanes) and `wiki/sqlite-knowledge-index.ts` (`sync`).
- **Tests:** extend `packages/cli/test/text-ingest.test.ts` and add a
verbatim-focused test file covering the acceptance criteria above.
## Benchmark context (motivation only)
Spider 2.0-Lite ships 8 external-knowledge markdown docs (RFM bucket
definitions, the haversine formula, F1 overtake rules, …). Gold SQL was
authored against their **exact** text; an LLM paraphrase that drops a bucket
boundary or rounds a constant loses the corresponding question. The current
workaround is hand-writing frontmatter and copying files into `wiki/global/`.
Verbatim mode turns that manual step into a supported **ktx** workflow, and
composes with the connection scoping from spec 01 so a doc relevant to exactly
one of the benchmark's ~30 SQLite databases does not surface for the other 29.
## Implementation notes
Shipped on branch `write-feature-spec-wiki`. All acceptance criteria are covered
by tests and verified end-to-end through the linked `ktx-dev` binary.
**What was built**
- New module `packages/cli/src/verbatim-ingest.ts`: `createLocalProjectVerbatimIngestor`
+ `LocalVerbatimIngestor`, plus the pure helpers `splitInputDocument`,
`deriveVerbatimPageKey`, `deriveDegradedSummary`, and `buildVerbatimFrontmatter`
(the last four are `@internal` exports for unit testing).
- `--verbatim` flag added to `ktx ingest` in `commands/ingest-commands.ts`, with a
guard that rejects `--verbatim` without `--text`/`--file`. The flag is threaded
into `KtxTextIngestArgs.verbatim`.
- `text-ingest.ts` now tags each loaded item with an `origin`
(`file` / `text` / `stdin`) and, when `verbatim` is set, constructs the verbatim
ingestor once and branches the per-item loop to a code-driven write instead of
`memoryIngest.ingest(...)`. The shared view, exit-code aggregation, and
`--fail-fast` handling are reused.
**Deviations from the literal spec (design refinements, per "implementer owns the design")**
- *Metadata call.* The spec suggested raw AI SDK v6 `generateObject`. The
implementation routes through the existing `KtxLlmRuntimePort.generateObject`
instead — it is implemented by all three backends (ai-sdk, claude-code, codex),
and the ai-sdk one already wraps `generateText` + `Output.object({schema})`.
This realizes the spec's "single constrained structured-output call" intent via
the canonical cross-backend path rather than forking a second LLM entry point.
- *Reindex (requirement 9).* In the standalone CLI, `searchLocalKnowledgePages`
rebuilds the SQLite index from disk on every call (recomputing embeddings for
changed pages), so a written page is findable without a dedicated reindex step.
The write still goes through the shared `KnowledgeWikiService.writePage` +
`syncSinglePage` path, so the page is also eagerly indexed.
- *Gap-fill optimization.* The LLM is skipped entirely when the input frontmatter
already supplies `summary`, `tags`, and `sl_refs` (generated metadata only fills
absent fields, so there is nothing to generate). A fully specified document thus
ingests with a configured backend without any LLM call.
**Tests**
- `packages/cli/test/verbatim-ingest.test.ts` — helper units + ingestor integration
against a real `initKtxProject` git repo (byte-identical body hash, >48k no-clip,
idempotency, conflict hard-error, frontmatter passthrough, explicit-summary
preservation, degraded mode, connection scoping + unknown-id rejection +
ambiguity error, no-heading inline error, LLM gap-fill, LLM-failure-fails-item,
lexical + semantic findability).
- `packages/cli/test/text-ingest.test.ts` — verbatim routing, origin tagging,
connection-id forwarding, fail-fast.
- `packages/cli/test/index.test.ts` — `--verbatim` flag threading and the
requires-`--text`/`--file` guard.
**Docs**
- `docs-site/content/docs/cli-reference/ktx-ingest.mdx` (flag, "Verbatim ingest"
section, examples, common errors) and
`docs-site/content/docs/guides/writing-context.mdx` (authoritative-document
workflow).
**Verification**
- Full CLI suite: 2959 passed, 1 skipped. `pnpm run build` and `pnpm run dead-code`
(Biome + Knip default + production) clean; pre-commit clean on changed files.
A pre-existing, unrelated type error in `test/mcp-server-factory.test.ts` is
untouched — it predates this work.

361
spider2-specs/specs/06-scan-tolerate-broken-objects.md
View file

@ -1,361 +0,0 @@
# Schema scan tolerates individual objects that fail introspection
> Refined spec. Intake draft: `todo/06-scan-tolerate-broken-objects.md`.
## Problem
A single broken or inaccessible object zeroes out an entire connection's
context. Schema introspection iterates objects with no per-object error
handling, so one throw aborts the whole scan, the live-database adapter's
`fetch()` rejects, and the connection ends with **no semantic layer at all** —
even when every other object was healthy.
The failure surfaces in two phases, and the contract must hold in both:
- **Metadata read (sqlite).** `connectors/sqlite/connector.ts` does
`rawTables.map((t) => this.readTable(...))` (≈ line 171) with no try/catch.
`readTable` runs `PRAGMA table_info(<object>)`, which *executes* a view's
body to resolve its columns — so a view over a dropped/renamed column (the
`oracle_sql` case: `emp_hire_periods_with_name` selecting `ehp.start_date`
from a base table that has no such column) raises `no such column:
ehp.start_date` and aborts introspection of all ~48 healthy objects.
- **Profiling read (warehouse drivers).** postgres/mysql/clickhouse/sqlserver/
bigquery/snowflake read metadata in bulk from catalog / `information_schema`
(a broken view rarely breaks that), then fail when a per-object profiling or
sampling `SELECT` runs against a broken object. Enrichment sampling is
*already* isolated (`description-generation.ts` wraps `sampleTable` in
try/catch → `sampling_failed`), but mandatory introspection-phase reads are
not uniformly isolated across drivers.
A second, related defect blocks the documented escape hatch. Setting
`enabled_tables: ["main.customers"]` on a sqlite connection produces a
different hard failure — `Adapter "database schema" did not recognize fetched
source output`. Root cause: the sqlite connector emits every object as
`{ db: null }` and filters the scope with `scopedTableNames(scope, { db: null })`
(`context/scan/table-ref.ts` ≈ line 47, `if (ref.db !== wantDb) continue`), but
`"main.customers"` parses to `{ db: "main", name: "customers" }`
(`context/scan/enabled-tables.ts`, `parseDottedTableEntry`). `"main" !== null`,
so the entry matches **nothing**, zero table files are written, and
`detectLiveDatabaseStagedDir` (`stage.ts` ≈ line 138) returns false, tripping
the generic "did not recognize fetched source output" error at
`context/ingest/local-stage-ingest.ts` (≈ line 291). The bare form
`enabled_tables: ["customers"]` would have worked; the `main.`-qualified form
silently matches nothing.
## Generic use case
Real warehouses routinely contain broken or inaccessible objects: views over
dropped/renamed columns, views referencing tables the connection role can't
read, permission-denied tables, and vendor system views that error on read.
**ktx** should ingest everything it *can* and skip what it can't, so one bad
object never zeroes out an entire connection's context. This is baseline
production robustness, independent of any benchmark — the same tolerance a
33-warehouse fleet needs the first time one of its databases has a stale view.
## Design
The unit of failure is **one object** (table or view). Introspecting or
profiling an object is an operation that can fail independently; a failure skips
that object, records a recoverable warning, and the scan continues from the
objects that succeeded.
Because seven Node connectors and the Python daemon each introspect differently
(sqlite reads metadata per-object via `PRAGMA`; warehouse drivers read metadata
in bulk and fail per-object during profiling), the **semantics** of "skip /
warn / total-failure" are defined **once** and every connector routes through
them — rather than seven copies of the same try/catch that drift apart:
- A shared per-object helper in the `scan/` layer — the sibling of the existing
`tryConstraintQuery` (`context/scan/constraint-discovery.ts`) — wraps a single
object read and returns `{ ok: true, table } | { ok: false, warning }`, with a
standard warning code (e.g. `object_introspection_failed`).
- A shared post-check enforces the total-failure rule (R3) uniformly.
- Each connector keeps its **natural** shape: sqlite routes each `readTable`
through the helper; bulk-read drivers route their per-object profiling reads
through it. The contract is uniform; the loop is not forced to be.
- The Python daemon implements the **same contract** in its own helper, adds a
`warnings` field to `DatabaseIntrospectionResponse`, and the Node adapter maps
those warnings into `KtxSchemaSnapshot` (`daemon-introspection.ts`).
The warning channel already exists end to end on the Node side
(`KtxSchemaSnapshot.warnings`, the `KtxScanWarning` shape with `table`/`column`/
`recoverable`, the `KtxScanWarningCode` enum, and the staged `warnings.json`
artifact written by `writeLiveDatabaseSnapshot`); sqlite simply never populates
it. This spec makes that channel carry object-skip warnings and surfaces them in
the ingest summary, the persisted report body, and `ktx status`.
## Requirements
### R1 — Per-object isolation (the contract)
If introspecting or profiling one object throws, the scan **MUST** skip that
object, record a `KtxScanWarning` (object name, the error message, and any
schema/catalog qualifier; `recoverable: true`), and continue with the remaining
objects. No single object may abort the scan.
- The contract holds in **both** phases: the mandatory metadata read *and* any
profiling/row-count/sample read performed during introspection.
- It holds for **all seven Node connectors**
(`packages/cli/src/connectors/<driver>/`) and the **Python daemon** postgres
path (R6).
- The semantics are defined once (the shared helper + warning code from the
Design section) and every connector routes through them. Do not inline a
divergent per-driver copy.
- Warnings **MUST NOT** carry secrets or full SQL bodies; record the object
identifier and the database's error text, redacted through the existing
`redactKtxSensitiveMetadata` path that `warnings.json` already uses.
### R2 — Surface, don't hide
Skipped objects **MUST** be reported both at ingest time and in the durable
status view:
- **Ingest summary.** The `ktx ingest` run summary (human-facing output) reports
a count plus the object name and a short reason for each skip — e.g.
`Skipped 1 object — emp_hire_periods_with_name: no such column ehp.start_date`.
- **Run report.** Object skips land in the run report's `warnings.json` artifact
(already written) and in the persisted report body (`IngestReportBody`), whose
natural home is the existing `fetch?: SourceFetchReport` field — the fetch
phase *is* introspection.
- **`ktx status`.** `ktx status` shows a per-connection skipped-objects line for
the connection's latest ingest — e.g. `oracle_sql: 1 object skipped —
emp_hire_periods_with_name: no such column ehp.start_date`. This is **derived
from the latest persisted report, not new persisted state**: the report body
is already stored whole as a JSON blob (`local_ingest_reports.body_json`), so
surfacing it requires **no `.ktx/db.sqlite` schema migration** — `status`
reads and renders the skip info already present in the latest report body. A
connection whose latest ingest skipped nothing shows no such line.
### R3 — Failure semantics (partial vs total)
Per-object skipping is **unconditional** — there is **no new config knob**, and
the existing `ingest.workUnits.failureMode` (which governs the later LLM
work-unit stage, not introspection) is untouched and orthogonal. Outcomes are
derived from object counts, not from a mode:
| Scope | Objects discovered / matched | Introspection outcome | Result |
| --- | --- | --- | --- |
| none | 0 | n/a (legitimately empty DB) | **success**, empty layer |
| none | N > 0 | ≥ 1 succeeds | **success** + warnings for the rest |
| none | N > 0 | all N fail | **connection failure** (clear error) |
| `enabled_tables` | matches 0 objects | n/a | **clear scope error** (R5) |
| `enabled_tables` | matches M > 0 | ≥ 1 succeeds | **success** + warnings |
| `enabled_tables` | matches M > 0 | all M fail | **connection failure** |
- "Connection failure" means the connector / `fetch()` raises a **clear,
actionable error** for that connection. It **MUST NOT** surface as the generic
`did not recognize fetched source output` (that message is reserved for a
genuinely unrecognized staged dir, not an empty/total-failure result).
- A total failure of one connection follows existing per-connection ingest
orchestration for whether sibling connections continue; this spec does not
change cross-connection behavior.
### R4 — A broken view never blocks base tables
A broken view **MUST NEVER** prevent base-table ingest.
- View introspection failures are isolated exactly like any other object (R1).
- Mandatory introspection **MUST** prefer reading an object's structure from the
catalog where possible over executing the object's body, and **MUST NOT** run
a data-reading query (row count, sample) against a view as a required step.
(sqlite already skips `COUNT(*)` for views; the remaining gap is isolating the
metadata read that executes the view definition.)
### R5 — `enabled_tables` allowlist works
The documented allowlist escape hatch **MUST** reliably restrict the scan to the
listed objects, with no spurious adapter error:
- **sqlite qualification.** The schema-qualified form `"main.<name>"` **MUST**
resolve to the same object as the bare form `"<name>"` (sqlite's sole schema
is `main`; the connector emits `db: null`). Both forms select the object;
neither silently matches nothing.
- **Documented format.** The accepted qualification forms for each driver
(`catalog.db.name` / `db.name` / `name`) and the sqlite-specific `main`
equivalence **MUST** be documented where `enabled_tables` is described
(`context/project/driver-schemas.ts` and the user-facing config docs).
- **Zero-match is a clear error.** A non-empty `enabled_tables` that resolves to
**zero** matched objects **MUST** fail with an actionable error naming the
connection, the unmatched entries, and the available object names — **not** the
generic `did not recognize fetched source output`. This is distinct from a
legitimately empty database (R3 row 1) and from a matched-but-all-broken scope
(R3 last row).
- **Any subset works.** An `enabled_tables` matching M > 0 objects ingests
**exactly** those M objects (minus any that fail per R1), with no adapter
recognition error regardless of how small or edge-case the set is.
### R6 — Python daemon parity
The daemon's postgres introspection path **MUST** honor the same contract:
- Add a `warnings` field to `DatabaseIntrospectionResponse`
(`python/ktx-daemon/src/ktx_daemon/database_introspection.py`) carrying the
same shape Node expects (code, message, object identifier, recoverable).
- Isolate per-object failures in the daemon's introspection so one broken object
does not abort the response; apply the R3 total-failure rule there too.
- Map daemon warnings into `KtxSchemaSnapshot.warnings` in
`mapDaemonSnapshot` (`context/ingest/adapters/live-database/daemon-introspection.ts`),
which currently drops them.
- The Node and Python warning shapes **MUST** stay in parity (the codebase
already mirrors Node↔Python schemas for telemetry; follow the same discipline
so the daemon cannot emit a code Node can't render).
## Acceptance criteria
- Ingesting a sqlite DB with one broken view + N healthy tables yields a
semantic layer for the N healthy tables and **exactly one** warning naming the
broken view and its error; exit is **success**.
- The skipped object appears in the `ktx ingest` summary output, in the run's
`warnings.json`, and in `ktx status` as a per-connection skipped-objects line
on the connection's latest ingest.
- A sqlite DB in which **every** discovered object fails introspection (and the
file opens) exits as a **connection failure** with a clear error — not an
empty "success" and not `did not recognize fetched source output`.
- A genuinely empty sqlite DB (zero objects) exits **success** with an empty
layer (not a failure).
- `enabled_tables: ["main.customers"]` and `enabled_tables: ["customers"]` both
ingest exactly the `customers` object on a sqlite connection.
- `enabled_tables` restricted to a valid subset of M objects ingests exactly
that subset, with **no** adapter-output error.
- `enabled_tables` that matches zero objects fails with an error naming the
connection, the unmatched entries, and available objects — distinguishable
from the empty-DB and all-broken cases.
- A broken view does not prevent ingest of base tables in the same connection
(regression test with a view that errors on read alongside a healthy table).
- The daemon's `DatabaseIntrospectionResponse` carries a `warnings` array, and a
per-object failure in the daemon path produces a warning mapped into
`KtxSchemaSnapshot.warnings` (Node↔Python parity test).
- A warehouse-driver object whose profiling/sample read fails is skipped with a
warning and does not abort introspection of its siblings.
- Existing healthy-only ingests (no broken objects, no `enabled_tables`) behave
identically before/after — no warnings, same semantic layer.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the design.
- **Shared semantics:** `context/scan/constraint-discovery.ts`
(`tryConstraintQuery` / `constraintDiscoveryWarning` — the precedent to mirror
for the per-object helper), `context/scan/types.ts`
(`KtxSchemaSnapshot.warnings`, `KtxScanWarning`, `KtxScanWarningCode` — add the
new object-skip code here).
- **Node connectors:** `packages/cli/src/connectors/<driver>/connector.ts` and
each `live-database-introspection.ts`. sqlite's loop is
`connectors/sqlite/connector.ts` `introspect` (≈ line 158) → `readTable`
(≈ line 306); the missing try/catch is the `rawTables.map(...)` at ≈ line 171.
Existing per-table sample isolation precedent: `description-generation.ts`
(≈ line 867, `sampling_failed`).
- **Driver dispatch:** `packages/cli/src/local-adapters.ts` (≈ lines 122-156)
routes every driver to its Node connector; the daemon is the `else` fallback.
- **`enabled_tables` matching:** `context/scan/enabled-tables.ts`
(`resolveEnabledTables`, `parseDottedTableEntry`), `context/scan/table-ref.ts`
(`scopedTableNames`, the `ref.db !== wantDb` filter ≈ line 47),
`context/project/driver-schemas.ts` (`enabled_tables` schema + description).
- **Staging / detect / error surface:**
`context/ingest/adapters/live-database/stage.ts`
(`writeLiveDatabaseSnapshot`, `warningArtifact` ≈ line 94,
`detectLiveDatabaseStagedDir` ≈ line 138),
`context/ingest/local-stage-ingest.ts` (the
`did not recognize fetched source output` throw ≈ line 291 — must stop being
the surface for empty-scope and total-failure).
- **Ingest summary:** `packages/cli/src/ingest.ts` (`writeReportStatus`
≈ line 202), `context/ingest/memory-flow/summary.ts`
(`formatMemoryFlowFinalSummary`) — thread object skips into the human-facing
summary.
- **Report body + `ktx status`:** `context/ingest/reports.ts` (`IngestReportBody`;
`SourceFetchReport` as the home for scan warnings),
`context/ingest/sqlite-local-ingest-store.ts` (the report body is persisted
whole as `body_json` ≈ line 90 — no migration needed), `status-project.ts`
(`buildLocalStatsStatus` reads `local_ingest_reports`; parse the latest body
per connection and render the skipped line via `renderLocalStatsAsLines`).
- **Daemon path:** `python/ktx-daemon/src/ktx_daemon/database_introspection.py`
(`DatabaseIntrospectionResponse` ≈ line 165, `introspect_database_response`
≈ line 323, `_load_postgres_rows` ≈ line 227, `_map_rows_to_tables`
≈ line 267), and the Node mapping in
`context/ingest/adapters/live-database/daemon-introspection.ts`
(`mapDaemonSnapshot` ≈ line 209).
## Benchmark context (motivation only)
`oracle_sql` (8 of the 135 local sqlite questions) currently has **no** semantic
layer because of its one broken view, so those questions fall back to raw
`sql`-tool introspection instead of ktx's enriched context. Tolerant scanning
restores enriched context for that database. The same robustness is required for
the full Spider 2.0-Lite run across BigQuery and Snowflake, where broken or
permission-restricted objects are common and a single one must not zero out a
warehouse's context.
## Implementation notes
Shipped on branch `write-feature-spec-wiki`. All requirements implemented;
verified with `pnpm --filter @kaelio/ktx run test` (2981 passing),
`pnpm run dead-code`, `uv run pytest python/ktx-daemon/tests` (97 passing),
`uv run pre-commit`, and `pnpm run build && pnpm run link:dev`.
**Shared semantics (R1).** New `context/scan/object-introspection.ts` exposes
`tryIntrospectObject(ctx, fn)` (sibling of `tryConstraintQuery`), returning
`{ ok, table } | { ok: false, warning }` and building an
`object_introspection_failed` warning (object name + redactable DB error). It
rethrows native programming faults (`isNativeProgrammingFault`) so a ktx bug is
never masked as an object skip. The new warning code was added to
`KtxScanWarningCode` (`scan/types.ts`), the `scanWarningCodes` allowlist
(`local-structural-artifacts.ts`, plus a new exported `isKtxScanWarningCode`
validator), and `describeWarningGroup` (`scan.ts`).
**Per-object isolation, where it actually exists (R1/R4).** Only sqlite
(`readTable` via `PRAGMA`) and bigquery (`tableRef.get()` per dataset) do
per-object reads during *mandatory* introspection; both now route each object
through `tryIntrospectObject`. The other five Node connectors (postgres, mysql,
clickhouse, sqlserver, snowflake) read metadata in bulk from the catalog/
`information_schema` (already object-safe at this phase) and isolate per-object
profiling/sampling in the enrichment phase (`description-generation.ts`,
`sampling_failed`), so no divergent per-driver try/catch was added there. sqlite
also tolerates a `COUNT(*)` (profiling) failure without dropping a
structurally-readable table, and a broken view's metadata read is isolated so it
never blocks base tables (R4).
**Single-source outcome decision (R3/R5).** New
`adapters/live-database/scan-outcome.ts#assertLiveDatabaseScanOutcome` runs once
in `LiveDatabaseSourceAdapter.fetch()` — the one path every driver (and the
daemon) routes through — and derives the outcome from the snapshot + scope:
≥1 object → success (skips ride along as warnings); all matched objects failed →
clear `KtxExpectedError`; non-empty `enabled_tables` matched nothing → clear
zero-match error naming the connection, the requested entries, and the available
objects (sqlite/bigquery attach the discovered inventory via
`metadata.discovered_object_names`); empty database (no scope) → success with an
empty layer. `detectLiveDatabaseStagedDir` no longer requires table files, so a
valid empty staging is recognized; total-failure/zero-match now throw a clear
connection error before staging instead of surfacing the generic
`did not recognize fetched source output`.
**`enabled_tables` matching (R5).** Normalized at the scope boundary in
`resolveEnabledTables` using `connection.driver`: for sqlite, `main.<name>` →
`{ db: null }`, so `"main.customers"` and `"customers"` select the same object.
`table-ref.ts` stayed generic. Documented in `driver-schemas.ts` and
`docs-site/.../configuration/ktx-yaml.mdx`.
**Surfacing (R2).** Deviation from the spec's orientation: live-database schema
ingest runs through the **stage-only** path (`runLocalStageOnlyIngest` →
`local_ingest_reports`), not the bundle runner, so the home for scan warnings is
`LocalIngestRunRecord.fetch` (a new `SourceFetchReport` field; `body_json` is
persisted whole, so **no migration**), not the bundle-only
`IngestReportBody.fetch`. Both ingest paths read `adapter.readFetchReport`
(`live-database/fetch-report.ts` derives skips from the existing `warnings.json`).
The ingest summary is already rendered by `runKtxScan` from `report.warnings`
(the new `describeWarningGroup` case), and `ktx status`
(`status-project.ts#buildLocalStatsStatus`/`renderLocalStats`) now parses the
latest report body per connection and prints a per-connection
`N object(s) skipped — name: reason` line.
**Daemon parity (R6).** `database_introspection.py` adds a `warnings` field to
`DatabaseIntrospectionResponse` and a `DatabaseIntrospectionWarning` model,
isolates per-object failures in `_map_rows_to_tables`, and shares the
`OBJECT_INTROSPECTION_FAILED_CODE = "object_introspection_failed"` string with
Node. `mapDaemonSnapshot` maps `raw.warnings` into `KtxSchemaSnapshot.warnings`,
dropping any code Node cannot render (validated via `isKtxScanWarningCode`).
Deviation: the daemon does **not** re-enforce the R3 total-failure rule — the
shared Node post-check (`assertLiveDatabaseScanOutcome`) owns it for every driver
including the daemon, avoiding a divergent second implementation. Parity is
covered by a Node test (daemon-shaped warning round-trips) and a pytest
(per-object failure → warning with the shared code).

363
spider2-specs/specs/07-analytics-skill-sql-craft.md
View file

@ -1,363 +0,0 @@
# Add universal SQL-authoring craft to the ktx-analytics skill
> Refined spec. Intake draft: `todo/07-analytics-skill-sql-craft.md`.
## Problem
The shipped `ktx-analytics` skill
(`packages/cli/src/skills/analytics/SKILL.md`) is an *orchestration* guide: its
`<workflow>` and `<rules>` tell the agent **which ktx tools to call and in what
order** (`discover_data` → `entity_details`/`sl_read_source` →
`sl_query`/`sql_execution` → validate → `memory_ingest`). It says almost nothing
about **writing correct SQL**.
That gap shows up as a specific failure shape: the agent reliably produces
*runnable* SQL but *wrong* results. The recurring defects are universal
analytics-engineering mistakes, not ktx-specific ones:
- comparing a string column to a numeric literal (or vice versa), which can
silently match zero rows;
- rounding inside intermediate CTEs, so the final number is off;
- ranking/“first”/“most recent” windows with no deterministic tie-breaker, so
results flicker run to run;
- filtering *before* a window function for sequence/“since”/“first” questions,
truncating the partition the window should see;
- returning a full ranked list for a “top/highest” question, or collapsing a
“per X” question to a single value;
- dropping the inputs (or the entity identifier) a derived value was built from.
These are correctness defects every ktx user hits on a live database. They
belong in the shipped skill — fixing them once improves ktx for everyone, rather
than living in any individual caller’s prompt.
## Generic use case
An analyst (human or agent) points ktx at a **live, production** database and
asks a real analytical question — “what’s the most recent order per customer”,
“top region by margin”, “average order value by month”. The schema is unfamiliar
(unknown date encodings, nullable join keys, string-typed numeric columns), the
question carries grain and ranking intent in its wording, and the answer must be
*correct and deterministic*, not merely executable. The skill should encode the
analytics-engineering craft that makes the difference between a query that runs
and a query that’s right — independent of any benchmark.
## Model
The change is **additive content in one Markdown file**, governed by these
invariants. They constrain the implementer; the exact prose is theirs.
### Inline-only delivery (this is a hard constraint, not a style preference)
All new guidance lives **inside `skills/analytics/SKILL.md`**. A bundled
`reference/*.md` file (the progressive-disclosure pattern Anthropic’s
skill-authoring guide recommends for large skills) **MUST NOT** be used here,
because the delivery mechanism ships only `SKILL.md`:
- `setup-agents.ts` installs the analytics skill via `readAnalyticsSkillContent()`,
which reads **only** `./skills/analytics/SKILL.md` and writes a **single** file
per target: `.claude/skills/ktx-analytics/SKILL.md` (Claude Code), the Codex /
universal `.agents` equivalent, a **flattened** single rules file for Cursor
(`.cursor/rules/ktx-analytics.mdc`) and OpenCode
(`.opencode/commands/ktx-analytics.md`), and a Claude Desktop **zip that
contains only `ktx-analytics/SKILL.md`** (`writeClaudeDesktopSkillBundle`).
- Nothing copies sibling files or subdirectories. A reference file would dangle
on every target, and the Cursor/OpenCode flatten-to-one-file shape cannot
represent a multi-file skill at all.
The skill is small enough that inline costs nothing meaningful: ~67 lines today
plus ~60 of craft is well under the 500-line budget. And this craft is **core
content** — consulted on every SQL-authoring turn — so even if multi-file delivery
existed it would still belong inline: progressive disclosure only pays off for
large, *conditionally-relevant* reference material loaded on demand, not for
always-needed craft.
Multi-file skill *delivery* is a legitimate future enhancement, but it must be
**pulled by a concrete need, not built ahead of one** — no shipped skill today
exceeds the budget (largest is ~346 lines) or uses a bundled reference. The first
real trigger is the **per-dialect SQL syntax follow-up**
(`todo/08-per-dialect-sql-syntax-notes.md`), whose load-on-demand
`reference/<dialect>.md` content is a genuine progressive-disclosure fit. When
that work is scoped, note that multi-file delivery is **not** a simple directory
copy: `setup-agents.ts` flattens the skill to a *single* file for Cursor
(`.mdc`) and OpenCode (`.md`), so those targets need a concatenation transform,
and uninstall needs per-file manifest entries. Recording the constraint here so a
future implementer does not “improve” this inline content into a bundled
reference that dangles on every target.
### Heuristics with a generic *why*, not a wall of MUSTs
The new rules are phrased as **heuristics with a one-line, universal rationale**,
because SQL authoring is a high-freedom task (many valid approaches, choice
depends on the question and the data). A bare imperative overfits; a rule plus
its *why* lets the model apply judgment and generalize. This follows Anthropic’s
own skill-authoring guidance (“if you find yourself writing ALWAYS/NEVER in all
caps or rigid structures, reframe and explain the reasoning”).
This **reconciles the draft’s “behavior only, no rationale” instruction**: the
prohibition is specifically on rationale that references a **grader, gold answer,
or the benchmark**. *Generic analytics-engineering rationale is required* — e.g.
“…so `RANK`/`ROW_NUMBER` results don’t flicker across runs”, “…a string-vs-number
compare can silently match nothing”. That is a universal truth, not a
grader reference.
### Dialect-agnostic
Every rule must read correctly on any SQL dialect a ktx connection might use.
**No dialect-specific syntax** — not `QUALIFY` (Snowflake/BigQuery/DuckDB only),
not `strftime`/`julianday` (sqlite), not backtick/`DB.SCHEMA.TABLE` FQTNs.
Per-dialect syntax notes are a **separate follow-up** living in a dialect-aware
(per-driver) location, explicitly out of scope here.
### Discovery craft attaches to discovery; authoring craft to query/validate
Two of the draft’s rules (inspect sample rows; cast before comparing) are
*schema-discovery* concerns that happen **before** SQL is composed. They belong
with the discovery steps of the existing workflow, not only at the query step.
The rest (composition, window correctness, precision, completeness) belong with
the query/validate steps. The draft’s “extend step 5/6” is the right home for
most rules but is slightly off for the discovery pair; this spec corrects that.
### Additive only
The existing `<workflow>`, `<rules>`, and `<examples>` — compact result tables,
summaries, clarification prompts, the tool-order workflow, the `connectionId`
scoping rules — are preserved unchanged. The skill must still read well for an
interactive, human-facing analysis session.
## Requirements
### 1. Placement and structure
Add a dedicated, scannable craft section to `SKILL.md`:
- A new top-level block — `<sql_craft>` (sibling to `<workflow>`/`<rules>`) — with
**five sub-headings**: *Schema discovery*, *Composition*, *Window functions*,
*Numeric precision*, *Answer completeness*. Sub-headings keep the block
scannable (the draft’s “group under clear sub-headings” goal).
- **Pointers, not duplication.** Step 5 (“Query”) and step 6 (“Validate and
explain”) each gain a **one-line pointer** into `<sql_craft>` rather than
inlining the rules (state each rule once; Anthropic’s “consistent terminology /
don’t repeat” guidance). The schema-discovery pair is additionally reflected as
a brief cue in the discovery steps (step 2 “Inspect” / step 4 “Plan”), pointing
to the same block.
- No new tool, flag, or config. This is content only.
### 2. The craft rules (all fourteen behaviors, grouped)
Every behavior from the intake draft must be represented. Tightly-related ones
**may** be merged into a single bullet where that reads better; none may be
dropped. Each carries a generic *why* (per Model). Dialect-agnostic throughout.
**Schema discovery** (cue in steps 2/4; lives in `<sql_craft>`)
1. Inspect representative **sample rows** of each table before composing SQL —
confirm date/time encoding (`YYYYMMDD` vs ISO vs epoch), null prevalence in
join/filter keys, and the real set of categorical/enum values
(`entity_details` + a small `sql_execution` sample). *Why:* assumptions about
encoding and nullability are the most common source of silently-wrong filters.
2. **Cast a column to its real type before comparing** it in `WHERE`/`JOIN`. A
string column compared to a numeric literal (or vice versa) can silently match
nothing.
**Composition**
3. Build complex queries **incrementally** — one CTE at a time, verifying each
layer’s output on a small sample before stacking the next. *Why:* a wrong
intermediate layer is far cheaper to catch early than to debug in the final
result.
4. **Avoid fan-out joins.** Add columns only from tables already at the target
grain, or **pre-aggregate** to that grain before joining. *Why:* a join that
multiplies rows quietly inflates every downstream `SUM`/`COUNT`.
**Window functions**
5. Give every ranking/ordering window function a **complete, deterministic
tie-breaker** (append unique key columns to `ORDER BY`), so
`RANK`/`ROW_NUMBER`/`LAG` are stable rather than flickering across runs.
6. For sequence / “first” / “most recent” / “since” questions, **filter after the
window**, not before: compute over the full partition, then keep the rows you
want. *Why:* a pre-filter shrinks the partition the window ranks over, so
“first”/“most recent” is computed against the wrong set. (See the worked
example, requirement 3.)
**Numeric precision**
7. Compute at **full precision; round only in the final projection**, never inside
intermediate CTEs.
8. Be **explicit about truncation** — `CAST AS INT` truncates; use explicit
rounding when rounding is intended. (May merge with rule 7.)
9. Distinguish **macro vs micro averages** based on the question’s wording:
“average of per-group averages” = `AVG(group_metric)`; “overall/weighted
average” = `SUM(numerator)/SUM(denominator)`.
**Answer completeness / interpretation**
10. “top / highest / most / lowest” → return only the **winning row(s)** (keep the
top-ranked row via the window result), not the full ranked list, unless a list
is asked for. *(Phrase the mechanism dialect-agnostically — do not name
`QUALIFY`.)*
11. “for each X / per X / by X” → **exactly one row per X**; don’t collapse to a
single value unless the question says “overall” or “total across X”.
12. When a question asks for inputs and a derived value (“X, Y, and their ratio”),
**include the inputs as columns** alongside the derived value.
13. When grouping by a human-readable label (a name), also **expose the entity’s
identifier** — identity, not just the label, is part of the result (and
disambiguates duplicate names).
14. When a result is **unexpectedly empty, relax filters one at a time** to find
which predicate removed the rows. *Why:* this is the validation feedback loop
that turns a silent empty result into a diagnosable one.
### 3. One worked example (dialect-agnostic)
Add **exactly one** compact before/after example to the skill, demonstrating the
**window-then-filter** rule (rule 6) — the subtlest and highest-value of the set.
It shows the wrong shape (filter inside, then rank) and the right shape (rank over
the full partition in a CTE, then filter to the top rank in the outer query),
using generic table/column names and standard SQL only (no `QUALIFY`, no
dialect functions). Keep it ~6–10 lines. Do not add a second example; the
existing three tool-orchestration examples stay as the primary example set.
*(Superseded by spec 09: the skill now carries a second `sql` worked example —
the multi-hop fan-out case — so the one-example constraint applies to spec 07's
window-then-filter example only.)*
### 4. Explicit exclusions
None of the following may appear in the skill (they are application/consumer
concerns, or actively wrong for live data):
- **Output-shape contracts** (“return a bare result set with exactly these
columns, no prose”). The skill is for interactive analysis and already favors
readable tables + summaries; a caller needing a strict shape specifies that
itself.
- **Anchoring relative time to `MAX(date)` of the data.** On a live database
“recent” / “past N months” means relative to *now*; `MAX(date)` anchoring is
only valid for static snapshots and must not be baked into the product.
- **Any advice justified by a grader, gold answer, or scoring comparator.**
- **Dialect-specific syntax** (deferred to the per-driver follow-up).
### 5. Coordination with spec 03
`03-multi-connection-routing-in-analytics-skill` also edits this same file (it
adds a connection-routing “step 0” to `<workflow>` and threads `connectionId`
through the tool calls). Spec 07’s additions are **orthogonal**: they live in a
new `<sql_craft>` block and in step 5/6 pointers, and must not rewrite the
`<workflow>` routing or the `<rules>` `connectionId` scoping that spec 03 owns.
If both land, the result is one coherent skill: routing in `<workflow>`/`<rules>`,
SQL craft in `<sql_craft>`.
## Acceptance criteria
- The shipped `analytics/SKILL.md` contains all fourteen behaviors above, grouped
under the five sub-headings, each phrased as a heuristic with a generic
rationale.
- **Zero references** to any benchmark, gold answer, grader, or scoring
comparator anywhere in the skill.
- **Dialect-agnostic:** the skill contains no `QUALIFY`, no `strftime`/`julianday`,
no backtick/`DB.SCHEMA.TABLE` FQTN syntax, and no other single-dialect
construct — including in the worked example.
- The existing interactive guidance is intact: the `<workflow>` steps, the
`<rules>` (compact tables, summaries, clarification prompt, `connectionId`
scoping), and the three existing examples all still read correctly and were not
removed or contradicted.
- **None of the excluded items** (output-shape contract, `MAX(date)` anchoring of
“recent”, grader-driven advice, dialect syntax) appear.
- Exactly **one** new worked example is present, demonstrating window-then-filter,
in standard dialect-agnostic SQL. *(Superseded by spec 09, which adds a second
`sql` worked example for the multi-hop fan-out case; the shipped skill then
contains two worked examples and the content test asserts two `sql` fences.)*
- The craft is **inline in `SKILL.md`** — no bundled reference file is introduced,
and the skill still installs as a single file through `setup-agents.ts` for all
targets (Claude Code, Codex, Cursor, OpenCode, universal, Claude Desktop zip).
- The skill stays **scannable and within a reasonable size** (comfortably under
the 500-line budget).
- The frontmatter (`name`, `description`) is unchanged and still parses through
`SkillsRegistryService.parseFrontmatter`.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the prose.
- **The skill file:** `packages/cli/src/skills/analytics/SKILL.md`. Add the
`<sql_craft>` block; add one-line pointers in steps 5/6 and a discovery cue in
steps 2/4; add the single worked example. Keep `<workflow>`/`<rules>`/`<examples>`
otherwise intact.
- **Delivery (why inline is mandatory):** `packages/cli/src/setup-agents.ts`
(`readAnalyticsSkillContent`, `installTarget`, `writeClaudeDesktopSkillBundle`,
`plannedKtxAgentFiles`). Each target gets a single file derived from
`SKILL.md`; Cursor/OpenCode flatten to one rules file; Claude Desktop zips only
`ktx-analytics/SKILL.md`. No change to `setup-agents.ts` is required by this
spec — confirm the skill still installs unchanged.
- **Coordination:** `03-multi-connection-routing-in-analytics-skill` edits the
same file; keep the changes non-overlapping (see requirement 5).
- **Tests:** a content assertion over the shipped `analytics/SKILL.md` is the
right level (this is prompt content, not executable logic). Assert the skill
text contains the craft sub-headings / representative rule phrases, contains the
worked example, and contains none of the banned constructs: the literal tokens
`QUALIFY`/`strftime`/`julianday`, grader/benchmark words (`spider`, `benchmark`,
`gold`, `grader`), and — checked as a phrase, not a raw `MAX(` grep, since
`MAX()` is a legitimate aggregate — any instruction anchoring relative time
(“recent”, “past N months”) to the data’s maximum date. The existing
`SkillsRegistryService` frontmatter-parse test must still pass. The standalone
`ktx-dev` binary should be rebuilt/re-linked (`pnpm run build && pnpm run
link:dev`) so the playground picks up the updated skill.
## Benchmark context (motivation only)
On the Spider 2.0-Lite sqlite subset the solver produced **0 execution errors but
~50 result mismatches**, and a large share traced to exactly these gaps:
premature rounding, string-vs-number compares, non-deterministic window ordering,
returning full lists for “top” questions, and dropping the inputs to derived
values. These are generic SQL-authoring defects — fixing them in the skill
improves ktx for every user querying a live database, and improving the benchmark
score is a side effect, not the goal. The skill itself must contain no trace of
the benchmark.
## Implementation notes
Implemented on branch `write-feature-spec-wiki`.
**What was built**
- Added a new `<sql_craft>` block to `packages/cli/src/skills/analytics/SKILL.md`
(sibling to `<workflow>`/`<rules>`, placed just before `<examples>`), with the
five sub-headings — *Schema discovery before writing SQL*, *Composition*,
*Window functions*, *Numeric precision*, *Answer completeness / interpretation* —
and a one-line opener framing the bullets as heuristics-with-a-why.
- All fourteen behaviors are represented. Rules 7 and 8 (round-at-the-end /
truncation) are merged into one "Round only at the end" bullet, as the spec
permitted. Each bullet carries a generic analytics-engineering rationale; none
references a benchmark, grader, or gold answer.
- Exactly one worked example (a fenced `sql` block inside `<sql_craft>`)
demonstrates the window-then-filter rule, and incidentally the deterministic
tie-breaker: the *wrong* shape filters before the window; the *right* shape
ranks the full partition in a CTE, then filters in the outer query. Standard
SQL only — no `QUALIFY`, no dialect functions.
- Step pointers added without duplicating the rules: a schema-discovery cue in
steps 2 and 4, an authoring pointer in step 5, and a validation pointer in
step 6, each pointing into `<sql_craft>`.
- The existing `<workflow>` / `<rules>` / `<examples>` (compact tables,
summaries, clarification prompt, `connectionId` scoping, the three
orchestration examples) are unchanged. Delivery is unchanged: still a single
`SKILL.md` per target via `readAnalyticsSkillContent`; no bundled `reference/`
file was introduced.
**Tests** — added `packages/cli/test/skills/analytics-skill-content.test.ts`, a
content assertion over the source `SKILL.md`: the five sub-headings, a
representative phrase for each behavior, exactly one `sql` worked example, the
preserved interactive guidance, and the absence of banned constructs
(`QUALIFY` / `strftime` / `julianday`, `spider` / `benchmark` / `gold` /
`grader`, a backtick three-part FQTN, and a phrase-level guard against anchoring
relative time to a `MAX(...)` date). The existing `setup-agents.test.ts` content
assertions and the `SkillsRegistryService` frontmatter test still pass (77/77
across the three relevant files). Rebuilt and re-linked `ktx-dev`
(`pnpm run build && pnpm run link:dev`); the craft block is present in the
shipped `dist` asset.
**Deviations / notes**
- The worked example runs ~18 lines including comments rather than the spec's
"~6–10"; a faithful before/after with a CTE needs the extra lines, and the
skill stays well within budget (~117 lines total).
- `pnpm run type-check` currently reports one **pre-existing, unrelated** error
in `test/mcp-server-factory.test.ts` (MCP server deps typing), committed on
this branch ahead of `origin/main`. The src type-check and `pnpm run build`
are green; this change does not touch any MCP file.
- Per-dialect SQL syntax stays out of scope here (deferred to
`todo/08-per-dialect-sql-syntax-notes.md`), so the skill remains
dialect-agnostic. No dialect-tool pointer was added to `SKILL.md` yet — that
belongs with spec 08's channel so the skill never references a tool that does
not exist.

395
spider2-specs/specs/08-per-dialect-sql-syntax-notes.md
View file

@ -1,395 +0,0 @@
# Per-dialect SQL syntax notes, served on demand and scoped to the connection
> Refined spec. Intake draft: `todo/08-per-dialect-sql-syntax-notes.md`. Companion
> to `specs/07-analytics-skill-sql-craft.md`, which kept the analytics SQL craft
> dialect-agnostic and explicitly deferred per-dialect syntax to this spec.
## Problem
Spec 07 added universal, **dialect-agnostic** SQL-authoring craft to the
`ktx-analytics` skill (`packages/cli/src/skills/analytics/SKILL.md`). That craft
deliberately excludes anything that reads correctly on only one engine — no
`QUALIFY`, no `strftime`/`julianday`, no backtick or `DB.SCHEMA.TABLE` FQTNs —
because the flat skill is installed verbatim and an agent querying sqlite must
never see Snowflake syntax.
But a large share of *real* correctness depends on exactly that excluded,
engine-specific syntax:
- **Snowflake:** `DATABASE.SCHEMA.TABLE` FQTNs, double-quoted case-sensitive
identifiers (unquoted folds to upper-case), VARIANT colon-paths
(`col:field.sub::type`), `QUALIFY`.
- **BigQuery:** backtick FQTNs (`` `project.dataset.table` ``), `_TABLE_SUFFIX`
for sharded/wildcard tables, `QUALIFY`, `JSON_VALUE`/`JSON_EXTRACT`.
- **sqlite:** `strftime`/`julianday`/`date()` for dates, no `QUALIFY`,
`json_extract`.
- and the remaining supported engines (`postgres`, `mysql`, `clickhouse`,
`sqlserver`/`tsql`), each with its own FQTN, quoting, date, top-N, and
JSON conventions.
This guidance is genuinely useful to an agent writing SQL against a live
database, but it must **not** pollute the flat dialect-agnostic skill. It belongs
in a **dialect-aware** channel, surfaced only for the dialect the active
connection actually uses, and selected from the project's own configured state —
not guessed, not shown all at once.
## Generic use case
Any **ktx** project whose connections span more than one warehouse engine — a
Snowflake warehouse plus a BigQuery export plus a local sqlite extract, say. When
the agent (or a human analyst the agent assists) writes SQL for a given
connection, it should receive *that engine's* syntax conventions — FQTN form,
identifier quoting, date functions, top-N idiom, semi-structured access — and
nothing for the engines it is not querying. The need is independent of any
benchmark: it is what "write correct SQL against this specific warehouse" requires
on every multi-engine stack.
## Model
The change adds a **dialect-aware channel** alongside spec 07's flat skill. The
following decisions are committed by this refinement; the implementer owns the
exact prose and code.
### Delivery: a dynamic MCP tool (decision committed)
The draft posed two delivery mechanisms and asked the refinement to "weigh them
before committing." This spec commits to **dynamic MCP delivery**: a new
read-only MCP tool returns the syntax notes for a given `connectionId`, with the
dialect resolved server-side from the connection's configured `driver`. The flat
skill gains a one-line pointer to that tool. **No install-mechanism change is
required.**
The alternative — **multi-file skill delivery** (bundle `reference/<dialect>.md`
files and point the skill at the matching one) — is **rejected** for **ktx**, for
reasons that hold regardless of how the skill is otherwise authored:
1. **It cannot scope on two of the six install targets.** Cursor
(`.cursor/rules/ktx-analytics.mdc`) and OpenCode
(`.opencode/commands/ktx-analytics.md`) are physically **single-file**;
`setup-agents.ts` flattens the skill to one file there. A bundled `reference/`
directory degenerates to "concatenate every dialect into one file," so a
sqlite agent would see Snowflake VARIANT syntax — **failing this spec's core
no-leak criterion on those targets**, and defeating progressive disclosure
(everything is in context at once). The MCP tool behaves **identically on all
six targets** because it is a tool call, not an installed file.
2. **Selecting the dialect is a deterministic operation, so it belongs in code,
not model judgment.** Anthropic's skill-authoring guidance explicitly says to
*"prefer scripts [tools] for deterministic operations."* With bundled files the
**model** must infer that connection X is Snowflake and open the right file —
and on a multi-connection project it can open the wrong one. With the tool, the
**server** resolves `driver → dialect` from `ktx.yaml` state and returns
exactly the right notes.
3. **It needs a delivery subsystem that the tool does not.** Multi-file delivery
requires reworking `readAnalyticsSkillContent`, `installTarget`,
`plannedKtxAgentFiles`, the install manifest (a directory variant),
`removeKtxAgentInstall`, and `writeClaudeDesktopSkillBundle`, plus a
concatenation transform for the single-file targets. The MCP tool requires one
read-only handler and one skill pointer.
4. **The dependency is free.** The `ktx-analytics` skill already hard-depends on
the **ktx** MCP server — its entire workflow is calling `discover_data`,
`entity_details`, `sql_execution`, and so on. Wherever the server is down, the
skill is already non-functional; the tool adds **no new dependency**.
5. **Dropping Cursor/OpenCode does not change this.** Removing those targets would
make multi-file delivery *possible*, but it would not make it better: reasons
2–4 stand, and the drop is a disproportionate cost (Cursor is a major target)
to neutralize a constraint the tool handles for free. Whether **ktx** supports
those targets is a separate product decision and is out of scope here.
This is consistent with Anthropic's progressive-disclosure goal — load the
relevant material on demand, at zero context cost until needed — which the tool
satisfies (its output costs context only when called) while resolving *which*
dialect from state rather than from a model guess. Reference:
[Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices).
### Scope derived from state, through the one existing resolver
Which dialect's notes the agent sees is **derived** from the connection's
configured `driver`, via the resolver the rest of the system already uses —
`sqlAnalysisDialectForDriver(driver)` in
`packages/cli/src/context/sql-analysis/dialect.ts`. The same function already
selects the dialect for `sql_execution`, `sl_query`, and the Python SQL-analysis
daemon. This spec **must not** introduce a second driver→dialect map. The notes
are **keyed by the resolved `SqlAnalysisDialect`** (so the SQL Server entry is
keyed `tsql`, not `sqlserver`), tying the note key-space to the resolver's
codomain so the two cannot drift.
### Authored per-engine notes are sanctioned static content
Enumerating syntax notes per engine is **not** a rotting denylist of bad
specifics; FQTN form and identifier quoting are genuine, stable invariants of each
engine — the kind of universal fact **ktx**'s design rules explicitly permit as
static content. What must stay derived-from-state is note *selection* (the active
dialect) and note *coverage* (every configured driver must resolve to notes that
exist), both of which this spec ties to the connector registry.
### The flat skill stays dialect-agnostic (spec 07 invariant preserved)
This work adds a *separate* channel. It does **not** amend spec 07's `<sql_craft>`
block or inline any dialect syntax into `SKILL.md`. Spec 07's acceptance criterion
— no `QUALIFY`/`strftime`/`julianday`/backtick-FQTN/etc. in the flat skill — stays
green. The only `SKILL.md` change is the pointer in requirement 3, which names the
tool and contains no dialect syntax.
## Requirements
### 1. A read-only `sql_dialect_notes` MCP tool
Register a new tool beside the existing context tools
(`packages/cli/src/context/mcp/context-tools.ts`). The tool name is the
implementer's to finalize but should follow the existing snake_case convention
(`entity_details`, `sql_execution`); `sql_dialect_notes` is the suggested name.
- **Input:** `{ connectionId }`, **required** — matching its siblings
`entity_details`/`sql_execution`, which always take an explicit connection.
- **Output:** `{ connectionId, dialect, notes }` where `dialect` is the resolved
`SqlAnalysisDialect` and `notes` is the markdown guidance for that dialect.
- **Resolution:** `connectionId → connection.driver →
sqlAnalysisDialectForDriver(driver) → notes[dialect]`, reusing the existing
resolver. Do not duplicate the driver→dialect map.
- **Guards:**
- A **non-SQL context-source** connection (driver `metabase`, `looker`,
`lookml`, `notion`, `dbt`, `metricflow`) returns a **clear "not a SQL
warehouse connection" error**, not postgres notes. Gate on the existing
`isDatabaseDriver()` (`packages/cli/src/connection-drivers.ts`).
- For any **SQL warehouse** connection the resolver always yields a dialect with
notes (all seven warehouse drivers are covered — requirement 2); its built-in
`postgres` default is a safety floor, so the tool never errors for a SQL
connection and never emits a single-engine dialect (e.g. Snowflake) by
accident.
- **Annotations:** read-only and idempotent, consistent with the other read
tools.
- **Description (docs-grade, third person, states what and when):** e.g.
*"Returns the SQL syntax conventions for a connection's dialect — FQTN form,
identifier quoting and case-folding, date/time functions, top-N idiom, and
semi-structured access. Use before authoring raw SQL against a connection so the
SQL matches that engine."* The description drives the agent's decision to call
the tool, so it must be specific.
### 2. Per-dialect note content
Author concise notes for each supported dialect against a **fixed rubric**, so
every dialect answers the same questions. Each facet is a line or two of timeless,
engine-true convention (no version-dated "as of vX" content), phrased as
guidance with the engine reason where it helps — inheriting spec 07's
heuristics-with-a-why tone. The rubric facets:
1. **FQTN form** — how to fully-qualify a table on this engine.
2. **Identifier quoting & case-folding** — quote character and how unquoted
identifiers fold.
3. **Date/time** — the engine's date functions and common date-encoding idioms.
4. **Top-N / window-filtering idiom** — `QUALIFY` where supported; a CTE +
outer-filter form where it is not; `TOP` for `tsql`.
5. **Semi-structured / JSON access** — VARIANT colon-paths, `JSON_VALUE`/
`JSON_EXTRACT`, `->`/`->>`, `json_extract`, as applicable.
6. **Sharded / partition idiom** where the engine has one (e.g. BigQuery
`_TABLE_SUFFIX`).
Constraints on the content:
- **Coverage = the reachable dialect set.** Every driver in the connector registry
must resolve to a dialect that has non-empty notes. The reachable set is
`postgres`, `mysql`, `snowflake`, `bigquery`, `sqlite`, `clickhouse`, and
`tsql` (from `sqlserver`). Do **not** author notes for `duckdb`/`databricks`:
they appear in the resolver map but no connector can produce them, so they are
unreachable — matching the draft's "don't author for nonexistent drivers."
- **Keyed by `SqlAnalysisDialect`** (see Model).
- **Storage is the implementer's choice.** The notes MAY live as per-dialect
markdown files inside the package (e.g. under the skill's directory) served by
the tool, or as a typed map. If files are used they are **package-internal** —
served by the tool, never installed onto an agent target — and already ship via
the recursive `src/skills → dist/skills` copy
(`packages/cli/scripts/copy-runtime-assets.mjs`); no `setup-agents.ts` change.
- **No benchmark, gold-answer, grader, or scoring references** anywhere in the
notes.
The implementer must verify each engine's specifics against current official
documentation (the well-known anchors above are starting points, not a
substitute for checking the engine's docs).
### 3. The `SKILL.md` pointer (completes spec 07's deferral)
Add a **single one-line pointer** to the SQL-authoring step (step 4 "Plan" / step
5 "Query") of `packages/cli/src/skills/analytics/SKILL.md`, directing the agent to
call the tool before writing raw SQL against a connection — e.g. *"Before writing
raw `sql_execution` SQL, call `sql_dialect_notes` with the connection's id to get
that engine's syntax conventions."* This is the pointer spec 07 deliberately did
not add because the tool did not yet exist.
- The pointer **names the tool only**; it contains **no dialect syntax**, so the
flat skill stays dialect-agnostic.
- Follow the skill's existing tool-reference convention. The skill currently names
MCP tools by **bare** name (`discover_data`, `sql_execution`). Anthropic's
guidance recommends **fully-qualified** `ServerName:tool` names to avoid
"tool not found" when multiple MCP servers are present. Whether to fully-qualify
the new pointer (and optionally retrofit the existing bare references) is a
small, separable decision flagged for the maintainer — **not** a rename sweep
this spec mandates.
### 4. Coverage is enforced from state, not by hand
A test must **derive** the required coverage from the connector registry rather
than hardcoding a dialect list: enumerate the configured warehouse drivers
(`warehouseDrivers` in `driver-schemas.ts` / `KTX_DATABASE_DRIVER_IDS` in
`connection-drivers.ts`), resolve each through `sqlAnalysisDialectForDriver`, and
assert each result has non-empty notes. Adding a connector later then **fails this
test** until its dialect gets notes — the allowlist-from-state discipline, not a
hand-maintained list.
### 5. No dialect syntax leaks into the flat skill
Spec 07's content assertion over `analytics/SKILL.md` stays green: the flat skill
(and its worked example) still contain no `QUALIFY`, `strftime`, `julianday`,
backtick/`DB.SCHEMA.TABLE` FQTN, or other single-engine construct. This spec adds
a tool and a tool-pointer; it does not move dialect syntax into the skill.
### 6. Delivery is unchanged
`setup-agents.ts` (`readAnalyticsSkillContent`, `installTarget`,
`writeClaudeDesktopSkillBundle`, `plannedKtxAgentFiles`) needs **no change**. The
skill still installs as a single `SKILL.md` per target. Confirm the channel works
on all six targets — Claude Code, Claude Desktop (zip), Codex, universal
`.agents`, Cursor (`.mdc`), OpenCode (`.md`) — by virtue of being a tool call,
including the single-file targets where multi-file delivery could not scope.
### 7. Coordination with specs 07 and 03
- **Spec 07** owns the dialect-agnostic `<sql_craft>` block. This spec must not
amend it; it adds the tool, the pointer, and the notes.
- **Spec 03** (`03-multi-connection-routing-in-analytics-skill`) threads
`connectionId` through the skill's tool calls. The `sql_dialect_notes` pointer
is `connectionId`-scoped and fits that routing; keep the pointer consistent with
spec 03's `connectionId` rules and do not rewrite the routing it owns.
## Acceptance criteria
- An agent querying a **sqlite** connection gets sqlite date idioms and **never**
sees Snowflake/BigQuery-only syntax; an agent querying **Snowflake** gets
FQTN / identifier / VARIANT guidance.
- The dialect shown is **derived from the connection's configured `driver`** via
the existing `sqlAnalysisDialectForDriver`, not hardcoded per project and not
guessed. No second driver→dialect map is introduced.
- **Every configured warehouse driver** (`postgres`, `mysql`, `snowflake`,
`bigquery`, `sqlite`, `clickhouse`, `sqlserver`) resolves to a dialect with
non-empty notes, and the coverage test derives this from the registry.
- A **non-SQL context-source** connection (e.g. `metabase`, `notion`) yields a
clear "not a SQL warehouse" response, **not** postgres notes.
- `analytics/SKILL.md` remains dialect-agnostic — spec 07's criteria are
unaffected. The new pointer references the tool only and adds no dialect syntax.
- The channel installs/serves correctly across **all six** agent targets,
including the single-file Cursor/OpenCode shape, with **no `setup-agents.ts`
change**.
- The notes contain **no** benchmark/gold/grader/scoring references and **no**
time-sensitive ("as of version X") content.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the design.
- **Dialect resolver (reuse, do not duplicate):**
`packages/cli/src/context/sql-analysis/dialect.ts` —
`sqlAnalysisDialectForDriver(driver)`, returning `SqlAnalysisDialect`
(`./ports.ts`), default `postgres`.
- **Connector registry (drives coverage):**
`packages/cli/src/connection-drivers.ts` (`KTX_DATABASE_DRIVER_IDS`,
`isDatabaseDriver`) and `packages/cli/src/context/project/driver-schemas.ts`
(`warehouseDrivers`, the per-driver `connectionConfigSchema`).
- **MCP tool registration:** `packages/cli/src/context/mcp/context-tools.ts`
(register beside `connection_list`, `entity_details`, `sql_execution`); the
`connectionId → driver → dialect` resolution already exists for `sql_execution`
in `packages/cli/src/context/mcp/local-project-ports.ts` — route the new tool
through the same path.
- **The skill (one-line pointer only):**
`packages/cli/src/skills/analytics/SKILL.md` — add the tool pointer in step 4/5;
leave `<workflow>`/`<rules>`/`<sql_craft>`/`<examples>` otherwise intact.
- **Note storage (if files):** under the skill directory, shipped by
`packages/cli/scripts/copy-runtime-assets.mjs`'s recursive copy; served by the
tool, never installed.
- **Delivery (confirm unchanged):** `packages/cli/src/setup-agents.ts`.
- **Tests:** unit tests for resolution (including `sqlserver → tsql`, unknown →
`postgres`, and non-warehouse rejection); a registry-derived coverage test
(requirement 4); a content test that each dialect's notes cover the rubric
facets and contain no banned tokens; and an extension of spec 07's
`analytics/SKILL.md` content test asserting the new pointer is present and the
flat skill is still dialect-clean. Rebuild and re-link the dev binary so the
playground picks up the change: `pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation only)
The Spider 2.0-Lite v9 harnesses' only per-dialect content was Snowflake
(`DB.SCHEMA.TABLE` FQTNs, double-quoted lower-case columns, VARIANT colon-paths),
BigQuery (backtick FQTNs, `_TABLE_SUFFIX` for sharded tables), and sqlite
(`strftime`/`julianday`). That content is real and useful but engine-specific;
spec 07 kept it out of the flat skill and deferred it here so the dialect-agnostic
rules stay clean. Delivering it through a dialect-scoped **ktx** tool generalizes
the same correctness benefit to every multi-engine **ktx** project — improving the
benchmark score is a side effect, not the goal, and the shipped skill contains no
trace of the benchmark.
## Implementation notes
Implemented on branch `write-feature-spec-wiki`, alongside spec 07. The committed
decision (dynamic MCP delivery, not multi-file skill bundling) was implemented as
specified — no `setup-agents.ts` change.
**What was built**
- Per-dialect notes are markdown files under
`packages/cli/src/context/sql-analysis/dialects/<dialect>.md` (one each for
`postgres`, `mysql`, `snowflake`, `bigquery`, `sqlite`, `clickhouse`, `tsql`),
served by `sqlDialectNotes(dialect)` in `sql-analysis/dialect-notes.ts` (lazy
read + cache, `postgres` fallback floor; the authored set is the
`DIALECTS_WITH_NOTES` const). `duckdb`/`databricks` are intentionally unauthored
(unreachable from any connector). Each note answers the fixed rubric — FQTN,
identifier quoting/case-folding, date/time, top-N/window idiom,
JSON/semi-structured, plus a sharded-table line for BigQuery. Engine specifics
were verified against current docs via Context7 (Snowflake VARIANT colon-paths
and unquoted→UPPER case-folding; BigQuery `_TABLE_SUFFIX`, `QUALIFY`,
`JSON_VALUE`; ClickHouse `LIMIT n BY` and `JSONExtract*`, with no `QUALIFY`). The
files are package-internal — `copy-runtime-assets.mjs` ships them to `dist`; they
are never installed onto an agent target.
- New read-only MCP tool `sql_dialect_notes` (`context-tools.ts`): input
`{ connectionId }` (required), output `{ connectionId, dialect, notes }`, read-only
+ idempotent annotations. It resolves through the **existing**
`connectionId → connection.driver → sqlAnalysisDialectForDriver` path (no second
driver→dialect map), implemented as the unconditional `dialectNotes` port in
`local-project-ports.ts` via an extracted `resolveDialectNotesForConnection`. A
non-SQL context source (gated by `isDatabaseDriver`) throws `KtxExpectedError`
("not a SQL warehouse"), not postgres notes — so the expected agent mistake stays
out of Error Tracking.
- `connection-drivers.ts`: `KTX_DATABASE_DRIVER_IDS` is now an exported (`@internal`)
readonly tuple so the coverage test derives required coverage from the registry;
`isDatabaseDriver` behavior is unchanged.
- `skills/analytics/SKILL.md`: a single dialect-agnostic pointer in step 5 ("call
`sql_dialect_notes` … to get that engine's FQTN, identifier-quoting, date, top-N,
and JSON conventions"). It names the tool only; spec 07's `<sql_craft>` block and
its dialect-clean content test are untouched.
**Tests**
- `test/context/mcp/dialect-notes.test.ts`: registry-derived coverage (a future
connector fails the test until its dialect has notes), the full rubric per dialect,
leak isolation (sqlite shows `strftime` and never `VARIANT`/`_TABLE_SUFFIX`;
`QUALIFY` only on snowflake/bigquery; engine-exclusive markers stay put), no
benchmark/grader or version-dated content, the postgres fallback, and
`resolveDialectNotesForConnection` resolving sqlite / snowflake / `sqlserver→tsql`
and rejecting a non-SQL source / unknown connection with `KtxExpectedError`; plus a
guard that the `DIALECTS_WITH_NOTES` const and the `dialects/*.md` files stay in sync.
- `test/context/mcp/server.test.ts`: `sql_dialect_notes` added to the retained tool
set + annotations assertion + a handler-routing test, and the regenerated
`__snapshots__/mcp-tools-list.json`.
- `test/skills/analytics-skill-content.test.ts`: asserts the new pointer is present
and the flat skill stays dialect-clean.
**Verification** — `tsc -p tsconfig.json` (src) clean; full default suite 393 files /
3001 passing; slow suite green (incl. `local-project-ports.test.ts`); all three
`dead-code` checks clean; the `dialects/*.md` files copy into `dist`. Rebuilt and
re-linked `ktx-dev`.
**Deviations / notes**
- Notes are stored as per-dialect markdown files (not a typed map, and not bundled
`reference/*.md` skill files) — all sanctioned by the spec; plain markdown is the
most maintainable to edit. They are served by the tool and ship via a
`copy-runtime-assets.mjs` entry (`src/context/sql-analysis/dialects → dist/…`); no
`setup-agents.ts` change.
- `pnpm run type-check` still reports one pre-existing, unrelated error in
`test/mcp-server-factory.test.ts` (committed in-flight MCP work on this branch);
this change adds zero new type errors and does not touch that file.

362
spider2-specs/specs/09-fan-out-safe-multi-hop-aggregation.md
View file

@ -1,362 +0,0 @@
# Strengthen fan-out-join safety for multi-hop aggregation in the analytics skill
> Refined spec. Intake draft: `todo/09-fan-out-safe-multi-hop-aggregation.md`.
> Extends spec 07 (`specs/07-analytics-skill-sql-craft.md`), which shipped the
> `<sql_craft>` block. Additive, content-only.
## Problem
The shipped `ktx-analytics` skill
(`packages/cli/src/skills/analytics/SKILL.md`) already carries a single-hop
fan-out rule in `<sql_craft>` → **Composition**:
> **Avoid fan-out joins.** Add columns only from tables already at the target
> grain, or pre-aggregate to that grain before joining. A join that multiplies
> rows quietly inflates every downstream `SUM`/`COUNT`.
In practice the agent honors that on a single join but still **silently
fans out on multi-hop join chains**, where the inflation is one or two joins
removed from the aggregate and therefore much harder to notice.
The failure shape: a measure that lives at a *coarse* grain (one row per parent
record) is counted/summed *after* the parent has been joined down to a *finer*
grain (one row per child line). Every parent-level value is then duplicated by
its child fan-out, so `COUNT(*)` / `SUM(amount)` over-counts by a data-dependent
amount — runnable SQL, plausible-looking number, quietly wrong.
The rule today is stated only as a **prohibition** ("Avoid…"). It needs two
upgrades: (a) generalize it so the danger is understood as *cumulative across a
whole join chain*, not a single join; and (b) pair it with an **affirmative
verification habit** the agent runs while composing, so a grain change is
detected and fixed rather than merely warned against.
## Generic use case (independent of any benchmark)
An analyst on any production warehouse asks a counting/summing question whose
path runs through several one-to-many hops — e.g. *"how many orders per region
contain a returned item?"* where the path is `region → store → order →
order_line`. The honest answer counts each order once. The naïve join chain joins
`order_line` (to apply the line-level condition) and then counts orders, so an
order with three returned lines is counted three times. The inflation happens
**three joins below the `COUNT`**, where it is easy to miss. This is one of the
most common silently-wrong analytics mistakes on normalized schemas — not
specific to any dataset, dialect, or benchmark.
## Model (invariants — the implementer owns the prose)
These constrain the change; the exact wording is the implementer's. Each is
grounded in Anthropic's skill-authoring and prompt-engineering guidance so the
addition stays consistent with how spec 07 was written.
### Additive, inline-only, dialect-agnostic (inherited from spec 07)
The change is **additive content inside `skills/analytics/SKILL.md`** only — no
bundled `reference/*.md` file (the delivery path ships a single `SKILL.md` per
target; see spec 07 §Model "Inline-only delivery"). No new tool, flag, or config.
Every addition must read correctly on any dialect: **no** `QUALIFY`,
`strftime`/`julianday`, backtick/`DB.SCHEMA.TABLE` FQTNs, or other single-dialect
construct — including in the worked example. The existing `<workflow>`, `<rules>`,
`<examples>`, and the other four `<sql_craft>` sub-headings are preserved
unchanged.
### Heuristic-plus-*why*, because SQL authoring is a high-freedom task
Anthropic's "set appropriate degrees of freedom" guidance classifies tasks with
many valid approaches where decisions depend on context as **high freedom →
text-based heuristics**, the "open field, many paths" case (versus low-freedom,
fragile operations that need an exact script). SQL authoring is squarely
high-freedom. So the new content is phrased as **heuristics with a one-line,
universal rationale**, never as bare `ALWAYS`/`NEVER` imperatives — matching the
existing `<sql_craft>` style and Anthropic's "add context / explain why so Claude
generalizes" principle.
### Affirmative framing for the verification step (do, not don't)
Anthropic's prompt-engineering guidance is explicit: **"Tell Claude what to do
instead of what not to do."** The draft's requirement for "a detect-and-fix
*habit*, not just a prohibition" is the same principle. Therefore:
- The **generalized rule keeps the established `Avoid fan-out joins` lead and the
term `fan-out`** — it is spec 07's consistent terminology and the existing
content test references that phrase; reframing it would churn shared vocabulary
for no gain.
- The **new verification step is phrased affirmatively** (e.g. *"Verify the grain
holds across each join"*) — an action the agent performs while composing, not a
warning. The two together satisfy both principles: a recognized anti-pattern
name *and* a positive habit.
### One default with an escape hatch, not two equal options
Anthropic: **"Avoid offering too many options… provide a default with an escape
hatch."** The fix for an inflated aggregate is presented as exactly that:
- **Default: pre-aggregate the measure to its own grain in a CTE, then join the
already-aggregated result.** This is the single-hop fix generalized, and it is
the *only* correct fix for `SUM`/`AVG` — you cannot de-duplicate a summed
measure with `DISTINCT` (two legitimately-equal amounts would collapse).
- **Escape hatch: `COUNT(DISTINCT key)` — for a pure count only.** It rescues an
inflated count in one line, but must be stated as count-only, not as a general
remedy.
This is the deepest correctness point in the spec and the easiest to get wrong; a
naïve blanket "just use `COUNT(DISTINCT)`" is silently wrong for sums.
### Consistent terminology
Anthropic: **"Choose one term and use it throughout."** Reuse spec 07's existing
vocabulary verbatim — **`grain`**, **`fan-out`**, **`pre-aggregate`** — do not
introduce synonyms (e.g. do not rename the concept "row blow-up" or
"multiplication factor"). Prose may vary, but the named concepts stay fixed.
### Concise — the addition must justify its token cost
Anthropic: **"Concise is key… does this paragraph justify its token cost?"** and
"Claude is already very smart." The agent knows what a join and a `GROUP BY` are;
the addition explains only the non-obvious trap (cumulative grain inflation) and
shows the fix. Net addition is roughly one rewritten bullet, one new bullet, and
one worked example — the skill stays comfortably under the 500-line budget
(~117 lines today).
### Examples over descriptions — exactly one
Anthropic's "examples pattern": **"Examples help Claude understand the desired
style and level of detail more clearly than descriptions alone"** and
"examples are concrete, not abstract." The multishot guidance favors 3–5 examples
in general, but here **conciseness and spec 07's one-example-per-rule economy
win**: the skill already carries the window-then-filter example, so this adds
**exactly one** compact wrong-vs-right example. The wrong/right contrast inside
that single example supplies the diversity multishot calls for, at one example's
token cost.
### Leak-safety (hard constraint)
The worked example must be a **synthetic, generic schema invented for teaching** —
not the tables, column names, query, or numeric results of any Spider 2.0-Lite
question. It demonstrates the *pattern* (a coarse-grain measure aggregated after a
one-to-many join), which is universal and reconstructable from first principles. A
reviewer must find nothing in it that ties it to a specific benchmark instance.
See "Leak-safety" below.
## Requirements
All four land in the **Composition** sub-heading of `<sql_craft>` in
`packages/cli/src/skills/analytics/SKILL.md`. Structure (chosen design): rewrite
the existing fan-out bullet, add one affirmative verification bullet, add one
worked example. Do not touch the other four sub-headings or `<workflow>`/`<rules>`/
`<examples>`.
### 1. Generalize the fan-out rule to multi-hop chains
Rewrite the existing **`Avoid fan-out joins.`** bullet so it makes explicit that
the danger is **cumulative**: *any* one-to-many hop on the path between a measure's
owning table and the aggregate inflates that measure, **even when the offending
join is several hops away from the `SUM`/`COUNT`**. The fix is the same as the
single-hop case — **pre-aggregate the measure to its own grain in a CTE, then join
the already-aggregated result** — but the agent must apply it **per
measure-owning table along the whole chain**, not just at the final join. Keep the
`fan-out` term and the one-line *why*.
### 2. Add an affirmative grain-verification habit
Add a companion bullet, phrased as an action the agent performs **while
composing** (not a prohibition):
- Confirm that a join intended to be one-to-one / many-to-one **did not change the
grain** it aggregates at — e.g. check that the row count (or the count of the
aggregate's key) is unchanged across that join.
- When a join is genuinely one-to-many, **reach for the default fix
(pre-aggregate to grain)**; for a **pure count**, `COUNT(DISTINCT key)` is an
acceptable escape hatch.
- State the caveat once: **`SUM`/`AVG` of a fanned-out measure must pre-aggregate**
— `DISTINCT` cannot de-duplicate a sum.
This is spec 07's "build incrementally and check each layer" discipline pointed
specifically at grain preservation, in affirmative form.
### 3. One concrete, generic multi-hop worked example
Add **exactly one** compact wrong-vs-right `sql` example inside `<sql_craft>`
demonstrating the multi-hop inflation and the pre-aggregate fix. It is the
**second** `sql` fence in the skill (the first is spec 07's window-then-filter
example).
**Required properties** (these are the constraints; the SQL below is orientation):
- **Multi-hop chain** where the inflating one-to-many hop is **≥1 join removed**
from the aggregate (not the single-hop case spec 07 already covers).
- **Unambiguous attribution**: each counted entity maps to **exactly one** group,
so the honest answer is well-defined. (This rules out "coarse measure attributed
to a fine dimension reached by descending," where one entity spans several
groups and the correct number is itself ambiguous — that would teach a murky
pattern.)
- **Motivated descent**: the finer-grain table is joined for a real reason (a
line-level filter or a needed line-level value), so the reader sees *why* the
fan-out join is there.
- **Plain `COUNT`/`SUM`**, not `AVG` — averaging collides with the existing
*Macro vs micro average* bullet and would muddy the fan-out lesson.
- The **RIGHT side demonstrates the default fix** (pre-aggregate to grain in a
CTE) and is **actually correct**, not merely runnable — its number must equal the
honest answer, not just avoid an error.
- Generic invented schema, standard dialect-agnostic SQL (no `QUALIFY`, no dialect
functions), no benchmark identifiers or values.
**Recommended sketch** (implementer may adjust within the properties above):
```sql
-- "How many orders per region contain a returned item?"
-- WRONG: joining order_lines to apply the line-level filter multiplies orders —
-- an order with two returned lines is counted twice, three joins below the COUNT.
SELECT r.region_id, COUNT(*) AS n_orders
FROM regions r
JOIN stores s ON s.region_id = r.region_id
JOIN orders o ON o.store_id = s.store_id
JOIN order_lines l ON l.order_id = o.order_id
WHERE l.status = 'returned'
GROUP BY r.region_id;
-- RIGHT: collapse order_lines to one row per qualifying order first, then join up.
WITH returned_orders AS (
SELECT order_id FROM order_lines WHERE status = 'returned' GROUP BY order_id
)
SELECT r.region_id, COUNT(*) AS n_orders
FROM regions r
JOIN stores s ON s.region_id = r.region_id
JOIN orders o ON o.store_id = s.store_id
JOIN returned_orders ro ON ro.order_id = o.order_id
GROUP BY r.region_id;
-- A pure count could also use COUNT(DISTINCT o.order_id); a SUM/AVG of an
-- order-level measure fanned out this way must pre-aggregate — DISTINCT can't
-- de-duplicate a sum.
```
### 4. Placement and structure
- Both bullets live under the existing **Composition** sub-heading; the example
follows them. The five-sub-heading structure spec 07 established is unchanged.
- **State each rule once** (Anthropic "consistent terminology / don't repeat"):
do not also restate the multi-hop rule in `<workflow>` steps 5/6 — those already
carry a one-line pointer into `<sql_craft>`, which is sufficient.
### 5. Coordination with spec 07 (supersession)
Spec 07's requirement 3 and acceptance criteria say the skill contains **exactly
one** worked example and "Do not add a second example." **This spec supersedes
that constraint**: the skill now carries **two** `sql` worked examples
(window-then-filter from spec 07, plus this multi-hop fan-out example). Annotate
spec 07 at those two spots with a one-line "superseded by spec 09" note so the two
permanent specs do not contradict. No other spec 07 content changes.
## Leak-safety (hard constraint on this spec and its example)
The benchmark's gold answers must never appear in ktx. The worked example must be
a **synthetic, generic schema invented for teaching** — not the tables, column
names, query, or numeric results of any Spider 2.0-Lite question. The example
demonstrates the *pattern* (a coarse-grain measure counted after a one-to-many
join), which is universal; it must be reconstructable from first principles by
anyone, with zero reference to benchmark data. A reviewer should be able to read
the example and find nothing that ties it to a specific benchmark instance.
## Acceptance criteria
- The `<sql_craft>` **Composition** section states the **multi-hop generalization**
of the fan-out rule (cumulative danger across the chain; pre-aggregate per
measure-owning table) and an **affirmative grain-verification habit**, inline and
dialect-agnostic.
- The fix is presented as **default (pre-aggregate to grain) + escape hatch
(`COUNT(DISTINCT key)`, count-only)**, with the explicit caveat that `SUM`/`AVG`
of a fanned-out measure must pre-aggregate.
- Exactly **one** new, **generic** worked example (wrong vs. pre-aggregated-right)
using an invented schema, with no benchmark-derived identifiers or values, whose
RIGHT side is actually correct (unambiguous attribution; honest number).
- The skill now contains **two** `sql` worked examples total; the existing content
test's fence-count assertion is updated `1 → 2` and new assertions cover the
multi-hop rule phrase and the grain-verification-habit phrase.
- Terminology is consistent with spec 07 (`grain`, `fan-out`, `pre-aggregate`); no
synonyms introduced.
- **No new tool, flag, or config.** Skill-content only; additive to spec 07.
- All spec 07 invariants still hold: the skill remains dialect-agnostic (no
`QUALIFY`/`strftime`/`julianday`, no backtick three-part FQTN, no relative-time
anchoring to a `MAX(...)` date) and free of any benchmark/grader/gold reference,
including in the new example; `<workflow>`/`<rules>`/`<examples>` and the other
four sub-headings are intact; frontmatter still parses through
`SkillsRegistryService.parseFrontmatter`; the skill stays under 500 lines.
- Spec 07's "exactly one example" constraint is annotated as superseded (no
contradiction between the two permanent specs).
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the prose.
- **The skill file:** `packages/cli/src/skills/analytics/SKILL.md` →
`<sql_craft>` → **Composition**. Rewrite the `Avoid fan-out joins` bullet, add
the affirmative grain-verification bullet, add the one worked example after them.
Leave the other four sub-headings, `<workflow>`, `<rules>`, and `<examples>`
unchanged.
- **Tests:** `packages/cli/test/skills/analytics-skill-content.test.ts`. Update the
"ships exactly one … worked example" test: `match(/```sql/g)` length `1 → 2`,
add an assertion for the new fan-out example's distinctive tokens (e.g.
`WITH returned_orders AS`), add the multi-hop-rule and grain-verification-habit
phrases to the behavior-presence list, and keep all banned-construct and
size-budget guards. This is a content assertion over the source `SKILL.md` — the
right level for prompt content.
- **Spec 07 annotation:** add a one-line "superseded by spec 09" note at spec 07's
requirement 3 and at its "Exactly one new worked example" acceptance bullet.
- **Rebuild/re-link** the dev binary so the playground picks up the change:
`pnpm run build && pnpm run link:dev` (provides `ktx-dev`).
## Benchmark context (motivation only)
Multi-hop aggregation questions (counting/averaging a coarse-grained measure
reached through several one-to-many joins) are a recurring source of
result-mismatch failures in the SQLite subset: the agent produces runnable SQL
with the right tables but a fan-out-inflated number. These are correctness
failures, not knowledge or schema-discovery failures (zero execution errors in the
latest run), so the fix belongs in the product's authoring craft — where it also
helps any real analyst — not in a benchmark-specific prompt. The skill itself must
contain no trace of the benchmark.
## Implementation notes
Shipped as specified — additive, content-only, no new tool/flag/config.
- **`packages/cli/src/skills/analytics/SKILL.md`** → `<sql_craft>` → **Composition**:
- Rewrote the `Avoid fan-out joins` bullet to `**Avoid fan-out joins — the
danger is cumulative.**`, generalizing to multi-hop chains: any one-to-many
hop between a measure's owning table and the aggregate inflates that measure
even when several hops below the `SUM`/`COUNT`; fix is pre-aggregate per
measure-owning table along the whole chain. Kept the `fan-out` term and the
one-line *why*.
- Added the affirmative `**Verify the grain holds across each join.**` bullet:
confirm a one-to-one / many-to-one join did not change the grain (row/key
count unchanged); default fix is pre-aggregate to grain, escape hatch is
`COUNT(DISTINCT key)` for a pure count only; stated once that `SUM`/`AVG` of a
fanned-out measure must pre-aggregate because `DISTINCT` cannot de-duplicate a
sum.
- Added one generic wrong-vs-right worked example (orders→regions via
stores/order_lines, `WITH returned_orders AS …`) — the second `sql` fence in
the skill. The inflating hop is three joins below the `COUNT`; the RIGHT side
pre-aggregates `order_lines` to one row per qualifying order so each order is
counted once (honest answer), and the trailing comment names the count-only
`COUNT(DISTINCT o.order_id)` escape hatch plus the `SUM`/`AVG` caveat. Invented
schema, dialect-agnostic SQL, no benchmark identifiers/values.
- The other four sub-headings and `<workflow>`/`<rules>`/`<examples>` are
untouched. Skill is 147 lines (well under the 500-line budget).
- **`packages/cli/test/skills/analytics-skill-content.test.ts`**: sql-fence count
`1 → 2`; added the multi-hop phrase (`the danger is cumulative`) and the
grain-verification phrase (`Verify the grain holds across each join`) to the
behavior-presence list; added new-example token assertions
(`WITH returned_orders AS`, `COUNT(DISTINCT o.order_id)`). All banned-construct,
relative-time, and size-budget guards retained. Test file passes (9/9).
- **Spec 07** annotated as superseded at requirement 3 and at its "exactly one
worked example" acceptance bullet — no contradiction between the two permanent
specs.
**Verification:** `vitest run test/skills/analytics-skill-content.test.ts` → 9
passed. `pnpm run build` (src `tsc -p tsconfig.json`) succeeds and the built
`dist/skills/analytics/SKILL.md` carries the new content; `pnpm run link:dev`
re-linked `ktx-dev`. A pre-existing, unrelated type error in
`test/mcp-server-factory.test.ts` (`KtxMcpContextPorts`/`context_tool`, last
touched in commit `2677b3ef`) surfaces under the full `type-check`'s
`tsconfig.test.json` pass; it is outside this change's surface and not introduced
here.

289
spider2-specs/specs/10-panel-completeness-spine.md
View file

@ -1,289 +0,0 @@
# Panel/period completeness — emit the full set of groups, not only the populated ones
> Refined spec. Intake draft: `todo/10-panel-completeness-spine.md`.
## Problem
When a question asks for a result *per period* or *per category* ("orders for
each month of 2023", "revenue by region", "count per status"), a plain `GROUP BY`
only returns groups that actually have rows. Periods or categories with **zero**
activity silently vanish, so a "12 months" answer comes back with 9 rows and the
three that should read `0` are simply absent. The SQL is runnable and the
aggregate is right, but the **panel is incomplete** — and a monthly report with
missing months or a category breakdown missing its empty categories is wrong for
any analyst, on any database.
The existing `<sql_craft>` "Answer completeness / interpretation" group already
carries a *"For each X / per X / by X returns exactly one row per X"* rule, but
that rule only governs **grain** (don't collapse to a single value). It says
nothing about the **domain**: "one row per X" today means one row per *observed*
X, so empty groups still drop. This spec sharpens that rule from grain-only to
grain-and-completeness.
## Generic use case (independent of any benchmark)
"How many orders were placed in each month of 2023?" must return **12 rows** even
if March had no orders (March = 0), not 11. "Sales per region" should include
regions with no sales when the question asks for *each* region. Both are
bread-and-butter reporting for any analyst on any warehouse, with no benchmark in
sight.
## Model
The feature splits across **two surfaces**, each holding the half it is suited
for. This split is the central design decision and exists to satisfy spec 07's
hard dialect-agnostic invariant without weakening it.
### Why two surfaces (the dialect-agnostic reconciliation)
The draft asked for a *"recursive-CTE date spine"* worked example. But a real
date/number series is **inherently dialect-specific** — Postgres `generate_series`,
SQLite recursive `date(d,'+1 month')`, BigQuery `GENERATE_DATE_ARRAY`, Snowflake
`GENERATOR`+`DATEADD` — and spec 07 made `<sql_craft>` strictly dialect-agnostic
(the analytics-skill content test bans single-dialect constructs). Inlining a date
spine would violate that invariant; carving out a test exception would erode it.
ktx already has the canonical home for engine-specific syntax: the per-dialect
notes in `packages/cli/src/context/sql-analysis/dialects/<dialect>.md`, served by
the `sql_dialect_notes` MCP tool (spec 08). Those files answer a fixed rubric
(FQTN / Identifiers / Date-time / Top-N / JSON) — but **series/spine generation is
not in that rubric yet**. So the date-spine syntax belongs *there*, alongside the
other per-dialect idioms, and the dialect-agnostic skill points to it. This
routes the dialect-specific half through the existing channel rather than
standing up a parallel dialect-specific recipe inside the skill.
Surface 1 (skill) carries the **pattern**; surface 2 (dialect notes) carries the
**concrete series syntax**.
### Additive, inline, heuristic-with-a-why
Consistent with spec 07: the skill change is **additive content in one Markdown
file** (`skills/analytics/SKILL.md`), inline (no bundled `reference/` file — the
delivery mechanism in `setup-agents.ts` ships only `SKILL.md`), dialect-agnostic,
and phrased as a **heuristic with a one-line generic rationale**, not a wall of
MUSTs. The dialect-notes change is additive content in the seven existing
`dialects/*.md` files. No new tool, flag, or config on either surface.
## Requirements
### 1. Skill surface — `<sql_craft>` "Answer completeness / interpretation"
Add the panel-completeness rule to the existing group (it extends, and should sit
adjacent to, the *"For each X / per X / by X"* bullet). It must cover:
1. **Recognize the full-panel cue.** *each / every / all / per <period> / for all
<category> / by month* signals that the answer's row set should be the
**complete expected domain** of periods or categories in scope, not just those
present in the filtered fact rows. *Why:* a plain inner `GROUP BY` can only emit
groups that have at least one fact row.
2. **Spine → LEFT JOIN → COALESCE.** Build the full set of expected groups (the
**spine**), then LEFT JOIN the aggregated facts onto it:
- **Category/dimension spine:** the distinct values from the **domain-defining
dimension/entity table** (e.g. all regions from a `regions` table), *not*
`SELECT DISTINCT region FROM facts` — the latter yields only categories that
already occur, so a zero-activity category still drops. When no dimension
table exists, the distinct values from the **unfiltered** fact table are the
best available domain (with the residual caveat that a category which never
occurs at all cannot surface).
- **Period/number spine:** generate the series for the question's stated range
(e.g. each month of 2023 → Jan..Dec 2023). The series bounds come from the
question's explicit range; when the range is "all periods present," derive
bounds from `MIN`/`MAX` over the **unfiltered** facts. The concrete
series-generation syntax is per-dialect — the rule points the author to
`sql_dialect_notes` (see requirement 2) and shows no inline series SQL.
3. **COALESCE by measure additivity.** Default missing measures with
`COALESCE(metric, 0)` for **additive** measures (a `COUNT` or `SUM` of events
or amounts — "no activity" genuinely reads as 0). Leave **non-additive**
measures (`AVG`, a running balance, a price, a rate, a ratio) as **NULL** —
absence is "no data," and 0 would be a wrong reading. *Why:* 0 is a real value
only for additive measures.
4. **Don't over-apply (the each-vs-which guard).** When the question asks only
about groups that exist ("*which* months had orders", "regions that made a
sale"), the spine is unnecessary and wrong — emit only observed groups. The cue
is *each / all / every* (complete domain) vs *which / that have* (observed
subset).
5. **One worked example — the category spine, fully portable.** Add **exactly
one** compact before/after example demonstrating the pattern with a
**distinct-dimension spine**: the wrong shape (`GROUP BY` over facts, empty
groups missing) and the right shape (`SELECT DISTINCT` domain from the
dimension table → LEFT JOIN aggregated facts → `COALESCE(metric, 0)`). Generic
table/column names, standard SQL only — no series generation, no dialect
functions, so the example stays dialect-clean. The period-spine variant is
described in prose (requirement 2) and delegated to `sql_dialect_notes`; it
gets **no** inline example. This is the **third** worked `sql` example in the
skill (after spec 07's window-then-filter and spec 09's multi-hop fan-out).
6. **Step pointer, no duplication.** The validate/explain step (and/or the query
step) already points into `<sql_craft>` for answer-completeness; extend that
existing pointer's wording if needed, but state the rule **once** inside
`<sql_craft>`. The step-5 pointer that lists what `sql_dialect_notes` provides
("FQTN, identifier-quoting, date, top-N, and JSON conventions") should also
name the **series/calendar** convention now that it exists.
### 2. Dialect-notes surface — `dialects/*.md`
Add a **"Series"** (date/number range) line to **each** of the seven authored
dialect files, giving that engine's idiomatic way to generate a contiguous
date or integer series for use as a spine. Each note is engine-exclusive — a
SQLite analyst gets the SQLite idiom and never another engine's construct, per the
existing dialect-notes leak guards. Orientation (exact syntax is the
implementer's):
- **postgres:** `generate_series('2023-01-01'::date, '2023-12-01'::date, interval '1 month')`.
- **sqlite:** recursive CTE — `WITH RECURSIVE m(d) AS (SELECT '2023-01-01' UNION ALL SELECT date(d,'+1 month') FROM m WHERE d < '2023-12-01')`.
- **bigquery:** `UNNEST(GENERATE_DATE_ARRAY('2023-01-01','2023-12-01', INTERVAL 1 MONTH))` (and `GENERATE_ARRAY` for integers).
- **snowflake:** `TABLE(GENERATOR(ROWCOUNT => n))` with `DATEADD('month', SEQ4(), start)`, or a recursive CTE.
- **mysql:** recursive CTE (8.0+) with `DATE_ADD(d, INTERVAL 1 MONTH)`.
- **clickhouse:** `numbers(n)` / `range(n)` with `addMonths(start, number)` (or `arrayJoin`).
- **tsql:** recursive CTE with `DATEADD(month, …)`, or a numbers/tally table.
This line is what makes the period spine usable from the dialect-agnostic skill,
and it is also consumed by **spec 11** (rolling-window-over-gappy-dates needs the
same date spine) — so it is foundational, not scope creep.
### 3. Coordination with spec 11
Spec 11 (time-series window recipes) explicitly depends on this date spine for the
gappy-rolling case ("build a complete date spine first (see spec 10)"). Spec 10
establishes the spine concept in the Answer-completeness group and the
series syntax in the dialect notes; spec 11 reuses both from the Window-functions
group. Keep the two non-overlapping: spec 10 owns the spine; spec 11 references it.
## Leak-safety (hard constraint)
Any worked example or note must use a **synthetic generic schema** (e.g. an
`orders` table with an `order_date`, a `regions` dimension) and demonstrate only
the *pattern* (spine + LEFT JOIN + COALESCE). **No** benchmark table names, SQL,
or result values on either surface. The dialect-notes additions, like the existing
notes, carry no benchmark/grader/version-dated content. The behavior is
reconstructable from first principles and tied to no specific instance.
## Acceptance criteria
- `<sql_craft>` "Answer completeness / interpretation" states: the full-panel cue,
the spine → LEFT JOIN → COALESCE recipe, the additive-vs-non-additive COALESCE
discriminator (0 vs NULL), and the each-vs-which over-application guard —
inline, dialect-agnostic, each with a generic *why*.
- Exactly **one** new worked `sql` example is present, a portable
distinct-dimension spine (`SELECT DISTINCT` domain → LEFT JOIN → `COALESCE`),
with no series generation and no dialect-specific syntax. The skill then carries
**three** `sql` worked examples total.
- Each of the seven `dialects/*.md` files gains a **Series** (date/number range)
line in its engine's own idiom; no engine leaks another engine's construct, and
the additions contain no benchmark/grader/version-dated content.
- The skill remains dialect-clean: no `QUALIFY`, `strftime`, `julianday`,
`generate_series`, `GENERATE_DATE_ARRAY`, backtick three-part FQTN, or other
single-dialect construct anywhere in `SKILL.md`, including the new example.
- The existing interactive guidance (`<workflow>`, `<rules>`, the other examples)
and the existing dialect-note rubric lines are intact and uncontradicted.
- No grader/benchmark reference, no output-shape contract, and no anchoring of
*relative* time ("recent" / "past N months") to a `MAX(date)` over the data
appears (period-spine bounds derive from the question's explicit range or, for
"all periods present," from `MIN`/`MAX` over the facts — which is range
derivation, not relative-time anchoring).
- The skill stays scannable and comfortably under the 500-line budget; frontmatter
still parses as `ktx-analytics`.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the prose.
- **Skill:** `packages/cli/src/skills/analytics/SKILL.md` — add the
panel-completeness bullets to the Answer-completeness group, the single category
spine example, and extend the existing step pointer / dialect-notes provision
list to name the series convention. Leave `<workflow>`/`<rules>`/other examples
intact. Delivery is unchanged (single `SKILL.md` per target via
`readAnalyticsSkillContent` in `setup-agents.ts`) — confirm, no change required.
- **Dialect notes:** the seven files under
`packages/cli/src/context/sql-analysis/dialects/`. The list is kept in sync with
`DIALECTS_WITH_NOTES` (`dialect-notes.ts`) and shipped to `dist` by
`copy-runtime-assets.mjs` — no plumbing change, content only.
- **Tests:**
- `packages/cli/test/skills/analytics-skill-content.test.ts` — add a
representative phrase for the completeness rule; bump the `sql`-fence count
assertion **2 → 3**; assert the spine + LEFT JOIN + `COALESCE` shape; the
existing dialect-clean guards already cover the no-inline-series requirement
(the example is `SELECT DISTINCT`, so they pass unchanged).
- `packages/cli/test/context/mcp/dialect-notes.test.ts` — extend the rubric loop
(the "answers the full rubric for every dialect" test) so every dialect must
also answer a **Series** line, e.g. `expect(notes).toMatch(/\*\*Series/)`.
Coverage is derived from `DIALECTS_WITH_NOTES`, so the new assertion enforces
all seven without a hand-maintained list.
- Rebuild and re-link the dev binary so the playground picks up both surfaces:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation only)
Per-period / per-category questions where some periods are empty produce
short-row result mismatches in the SQLite subset, and the related rolling/cumulative
cluster (spec 11) needs a complete date spine to be correct at all. The fix is a
universal reporting habit (complete panels) plus the per-dialect series syntax
that makes it executable — both belong in the product, where they help real
analysts. Improving the benchmark score is a side effect; the skill and the
dialect notes contain no trace of the benchmark.
## Implementation notes
Shipped on branch `write-feature-spec-wiki`. Content-only across two surfaces, no
new tool/flag/config, no plumbing change.
**Surface 1 — skill (`packages/cli/src/skills/analytics/SKILL.md`):**
- Added a **"Complete the panel for 'each / every / all / per <period or
category>'"** bullet to the `<sql_craft>` "Answer completeness / interpretation"
group, directly after the *"For each X / per X / by X"* bullet, with three
sub-bullets carrying the rest of the rule each with its generic *why*: **Spine
source** (distinct domain from the dimension/entity table — not `SELECT DISTINCT`
over the facts; period/number series across the question's stated range, bounds
from `MIN`/`MAX` over the *unfiltered* facts for "all periods present"; series
syntax delegated to `sql_dialect_notes`), **Default by additivity**
(`COALESCE(metric, 0)` for additive measures, `NULL` for non-additive), and
**Don't over-apply** (the each-vs-which guard).
- Added **one** worked `sql` example at the end of the Answer-completeness group: a
portable distinct-dimension spine (`SELECT DISTINCT region_id FROM regions` →
`LEFT JOIN` aggregated facts → `COALESCE(ro.n_orders, 0)`), wrong-vs-right,
standard SQL only, no series generation, no dialect functions. The skill now
carries **three** `sql` worked examples.
- Extended the step-5 dialect-notes pointer to name the **series/calendar**
convention alongside FQTN / identifier-quoting / date / top-N / JSON.
- Delivery unchanged: `readAnalyticsSkillContent` in `setup-agents.ts` ships the
single `SKILL.md` per target — confirmed, no change.
**Surface 2 — dialect notes (`packages/cli/src/context/sql-analysis/dialects/*.md`):**
- Added a `- **Series:**` line to all seven authored files (postgres, sqlite,
bigquery, snowflake, mysql, clickhouse, tsql), each in that engine's own idiom
(`generate_series`; recursive CTE with `date(d,'+1 month')`;
`UNNEST(GENERATE_DATE_ARRAY(...))`; `GENERATOR`/`SEQ4`/`DATEADD`; recursive CTE
with `DATE_ADD`; `numbers(n)`/`addMonths`; recursive CTE with `DATEADD` +
`MAXRECURSION`), placed right after each file's Date/time line. No cross-engine
leak, no version-dated/benchmark content. Shipped to `dist` unchanged by
`copy-runtime-assets.mjs`; coverage stays derived from `DIALECTS_WITH_NOTES`.
**Tests:**
- `test/skills/analytics-skill-content.test.ts`: added the `Complete the panel`
and `Default by additivity` phrases; renamed the worked-examples test and bumped
the `sql`-fence count **2 → 3**; asserted the spine + `LEFT JOIN` + `COALESCE`
shape. Also added `generate_series` and `GENERATE_DATE_ARRAY` to the
dialect-clean banned list — a deliberate **strengthening** beyond the spec's
test orientation so the "no inline series" acceptance criterion is *enforced*,
not merely incidentally true of a `SELECT DISTINCT` example.
- `test/context/mcp/dialect-notes.test.ts`: extended the "answers the full rubric
for every dialect" loop with `expect(notes).toMatch(/\*\*Series/)`, so all seven
dialects are required to answer a Series line (coverage derived from
`DIALECTS_WITH_NOTES`, no hand-maintained list).
**Verification:** both affected test files pass (19 tests). `src` type-check and
`pnpm run build` are clean, and `copy-runtime-assets.mjs` placed the Series line in
all seven `dist` dialect files; `pnpm run link:dev` re-linked `ktx-dev`. Note: an
unrelated, pre-existing `tsconfig.test.json` type error in
`test/mcp-server-factory.test.ts` exists on this branch — untouched by this work
and outside its scope.
**Coordination with spec 11:** the per-dialect Series line is the foundational
date spine that spec 11 (rolling/cumulative windows over gappy dates) references.
Spec 10 owns the spine (Answer-completeness group + dialect Series notes); spec 11
will reference it from the Window-functions group. No overlap introduced.

391
spider2-specs/specs/11-time-series-window-recipes.md
View file

@ -1,391 +0,0 @@
# Time-series window craft — running totals, rolling-over-time (min-periods), period-over-period
> Refined spec. Intake draft: `todo/11-time-series-window-recipes.md`.
## Problem
A large share of analytics questions are time-series shaped: a **running /
cumulative balance**, a **rolling N-day average**, or **period-over-period
growth**. The agent already knows window functions exist — spec 07 gave the
`<sql_craft>` "Window functions" group its determinism and window-then-filter
rules, and spec 10 added panel/period completeness — but it still gets the
*time-series specifics* wrong:
- a cumulative balance computed **without an explicit unbounded-preceding
frame**, or with the implicit frame misbehaving when there are **ties on the
order key**;
- "rolling 30 days" implemented as `ROWS BETWEEN 29 PRECEDING` over **gappy**
daily data, so the window spans the wrong calendar span when days are missing;
- no **minimum-periods** handling — a rolling average reported before the window
is actually full;
- "growth vs the previous period" written **without `LAG`** (or against the wrong
neighbor), with an **unguarded** `(cur - prev) / prev` that breaks on a zero or
absent prior.
These are runnable-but-wrong: the structure is close, the edge case diverges.
It is the same failure shape spec 07 addressed at the general level; this spec
adds the time-series specifics to the **same Window-functions group**, building
on the rules already there rather than restating them.
## Generic use case (independent of any benchmark)
- "Each account's month-end running balance over 2023" — a cumulative sum of
monthly net over an ordered window.
- "30-day rolling average of daily revenue, only once 30 days of history exist."
- "Month-over-month revenue growth rate."
All three are bread-and-butter for any analyst on any time-series table, with no
benchmark in sight. The methodology is universal analyst craft, so it belongs in
the shipped skill — it transfers to every ktx user querying a live database.
## Model
The change is **additive content across two surfaces** — the same split spec 10
made, and for the same reason. The split is the central design decision; it
satisfies spec 07's hard dialect-agnostic invariant for `<sql_craft>` without
weakening it.
### Why two surfaces (the dialect-agnostic reconciliation)
Two of the three recipes are **pure standard SQL** and stay entirely in the
dialect-agnostic skill:
- **Cumulative / running total** — `SUM(x) OVER (... ROWS BETWEEN UNBOUNDED
PRECEDING AND CURRENT ROW)` is standard on every engine.
- **Period-over-period** — `LAG(metric) OVER (...)`, the growth ratio, and a
`NULLIF`-style divide-by-zero guard are standard on every engine.
The third recipe — a **rolling window over calendar time** — has one piece that
is genuinely dialect-divergent: the **calendar-range window frame**. A native
range frame such as `RANGE BETWEEN INTERVAL '29 days' PRECEDING AND CURRENT ROW`
exists on some engines (e.g. postgres, mysql 8) but **not others** — sqlite has
no date-interval range frame, and SQL Server has **no offset `RANGE` frames at
all**; bigquery's `RANGE` frames are numeric-only. So a portable skill cannot
inline a range frame any more than it could inline a date-series generator.
ktx already routes that kind of engine-specific syntax through the per-dialect
notes in `packages/cli/src/context/sql-analysis/dialects/<dialect>.md`, served by
the `sql_dialect_notes` MCP tool (spec 08). Spec 10 established the precedent
exactly: series/spine generation was not in the dialect rubric, so it was added
there (the **Series** line) and the dialect-agnostic skill points to it.
Rolling-window framing is the next construct in that same position — not in the
rubric yet, dialect-specific — so the **rolling-window idiom belongs in the
dialect notes**, and the skill points to it.
Surface 1 (skill) carries the **pattern** (calendar range, not a row count; the
min-periods guard; the spine-or-range choice). Surface 2 (dialect notes) carries
the **concrete rolling-window frame syntax** per engine.
### Additive, inline, heuristic-with-a-why
Consistent with specs 07 and 10: the skill change is **additive content in one
Markdown file** (`skills/analytics/SKILL.md`), inline (no bundled `reference/`
file — `setup-agents.ts` ships only `SKILL.md`), dialect-agnostic, and phrased as
**heuristics with a one-line generic rationale**, not a wall of MUSTs. The
dialect-notes change is additive content in the seven existing `dialects/*.md`
files. No new tool, flag, or config on either surface.
### Build on the rules already present; do not restate them
The Window-functions group already carries **"Make the ordering deterministic"**
(complete tie-breaker) from spec 07, and the Numeric-precision group carries
**"Round only at the end."** The cumulative and period-over-period recipes
**reference** these rather than repeat them (state each rule once — Anthropic's
"consistent terminology / don't repeat" guidance, already followed in spec 07).
Spec 10's **Series** dialect line is likewise **referenced** by the rolling
recipe's spine fallback, not duplicated.
## Requirements
### 1. Skill surface — `<sql_craft>` "Window functions" group (three recipes)
Add three recipes to the **existing** "Window functions" group, after its two
current bullets (deterministic ordering; filter-after-the-window). Each is a
heuristic with a generic *why*, dialect-agnostic.
1. **Cumulative / running total.** Use an **explicit frame** — `SUM(x) OVER
(PARTITION BY k ORDER BY t ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)` —
with a **complete tie-breaker** on the `ORDER BY` (per the group's existing
deterministic-ordering rule; reference it, do not restate). *Why:* a bare
`ORDER BY` defaults to a `RANGE … CURRENT ROW` frame, which on **ties in the
order key** folds every tied peer into the same cumulative value — it runs and
looks plausible, but the running total jumps at each tie boundary.
2. **Rolling window over calendar time, plus minimum periods.** "Rolling N
days/months" must span a **calendar range**, not a fixed row count: a `ROWS
BETWEEN n-1 PRECEDING` frame silently measures the wrong span when days are
missing. Two sanctioned techniques:
- **Spine + `ROWS` (portable).** Build a gap-free date spine first (spec 10's
**Series**, via `sql_dialect_notes`) so the data has one row per calendar
unit; then a `ROWS BETWEEN n-1 PRECEDING AND CURRENT ROW` frame equals the
intended calendar span. This path is fully dialect-agnostic.
- **Native range frame or date-keyed self-join (engine-specific).** Where the
engine supports it, a calendar **range frame** expresses the window directly;
otherwise a self-join keyed on the date does. Both use engine-specific
syntax — get the **rolling-window** idiom from `sql_dialect_notes` (see
requirement 3); show no inline range frame in the skill.
**Minimum periods.** When the question says "only after N periods of data" (or
a rolling metric implies it), emit `NULL` / skip until the window is actually
full — guard on a window count, e.g. `COUNT(*) OVER (<same frame>) = N`. On a
gap-free spine, `COUNT(*)` counts calendar slots; count the **non-null
observations** instead when "N periods" means N data points rather than N
calendar units. *Why:* a row-count frame over missing dates measures the wrong
span, and a partial early window is not the requested metric.
3. **Period-over-period.** Use `LAG(metric) OVER (PARTITION BY k ORDER BY period)`
for the prior-period comparison; compute growth as `(cur - prev) / prev` at
**full precision**, rounding only in the final projection (per the existing
"Round only at the end" rule), and **guard divide-by-zero / NULL prev**
(e.g. divide by `NULLIF(prev, 0)`). *Why:* without `LAG` — or ordered against
the wrong neighbor — the comparison lands on the wrong period, and an unguarded
ratio errors or returns garbage when the prior period is zero or absent.
**Step pointer (no duplication).** The step-5 `sql_dialect_notes` provision list
(currently "FQTN, identifier-quoting, date, top-N, series/calendar, and JSON
conventions") should also name the **rolling-window** convention now that it
exists. State each rule once inside `<sql_craft>`; the workflow steps only point
to it.
### 2. One worked example — cumulative running total (dialect-agnostic)
Add **exactly one** new compact before/after `sql` example, demonstrating the
**cumulative running total** — the subtlest of the three (the implicit-frame trap
runs fine and is wrong only at tie boundaries) and the highest-value to show.
Use a synthetic generic schema (e.g. `account_txns(account_id, txn_date, net)`):
- **Wrong:** `SUM(net) OVER (PARTITION BY account_id ORDER BY txn_date)` — the
implicit `RANGE` frame makes two txns on the same date share one inflated
running balance.
- **Right:** the same with an explicit `ROWS BETWEEN UNBOUNDED PRECEDING AND
CURRENT ROW` frame and a complete tie-breaker (`ORDER BY txn_date, txn_id`).
Standard SQL only — no `QUALIFY`, no dialect functions, no series generation, no
`RANGE … INTERVAL`. Keep it ~10–14 lines. The **rolling-over-time** recipe gets
**no** inline example (its correct form needs the engine-specific frame/spine,
delegated to `sql_dialect_notes`, exactly as spec 10's period-spine variant was
prose-only); the **period-over-period** recipe is self-evident from its bullet
and also gets no example. This is the **fourth** worked `sql` example in the
skill, after spec 07 (window-then-filter), spec 09 (multi-hop fan-out), and
spec 10 (panel-completeness spine).
### 3. Dialect-notes surface — `dialects/*.md` (rolling window)
Add a **rolling-window-over-time** idiom line to **each** of the seven authored
dialect files, parallel to spec 10's **Series** line. Each note is
engine-exclusive — a SQLite analyst gets the SQLite idiom and never another
engine's construct, per the existing dialect-notes leak guards. Each note either
gives the engine's native calendar-range frame **or** references its own
**Series** line for the spine + `ROWS` fallback (a cross-reference within the
file, not a duplicate of the Series line).
Orientation only — **`RANGE`-frame support genuinely varies by engine and
version, so the implementer must verify each engine's current support against
authoritative docs (context7 / the engine's manual) rather than assert it from
memory.** Starting points:
- **postgres:** native — `... OVER (ORDER BY day RANGE BETWEEN INTERVAL '29 days'
PRECEDING AND CURRENT ROW)`.
- **mysql (8.0+):** native — `RANGE BETWEEN INTERVAL 29 DAY PRECEDING AND CURRENT
ROW` over a temporal order key.
- **bigquery:** `RANGE` frames are **numeric** — range over an integer day key
(e.g. `UNIX_DATE(day)`) with `RANGE BETWEEN 29 PRECEDING AND CURRENT ROW`, or
build a spine (see **Series**) and use a `ROWS` frame.
- **sqlite:** **no** date-interval range frame — build a date spine (see
**Series**) and use a `ROWS` frame.
- **tsql (SQL Server):** **no** offset `RANGE` frames at all — build a spine (see
**Series**) and use a `ROWS` frame, or a date-keyed self-join.
- **snowflake / clickhouse:** range-frame support over dates is limited — verify;
default to a spine (see **Series**) + `ROWS` frame where a native calendar range
frame is unavailable.
This line is what makes the rolling-over-time recipe executable from the
dialect-agnostic skill. It is **distinct** from spec 10's Series line (Series =
how to *generate* a spine; Rolling window = how to compute a *moving
calendar-range aggregate*, natively or via that spine), and it cross-references
the Series line rather than overlapping it.
### 4. Explicit constraints / exclusions
None of the following may appear (consistent with specs 07 and 10):
- **No inline dialect-specific range-frame syntax in the skill** — no
`RANGE … INTERVAL` frame, no series generator, no dialect function. The skill
stays dialect-clean; the range frame lives only in the dialect notes.
- **No anchoring of relative time to `MAX(date)`.** "Recent" / "past N months"
means relative to *now* on a live database. A range *bound* may be derived from
the question's explicit range or, for "all periods present," from `MIN`/`MAX`
over the **unfiltered** facts (range derivation, per spec 10) — but the metric
must never silently redefine "recent" as the data's maximum date.
- **No grader / gold-answer / benchmark reference**, and no output-shape contract
(the skill is for interactive analysis).
### 5. Coordination with specs 07 and 10
All three recipes live in the **existing** `<sql_craft>` "Window functions"
group; the two current bullets and the spec-07 window-then-filter example must
stay intact and uncontradicted.
- **Spec 07** owns the deterministic-ordering rule (Window functions) and the
round-at-the-end rule (Numeric precision). Spec 11 **builds on** both —
references them, never restates them.
- **Spec 10** owns the spine concept and the dialect **Series** line. Spec 11
**references** the spine for the gappy-rolling fallback and adds the **distinct**
rolling-window dialect line. Keep them non-overlapping: spec 10 = how to make a
spine; spec 11 = how to compute a moving calendar-range aggregate (native frame
or spine + `ROWS`).
## Leak-safety (hard constraint)
Every worked example or note uses a **synthetic generic schema** (e.g.
`daily_revenue(day, amount)` or `account_txns(account_id, txn_date, net)`) and
shows only the *pattern*. **No** benchmark table names, SQL, or result values on
either surface. The dialect-notes additions, like the existing notes, carry no
benchmark / grader / version-dated content. The behavior is reconstructable from
first principles and tied to no specific instance.
## Acceptance criteria
- The `<sql_craft>` "Window functions" group states the three recipes — inline,
dialect-agnostic, each with a generic *why*, and each **building on** (not
restating) the deterministic-ordering and round-at-the-end rules:
- **cumulative / running total** with an explicit `ROWS BETWEEN UNBOUNDED
PRECEDING AND CURRENT ROW` frame and a complete tie-breaker;
- **rolling window over calendar time + minimum periods** — calendar range not
row count, the spine-or-range choice, the min-periods `COUNT(*) OVER (...)`
guard — delegating the engine's range-frame syntax to `sql_dialect_notes`;
- **period-over-period** via `LAG`, with full-precision growth and a
divide-by-zero / NULL-prev guard.
- Exactly **one** new worked `sql` example: the cumulative running total,
wrong-vs-right, with the explicit `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT
ROW` frame and a complete tie-breaker, in standard dialect-agnostic SQL. The
skill then carries **four** `sql` worked examples total.
- Each of the seven `dialects/*.md` files gains a **rolling-window-over-time**
idiom line in its engine's own idiom (native calendar-range frame where
supported, otherwise a spine + `ROWS` fallback that references its **Series**
line); no engine leaks another engine's construct, and the additions contain no
benchmark / grader / version-dated content.
- The skill remains **dialect-clean:** no `QUALIFY`, `strftime`, `julianday`,
`generate_series`, `GENERATE_DATE_ARRAY`, backtick three-part FQTN, **and no
inline `RANGE … INTERVAL` frame**, anywhere in `SKILL.md` including the new
example.
- The step-5 `sql_dialect_notes` provision list names the **rolling-window**
convention alongside FQTN / identifier-quoting / date / top-N / series/calendar /
JSON.
- The existing interactive guidance (`<workflow>`, `<rules>`, the other
examples), the two existing Window-functions bullets, the window-then-filter
example, and the existing dialect-note rubric lines (including **Series**) are
intact and uncontradicted.
- No grader / benchmark reference, no output-shape contract, and no anchoring of
*relative* time ("recent" / "past N months") to a `MAX(date)` over the data.
- The skill stays scannable and comfortably under the 500-line budget; frontmatter
still parses as `ktx-analytics`.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the prose.
- **Skill:** `packages/cli/src/skills/analytics/SKILL.md` — add the three recipes
to the "Window functions" group (after its two existing bullets), the single
cumulative worked example, and extend the step-5 dialect-notes provision list to
name the rolling-window convention. Leave `<workflow>` / `<rules>` / the other
examples and the two existing window bullets intact. Delivery is unchanged
(single `SKILL.md` per target via `readAnalyticsSkillContent` in
`setup-agents.ts`) — confirm, no change required.
- **Dialect notes:** the seven files under
`packages/cli/src/context/sql-analysis/dialects/`. The list is kept in sync with
`DIALECTS_WITH_NOTES` (`dialect-notes.ts`) and shipped to `dist` by
`copy-runtime-assets.mjs` — no plumbing change, content only. **Verify each
engine's actual `RANGE`-frame support against authoritative docs before writing
the idiom; do not assert from memory.**
- **Tests:**
- `packages/cli/test/skills/analytics-skill-content.test.ts` — add a
representative phrase for each of the three recipes; bump the `sql`-fence count
assertion **3 → 4**; assert the cumulative example shape (e.g. `ROWS BETWEEN
UNBOUNDED PRECEDING AND CURRENT ROW`); and **strengthen** the dialect-clean
guard with a no-inline-`RANGE … INTERVAL` assertion (mirroring spec 10 adding
`generate_series` / `GENERATE_DATE_ARRAY` to the banned list, so the
"range frame lives only in the dialect notes" criterion is *enforced*, not
incidentally true).
- `packages/cli/test/context/mcp/dialect-notes.test.ts` — extend the "answers the
full rubric for every dialect" loop with the rolling-window assertion, e.g.
`expect(notes).toMatch(/\*\*Rolling/)`, so every dialect must answer it.
Coverage is derived from `DIALECTS_WITH_NOTES`, so the new assertion enforces
all seven without a hand-maintained list.
- Rebuild and re-link the dev binary so the playground picks up both surfaces:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation only)
Running-balance / rolling / period-over-period questions are the single largest
result-mismatch cluster in the SQLite subset (financial-transactions-style DBs):
cumulative balances with the wrong frame on ties, rolling windows that mis-span
gappy dates, partial early windows, and unguarded period-over-period ratios. The
methodology is universal analyst craft, so it belongs in the product's skill
(where it helps every real user) plus the per-dialect rolling-window syntax that
makes it executable — not in a benchmark-specific prompt. Depends on spec 10 (the
date spine) for the gappy-rolling fallback. Improving the benchmark score is a
side effect; the skill and the dialect notes contain no trace of the benchmark.
## Implementation notes
Shipped as additive content across the two surfaces the spec specified — no new
tool, flag, or config.
**Skill (`packages/cli/src/skills/analytics/SKILL.md`).** Added the three recipes
to the existing `<sql_craft>` "Window functions" group, after its two bullets and
the spec-07 window-then-filter example: **Cumulative / running total** (explicit
`ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` + a tie-breaker, referencing
the deterministic-ordering rule), **Rolling window over calendar time, plus
minimum periods** (calendar range not row count; spine-or-native-range choice
delegated to `sql_dialect_notes`; the `COUNT(*) OVER (<same frame>) = N`
min-periods guard), and **Period-over-period** (`LAG` + full-precision growth +
`NULLIF` divide guard, referencing the round-at-the-end rule). Added one worked
`sql` example — the cumulative running total, wrong-vs-right, using
`account_txns(account_id, txn_id, txn_date, net)` — bringing the skill to four
worked examples. Extended the step-5 `sql_dialect_notes` provision list to name
the rolling-window convention. No inline `RANGE … INTERVAL` frame anywhere in the
skill; it stays dialect-clean.
**Dialect notes (`packages/cli/src/context/sql-analysis/dialects/*.md`).** Added a
**Rolling window over time** line to all seven files, parallel to the spec-10
**Series** line and cross-referencing it for the spine fallback.
**Deviation — `RANGE`-frame support verified against authoritative docs (the
spec's hard requirement), which corrected two of its starting points:**
- **postgres** — native interval frame: `RANGE BETWEEN INTERVAL '29 days'
PRECEDING AND CURRENT ROW` (as the spec guessed).
- **mysql** — native interval frame over a temporal key: `RANGE BETWEEN INTERVAL
29 DAY PRECEDING AND CURRENT ROW` (as guessed).
- **bigquery** — `RANGE` is numeric-only: range over `UNIX_DATE(day)` with
`RANGE BETWEEN 29 PRECEDING AND CURRENT ROW`, or spine + `ROWS` (as guessed).
- **snowflake** — **corrected:** the spec said "limited; default to a spine," but
Snowflake *does* support a native interval `RANGE` frame over a date/timestamp
key and it is gap-tolerant, so the note gives the native frame
(`RANGE BETWEEN INTERVAL '29 days' PRECEDING AND CURRENT ROW`), no spine needed.
- **clickhouse** — **corrected:** the spec said "limited; default to a spine," but
ClickHouse supports a numeric `RANGE` offset over a `Date` column (counts in
days, gap-tolerant); the `INTERVAL` form is unsupported (use seconds for
`DateTime`). The note gives the numeric `RANGE` frame, with spine + `ROWS` as
the fallback.
- **sqlite** — no date-interval range frame (no native date type): spine + `ROWS`
(as guessed).
- **tsql** — `RANGE` supports only `UNBOUNDED`/`CURRENT ROW` (no offset frame):
spine + `ROWS`, or a date-keyed self-join (as guessed).
**Tests.** `test/skills/analytics-skill-content.test.ts` — added a representative
phrase per recipe (plus `minimum periods`), bumped the `sql`-fence count 3 → 4,
asserted the cumulative example shape (`ROWS BETWEEN UNBOUNDED PRECEDING AND
CURRENT ROW` and the `ORDER BY txn_date, txn_id` tie-breaker), and strengthened
the dialect-clean guard with a no-inline-`RANGE … INTERVAL` regex.
`test/context/mcp/dialect-notes.test.ts` — extended the per-dialect rubric loop
with `expect(notes).toMatch(/\*\*Rolling/)`, so every dialect (derived from
`DIALECTS_WITH_NOTES`) must answer the rolling-window rubric.
**Verification.** Full `@kaelio/ktx` vitest suite green (3001 passed, 1 skipped);
`pnpm run build` mirrors both surfaces into `dist`; `pnpm run link:dev` refreshed
`ktx-dev`. Pre-existing, unrelated note: `tsc -p tsconfig.test.json` reports one
error in `test/mcp-server-factory.test.ts` (a `KtxMcpContextPorts` cast) that is
present in committed branch code and untouched by this work.

405
spider2-specs/specs/12-parse-text-encoded-numbers.md
View file

@ -1,405 +0,0 @@
# Parse text-encoded numeric columns before doing math on them
> Refined spec. Intake draft: `todo/12-parse-text-encoded-numbers.md`.
## Problem
Numeric measures are often stored as **text** with human formatting: unit
suffixes (`"1.2K"`, `"3M"`, `"4B"`), currency symbols and thousands separators
(`"$1,200"`), percent signs (`"12%"`), or non-numeric sentinels for missing/zero
(`"-"`, `"N/A"`, `""`). Aggregating or comparing such a column directly is
**silently wrong**: a string comparison orders `"100" < "9"`, and a naive
`CAST(x AS REAL)` yields `0`/NULL/partial on the formatted values rather than the
intended number. The query runs, the shape looks right, the number is garbage.
The agent already samples schemas before composing — spec 07 gave the
`<sql_craft>` "Schema discovery before writing SQL" group its *"Sample before you
compose"* and *"Cast to the real type before comparing"* rules. But those rules
guard **encoding** (date format, nullability) and **type-mismatch in `WHERE`**;
they say nothing about a column whose declared/affinity type is text yet whose
*meaning* is numeric. When the agent sees a "numeric-looking" column it tends to
assume a real number type and skips the parse, so the arithmetic runs on the raw
strings. This spec adds the detect → parse/scale → verify habit to that same
group, building on the two rules already there rather than restating them.
## Generic use case (independent of any benchmark)
- A `trade_volume` column stored as `"1.2K" / "3M" / "-"` must become
`1200 / 3000000 / 0` before you can sum it or compute a daily change.
- A `price` stored as `"$1,299.00"` must become `1299.00` before averaging.
- A `conversion_rate` stored as `"12%"` must become `0.12` before weighting it.
This is routine data hygiene on real, messy production tables — every analyst
hits text-encoded measures on some warehouse, with no benchmark in sight. The
methodology is universal craft, so it belongs in the shipped skill; it transfers
to every ktx user querying a live database.
## Model
The change is **additive content across two surfaces** — the same split specs 10
and 11 made, and for the same reason. The split is the central design decision;
it satisfies spec 07's hard dialect-agnostic invariant for `<sql_craft>` without
weakening it.
### Why two surfaces (the dialect-agnostic reconciliation)
The **detect → parse → scale** half is **pure portable SQL** and stays entirely
in the dialect-agnostic skill:
- Stripping `$` / `,` / `%` is a portable chained `REPLACE` over a small, known
set of literal characters — no regex needed.
- Suffix scaling (K=10³, M=10⁶, B=10⁹) is a portable `LIKE`/`CASE` expression.
- Sentinel mapping (`-` / `N/A` / empty → `0` or `NULL`) is a portable `CASE`.
- The final cast to a numeric type is `CAST(... AS DECIMAL)`, broadly portable.
The **verify** half has one piece that is genuinely dialect-divergent: a
**failure-detecting numeric cast** — a cast that signals (rather than silently
swallows) a value that did not parse. This is exactly what requirement 3
("confirm coverage") needs, and it cannot be written portably:
- **bigquery:** `SAFE_CAST(x AS FLOAT64)` → `NULL` on failure.
- **snowflake:** `TRY_TO_NUMBER(x)` / `TRY_CAST` → `NULL` on failure.
- **tsql (SQL Server):** `TRY_CAST(x AS DECIMAL(...))` / `TRY_CONVERT` → `NULL`.
- **clickhouse:** `toFloat64OrNull(x)` / `toDecimalOrNull(...)` → `NULL`.
- **postgres / mysql:** no `TRY_CAST` — guard with a numeric pattern test before
casting (e.g. `CASE WHEN x ~ '^-?[0-9.]+$' THEN x::numeric END`).
- **sqlite (the gotcha):** a plain `CAST('abc' AS REAL)` returns **`0.0`** and
`CAST('12abc' AS REAL)` returns **`12.0`** — it neither errors nor NULLs, so an
`IS NULL` coverage check is **silently broken**. Detecting a failed parse needs
a `GLOB`/`typeof` pattern guard.
So a portable skill cannot inline a safe cast any more than spec 10 could inline a
date-series generator or spec 11 a calendar range frame. ktx already routes that
kind of engine-specific syntax through the per-dialect notes in
`packages/cli/src/context/sql-analysis/dialects/<dialect>.md`, served by the
`sql_dialect_notes` MCP tool (spec 08). Specs 10 and 11 set the exact precedent:
a construct not yet in the dialect rubric, genuinely engine-specific, was added
there (the **Series** line; the **Rolling window** line) and the dialect-agnostic
skill points to it. The failure-detecting cast is the next construct in that same
position, so the **safe-cast idiom belongs in the dialect notes**, and the skill
points to it.
Surface 1 (skill) carries the **pattern** (detect the text encoding; parse/scale
in an early CTE; verify with a failure-detecting cast). Surface 2 (dialect notes)
carries the **concrete safe-cast syntax** per engine, including the sqlite
`CAST`-returns-0 gotcha.
The regex character-*strip* is deliberately **not** promoted to the dialect
notes: a portable chained `REPLACE` over a known character set is the opinionated
default, so there is no need for a per-dialect strip line (derive from need; one
default). The dialect surface gains exactly one thing — the safe cast — because
that is the only piece the portable path genuinely cannot express.
### Additive, inline, heuristic-with-a-why
Consistent with specs 07, 10, and 11: the skill change is **additive content in
one Markdown file** (`skills/analytics/SKILL.md`), inline (no bundled
`reference/` file — `setup-agents.ts` ships only `SKILL.md`), dialect-agnostic,
and phrased as **heuristics with a one-line generic rationale**, not a wall of
MUSTs. The dialect-notes change is additive content in the seven existing
`dialects/*.md` files. No new tool, flag, or config on either surface.
### Build on the rules already present; do not restate them
- The Schema-discovery group already carries **"Sample before you compose"** and
**"Cast to the real type before comparing"** (spec 07). The detect rule
**extends** the first (distinct-value sampling to learn the encoding) and the
parse rule **complements** the second (text-meaning-numeric, not just
text-vs-numeric literal mismatch) — reference them, do not repeat them.
- The sentinel **0-vs-NULL** choice is the **same additive-vs-non-additive
judgment** spec 10 established in its *"Default by additivity"* rule (0 only
when "no value" genuinely reads as 0; NULL otherwise). **Reference** that rule
rather than restating the discriminator (state each rule once).
## Requirements
### 1. Skill surface — `<sql_craft>` "Schema discovery before writing SQL"
Add the text-encoded-numeric guidance to the **existing** group, after its two
current bullets. Phrase as heuristics, each with a generic *why*, dialect-agnostic.
It must cover:
1. **Detect text-encoded numerics during sampling.** When a column the question
treats as a number is stored as text, sample its **distinct** values to learn
the encodings actually present — unit suffixes (`K`/`M`/`B`), currency
symbols, thousands separators, percent signs, and non-numeric sentinels
(`-`, `N/A`, empty) — **before** composing. Never infer the format from the
column name. *Why:* compared/aggregated as-is, the text sorts lexically
(`'100' < '9'`) and a naive cast collapses formatted values to `0`/NULL —
producing a silently wrong result instead of an error.
2. **Parse and scale in an early CTE.** Strip currency/separator/percent
characters, multiply by the suffix scale (K=10³, M=10⁶, B=10⁹), map sentinels
to `0` **or** `NULL` per the question's intent, then cast to a numeric type —
all in **one early CTE**, so every downstream layer sees clean numbers. The
`0`-vs-`NULL` choice for sentinels follows spec 10's **additive-vs-non-additive**
rule (reference it; do not restate). *Why:* a string column aggregated as-is
sorts lexically and casts to 0, so the math is silently wrong.
3. **Confirm coverage (verify).** After parsing, sanity-check that **no
intended-numeric value silently failed to parse** — a failed parse should
surface as `NULL`, which is only visible with a **failure-detecting cast**.
Note the divergence: a plain `CAST` errors on some engines and, on sqlite,
returns `0`/partial rather than NULL — so use the engine's safe-cast idiom from
`sql_dialect_notes` (requirement 3), then count residual NULLs among
non-sentinel rows. *Why:* an encoding the sample missed would otherwise vanish
as `0`/NULL instead of being caught.
### 2. One worked example — parse/scale, fully portable
Add **exactly one** new compact before/after `sql` example demonstrating the
parse-and-scale pattern on a synthetic generic schema
(e.g. `metrics(label, value_text)` with values like `'1.2K'`, `'$1,200'`, `'-'`):
- **Wrong:** `SUM(CAST(value_text AS REAL))` (or summing the raw strings) — the
formatted values collapse to `0`/partial, so the total is silently wrong.
- **Right:** an early CTE that strips symbols with chained `REPLACE`, applies a
`CASE` for the K/M/B suffix scale, maps `'-'`/`'N/A'`/`''` to `0`, casts to
`DECIMAL`, then `SUM`s the parsed column.
**Standard, portable SQL only** — no `REGEXP_REPLACE`, `SAFE_CAST`, `TRY_CAST`,
`TRY_TO_NUMBER`, `toFloat64OrNull`, `GLOB`, or any dialect function — so the
example stays dialect-clean. Keep it ~12–16 lines. The **verify** step gets **no**
inline example (its correct form needs the engine-specific safe cast, delegated to
`sql_dialect_notes`, exactly as spec 10's period-spine and spec 11's
rolling-window variants were prose-only).
This adds **one** worked `sql` example to the skill. Spec 11 independently adds
one as well; **do not hardcode the resulting total** — increment from the current
state. As of this writing the skill carries **three** examples (spec 07
window-then-filter, spec 09 multi-hop fan-out, spec 10 panel spine), so this is
the **fourth**; if spec 11 ships first it is the **fifth**. The fence-count test
assertion is incremented by one from its current value (see Acceptance criteria).
### 3. Dialect-notes surface — `dialects/*.md` (safe cast)
Add a **"Safe cast"** idiom line to **each** of the seven authored dialect files,
parallel to spec 10's **Series** line and spec 11's **Rolling window** line. Each
line gives that engine's **failure-detecting numeric cast** — a cast that returns
`NULL` (or is detectably invalid) on a non-numeric input — which is what makes the
verify step correct on that engine. Each note is engine-exclusive (a SQLite
analyst gets the SQLite idiom and never another engine's construct, per the
existing dialect-notes leak guards). Orientation only — exact syntax is the
implementer's; verify against authoritative docs (context7 / the engine manual)
rather than asserting from memory:
- **postgres:** no `TRY_CAST` — guard with a numeric pattern before casting,
e.g. `CASE WHEN x ~ '^-?[0-9.]+$' THEN x::numeric END`. (`regexp_replace` is
available for the strip, but chained `REPLACE` is the portable default.)
- **mysql (8.0+):** no `TRY_CAST` — guard with `x REGEXP '^-?[0-9.]+$'` before
`CAST(... AS DECIMAL)`; `REGEXP_REPLACE` is available for the strip.
- **bigquery:** `SAFE_CAST(x AS FLOAT64)` (or `SAFE_CAST(... AS NUMERIC)`) →
`NULL` on failure.
- **snowflake:** `TRY_TO_NUMBER(x)` / `TRY_TO_DECIMAL(x, p, s)` / `TRY_CAST` →
`NULL` on failure.
- **clickhouse:** `toFloat64OrNull(x)` / `toDecimalOrNull(...)` → `NULL`.
- **tsql (SQL Server):** `TRY_CAST(x AS DECIMAL(18,4))` / `TRY_CONVERT` → `NULL`.
- **sqlite (the gotcha):** a plain `CAST` returns `0`/partial, **not** NULL or an
error, so a coverage check must use a pattern guard such as
`CASE WHEN cleaned GLOB '...' THEN CAST(cleaned AS REAL) END` (or a `typeof`
check) to detect a value that did not parse.
This line is what makes the verify step executable from the dialect-agnostic
skill. It is **distinct** from the Series and Rolling-window lines (those generate
or window over a calendar; this detects a failed numeric parse). Phrase any
version note as `8.0+`-style, **not** "as of version …" (the dialect-notes test
bans version-dated wording).
### 4. Explicit constraints / exclusions
None of the following may appear (consistent with specs 07, 10, and 11):
- **No inline dialect-specific cast/regex syntax in the skill** — no `SAFE_CAST`,
`TRY_CAST`, `TRY_TO_NUMBER`, `REGEXP_REPLACE`, `toFloat64OrNull`,
`replaceRegexpAll`, or `GLOB` anywhere in `SKILL.md`. The portable strip is
chained `REPLACE`; the failure-detecting cast lives only in the dialect notes.
- **No regex-strip dialect line.** The character strip stays the portable
chained-`REPLACE` default; the dialect notes gain only the **safe cast**.
- **No grader / gold-answer / benchmark reference**, and no output-shape contract
(the skill is for interactive analysis).
### 5. Coordination with specs 07, 08, 10, and 11
- **Spec 07** owns the Schema-discovery group and its two existing bullets
(*"Sample before you compose"*, *"Cast to the real type before comparing"*).
Spec 12 **extends** that group and **builds on** both bullets — references them,
never restates them; they must stay intact and uncontradicted.
- **Spec 08** owns the dialect-notes channel and its leak guards. Spec 12 adds one
rubric line through that channel; the engine-exclusivity guards apply unchanged.
- **Spec 10** owns the additive-vs-non-additive discriminator (Answer
completeness) and the dialect **Series** line. Spec 12 **references** the
additivity rule for the sentinel `0`-vs-`NULL` choice; do not duplicate it.
- **Spec 11** independently adds the dialect **Rolling window** line, one `sql`
example, and the **rolling-window** entry to the step-5 provision list. Spec 12
touches the **same** three places (the dialect-notes rubric loop, the example
count, and the step-5 list). Both are independent and additive — **add to the
current state, do not assume an order**: name **safe-cast** in the step-5 list
without removing rolling-window/series; increment the example count by one from
whatever it is; add `/\*\*Safe cast/` to the rubric loop alongside any
`/\*\*Rolling/` assertion.
### 6. Step pointer (no duplication)
The step-5 `sql_dialect_notes` provision list (currently "FQTN,
identifier-quoting, date, top-N, series/calendar, and JSON conventions"; spec 11
also names rolling-window) should additionally name the **safe-cast** convention
now that it exists. State each rule once inside `<sql_craft>`; the workflow steps
only point to it.
## Leak-safety (hard constraint)
Every worked example or note uses a **synthetic generic schema** (e.g.
`metrics(label, value_text)`) and made-up values (`'1.2K'`, `'$1,200'`, `'-'`),
showing only the *pattern*. **No** benchmark table names, SQL, or result values on
either surface. The dialect-notes additions, like the existing notes, carry no
benchmark / grader / version-dated content. The behavior is reconstructable from
first principles and tied to no specific instance.
## Acceptance criteria
- The `<sql_craft>` "Schema discovery before writing SQL" group states the three
heuristics — inline, dialect-agnostic, each with a generic *why*, and each
**building on** (not restating) the existing *"Sample before you compose"* and
*"Cast to the real type before comparing"* bullets and spec 10's additivity rule:
- **detect** text-encoded numerics by sampling distinct values (suffixes,
symbols, separators, sentinels) — never from the column name;
- **parse and scale** in an early CTE (strip → suffix-scale → sentinel map →
cast), sentinel `0`-vs-`NULL` per spec 10's additivity rule;
- **confirm coverage** with a failure-detecting cast, delegating the engine's
safe-cast syntax to `sql_dialect_notes`.
- Exactly **one** new worked `sql` example: parse-and-scale, wrong-vs-right, using
chained `REPLACE` + `CASE` suffix scale + sentinel `CASE` + `CAST(... AS
DECIMAL)`, in standard portable SQL. The `sql`-fence count assertion is
incremented by **one** from its current value (3 today → 4; or 5 if spec 11
shipped first).
- Each of the seven `dialects/*.md` files gains a **"Safe cast"** idiom line in its
engine's own failure-detecting numeric-cast idiom (including the sqlite
`CAST`-returns-0 gotcha); no engine leaks another engine's construct, and the
additions contain no benchmark / grader / version-dated content.
- The skill remains **dialect-clean:** no `QUALIFY`, `strftime`, `julianday`,
`generate_series`, `GENERATE_DATE_ARRAY`, backtick three-part FQTN, inline
`RANGE … INTERVAL` frame, **and no `SAFE_CAST` / `TRY_CAST` / `TRY_TO_NUMBER` /
`REGEXP_REPLACE` / `toFloat64OrNull` / `GLOB`**, anywhere in `SKILL.md`
including the new example.
- The step-5 `sql_dialect_notes` provision list names the **safe-cast** convention
alongside FQTN / identifier-quoting / date / top-N / series-calendar /
rolling-window / JSON.
- The existing interactive guidance (`<workflow>`, `<rules>`, the other examples),
the two existing Schema-discovery bullets, and the existing dialect-note rubric
lines (including **Series** and, if present, **Rolling window**) are intact and
uncontradicted.
- No grader / benchmark reference, and no output-shape contract.
- The skill stays scannable and comfortably under the 500-line budget; frontmatter
still parses as `ktx-analytics`.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the prose.
- **Skill:** `packages/cli/src/skills/analytics/SKILL.md` — add the three
heuristics to the "Schema discovery before writing SQL" group (after its two
existing bullets), the single parse-and-scale worked example, and extend the
step-5 dialect-notes provision list to name the safe-cast convention. Leave
`<workflow>` / `<rules>` / the other examples and the two existing
schema-discovery bullets intact. Delivery is unchanged (single `SKILL.md` per
target via `readAnalyticsSkillContent` in `setup-agents.ts`) — confirm, no
change required.
- **Dialect notes:** the seven files under
`packages/cli/src/context/sql-analysis/dialects/`. The list is kept in sync with
`DIALECTS_WITH_NOTES` (`dialect-notes.ts`) and shipped to `dist` by
`copy-runtime-assets.mjs` — no plumbing change, content only. **Verify each
engine's actual safe-cast / try-cast support against authoritative docs before
writing the idiom; do not assert from memory** (in particular the sqlite
`CAST`-returns-0 behavior, which is the motivating gotcha).
- **Tests:**
- `packages/cli/test/skills/analytics-skill-content.test.ts` — add a
representative phrase for each of the three heuristics (e.g. a *detect*, a
*parse/scale*, and a *confirm-coverage* phrase) to the `represents every craft
behavior` list; bump the `sql`-fence count assertion **by one** from its
current value; assert the example shape (e.g. `REPLACE(` and `CAST(` and a
suffix-scale multiplier); and **strengthen** the dialect-clean guard by adding
`SAFE_CAST`, `TRY_CAST`, `TRY_TO_NUMBER`, `REGEXP_REPLACE`, `toFloat64OrNull`,
and `GLOB` to the banned list (mirroring spec 10 adding `generate_series` /
`GENERATE_DATE_ARRAY` and spec 11 adding the no-inline-`RANGE … INTERVAL`
guard, so the "safe cast lives only in the dialect notes" criterion is
*enforced*, not incidentally true).
- `packages/cli/test/context/mcp/dialect-notes.test.ts` — extend the "answers
the full rubric for every dialect" loop with the safe-cast assertion,
`expect(notes).toMatch(/\*\*Safe cast/)`, so every dialect must answer it.
Coverage is derived from `DIALECTS_WITH_NOTES`, so the new assertion enforces
all seven without a hand-maintained list. Do **not** add a false-exclusivity
assertion for `TRY_CAST` (it is shared by snowflake and tsql); requiring the
line per dialect is sufficient.
- Rebuild and re-link the dev binary so the playground picks up both surfaces:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation only)
At least one SQLite-subset question stores trading volume as suffix-encoded text
(`"K"`/`"M"`, `"-"` for zero) and fails because the agent aggregates the raw
strings — runnable, plausible, wrong. The sqlite `CAST`-returns-0 behavior makes
the failure especially insidious: there is no error to alert the agent, and a
naive `IS NULL` coverage check would not catch it either, which is precisely why
the safe-cast idiom belongs in the dialect notes. The fix — parse messy encodings
before math, then verify coverage with a failure-detecting cast — is universal
data hygiene that helps any analyst on any warehouse, so it belongs in the
product's craft (skill) plus the per-dialect safe-cast syntax that makes the
verify step executable, not in a benchmark-specific prompt. Improving the
benchmark score is a side effect; the skill and the dialect notes contain no trace
of the benchmark.
## Implementation notes
Shipped on branch `write-feature-spec-wiki`, on top of specs 10 and 11 (both already
applied in the working tree). Built from the current state per the "do not assume an
order" guidance — there were **four** worked examples (specs 07 window-then-filter,
09 multi-hop fan-out, 10 panel spine, 11 cumulative running total), so this is the
**fifth**, and step 5 already named `series/calendar, rolling-window`.
**Skill — `packages/cli/src/skills/analytics/SKILL.md`:**
- Added the three heuristics to the **"Schema discovery before writing SQL"** group,
after the two existing bullets: *Parse text-encoded numerics before doing math on
them* (detect by sampling distinct values, extending *Sample before you compose*,
never inferring from the column name), *Strip, scale, and cast in one early CTE*
(the *meaning-is-numeric* complement to *Cast to the real type before comparing*,
with the sentinel `0`-vs-`NULL` choice deferred to spec 10's *Default by
additivity* rule), and *Confirm the parse covered every value* (failure-detecting
cast from `sql_dialect_notes`). Each carries a one-line generic *why*; the existing
bullets and the additivity rule are referenced, not restated.
- Added **one** portable worked example (`metrics(label, value_text)` with `'1.2K'`,
`'3M'`, `'$1,200'`, `'-'`): wrong = `SUM(CAST(value_text AS REAL))`; right = an
early `parsed` CTE that strips with chained `REPLACE`, scales the K/M/B suffix with
a `CASE`, maps sentinels to `0`, casts to `DECIMAL(18,4)`, then `SUM`s. Standard
portable SQL only — no dialect functions, no inline safe cast.
- Step 5 dialect-notes provision list now names **safe-cast** alongside the others.
**Dialect notes — `packages/cli/src/context/sql-analysis/dialects/*.md`:** added a
**Safe cast** line to all seven files (after the *Rolling window* line), each giving
that engine's failure-detecting numeric cast: postgres/mysql use a numeric pattern
guard before casting (no `TRY_CAST`; mysql's bare `CAST` returns `0` with a warning);
bigquery `SAFE_CAST`; snowflake `TRY_TO_NUMBER`/`TRY_TO_DECIMAL`/`TRY_CAST`; tsql
`TRY_CAST`/`TRY_CONVERT`; clickhouse `toFloat64OrNull`/`toDecimal64OrNull` (the
`...OrZero` variants return `0`); sqlite documents the `CAST`-returns-`0.0`/partial
gotcha and a `GLOB` pattern guard. ClickHouse function names were verified against
the official docs via context7 (the spec's loose `toDecimalOrNull` is not a real
name — the `to<Type>OrNull` family requires a bit width, hence `toDecimal64OrNull`).
No version-dated wording.
**Tests:** `analytics-skill-content.test.ts` — added the three representative
phrases, bumped the `sql`-fence count 4 → 5 (and the test title), asserted the
example shape (`WITH parsed AS`, `REPLACE(`, `AS DECIMAL(`, `LIKE '%K' THEN 1000`),
and strengthened the dialect-clean banned list with `SAFE_CAST`, `TRY_CAST`,
`TRY_TO_NUMBER`, `REGEXP_REPLACE`, `toFloat64OrNull`, and `GLOB` (mirroring spec 10's
`generate_series` / spec 11's inline-`RANGE … INTERVAL` guards). `dialect-notes.test.ts`
— added `expect(notes).toMatch(/\*\*Safe cast/)` to the per-dialect rubric loop, so
all seven (derived from `DIALECTS_WITH_NOTES`) must answer it; no false-exclusivity
assertion for the shared `TRY_CAST`.
**Verification:** both affected test files pass (19 tests); broader `test/skills` +
`test/context/mcp` pass (65 tests); production type-check (`tsc -p tsconfig.json`)
is clean; `pnpm run build` copies both surfaces into `dist` (7 dialect files carry
*Safe cast*, the built `SKILL.md` carries the parse example) and `pnpm run link:dev`
relinks `ktx-dev`. One **pre-existing, unrelated** type error remains in the
test-only config (`test/mcp-server-factory.test.ts:152`, byte-identical to HEAD,
untouched here) — out of scope for this spec.

336
spider2-specs/specs/14-output-completeness-final-check.md
View file

@ -1,336 +0,0 @@
# Output completeness — answer every requested part, enforced by a final pre-emit check
> Refined spec. Intake draft: `todo/14-output-completeness-final-check.md`.
## Problem
The single largest correctness failure mode for the analytics skill is
**incomplete output**: the query runs and the methodology is roughly right, but
the projection is missing columns the question asked for. The SQL is runnable and
the aggregate is correct — the answer is simply *short by columns*. Three
recurring shapes:
1. **Multi-part questions answered partially.** A question that asks for several
things ("report the highest *and* the lowest month, each with its count and
average, *and* the difference") comes back with only the first clause — one
column where several were requested.
2. **Identity dropped.** Grouping by a human-readable name but not projecting the
entity's identifier (a product name without its product id, a customer name
without its customer id).
3. **Inputs to a derived value dropped.** Returning a ratio / percentage /
difference but not the underlying counts the question also asked for.
Shapes 2 and 3 are **already covered** by shipped `<sql_craft>` rules — spec 07's
*"Expose identity, not just the label"* and *"Keep the inputs to a derived
value"* — yet they are frequently **not applied**. So the gap is not missing
knowledge: these rules sit as passive heuristics in a list, and nothing makes the
agent reliably check them before finalizing. The fix is twofold: (a) add the
missing **multi-part-completeness** rule that generalizes shapes 1–3, and (b)
turn output-completeness into an **explicit final verification step** the agent
performs before emitting SQL, so the existing identity/inputs rules are actually
enforced rather than merely listed.
The failure is **model-independent**: a markedly stronger model produced the same
incomplete-output mistakes on these questions, which means it is a
craft/enforcement gap, not a capability gap — exactly the kind of universal
analyst craft that belongs in the shipped skill.
## Generic use case (independent of any benchmark)
An analyst is asked: *"For each region, report the highest and the lowest monthly
order count, and the difference between them."* A complete answer has a column for
the region's id and name, the highest count, the lowest count, and the difference
— five columns. Returning just the region and a single number answers only part
of the request. This is a universal expectation on any database: answer **every**
part of a multi-part request, identify the entities, and show the inputs behind
any derived figure — and answer *exactly* that, without padding the result with
columns the question never asked for.
## Model
The change is **additive content in one Markdown file**
(`skills/analytics/SKILL.md`), governed by the same invariants spec 07
established. They constrain the implementer; the exact prose is theirs.
### Additive, inline, heuristic-with-a-why
Consistent with specs 07 and 10: the change is additive content in
`skills/analytics/SKILL.md`, **inline** (no bundled `reference/` file — the
`setup-agents.ts` delivery ships only `SKILL.md` per target), dialect-agnostic,
and phrased as **heuristics with a one-line generic rationale**, not a wall of
MUSTs. The new rule extends the existing `<sql_craft>` "Answer completeness /
interpretation" group; the shipped bullets in that group (including the *identity*
and *inputs* rules this spec builds on) are preserved unchanged. No new tool,
flag, or config.
### The over-projection guard carries a *universal* why, not a grader reference
The intake draft frames "don't pad the result with extra columns" as
*grader-gaming*. The skill forbids **any** reference to a grader, gold answer, or
benchmark (spec 07's hard invariant; the content test bans the words). So the
guard must ship with a **universal analytics rationale** instead: columns the
question did not ask for add noise, mislead the reader into thinking they matter,
and make the result harder to consume — match the request exactly, neither short
nor padded. This is the same reconciliation spec 07 applied to the draft's
"behavior only, no rationale" instruction: generic *why* is required; only
grader/gold/benchmark rationale is banned.
### Completeness is a closed set — identity and inputs are *inside* it
"Expose identity" and "keep the inputs" tell the agent to add columns; the
over-projection guard tells it not to. These only contradict if the target is
left fuzzy, so this spec pins it down. A **complete projection** is exactly:
> {every requested metric/attribute} ∪ {the identifier of each grouped/named
> entity} ∪ {the inputs to each derived value}, at the grain the question
> specifies.
Identity and inputs are **members of that set** — part of completeness, never
"padding." **Under-projection** is any member missing (the failure this spec
attacks); **over-projection** is any column *outside* the set (what the guard
forbids). The implementer must phrase the rule and guard against this single
definition so they read as one coherent notion, not two competing instructions.
### Dialect-agnostic, additive-only, exclusions intact
Every addition reads correctly on any dialect — no dialect-specific syntax in the
rule text or the worked example. The existing `<workflow>`, `<rules>`, and the
other `<sql_craft>` bullets and examples (specs 07/09/10/11/12) are preserved and
uncontradicted. Spec 07's exclusions still hold: no output-shape contract, no
`MAX(date)` anchoring of relative time, no grader-driven advice, no dialect
syntax.
## Requirements
### 1. Multi-part / multi-output completeness — a new umbrella rule
Add a bullet to the `<sql_craft>` "Answer completeness / interpretation" group:
when a question requests several outputs — a **list** ("A, B, and C"), **paired
extremes** ("the highest *and* the lowest"), or a **value plus its components**
("X, Y, and their ratio") — the final projection must contain a column for
**each** requested output. *Why:* answering only the first clause is the most
common way a runnable query is still wrong; the grain and methodology can be
perfect yet the answer is short by columns.
This rule is the **umbrella** over the two shipped completeness rules: the
*inputs* rule (*"Keep the inputs to a derived value"*) is its "value + components"
instance, and the *identity* rule (*"Expose identity, not just the label"*) is its
"entity identity" instance. The new bullet should **name that relationship**
(so the three read as one notion) rather than restating either rule.
Keep this distinct from the row-selection rules in the same group: *"Top /
highest / most / lowest"* and *"For each X / per X / by X"* govern **which rows**
appear; multi-part completeness governs **which columns** appear. They compose
(e.g. "highest and lowest per region" needs one row per region *and* a column per
clause).
### 2. Final completeness check — the enforcement mechanism
The rule content lives **once** in `<sql_craft>`; the trigger is promoted to a
first-class line in `<workflow>` step 6.
- **Capstone bullet in `<sql_craft>`** (closing the "Answer completeness /
interpretation" group): *before emitting the final SQL, re-read the question and
confirm the projection covers* —
1. every named **metric / attribute** the question asks for (→ the multi-part
rule);
2. the **identifier** of every grouped or named entity (→ the *identity* rule);
3. every **input** to each derived value (→ the *inputs* rule);
4. all at the **grain** the question specifies (→ the *for each X* / panel
rules).
Each facet cross-references the rule it enforces, so the check is what makes
those passive rules active. Phrase it as a short, concrete "confirm the
projection covers…" checklist, not a wall of MUSTs.
- **Over-projection guard** (attached to the check): do **not** add columns the
question did not ask for "to be safe" — extra columns add noise, mislead, and
make the result harder to consume; match the request exactly. Carries the
**universal** why from the Model, **never** a grader/gold/benchmark reference.
- **`<workflow>` step 6 line** (the explicit ritual): step 6 ("Validate and
explain") gains a mandatory line directing the agent to **always** run the final
completeness check before emitting — re-read the question and verify every
requested output, each entity's identity, each derived value's inputs, and the
grain are all projected — pointing into the `<sql_craft>` capstone for the
detail. This **replaces the current conditional pointer's role** ("If a result
is unexpectedly empty or its grain looks wrong, work through the … rules"): the
empty/grain diagnostic stays available (it maps to the existing *"Diagnose empty
results"* and grain rules), but the completeness check fires **unconditionally**,
on every SQL-authoring turn, not only when a result looks off. The workflow line
names the ritual and the four facets; the rationale, guard, and example are
stated once in `<sql_craft>`, not duplicated into the workflow.
### 3. One worked example (dialect-agnostic)
Add **exactly one** compact before/after example to the "Answer completeness /
interpretation" group, demonstrating multi-part completeness on a **synthetic**
schema (`regions`, `region_monthly`):
- **WRONG:** answers only the first clause — `SELECT region_name,
MAX(monthly_orders) AS highest … GROUP BY region_name` — with no region id, no
lowest, no difference.
- **RIGHT:** one column per requested output plus the entity's identity, at the
region grain — `region_id, region_name`, the highest, the lowest, and the
difference, with `regions` joined to `region_monthly` and grouped by the region
id and name.
Standard dialect-clean SQL only (no `QUALIFY`, no dialect functions; `MAX`/`MIN`
are portable aggregates). Keep it tight. It teaches multi-clause coverage +
identity + derived-value inputs in one capstone, and is **distinct** from the
spec-10 `regions` panel example: that one is about missing **rows** (LEFT-JOIN
spine + `COALESCE`); this one is about missing **columns**. This is the **sixth**
worked `sql` example in the skill (after specs 07/09/10/11/12).
### 4. Coordination with specs 03 and 07/09/10/11/12
- **Spec 03** (multi-connection routing) owns `<workflow>` step 0 and the
`connectionId` threading/scoping. Spec 14 touches `<workflow>` only to add the
completeness-check line to **step 6** — it must not rewrite the routing or the
`<rules>` `connectionId` scoping. If both land, step 6 reads coherently: validate
+ the completeness ritual.
- **Specs 07/09/10/11/12** own their own bullets and worked examples in
`<sql_craft>`. Spec 14 is **additive** to the same "Answer completeness /
interpretation" group and adds one example; it must not remove or contradict
theirs.
## Leak-safety (hard constraint)
The example uses an **invented, generic schema** (`regions`, `region_monthly`) and
made-up columns — **no benchmark table names, SQL, or result values.** It teaches
the *pattern* (cover every requested output + identity + inputs, at grain, without
padding), which is universal and tied to no specific instance. The over-projection
guard's rationale is **universal** (noise/clarity/consumability), never
"grader-gaming" or any other scoring reference. No part of the addition mentions a
benchmark, gold answer, grader, or scoring comparator.
## Acceptance criteria
- `<sql_craft>` "Answer completeness / interpretation" states the **multi-part /
multi-output completeness** rule (a column per requested output; list / paired
extremes / value-plus-components), named as the umbrella over the shipped
*identity* and *inputs* rules — inline, dialect-agnostic, with a generic *why*.
- `<sql_craft>` states a concrete **final completeness check** (re-read the
question → confirm metrics + entity identity + derived-value inputs + grain are
projected), cross-referencing the existing identity/inputs/grain rules so they
are enforced, not merely listed.
- The check carries the **over-projection guard** with a **universal** rationale
(don't pad with unrequested columns — noise / misleading / harder to consume),
and the skill contains **zero** grader/gold/benchmark references anywhere.
- `<workflow>` **step 6** carries a mandatory line that runs the completeness
check **unconditionally** before emitting and points into the `<sql_craft>`
capstone; the rule content is **stated once** in `<sql_craft>` (no duplicated
rationale/guard in the workflow). The empty/grain diagnostic remains available.
- Exactly **one** new worked `sql` example is present (synthetic
`regions`/`region_monthly`, wrong vs complete), in standard dialect-agnostic SQL;
the skill then carries **six** `sql` worked examples total.
- The existing interactive guidance (`<workflow>` steps, `<rules>`, the other
`<sql_craft>` bullets and the five prior examples) is intact and uncontradicted;
the additive-only and dialect-clean invariants from specs 07/10 still hold.
- None of spec 07's excluded items appear (output-shape contract, `MAX(date)`
anchoring of "recent"/"past N", grader-driven advice, dialect syntax).
- The skill stays scannable and comfortably under the 500-line budget; the
frontmatter still parses as `ktx-analytics`.
- The analytics-skill **content test is updated** to cover the new rule and check
(see Implementation orientation).
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the prose.
- **Skill:** `packages/cli/src/skills/analytics/SKILL.md`.
- Add the multi-part-completeness bullet and the final-completeness-check
capstone (with the over-projection guard) to the `<sql_craft>` "Answer
completeness / interpretation" group; add the single
`regions`/`region_monthly` worked example.
- In `<workflow>` step 6, replace the current conditional answer-completeness
pointer with the mandatory completeness-check line (unconditional, names the
four facets, points into `<sql_craft>`); keep the empty/grain diagnostic.
- Leave `<workflow>` steps 0–5, `<rules>`, and the other `<sql_craft>`
bullets/examples intact. Delivery is unchanged (single `SKILL.md` per target
via `readAnalyticsSkillContent` in `setup-agents.ts`) — confirm, no change
required.
- **Tests:** `packages/cli/test/skills/analytics-skill-content.test.ts`.
- Add representative phrases to the "represents every craft behavior" list for
the multi-part rule, the final completeness check, and the over-projection
guard.
- Bump the worked-example `sql`-fence count assertion **5 → 6** (and update the
test name/comment), and assert the new example's shape (e.g. `region_monthly`,
`MAX(`, `MIN(`, the difference expression, `region_id`).
- The existing dialect-clean, grader/benchmark-clean, and relative-time
(`MAX(...)` anchoring) guards must still pass — the new example's `MAX`/`MIN`
lines carry no "recent"/"past N" wording, so the phrase-level guard is
unaffected. The `SkillsRegistryService` frontmatter test must still pass.
- Rebuild and re-link the dev binary so the playground picks up the updated skill:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation only)
On the latest SQLite-subset run, **incomplete output was the single largest
failure bucket (~13 of 51 voted failures)**: multi-part questions answered
partially, plus dropped identity / derived-value inputs — the latter two being
spec-07 rules that already exist but weren't applied. A probe with a much stronger
model reproduced the *same* incomplete-output failures, confirming this is a
craft-enforcement gap rather than a model-capability one. The fix — answer every
requested part, identify the entities, keep the inputs, and don't pad — is
universal analyst craft, so it belongs in the product skill (and transfers to real
users), enforced as a final pre-emit check rather than left as a passive hint.
Improving the benchmark score is a side effect; the skill contains no trace of the
benchmark.
## Implementation notes
Implemented as additive content in one Markdown file plus a test update.
- **Skill — `packages/cli/src/skills/analytics/SKILL.md`** (`<sql_craft>` "Answer
completeness / interpretation" group):
- Added the **"Answer every requested output"** umbrella bullet (list / paired
extremes / value-plus-components → a column per requested output, with a generic
*why*). It names *keep the inputs* and *expose identity* as its "value +
components" and "entity identity" instances, pins the closed-set definition of a
complete projection, and marks itself as governing *which columns* appear —
distinct from the *Top …* / *For each X* row-selection rules, with which it
composes. The two shipped instance rules are preserved verbatim.
- Added the **"Final completeness check"** capstone bullet: a four-facet
"before emitting, re-read the question and confirm the projection covers…"
checklist (metric/attribute → multi-part rule; identifier → *expose identity*;
inputs → *keep the inputs*; grain → *for each X* / *complete the panel*), run on
every query. It carries the **over-projection guard** with a universal rationale
(unrequested columns add noise, mislead, and are harder to consume — match the
request exactly), with **no** grader/gold/benchmark reference.
- Added one worked `sql` example (synthetic `regions` / `region_monthly`): WRONG
answers only the first clause (`SELECT region_name, MAX(monthly_orders) …`),
dropping the region id, the lowest, and the difference; RIGHT projects
`r.region_id, r.region_name`, `MAX` highest, `MIN` lowest, and the
`MAX − MIN` difference, joining `regions` to `region_monthly` and grouping by id
+ name. This is the **sixth** `sql` example, dialect-clean (portable `MAX`/`MIN`).
- `<workflow>` **step 6**: replaced the conditional answer-completeness pointer
with an unconditional *"Always run the final completeness check before emitting"*
line that names the four facets and points into the `<sql_craft>` capstone; the
empty/grain diagnostic is retained for diagnosis. Steps 0–5, `<rules>`, and the
other `<sql_craft>` bullets/examples are untouched.
- Delivery is unchanged: `readAnalyticsSkillContent` in
`packages/cli/src/setup-agents.ts` still ships the single `SKILL.md` per target
(confirmed, no change required).
- **Tests — `packages/cli/test/skills/analytics-skill-content.test.ts`:** added the
three representative phrases (`Answer every requested output`, `Final completeness
check`, `Don't over-project`); bumped the `sql`-fence count assertion 5 → 6 and
renamed that test; asserted the new example's shape (`region_monthly`,
`MAX(rm.monthly_orders)`, `MIN(rm.monthly_orders)`, the `MAX − MIN` difference, and
`r.region_id, r.region_name`). The dialect-clean, grader/benchmark-clean,
relative-time, and frontmatter guards still pass.
- **Verification:** `analytics-skill-content` 9/9 and `setup-agents` 46/46 pass;
production type-check (`tsconfig.json`, src) is clean; `pnpm run build` copied the
updated skill into `dist/skills/analytics/SKILL.md` (6 fences, all new content
present) and `pnpm -w run link:dev` re-linked `ktx-dev` so the playground picks it
up. The skill is 244 lines (< 500 budget) and the frontmatter still parses as
`ktx-analytics`.
- **Deviation (cosmetic):** the worked example uses alias `rm` and a difference
column named `order_count_range`; the intake draft sketched alias `m` and
`AS difference`. The spec leaves prose to the implementer, so the change is purely
naming.
- **Unrelated pre-existing issue:** `tsconfig.test.json` reports one type error in
`packages/cli/test/mcp-server-factory.test.ts` (a `KtxMcpContextPorts`/`contextTools`
mismatch introduced by the earlier connection-scoped-wiki commit `2677b3ef`). It is
untouched by this work and out of scope here.

405
spider2-specs/specs/15-mcp-server-structured-logging.md
View file

@ -1,405 +0,0 @@
# Structured, leveled logging for the ktx MCP server
> Refined spec. Intake draft: `todo/15-mcp-server-structured-logging.md`.
>
> **Scope: observability only.** This spec is about *seeing* what the MCP server
> does (which tool, what params, when, how long, outcome). *Preventing* a runaway
> query from blocking the server (off-event-loop / interruptible execution) is a
> separate concern — see "Non-goals".
## Problem
The ktx MCP server (`mcp-http-server.ts` + `mcp-stdio-server.ts`, both built
through `mcp-server-factory.ts` on raw `node:http` + the
`@modelcontextprotocol/sdk` transports) emits almost no operational logs. There
is no server-side record of **which MCP tool was called, with what parameters,
when, how long it took, or whether it succeeded** — nor of session open/close or
transport errors. When a tool call is slow, hangs, or a client connection drops
("Transport channel closed"), an operator has no trail to diagnose it and must
resort to process sampling / `lsof` / guesswork — and the offending input
(e.g. the exact SQL) is typically unrecoverable.
The hook to fix this already exists but is half-built: `instrumentMcpServer`
(`context/mcp/context-tools.ts`) wraps every tool handler and already times it,
but it emits **only on completion** (a sampled `mcp_request_completed` telemetry
event) and **never writes a start line and never writes to the server log**. A
call that never returns therefore leaves no trace at all.
## Generic use case (independent of any benchmark)
Anyone running a long-lived ktx MCP server — a developer's local instance
(stdio, launched by Claude Desktop / Cursor), a foreground HTTP server, or a
shared/hosted HTTP daemon — needs observability into tool-call activity to:
- diagnose slow or hung tool calls (which `sql_execution` ran, against which
connection, with what SQL, for how long);
- explain client-visible connection failures from the server side (session
lifecycle, transport-closed events);
- audit what agents asked the server to do;
- spot patterns (hot tools, slow connections, error rates).
This is standard production-server hygiene; the server currently provides none.
## Design decisions (resolved during refinement)
These resolve ambiguities the intake draft left open. They constrain the
implementer; the exact code is theirs.
### One `pino` logger, synchronous, written to **stderr**
Use `pino` — the de-facto standard structured-JSON logger for Node servers — as
a single shared instance. Two corrections to the draft's sketch:
- **stderr, not stdout.** The stdio transport reserves **stdout** for the
JSON-RPC protocol (`mcp-stdio-server.ts` deliberately no-ops `stdout.write`);
writing logs there would corrupt the protocol stream. The HTTP daemon already
redirects **both** child fds to `.ktx/logs/mcp.log`
(`managed-mcp-daemon.ts`: `stdio: ['ignore', log.fd, log.fd]`), so stderr lands
in the same log file (surfaced by `ktx mcp logs`). **stderr is therefore the
one universally-correct sink** for both transports.
- **Synchronous, no worker-thread transport.** `pino` writes through a
`DestinationStream` (`{ write(msg) }`) — the server's existing
`KtxCliIo.stderr` sink satisfies that interface directly. Configure pino with a
**synchronous** destination (`pino.destination({ sync: true })`, or the
pino-pretty stream below with `sync: true`). This is load-bearing: the
`tool.start` line **must** be flushed to the fd *before* the (possibly
blocking) handler runs, so a runaway synchronous `better-sqlite3` query that
pegs the event loop still leaves the start line on disk. A worker-thread
transport (`transport: { target: ... }`) buffers and can lose that exact line
on a hard crash — **do not use transport mode.**
### Format is derived from `stderr.isTTY`, not a config flag
One logger, two serializations chosen by the environment (the "behavior follows
from inputs" rule — not a user-visible knob):
- **TTY** (`ktx mcp start --foreground` or `ktx mcp stdio` run in a terminal) →
**`pino-pretty` as a synchronous in-process stream** (`pretty({ sync: true,
destination: <stderr sink> })`, colorized). A readable live dev view.
- **Not a TTY** (the detached daemon, whose stderr is the `.ktx/logs/mcp.log`
file fd) → **plain JSON line** via the synchronous pino destination. The log
*file* stays structured JSON so the incident workflow ("recover the hung query
with a one-line `grep` / `jq`") works — colorized ANSI in a file would defeat
it.
`KtxCliIo.stderr` has no `isTTY` field (`cli-runtime.ts`), so detect the terminal
from the underlying stream (`process.stderr.isTTY`) at logger construction, while
still writing *through* the `io.stderr` sink so tests can capture emitted lines.
### Single hook: extend `instrumentMcpServer`, do not fork a second wrapper
Tool-call logging is added to the existing `instrumentMcpServer`
(`context-tools.ts`), which already wraps `registerTool` and measures duration.
It receives the **raw** tool input (it wraps the schema-parsing handler from
`registerParsedTool`), so the params it logs include `sql` for `sql_execution`.
The existing telemetry emission stays unchanged; logging is **additive** beside
it. Because both transports build their server through `mcp-server-factory.ts` →
`registerKtxContextTools`, this single change gives **both HTTP and stdio**
tool-call logging for free.
### `sessionId` / `callId` provenance
- **`sessionId`** comes from the SDK's per-call handler context
(`RequestHandlerExtra.sessionId`; confirmed present in `@modelcontextprotocol/sdk`
`1.29.0`). It is populated for the HTTP StreamableHTTP transport and absent for
stdio (single session) — log it when present, omit otherwise. Add
`sessionId?: string` to `KtxMcpToolHandlerContext` (`context/mcp/types.ts`).
- **`callId`** is generated per invocation with `randomUUID()` (already imported
in `context-tools.ts`). It correlates a `tool.start` with its `tool.end`.
### No redaction in v1 (explicit)
v1 ships **no log redaction**. Rationale recorded here so it is a deliberate
choice, not an oversight: these logs are **local** (stderr → `.ktx/logs/mcp.log`),
**never transmitted off-box**, and sit at the **same trust boundary** as the
`ktx.yaml` / environment that already hold the connection credentials. Concretely:
- Request **headers are never logged** at all, so the bearer token
(`KTX_MCP_TOKEN`) simply isn't collected — this is "not logged," not "redacted."
- Errors are logged with their **full message and stack** via pino's standard
`err` serializer.
- SQL text and tool params are logged **verbatim** (they are not secrets).
Credential redaction (e.g. a DB URL embedded in a driver error string) is an
explicit **v1 non-goal**; revisit only if these logs are ever shipped off-box.
This drops the draft's "light redaction" requirement and the
`collectTelemetryRedactionSecrets` / scrubber reuse it implied.
## Requirements
### 1. One shared pino logger
- A single `pino` instance per server process, constructed once and threaded to
both the transport layer (for lifecycle events) and the tool layer (for
tool-call events). Level set from env (Requirement 7), default `info`.
- Synchronous destination bound to the server's stderr sink (see Design
decisions). Pretty (`pino-pretty`, sync stream) when `process.stderr.isTTY`,
otherwise plain JSON. Each line carries pino's standard `time` and `level`.
- No new dependency beyond `pino` and `pino-pretty`. No OpenTelemetry / metrics
stack, no async/worker transport, no in-app file rotation.
### 2. Per-session / per-call context via child loggers
Use pino child loggers so every line carries the relevant correlation fields:
a per-call child binds `{ tool, callId }` plus `sessionId` when present, so one
session's or one call's activity can be grepped from the log.
### 3. Tool-call logging — START before execute, END after
In `instrumentMcpServer`, for **every** MCP tool invocation:
- **On entry, before invoking the handler**, write `tool.start` with
`{ tool, callId, sessionId?, params }` at **`info`**. `params` is the raw tool
input; for `sql_execution` this includes the full **SQL text** (the single most
useful field). The write is synchronous so the line exists even if the handler
never returns.
- **On normal completion**, write `tool.end` with
`{ tool, callId, sessionId?, durationMs, outcome: "ok", resultSize }` at
**`info`** — *unless* it is a slow call (Requirement 4). `resultSize` is a
tool-agnostic size measure (byte length of the serialized result text content).
- **On error**, write `tool.end` with
`{ tool, callId, sessionId?, durationMs, outcome: "error", err }` at **`error`**,
where `err` is the serialized error (message + stack) per Requirement 6.
`tool.start` and `tool.end` share the **same correlation fields and the same
`info` level** (for the non-slow, non-error case) so that an **unmatched
`tool.start`** — a start with no `tool.end` for the same `callId` — is an
unambiguous "this call hung" signal. This is the property that makes a runaway
`sql_execution` identifiable from the log alone, with its exact SQL and
timestamp, no process sampling.
> **Deliberate change from the intake draft.** The draft put `tool.start` /
> `tool.end` at `debug` (suppressed at the default `info`). That defeats the
> motivating incident: a hang is unpredictable, so debug would have to be enabled
> *before* it occurs, which never happens. v1 logs start/end at **`info`** — an
> always-on access log — so the offending query is recoverable at the default
> level. `debug` is reserved for heavier detail (Requirement 7).
### 4. Slow-call warning
When a call **completes** with `durationMs` greater than the configured slow
threshold (Requirement 7), emit its `tool.end` at **`warn`** (carrying the same
fields plus the duration) instead of `info`. This makes a completed-but-slow call
stand out and keeps it visible even when the level is raised to `warn`.
### 5. Connection / session lifecycle and transport errors
- **HTTP** (`mcp-http-server.ts`, in `newTransport`): log `session.open` from
`onsessioninitialized` and `session.close` from `onsessionclosed` /
`transport.onclose`, each with `sessionId`, at `info`. **Wire the currently
unused `transport.onerror`** to log `transport.error` (the SDK's
closed-channel / "Transport channel closed" events) at `error`, so a
client-visible connection failure has a server-side counterpart.
- **stdio** (`mcp-stdio-server.ts`): route the existing raw
`transport.onerror` stderr string (it currently writes a plain string) through
the logger as a `transport.error` line at `error`. A single `session.open` /
`session.close` pair for the one stdio connection MAY be logged at `info`.
### 6. Structured error logging
Errors are logged as structured objects via pino's standard `err` serializer
(`pino.stdSerializers.err` or equivalent), carrying error class, message, and
stack — never a bare interpolated string. The existing telemetry exception
reporting in `instrumentMcpServer` / `registerParsedTool` is unchanged.
### 7. Configuration surface
- **`KTX_MCP_LOG_LEVEL`** — pino level (`error` | `warn` | `info` | `debug` |
…), default **`info`**. MCP-scoped name because the MCP server is the only
emitter today; naming it global (`KTX_LOG_LEVEL`) would imply a logging system
that does not exist.
- **`KTX_MCP_SLOW_TOOL_MS`** — slow-call threshold in milliseconds (Requirement
4), default **`10000`**. Justified as a real ops knob: "slow" differs sharply
between a local SQLite file and a remote warehouse.
- Level ladder that results from Requirements 3–5:
- `debug`: everything below **plus** heavier detail (e.g. result bodies,
progress notifications) — implementer's discretion on what extra to attach.
- `info` (default): `tool.start` / `tool.end`, session lifecycle, slow `warn`s,
errors.
- `warn`: slow-call `tool.end`s, `transport.error`, errored `tool.end`s — but
not routine tool traffic.
- `error`: errored `tool.end`s and `transport.error` only.
## Acceptance criteria
- At default level (`info`), invoking any MCP tool produces a `tool.start`
(`tool`, `callId`, `sessionId` when HTTP, `params`) and a matching `tool.end`
(`durationMs`, `outcome`, `resultSize`) line, as **JSON to stderr** when stderr
is not a TTY.
- A tool call that never returns (e.g. a runaway `sql_execution`) leaves a
`tool.start` line carrying its **exact SQL and timestamp** and **no** matching
`tool.end` for that `callId` — so the offending query is recoverable from the
log alone, with no process sampling.
- A completed call slower than `KTX_MCP_SLOW_TOOL_MS` emits its `tool.end` at
`warn` with its `durationMs`.
- Session open/close and transport-closed (`transport.error`) events are logged
with the `sessionId` (HTTP); the stdio transport error path goes through the
logger, not a raw `stderr.write`.
- At level `warn`, routine `tool.start` / `tool.end` are suppressed but
slow-call warnings, transport errors, and errored calls are present.
- When stderr is a TTY (`ktx mcp start --foreground` / `ktx mcp stdio` in a
terminal), output is human-readable colorized `pino-pretty`; the daemon log
file (`.ktx/logs/mcp.log`) is plain JSON. Both paths are synchronous.
- The bearer token never appears in any log line (headers are not logged); SQL
and tool params do appear.
- No worker-thread / async log transport is introduced; no OpenTelemetry /
metrics stack; the only new dependencies are `pino` and `pino-pretty`.
- The existing `mcp_request_completed` telemetry and exception reporting still
work unchanged.
## Non-goals
- **Preventing / interrupting runaway queries** (off-event-loop execution, query
timeouts, worker-thread isolation). A single synchronous query that fans out
into a massive nested-loop join can peg the single-threaded server for hours
and break new connections — observability surfaces *which* query, but the fix
is execution-model work in a separate spec. (This logging is also the
prerequisite for a future watchdog that detects a `tool.start` with no
`tool.end` past a threshold and recycles the server.)
- **Log redaction** (see Design decisions) — explicit v1 non-goal.
- **Pretty output as a worker-thread transport** — the TTY path uses pino-pretty
as a synchronous in-process stream only.
- Metrics / tracing / OpenTelemetry exporters.
- Forwarding logs to the MCP *client* via the protocol logging capability
(`notifications/message`, `logging/setLevel`) — a possible later enhancement,
distinct from operational stderr logging.
- A global `KTX_LOG_LEVEL` spanning non-MCP commands — out of scope until other
surfaces emit structured logs.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the design.
- **New module** — a small logger factory, e.g.
`packages/cli/src/context/mcp/logger.ts`: builds the shared pino instance from
the stderr sink + `KTX_MCP_LOG_LEVEL`, choosing the pino-pretty (sync) stream
when `process.stderr.isTTY` else `pino.destination({ sync: true })`, and
exposes a `slow-threshold` read from `KTX_MCP_SLOW_TOOL_MS`.
- **Tool-call logging** — `packages/cli/src/context/mcp/context-tools.ts`:
extend `instrumentMcpServer` (~line 585) to write `tool.start` before
`handler(...)` and `tool.end` after (ok / slow-`warn` / `error`); generate
`callId` via the already-imported `randomUUID`; read `sessionId` from the
handler `context`. Thread the logger via `RegisterKtxContextToolsDeps`
(~line 26) and `registerKtxContextTools` (~line 650). Leave `registerParsedTool`
and the existing telemetry emission intact.
- **Context type** — `packages/cli/src/context/mcp/types.ts`: add
`sessionId?: string` to `KtxMcpToolHandlerContext`; add the logger to
`KtxMcpServerDeps` / the register deps.
- **Server wiring** — `packages/cli/src/context/mcp/server.ts`
(`createDefaultKtxMcpServer` / `createKtxMcpServer`) and
`packages/cli/src/mcp-server-factory.ts` (`createKtxMcpServerFactory`): accept
and pass the logger down to `registerKtxContextTools`.
- **HTTP lifecycle** — `packages/cli/src/mcp-http-server.ts`: construct (or
receive) the logger; in `newTransport` (~line 186) log `session.open` /
`session.close` and add `transport.onerror` → `transport.error`.
- **stdio lifecycle** — `packages/cli/src/mcp-stdio-server.ts`: construct (or
receive) the logger; route the existing `transport.onerror` (~line 54) through
it.
- **Log destination is already captured** — `packages/cli/src/managed-mcp-daemon.ts`
redirects child stdout+stderr to `.ktx/logs/mcp.log`; `ktx mcp logs`
(`commands/mcp-commands.ts`) tails it. No change needed there.
- **Dependencies** — add `pino` and `pino-pretty` to
`packages/cli/package.json`. Verify Knip/Biome dead-code and bundle checks
still pass.
- **Tests** — extend `packages/cli/test/mcp-http-server.test.ts`,
`mcp-server-factory.test.ts`, `context/mcp/server.test.ts`, and
`commands/mcp-commands.test.ts`: assert (a) a `tool.start` JSON line is written
before a (mock) handler runs and carries `params`/`sql`; (b) a matching
`tool.end` with `durationMs`/`outcome`; (c) a hung-handler scenario yields a
`tool.start` with no `tool.end` for that `callId`; (d) a slow completion emits
`warn`; (e) session lifecycle + `transport.error` lines; (f) the bearer token
never appears. Inject a capturing `io.stderr` and parse the JSON lines.
*Note:* `mcp-server-factory.test.ts` carries a pre-existing
`KtxMcpContextPorts`/`contextTools` type error (from commit `2677b3ef`,
unrelated to this work) — do not let it mask new failures.
- After implementing, rebuild and re-link so the playground picks it up:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation, not a requirement)
Running Spider 2.0-Lite against the MCP server at concurrency, an
adversarial-reviewer-generated query degenerated into a massive nested-loop join;
synchronous `better-sqlite3` executed it on the event loop, pegging a server at
~100% CPU for hours and breaking new MCP connections ("Transport channel
closed"). We could not determine *which* query, because the server logs nothing
about tool calls — diagnosis required `sample` / `lsof` on the live process and
the exact SQL was never recovered. Structured tool-call logging — especially
`tool.start` written synchronously *before* execution, at the default level —
would have turned this into a one-line `grep` of the server log. Improving the
benchmark is a side effect; the logging is generic production-server hygiene.
## Implementation notes
Implemented on branch `write-feature-spec-wiki`. All requirements and acceptance
criteria are satisfied.
**What was built / where**
- **New module `packages/cli/src/context/mcp/logger.ts`** — `createMcpLogger(io,
{ isTTY? })` builds one synchronous `pino` (v10) instance written through the
`io.stderr` sink: plain JSON when stderr is not a TTY, a `pino-pretty` (v13)
synchronous in-process stream (`{ colorize: true, sync: true }`, wrapping the
sink in a `node:stream.Writable`) when it is. Also exports `mcpLogLevel`
(`KTX_MCP_LOG_LEVEL`, validated against pino levels, default `info`),
`mcpSlowToolMs` (`KTX_MCP_SLOW_TOOL_MS`, default `10000`), and
`serializeMcpError`. No worker/async transport; no global `KTX_LOG_LEVEL`.
- **Tool-call logging — `instrumentMcpServer` (`context/mcp/context-tools.ts`)** —
per invocation: `callId = randomUUID()`, a child logger bound to
`{ tool, callId, sessionId? }`, `tool.start { params }` written at `info`
**before** awaiting the handler (synchronous, so a runaway query still leaves it
on disk), and `tool.end` after: `info { durationMs, outcome:"ok", resultSize }`,
`warn` when `durationMs > KTX_MCP_SLOW_TOOL_MS`, or `error { outcome:"error",
err }`. `resultSize` is the UTF-8 byte length of the serialized text content.
The existing `mcp_request_completed` telemetry + `reportException` are unchanged
(`durationMs` is now computed once and shared); `registerParsedTool` is intact.
- **`sessionId` / logger plumbing** — `sessionId?: string` added to
`KtxMcpToolHandlerContext`; a single per-process logger threads from each
transport entrypoint through `createKtxMcpServerFactory` →
`createDefaultKtxMcpServer` → `createKtxMcpServer` → `registerKtxContextTools`
(`KtxMcpServerDeps.logger`, `RegisterKtxContextToolsDeps.logger`).
- **HTTP lifecycle (`mcp-http-server.ts`)** — `session.open` from
`onsessioninitialized`, `session.close` from `transport.onclose`, and the
previously-unused `transport.onerror` wired to `transport.error` at `error`.
- **stdio lifecycle (`mcp-stdio-server.ts`)** — the raw `transport.onerror`
string write is replaced by a `transport.error` log line; `session.open` /
`session.close` are logged for the single stdio session.
- **Deps** — `pino ^10.3.1`, `pino-pretty ^13.1.3` added to
`packages/cli/package.json`.
- **Tests** — `test/context/mcp/logger.test.ts` (factory, level/threshold env
parsing, error serializer, TTY vs JSON), a "MCP tool-call logging" block in
`test/context/mcp/server.test.ts` (start-before-handler, matching end with
`resultSize`, hung-handler leaves an unmatched start, slow→`warn`, `warn`-level
suppression with errored end still present, no-logger no-op), session lifecycle
+ bearer-token-never-logged in `test/mcp-http-server.test.ts`, and
`test/mcp-stdio-server.test.ts` for `transport.error`.
**Deviations / decisions**
- **In-band errors carry no stack (inherent).** `registerParsedTool` converts a
thrown handler error into an `{ isError: true }` result (and reports the full
error via telemetry) before it reaches `instrumentMcpServer`, so the original
stack is already gone. `tool.end` for such a result logs `outcome:"error"` with
`err.message` only; a genuine throw that escapes gets the full pino `err`
serialization (type + message + stack). The field is always `err` for
consistency. This honours "leave `registerParsedTool` intact."
- **`session.close` is logged from `transport.onclose`** (the universal close
signal for both clean DELETE and dropped connections) rather than
`onsessionclosed`, to avoid duplicate lines; `onsessionclosed` keeps its
session-map cleanup role.
- **The logger is optional throughout.** Production always wires one per process;
when absent (programmatic/test callers that inject `createMcpServer`), tool-call
logging is simply off — which keeps existing tests unchanged.
- `createMcpLogger` accepts an optional `isTTY` purely as a test seam; production
derives format from `process.stderr.isTTY`.
**Verification**
`pnpm --filter @kaelio/ktx exec vitest run` for the four touched/added MCP test
files: 57 passed. Full default `pnpm run test`: 3018 passed, 1 skipped — the only
2 failures are in `test/skills/analytics-skill-content.test.ts`, pre-existing and
unrelated to this change (in-progress analytics-skill work on this branch).
`pnpm run dead-code` (Biome + Knip default + Knip production) clean. `pnpm run
build` and `pnpm run link:dev` succeed. `pnpm run type-check` reports only the
one pre-existing, test-only error in `test/mcp-server-factory.test.ts` from commit
`2677b3ef` (documented above); all source and the new tests type-check clean.

493
spider2-specs/specs/16-bounded-query-execution-timeout.md
View file

@ -1,493 +0,0 @@
# Bounded query execution (deadline + non-blocking) for read SQL
> Refined spec. Intake draft: `todo/16-bounded-query-execution-timeout.md`.
>
> **Scope: bound and cancel a read query that runs too long.** This is the
> execution-model companion to spec 15 (MCP structured logging). Spec 15
> *surfaces* a runaway query in the log; it explicitly defers *preventing* one —
> "off-event-loop execution, query timeouts, worker-thread isolation … is
> execution-model work in a separate spec." This is that spec.
## Problem
Two compounding gaps on the read-query path (`executeReadOnly`), confirmed in the
current code:
1. **No execution deadline, handled divergently per connector.** A single
expensive query runs unbounded, and whether it is bounded at all depends
entirely on which driver the caller hit:
- **BigQuery** is the only connector with a real statement timeout — it sets
`jobTimeoutMs` on the query job from a per-connection config field
`job_timeout_ms` (`connectors/bigquery/connector.ts`, `query(...)` ~491–512).
- **ClickHouse** sets a hardcoded 30s *HTTP* `request_timeout` at client
creation (`connectors/clickhouse/connector.ts:602`) — a client-side give-up,
not a server-side `max_execution_time`; the server keeps working.
- **Snowflake, Postgres, MySQL, SQL Server** bound only pool/connection
*acquisition* (Snowflake `acquireTimeoutMillis: 60_000`; Postgres
`connectionTimeoutMillis: 10_000`; SQL Server `idleTimeoutMillis: 30000`;
MySQL pool size only) — nothing bounds statement *execution*.
- **SQLite** has nothing.
2. **In-process SQLite blocks the event loop and cannot be cancelled.** The
SQLite connector executes on the main thread via synchronous
`better-sqlite3 .prepare().all()` (`connectors/sqlite/connector.ts`,
`query(...)` 311–318, used by `executeReadOnly` 247–251). A slow query freezes
the whole MCP server — it cannot serve other requests, send progress, or write
`tool.end` — and there is no in-thread way to interrupt it: better-sqlite3 (v12)
exposes no interrupt/cancel API. Its documented mechanism for slow queries is a
**worker thread**, and the only way to stop a runaway synchronous query is to
**terminate the thread** executing it (context7 `/wiselibs/better-sqlite3`,
`docs/threads.md`).
The observed failure (Spider2-lite sqlite run, 2026-06-18): a single
`sql_execution` MCP call —
`SELECT MIN(time_id), MAX(time_id), COUNT(*) FROM profits` on `complex_oracle`,
where `profits` is a VIEW (`costs ⋈ sales`, 918,843 × 82,112 rows, joined on a
4-column key with no composite index) — degraded to an O(N×M) nested-loop scan,
pegged a worker at 100% CPU for 13+ minutes, never returned, produced a
`tool.start` with no matching `tool.end`, and stalled an eval shard until the
worker was killed by hand. A row cap (`maxRows`) does not help: it bounds returned
rows, not scan work, and the failing query returned a single aggregate row.
## Generic use case (independent of any benchmark)
Any data agent that lets an LLM author SQL will eventually issue an
accidentally-expensive query — an unindexed or cartesian join, an expensive VIEW,
a wide aggregate over a large fact table. A general-purpose context layer must
bound that and return a clean, fast "query exceeded Ns" error so the agent can
revise (add filters, query base tables, narrow the range) instead of hanging the
tool and the server. This matters for embedded/local warehouses (SQLite, and any
future DuckDB-style in-process driver) and remote ones alike, and is wholly
independent of any benchmark.
## Design decisions (resolved during refinement)
These resolve ambiguities the intake draft left open. They constrain the
implementer; the exact code is theirs.
### One canonical deadline, applied uniformly at the contract
The deadline is enforced for **every** `executeReadOnly` caller, not only the MCP
`sql_execution` path. `executeReadOnly` has 13 call sites beyond MCP (ingest query
executor, relationship profiling and composite-candidate probes, relationship
validation, historic-SQL probes, `ktx sql`); the contract is the single place to
bound all of them. A heavy ingest profiling probe over a giant unindexed join is
exactly as worth abandoning as an interactive one — those call sites are
best-effort and degrade gracefully, so a deadline `KtxQueryError` becomes "skip
this probe / mark unprofiled," not "fail the source." (Requirement 8 covers the
call sites that must treat the timeout as recoverable.)
> Rejected alternative: a caller-resolved deadline (short on the interactive path,
> longer/none for ingest). That introduces a second value source and the open
> question "what is the ingest budget," for no real gain — the 30s default already
> clears any normal profiling probe, and a probe that exceeds it is one to drop.
### Default 30s, configurable per-connection via one shared field
- **Default `30_000` ms.** Fast enough that an LLM agent gets a clean
"exceeded 30s" and revises within the same turn; generous headroom over any
indexed aggregate or normal profiling probe; a genuine pathological nested-loop
scan blows past it immediately.
- **One shared per-connection override**, honored by every connector:
`query_timeout_ms` in `ktx.yaml` (`queryTimeoutMs` in TS), a positive integer
in **milliseconds**. Milliseconds matches the BigQuery SDK and the field it
replaces; the user-facing error still reads in seconds.
- **BigQuery's `job_timeout_ms` config key is removed**, not kept alongside the
new field. BigQuery reads the shared `query_timeout_ms` and maps the resolved
value onto its SDK's `jobTimeoutMs`. ktx keeps no backward compatibility, so
there is exactly one way to set a query timeout — no parallel knob (intake
requirement 1).
- **Granularity is per-connection only.** No global all-connections override —
different warehouses have different performance envelopes, and a second
(global) knob would double the configuration surface for no stated need.
### The shared contract is a value + an error, not a base class
There is **no shared connector base class or factory** — each connector is
constructed independently; the only shared registry is the *dialect* factory
(`context/connections/dialects.ts:47–55`). So "defined once" (intake requirement
3) means a single shared module that owns:
- `DEFAULT_QUERY_TIMEOUT_MS = 30_000`;
- `resolveQueryDeadlineMs(connectionConfig)` → the validated `query_timeout_ms`
override, else the default — so the default and the override precedence live in
exactly one place;
- `queryDeadlineExceededError(deadlineMs)` → a `KtxQueryError` with the canonical
message `query exceeded ${Math.round(deadlineMs / 1000)}s`.
Each connector calls the resolver once (at construction; connectors already
receive their connection config) and stores `this.deadlineMs`. **Enforcement is
necessarily per-connector** — different engines cancel differently — but the
*value* and the *error message* are shared, so the agent sees one consistent,
actionable error regardless of driver.
### Real cancellation, not client-side give-up
Per intake requirement 5, the deadline must *stop the work*, not merely abandon
the promise while the query keeps running (which on a pooled driver also risks
returning a still-busy connection to the pool). So:
- **In-process (SQLite, and any future embedded driver):** run the query off the
main thread and enforce the deadline by **terminating the worker thread**. There
is no generic `Promise.race` outer wrapper — a `Promise.race` against a
synchronous in-thread `.all()` can never fire (the loop is blocked), and against
a pooled remote query it would poison the pool. Thread termination *is* the
cancellation.
- **Remote engines:** set the engine's **server-side statement timeout** so the
server itself aborts the query and frees the connection cleanly.
### Logging routes through spec 15's pino path — no second logger
The deadline cases are logged through the **existing** MCP tool-call logger
(spec 15's `instrumentMcpServer`, `context/mcp/context-tools.ts:644–730`), not a
new logging path threaded into the connector. Verified flow for a timeout:
`executeReadOnly` throws `queryDeadlineExceededError` (a `KtxQueryError`) →
`local-project-ports.ts` preserves it → `registerParsedTool` (:552) reports it
(`reportException` skips `$exception` for `KtxExpectedError`) and returns an
in-band `isError` result → `instrumentMcpServer` writes `tool.end` at **`error`**
with `outcome:"error"`, `err.message = "query exceeded {N}s"`, and the **same
`callId`** as the `tool.start`.
This is the central observability win and it requires **no new MCP logging code**:
spec 15 made a hang show up as a `tool.start` with *no* matching `tool.end`; this
spec turns it into a **matched `tool.start` → `tool.end(error)` pair** whose
`tool.end` names the deadline. The worker-termination (SQLite) and server-side
abort (remote) are internal enforcement mechanisms; their single observable signal
is that `tool.end`, so the connector does **not** get its own logger threaded
through `KtxScanContext` — that would fork a second path for one capability. The
"worker was actually reaped, not left spinning" guarantee is asserted by the
worker's `exit` event in tests (Requirement 3), not by a log line.
## Requirements
### 1. Shared deadline contract, defined once
A single new module (e.g. `packages/cli/src/context/connections/query-deadline.ts`)
exports `DEFAULT_QUERY_TIMEOUT_MS` (30_000), `resolveQueryDeadlineMs(connectionConfig)`,
and `queryDeadlineExceededError(deadlineMs)`. Every connector resolves its
deadline through this resolver; no connector hardcodes its own default or
duplicates the override-precedence logic.
### 2. Shared per-connection config field; BigQuery's removed
`query_timeout_ms` is added to the **shared** connection config schema (validated
as an optional positive integer, milliseconds) so every driver accepts it. The
BigQuery-specific `job_timeout_ms` config field and its dedicated reader
(`bigQueryJobTimeoutMsFromConnection`) are removed; BigQuery sources its timeout
from the shared field and applies it as `jobTimeoutMs`. A bad `query_timeout_ms`
(zero, negative, non-integer) is a clear config validation error, consistent with
how ktx validates `ktx.yaml`.
### 3. SQLite executes off the main thread, terminated on deadline
`executeReadOnly` on the SQLite connector MUST NOT block the MCP server event
loop:
- Read-only validation and the row-limit wrapper (`assertReadOnlySql` +
`limitSqlForExecution`) run **on the main thread** before dispatch — invalid SQL
fails instantly without spawning a worker, and read-only enforcement stays at
the boundary (Requirement 7).
- The validated, row-limited SQL (and any params) is dispatched to a **worker
thread** that opens the database `{ readonly: true, fileMustExist: true }`, runs
the query, and posts back `{ headers, rows, totalRows }` (all values are
structured-cloneable — primitives, `Buffer`, `BigInt`).
- The main thread arms a timer for `this.deadlineMs`; on expiry it calls
`worker.terminate()` and rejects with `queryDeadlineExceededError`. On a normal
message it clears the timer and resolves. On a worker error (SQLite rejected the
SQL) it rejects with that error, message preserved. A provided
`ctx.signal` (`KtxScanContext.signal`, already on the contract) also terminates
the worker, for external cancellation.
- **One short-lived worker per call**, terminated on completion or deadline — not
a persistent worker or pool. Terminate-on-deadline destroys the worker, so a
pool would need respawn/job-tracking for no benefit: `executeReadOnly` is
low-frequency (LLM-issued, serial per agent turn) and worker spawn cost is
negligible against query latency. The other SQLite paths (introspect, sample,
stats, distinct-values, row-count) stay on the main thread — they are
ktx-authored, bounded, and not on the `executeReadOnly` contract.
- The event loop stays responsive throughout, so `tool.end` is always written and
concurrent requests on the same port are served.
### 4. Remote engines set a real server-side statement timeout
Each remote connector applies `this.deadlineMs` as its engine's server-side
statement timeout, so the deadline stops server work rather than abandoning the
promise:
| Connector | Mechanism | Unit |
|------------|--------------------------------------------------------|---------------|
| BigQuery | `jobTimeoutMs` on the query job (replaces `job_timeout_ms`) | ms |
| Postgres | `statement_timeout` | ms |
| MySQL | session `max_execution_time` (applies to read-only SELECT — the only kind on this path) | ms |
| Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` (ALTER SESSION) | s (ceil) |
| ClickHouse | `max_execution_time` setting, with `request_timeout` aligned to the deadline so the HTTP client does not give up before the server aborts | s (ceil) |
| SQL Server | `mssql` `requestTimeout` (TDS attention cancels server-side) | ms |
ClickHouse's existing hardcoded 30s `request_timeout` is brought under this
contract (derived from the resolved deadline), not left as a parallel mechanism.
### 5. Timeout resolves as a `KtxQueryError` with the canonical message
On exceeding the deadline, the path resolves with a `KtxQueryError`
(`query exceeded {N}s`) — a finite, decision-reaching outcome, never an unbounded
hang. For SQLite the worker-termination path throws `queryDeadlineExceededError`
directly. For remote engines, each connector recognizes **its own** engine's
timeout signal (Postgres `57014`; MySQL errno `3024`; ClickHouse code `159`;
SQL Server `ETIMEOUT`; Snowflake and BigQuery timeout errors) and re-wraps it as
`queryDeadlineExceededError`, keeping the driver error as `cause`. Each connector
owns its driver's signal — there is no central denylist of error codes to
maintain.
### 6. MCP surfacing and logging via the existing pino path
The MCP `sql_execution` path already (a) maps any non-native driver error to
`KtxQueryError` (`context/mcp/local-project-ports.ts:78–88`, guarded by
`isNativeProgrammingFault`), (b) reports it through `reportException`, which skips
`$exception` Error Tracking for `KtxExpectedError`, and (c) writes `tool.start`
synchronously before the handler and `tool.end` in `instrumentMcpServer`
(`context/mcp/context-tools.ts:644–730`). The deadline cases MUST surface through
this path — the implementer verifies and tests them, but adds **no parallel
classification or logging path**:
- **Query exceeds the deadline (any driver):** a `tool.end` at **`error`** with
`outcome:"error"` and `err.message = "query exceeded {N}s"`, carrying the same
`callId` as the `tool.start`. Classified as an expected error, so it is absent
from `$exception` Error Tracking. The reason `tool.end` was previously missing
is solely the blocked event loop (Requirement 3); once the loop stays free and
the deadline throws, the existing instrumentation logs the matched pair — closing
spec 15's "`tool.start` with no `tool.end` = hang" gap for this case.
- **Completed-but-slow query (under the deadline, over `KTX_MCP_SLOW_TOOL_MS`):**
unchanged from spec 15 — its `tool.end` is emitted at **`warn`**. The deadline
(default 30s) and the slow threshold (default 10s) are independent knobs; a query
between 10s and 30s completes with a slow `warn`, one past 30s is killed with the
`error` above.
### 7. Read-only enforcement and `maxRows` unchanged
`assertReadOnlySql` and the `maxRows` row cap (`limitSqlForExecution`) behave
exactly as today. The deadline is additive. `maxRows` is not a substitute for it
(it bounds returned rows, not scan work).
### 8. Best-effort callers treat a deadline timeout as recoverable
The non-interactive `executeReadOnly` call sites that are best-effort —
relationship profiling, composite-candidate probes, relationship validation,
historic-SQL probes — MUST treat a deadline `KtxQueryError` as "skip this
probe / mark unprofiled" and continue, never as a source-fatal error. The
implementer confirms each such site already swallows query errors into a
graceful-skip and adds that handling where it does not, so the uniform deadline
(Requirement 1, applied to all callers) cannot abort an ingest run. A skipped
probe is logged at the skip site through that path's existing scan/ingest logger
(`KtxScanContext.logger`, `warn`/`debug`), never silently dropped — these callers
are off the MCP tool-call path, so their visibility comes from the logger they
already use.
## Acceptance criteria
- A read query that exceeds the deadline returns a `KtxQueryError`
(`query exceeded {N}s`) within roughly the deadline; the MCP worker stays
responsive (a concurrent tool call on the same server completes while the slow
query is still pending) and writes a matching `tool.end` with a non-ok outcome.
- **Logging:** a timed-out `sql_execution` produces a `tool.start` and a matching
`tool.end` (same `callId`) at `error` with `outcome:"error"` and
`err.message = "query exceeded {N}s"` — no unmatched `tool.start` remains. The
timeout does not raise a `$exception` Error Tracking event (it is a
`KtxExpectedError`). A completed query slower than `KTX_MCP_SLOW_TOOL_MS` but
under the deadline still emits its `tool.end` at `warn`. No new logger is
introduced — the lines come from the existing `instrumentMcpServer`.
- **SQLite specifically:** executing a deliberately pathological query (an
expensive VIEW or an unindexed cross join) on a fixture does not block the event
loop, is terminated at the deadline, and the worker exits (the off-main-thread
executor is killed, not left spinning) so CPU returns to idle.
- **One server-side-timeout driver (Postgres):** the connector applies
`statement_timeout` equal to the resolved deadline, and a `57014` cancellation
is mapped to the canonical `KtxQueryError`.
- `resolveQueryDeadlineMs` returns 30_000 by default, honors a `query_timeout_ms`
override, and rejects an invalid value (zero / negative / non-integer).
- **No regression:** normal fast queries return identical results; read-only
rejection still works; `maxRows` still bounds returned rows.
- The shared `query_timeout_ms` field is accepted by every connector; BigQuery's
former `job_timeout_ms` key is gone and BigQuery's timeout is driven by the
shared field.
## Non-goals
- **A row/byte/cost budget on returned data.** This spec bounds *time*, not result
size — `maxRows` already bounds rows, and BigQuery's `maximumBytesBilled` is a
separate, retained concern.
- **A global `KTX_QUERY_TIMEOUT_MS` or per-call user flag.** One opinionated
default plus a per-connection override; no per-call knob, no global knob.
- **A server watchdog that recycles the process on an unmatched `tool.start`.**
Spec 15 names this as a possible future mitigation; this spec prevents the hang
at the source, so the watchdog is out of scope here.
- **Moving SQLite introspection / sampling / stats off the main thread.** Only the
`executeReadOnly` (LLM-SQL) path needs worker isolation; the rest are bounded
ktx-authored queries.
- **Per-connection retry / backoff on timeout.** A timeout returns a clean error
for the agent to revise; ktx does not auto-retry.
- **A second logger threaded into the connector.** The deadline cases are logged
through spec 15's existing MCP tool-call logger; the connector gets no separate
pino instance and `KtxScanContext` gets no MCP-logger thread (see "Logging routes
through spec 15's pino path").
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the design.
- **Shared contract** — new `packages/cli/src/context/connections/query-deadline.ts`:
`DEFAULT_QUERY_TIMEOUT_MS`, `resolveQueryDeadlineMs`, `queryDeadlineExceededError`.
Error class is `KtxQueryError` (`packages/cli/src/errors.ts:25`).
- **Contract anchor** — `KtxScanConnector.executeReadOnly`
(`context/scan/types.ts:343`), `KtxReadOnlyQueryInput` (`types.ts:285`),
`KtxScanContext.signal` (`types.ts:176`, already present, currently unused on the
MCP path).
- **Config schema** — add `query_timeout_ms` to the shared connection config
(`context/project/config.ts`, `KtxProjectConnectionConfig` and its zod schema);
remove BigQuery's `job_timeout_ms` reader.
- **SQLite worker** — new `packages/cli/src/connectors/sqlite/read-query-worker.ts`
(constructed by path via `new URL('./read-query-worker.js', import.meta.url)`);
rework `connectors/sqlite/connector.ts` `executeReadOnly` (247–251) to validate
on the main thread then dispatch to the worker with a terminate-on-deadline
timer. Reuse `normalizeQueryRows` (`context/connections/query-executor.ts`) in
the worker. Register the worker as a dynamic entry in `knip.json` (it is
referenced by path, not import) and confirm the build copies it into `dist`.
- **Remote connectors** — apply the resolved deadline and recognize the engine's
timeout signal in each `executeReadOnly` / `query(...)`:
`connectors/bigquery/connector.ts` (~491–512, `jobTimeoutMs`),
`connectors/clickhouse/connector.ts` (~602/629–644, `max_execution_time` +
`request_timeout`), `connectors/snowflake/connector.ts` (~354–371/510–534,
`STATEMENT_TIMEOUT_IN_SECONDS`), `connectors/postgres/connector.ts` (~822–838,
`statement_timeout`), `connectors/mysql/connector.ts` (~774–793,
`max_execution_time`), `connectors/sqlserver/connector.ts` (~812–832,
`requestTimeout`).
- **MCP path + logging (verify only)** — `context/mcp/local-project-ports.ts:69–88`
(error mapping), the `sql_execution` registration (~915–943), and the logging in
`instrumentMcpServer` (`context/mcp/context-tools.ts:644–730`, which writes
`tool.start`/`tool.end` via the spec-15 pino logger `context/mcp/logger.ts`). No
new classification or logging code; confirm the timeout flows through as an
expected error producing a matching `tool.end(error)` with the canonical message.
- **Best-effort callers** — `context/scan/relationship-profiling.ts` (~227, 275),
`context/scan/relationship-composite-candidates.ts` (~365, 440),
`context/scan/relationship-validation.ts` (~259),
`context/ingest/historic-sql-probes/bigquery-runner.ts` (~97), and the
historic-sql clients: confirm a deadline `KtxQueryError` is swallowed into a
graceful skip.
- **Tests** — a SQLite fixture with a pathological query (tiny `query_timeout_ms`
as the test seam) asserting terminate-on-deadline, event-loop responsiveness
(a concurrent promise resolves while the query is pending), and worker exit; a
Postgres test asserting `statement_timeout` is set to the resolved deadline and
a `57014` error maps to `KtxQueryError`; resolver unit tests (default /
override / invalid); regression tests for normal results, read-only rejection,
and `maxRows`. Extend the MCP logging tests (alongside spec 15's, e.g.
`test/context/mcp/server.test.ts`) to assert a timed-out `sql_execution` yields a
matched `tool.start`/`tool.end(error)` pair carrying `query exceeded {N}s`.
- After implementing, rebuild and re-link so the playground picks it up:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation, not a requirement)
The Spider2-lite local set loads several warehouses into SQLite, some with
expensive VIEWs over large fact tables — e.g. `complex_oracle.profits` =
`costs ⋈ sales` on `(prod_id, time_id, channel_id, promo_id)`, 918,843 × 82,112
rows, no composite index, with `promo_id` (the index the optimizer picks) being
95.5% a single value. LLM-authored profiling queries (MIN/MAX/COUNT over such a
view) trigger O(N×M) nested-loop scans. Without a deadline these hang an eval
shard for 10+ minutes; with one, the agent gets a fast error and can scope the
query instead. Improving the benchmark is a side effect; the deadline is generic
production hygiene for any agent that lets an LLM author SQL.
## Implementation notes
Implemented on branch `write-feature-spec-wiki` (ktx worktree `tallinn-v2`). All
acceptance criteria are met; tests, type-check, dead-code, and build are green
for the changed surface.
### What was built, and where
- **Shared contract** — new `packages/cli/src/context/connections/query-deadline.ts`:
`DEFAULT_QUERY_TIMEOUT_MS = 30_000`, `resolveQueryDeadlineMs(connection)` (returns
the validated `query_timeout_ms` override else the default; throws on
zero/negative/non-integer), and `queryDeadlineExceededError(deadlineMs, options?)`
(a `KtxQueryError` reading `query exceeded ${round(ms/1000)}s`, carrying the
driver error as `cause`). Unit-tested in `test/context/connections/query-deadline.test.ts`.
- **Config field** — `query_timeout_ms` (optional positive integer, ms) added to
the **shared warehouse** schema. NOTE (spec drift): that schema lives in
`context/project/driver-schemas.ts` (`warehouseConnectionSchema`), not
`config.ts`. The warehouse schemas use `z.looseObject`, so the field had to be
declared explicitly to be *validated* (otherwise it would pass through
unvalidated). BigQuery's `job_timeout_ms` field and `bigQueryJobTimeoutMsFromConnection`
reader were removed; BigQuery now resolves the shared field. Every connector
resolves its deadline once at construction via `resolveQueryDeadlineMs`.
### Deviation from the spec's SQLite mechanism (worker thread → child process)
The spec mandated running SQLite read queries on a **worker thread** and enforcing
the deadline by `worker.terminate()`. This was **empirically disproven**:
`Worker.terminate()` cannot interrupt a CPU-bound synchronous `better-sqlite3`
scan — the native `sqlite3_step` loop never yields to V8, so terminate's promise
never even resolves (an 8s probe of the exact failing query shape confirmed the
thread keeps spinning). better-sqlite3 v12 exposes no `interrupt`/progress-handler
API, and `.iterate()` does not help because the failing query is a single
aggregate row produced only *after* the full scan.
The implemented mechanism is therefore **`child_process.fork` + `SIGKILL`**
(`packages/cli/src/connectors/sqlite/read-query-child.ts`, spawned from
`connector.ts`). SIGKILL lets the OS reclaim the whole process — a probe confirmed
the scan is interrupted in ~2 ms and CPU returns to idle. This satisfies *both*
SQLite requirements better than a thread (event loop stays free **and** the query
is genuinely cancellable). The child is self-contained (imports only
`better-sqlite3` + node builtins); validation/row-limiting (`limitSqlForExecution`)
and `normalizeQueryRows` stay on the main thread. One short-lived child per call,
killed on completion, deadline, or `ctx.signal` abort. Node v24's native
TS type-stripping lets the `.ts` child load under vitest; a `.js`-if-exists-else-`.ts`
URL resolver picks the compiled child in `dist`. Registered as a dynamic entry in
`knip.json`; `tsc` emits it to `dist` (verified, plus a dist-level end-to-end smoke).
### Remote connectors (server-side timeouts + own-signal mapping)
Each applies the resolved deadline server-side and re-wraps its own timeout signal
as `queryDeadlineExceededError(deadlineMs, { cause })`:
- **BigQuery** — `jobTimeoutMs` on the query job; maps a "Job timed out" / timeout-reason error.
- **Postgres** — `statement_timeout` via pool `options` (`-c statement_timeout=<ms>`); maps `57014`.
- **MySQL** — `SET SESSION max_execution_time = <ms>` before the read; maps errno `3024`.
- **Snowflake** — `ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = <ceil(s)>` in the pooled connection; maps code `604` / "reached its … timeout".
- **ClickHouse** — `max_execution_time` (ceil seconds) setting, with `request_timeout` set to `deadline + 5s` so the HTTP client outlasts the server abort (replaces the old hardcoded 30s); maps code `159`.
- **SQL Server** — `requestTimeout` on the `mssql` pool config (TDS attention cancels server-side); maps `ETIMEOUT`.
Each connector has a focused test asserting the timeout is applied and its signal
maps to `KtxQueryError` (Postgres is the spec's required acceptance test).
### Best-effort callers (Requirement 8)
Confirmed already graceful: relationship **profiling** (outer try/catch →
`profile_failed` warning) and **composite-candidate** detection
(`detectCompositeRelationships` → recoverable warning, returns `[]`). Historic-SQL
**probes** flow through `runHistoricSqlReadinessProbe`, which catches *any* error
into `{ ok: false }`. **Added** handling to relationship **validation**: a
`KtxQueryError` on the per-candidate coverage probe now sends that one candidate to
`review` (`validation_query_failed`, logged via `ctx.logger.warn`) instead of
aborting the whole validation pass. `ingest-query-executor.ts` is a generic
executor port whose callers own recoverability — left unchanged.
### MCP surfacing/logging
No new MCP classification or logging code. The deadline `KtxQueryError` flows
through the existing `local-project-ports` mapping → `reportException` (skips
`$exception` for `KtxExpectedError`; existing test `telemetry/exception.test.ts`
covers the skip for `KtxQueryError`) → `instrumentMcpServer`, which logs a matched
`tool.start` → `tool.end(error, level 50)` pair carrying `err.message = "query
exceeded {N}s"`. A test in `test/context/mcp/server.test.ts` asserts the matched
pair, closing spec 15's "`tool.start` with no `tool.end` = hang" gap for this case.
### Pre-existing branch issues encountered (not part of this feature)
- `test/mcp-server-factory.test.ts` had a type error (an `as` cast to a shape with
a fake `context_tool` key, introduced by branch commit `2677b3ef`) that broke
`tsc -p tsconfig.test.json`. Fixed with a clean single cast to keep the
type-check gate green; behavior unchanged.
- `test/skills/analytics-skill-content.test.ts` fails (2 cases: missing
`**Window functions**` heading and `Expose identity, not just the label` prose
in `src/skills/analytics/SKILL.md`). This is unrelated analytics-skill (spec
13/14) content drift committed earlier on the branch; **left untouched** — no
skill files were modified by this feature.

418
spider2-specs/specs/18-bigquery-cross-project-datasets.md
View file

@ -1,418 +0,0 @@
# BigQuery cross-project dataset introspection (foreign-hosted datasets, billed in own project)
> Refined spec. Intake draft: `todo/18-bigquery-cross-project-datasets.md`.
>
> **Scope: let the BigQuery connector introspect a dataset hosted in a *different*
> project than the one it bills jobs to.** A `dataset_ids` entry may be written
> fully-qualified as `project.dataset`; the connector introspects each entry in
> *its own* project while every job still runs in `credentials.project_id`. A
> bare `dataset` keeps today's single-project behavior unchanged.
>
> Out of scope (confirmed during refinement): the interactive `ktx setup` wizard
> is **not** expected to *discover* foreign datasets — you cannot enumerate
> datasets in a project you don't own, and the wizard doesn't know which foreign
> projects to probe. Users hand-write `project.dataset` entries (in `ktx.yaml` or
> at the dataset prompt); the connector must accept and introspect them. See
> *Non-goals*.
## Problem
**ktx**'s BigQuery connector derives a single `projectId` from
`credentials.project_id` and uses it for **both** job billing **and** schema
introspection. There is no way to introspect a dataset that lives in another
project, even though *querying* such a dataset already works (a cross-project
read in a `FROM` clause bills to the caller's project — that path is proven).
Confirmed in the current connector (`packages/cli/src/connectors/bigquery/connector.ts`):
- **`:294`** — `projectId` is read only from `credentials.project_id`. There is
no separate billing-vs-dataset project. `bigQueryConnectionConfigFromConfig`
(`:278`–`:301`) returns `datasetIds: string[]` — raw, unparsed.
- **`datasetIds()` (`:163`)** — returns `dataset_ids` / `dataset_id` verbatim;
it never parses a `project.` prefix.
- **`introspectDataset` (`:544`)** — calls `this.getClient().dataset(datasetId)`,
which resolves the dataset in the **client's (billing) project**, and labels
every table `catalog: this.resolved.projectId` (`:566`, `:574`) — including the
introspection-failure warning metadata (`:566`).
- **`primaryKeys` (`:591`)** — builds `INFORMATION_SCHEMA` SQL as
`` `<projectId>.<datasetId>.INFORMATION_SCHEMA.TABLE_CONSTRAINTS` `` using the
**billing** project.
- **`listTables` (`:453`)** — queries
`` `<projectId>`.`region-<region>`.INFORMATION_SCHEMA.TABLES `` against the
**billing** project and labels each row `catalog: this.resolved.projectId`.
- **`testConnection` (`:344`)** — calls `client.dataset(datasetId).get()` in the
billing project.
### Empirical confirmation (from the intake draft)
With a service account in project `ktx-spider2-lite`:
- ktx's call pattern `client.dataset("austin_311")` → **`404 NotFound`** (it looks
in `projects/ktx-spider2-lite/datasets/austin_311`).
- The cross-project form `dataset("austin_311", { projectId: "bigquery-public-data" })`
→ **succeeds** (public metadata is readable by any authenticated principal).
- There is **no config knob** to separate the introspection project from billing.
### Why the table `catalog` label is load-bearing, not cosmetic
The BigQuery dialect generates **three-part `catalog.db.name`** SQL
(`connectors/bigquery/dialect.ts:38` → `formatDialectTableName(..., 'three-part')`;
`context/connections/dialect-helpers.ts:27`–`32` emits `catalog.db.name`). The
`catalog` stored on each scanned table is therefore the project that *every*
later query targets — `sampleTable`, `sampleColumn`, `getColumnDistinctValues`,
and ref-based `executeReadOnly` all format the ref through the dialect. If a
foreign dataset's tables are labeled with the billing project, every one of those
queries becomes `` `billing-project`.`austin_311`.`table` `` → `404`. So labeling
the table `catalog` with the dataset's own project is a **correctness
requirement**, and it is the single lever that makes sampling, dictionary value
extraction, and `discover_data` all resolve once the snapshot is right.
### One introspection path, no divergence
`connectors/bigquery/live-database-introspection.ts` wraps
`KtxBigQueryScanConnector.introspect` directly, so the ingest and live-database
paths share **one** introspection implementation. The SDK already supports the
fix: `client.dataset(id, { projectId })` — `@google-cloud/bigquery@8.3.1`'s
`DatasetOptions` exposes `projectId?: string`.
## Generic use case (independent of any benchmark)
Analysts routinely introspect datasets they can **read but do not own and do not
bill to**: Google's `bigquery-public-data`, a partner's shared project, an
organization's central data project that a smaller team queries from its own
billing project. To make those connectable in **ktx** — so `discover_data`, the
semantic layer, dictionary sampling, and `sql_dialect_notes` all work — the
connector must introspect a foreign-hosted dataset while billing jobs in the
credentials' own project. This is a standard BigQuery deployment shape and is
wholly independent of any benchmark.
The class to design for is "the dataset's project ≠ the billing project," and it
must generalize beyond one example: a single connection may reference datasets in
**several** foreign projects at once (e.g. one slice mixing `bigquery-public-data`
and `isb-cgc-bq`), and two different projects may host datasets with the **same
name**. The design must keep those distinct.
## Design decisions (resolved during refinement)
These resolve ambiguities the intake draft left open. They constrain the
implementer; the exact code is theirs.
### Carry the project inline on each dataset entry — no separate knob
The introspection project is expressed **per dataset**, inline, as the optional
`project.` prefix on a `dataset_ids` / `dataset_id` entry. There is no new config
field.
> Rejected alternative: a separate connection-level `dataset_project` (or
> `introspection_project`) field. It is a speculative runtime knob (against the
> repo's opinionated-defaults rule) and, more decisively, it **cannot express the
> requirement**: one connection must span *multiple* foreign projects, which a
> single global field cannot represent. The inline form also derives scope from
> the user's own declared input rather than adding a parallel setting.
### Parse to canonical `{ project, dataset }` pairs at the config boundary
Each entry is parsed **once**, in `bigQueryConnectionConfigFromConfig` /
`datasetIds()`, into a canonical pair: the project (when no prefix is present,
default it to `credentials.project_id`) and the bare dataset id. Every
introspection-side call site reads the resolved pair; nothing downstream re-parses
a `project.dataset` string.
> Rejected alternative: keep `datasetIds: string[]` raw and split the prefix
> lazily at each use site (`introspectDataset`, `primaryKeys`, `listTables`,
> `testConnection`). That re-implements one rule in four places and is exactly the
> drift trap the repo's single-source-of-truth rule warns about — a later fix
> lands on one path and not another. Normalize at the boundary; carry the
> canonical form downstream.
The internal resolved-config type (`KtxBigQueryResolvedConnectionConfig.datasetIds`)
changes shape from `string[]` to a structured pair list. That is an internal type;
the connector internals and the connector test fixtures are the only consumers.
### Parsing rule (at the boundary)
- An entry contains **at most one `.`**.
- With a dot: the segment **before** the dot is the project, validated by the
existing `normalizeBigQueryProjectId` charset
(`context/connections/bigquery-identifiers.ts`); the segment **after** is the
dataset id (validated as a normal identifier).
- Without a dot: a bare dataset; the project defaults to `credentials.project_id`
(today's behavior).
- **More than one `.`** (e.g. a stray `proj.ds.table`) is a clear config error
raised at resolution time, naming the connection — not a silent
mis-introspection.
- Legacy domain-scoped project ids that contain `:` (e.g. `example.com:proj`) stay
**out of scope**, consistent with `normalizeBigQueryProjectId`'s current charset
(which already rejects `.` and `:` in a project id).
### Billing is never the dataset's project
The BigQuery client is still constructed with `projectId = credentials.project_id`
(`getClient()`, `:487`–`:495`), and `createQueryJob` always bills there. Only the
*introspection* surfaces switch to the per-dataset project. Cross-project reads in
a `FROM` clause already bill to the caller — unchanged and already proven.
### Dataset identity downstream is `(catalog, db)`
Scanned tables are keyed by `(catalog, db, name)` throughout
(`context/scan/table-ref.ts`; `context/scan/warehouse-catalog.ts:107`). Because
the table `catalog` now holds the dataset's own project, two foreign projects that
each host a `austin_311` dataset remain distinct with no extra work — provided the
snapshot's `scope` / `metadata` also preserve the project (Requirement 6).
### Setup-wizard scope: accept, don't discover
The connector's region-scoped `listTables` (`:453`) is consumed **only** by the
`ktx setup` wizard's table-selection step (`setup-databases.ts`); the
ingest / `discover_data` path reads persisted snapshot JSON via
`WarehouseCatalogService.listTables`, not the connector method. The wizard is not
expected to enumerate foreign datasets (you can't list a project you don't own).
A `project.dataset` value hand-entered at the dataset prompt, or written into
`ktx.yaml`, must be accepted, validated, and introspected. See *Non-goals* for the
region caveat that follows from this.
## Requirements
### R1 — Accept and parse `project.dataset` at the config boundary
`datasetIds()` / `bigQueryConnectionConfigFromConfig` resolve each
`dataset_ids` and `dataset_id` entry into a canonical `{ project, dataset }` pair
per the parsing rule above, defaulting `project` to `credentials.project_id` when
unprefixed. A malformed entry (more than one `.`, an empty project or dataset
segment, or a project/dataset that fails identifier validation) raises a clear
error at resolution time that names the connection id.
### R2 — Introspect each dataset in its own project
`introspectDataset` resolves the dataset via the **dataset's** project —
`client.dataset(datasetId, { projectId })` — for `getTables()` and each
`tableRef.get()`. This requires extending the `KtxBigQueryClient.dataset` port to
accept the project (e.g. `dataset(id, projectId)` / `dataset(id, { projectId })`)
and forwarding it from `DefaultBigQueryClientFactory`.
### R3 — Label table `catalog` with the dataset's project
Every table produced by `introspectDataset` is labeled `catalog: <dataset's
project>` (not the billing project), and the introspection-failure warning
metadata (`object` / `catalog`) likewise reflects the dataset's project. This is
what makes downstream sample/distinct-value/read queries resolve.
### R4 — Primary-key discovery targets the dataset's project
The `primaryKeys` `INFORMATION_SCHEMA.TABLE_CONSTRAINTS` /
`KEY_COLUMN_USAGE` SQL is built against
`` `<dataset's project>.<datasetId>.INFORMATION_SCHEMA…` ``. (This INFORMATION_SCHEMA
view is dataset-qualified and therefore region-independent.) Its existing
soft-fail-on-denied behavior (`tryConstraintQuery`, scan warning) is preserved.
### R5 — `listTables` lists each dataset in its own project
`listTables` returns rows labeled `catalog: <that dataset's project>` and queries
each referenced project's region `INFORMATION_SCHEMA.TABLES`. Because a connection
can now span projects, it queries per distinct project rather than assuming one.
(This is the setup-wizard surface — see the cross-region caveat in *Non-goals*.)
### R6 — Snapshot scope and metadata reflect multiple projects
`introspect`'s returned snapshot keeps `metadata.project_id` = the **billing**
project, but `scope.catalogs` becomes the **distinct set of dataset projects**
actually introspected. `scope.datasets` / `metadata.datasets` must stay
unambiguous when two projects share a dataset name (e.g. carry the qualified
`project.dataset`, or otherwise preserve the project). The scoped table-name
lookup that today passes `catalog: this.resolved.projectId` (`:359`) must pass
each dataset's own project so `tableScope` / `enabled_tables` filtering still
matches.
### R7 — `testConnection` resolves foreign datasets
`testConnection` validates each configured dataset via its own project
(`client.dataset(datasetId, { projectId }).get()`), so a connection pointing only
at foreign datasets reports success rather than a spurious `404`.
### R8 — Billing unchanged; bare dataset is a strict no-op
`createQueryJob` continues to bill in `credentials.project_id`. A connection whose
`dataset_ids` are all bare (no `project.` prefix) behaves **exactly** as before:
same resolved project, same `catalog` labels, same INFORMATION_SCHEMA targets, no
behavioral change.
### R9 — `getTableRowCount` honors the parsed entry
`getTableRowCount`'s default-dataset handling (`:431`, today
`this.resolved.datasetIds[0]`) resolves through the canonical pair so a foreign
default dataset is introspected in its own project.
### R10 — Docs reflect the qualified form
Document that a BigQuery `dataset_ids` / `dataset_id` entry may be written
`project.dataset` to introspect a dataset hosted in another project (billing stays
in `credentials.project_id`). Update the BigQuery rows/examples in
`docs-site/content/docs/configuration/ktx-yaml.mdx` and
`docs-site/content/docs/integrations/primary-sources.mdx` (and the dataset-scope
note in `docs-site/content/docs/cli-reference/ktx-setup.mdx`). Keep examples
copy-pasteable and follow the `fumadocs-mdx-structure` skill.
## Acceptance criteria
1. **Foreign single-project introspection.** With credentials in project
`ktx-spider2-lite` and `dataset_ids: ['bigquery-public-data.austin_311']`,
`ktx ingest <conn>` introspects the tables, enriches, and samples values;
`discover_data` / `dictionary_search` return them. Tables are labeled
`catalog: 'bigquery-public-data'`.
2. **Multi-project connection.** `dataset_ids: ['bigquery-public-data.x',
'other-project.y']` introspects **both**, each under its own project; the
snapshot's `scope.catalogs` contains both projects.
3. **Cross-project query still bills locally.** `sql_execution` of a
fully-qualified `project.dataset.table` query runs and bills in
`credentials.project_id`.
4. **Same dataset name, two projects.** `['proj-a.shared', 'proj-b.shared']`
yields two distinct dataset groups; tables do not collide.
5. **No regression.** `dataset_ids: ['my_dataset']` (or singular `dataset_id`)
behaves exactly as before — resolved under `credentials.project_id`, same
`catalog` labels and INFORMATION_SCHEMA targets.
6. **Malformed entry fails clearly.** `dataset_ids: ['proj.ds.table']` (or an
empty segment) raises a config error naming the connection, not a `404` at
scan time.
7. **Test coverage** (extend `packages/cli/test/connectors/bigquery/connector.test.ts`,
using the existing fake `clientFactory` harness):
- the fake `dataset()` is called with the dataset's project for a prefixed
entry, and with the billing project for a bare entry;
- a prefixed entry yields tables with `catalog: '<dataset project>'`;
- a mixed two-project `dataset_ids` introspects both;
- `bigQueryConnectionConfigFromConfig` rejects a multi-dot / empty-segment
entry;
- the existing single-project tests still pass unchanged.
## Non-goals
- **Foreign-dataset discovery in the setup wizard.** The wizard does not
enumerate datasets in projects the credentials don't own; users supply
`project.dataset` explicitly (scope decision A).
- **Cross-region `listTables`.** `listTables`' region-scoped
`region-<location>.INFORMATION_SCHEMA.TABLES` query uses the connection-level
`location`; a foreign dataset in a *different* region than the connection's
`location` will not be listed by that wizard-facing query. This does **not**
affect ingest/`discover_data`, whose introspection path
(`introspectDataset` REST metadata + dataset-qualified PK INFORMATION_SCHEMA) is
region-independent. A per-dataset region knob is a separate spec if ever needed.
- **Domain-scoped legacy project ids** containing `:` (e.g. `example.com:proj`),
already unsupported by `normalizeBigQueryProjectId`.
- **A separate billing/introspection config field** — explicitly rejected above.
## Implementation orientation
Pointers from exploration; line numbers may have drifted, and the implementer owns
the design.
- `packages/cli/src/connectors/bigquery/connector.ts`
- `datasetIds()` (`:163`) and `bigQueryConnectionConfigFromConfig` (`:278`) —
parse + canonicalize (R1); change `KtxBigQueryResolvedConnectionConfig.datasetIds`
shape.
- `KtxBigQueryClient.dataset` port (`:100`–`:110`) and
`DefaultBigQueryClientFactory.dataset` (`:130`–`:135`) — thread `projectId`
(R2). `getClient()` (`:487`) keeps the billing project (R8).
- `introspectDataset` (`:544`) — `dataset(id, { projectId })`, table `catalog`
+ warning metadata (R2, R3).
- `primaryKeys` (`:591`) — dataset-qualified INFORMATION_SCHEMA (R4).
- `listTables` (`:453`) — per-project region INFORMATION_SCHEMA + row catalog
(R5).
- `introspect` (`:352`) — `scope.catalogs`, `scope.datasets`, scoped-name lookup
(`:359`) (R6).
- `testConnection` (`:339`) (R7); `getTableRowCount` (`:431`) (R9).
- `packages/cli/src/connectors/bigquery/live-database-introspection.ts` — wraps
`introspect`; no separate change needed (it inherits the fix).
- `packages/cli/src/context/connections/bigquery-identifiers.ts` —
`normalizeBigQueryProjectId` is the project-segment validator.
- `packages/cli/src/context/connections/dialect-helpers.ts` /
`connectors/bigquery/dialect.ts` — three-part naming; no change, but this is
*why* R3 matters.
- After implementing, rebuild and re-link so the playground picks it up:
`pnpm run build && pnpm run link:dev`. Run
`pnpm --filter @kaelio/ktx run type-check` and the connector test suite.
## Benchmark context (motivation, not a requirement — do not encode benchmark specifics)
Spider 2.0-Lite's **BigQuery slice (~205 questions)** is otherwise unservable
faithfully: every one of its ~74 logical databases groups datasets hosted in
foreign public projects (`bigquery-public-data`, `isb-cgc-bq`,
`data-to-insights`, …), never in a project we own. Query execution already works
cross-project; ktx-only *discovery* is the sole blocker, and it is blocked exactly
because the connector can't introspect a foreign-hosted dataset. Of 74 BQ
databases only **one** spans more than one source project, so "let `dataset_ids`
carry `project.dataset` and introspect each in its own project" covers the
benchmark and the general case alike. None of these project names belong in the
code — they are derived from the user's own `dataset_ids` input.
## Implementation notes
Implemented on branch `write-feature-spec-wiki`. The whole change is contained in
the BigQuery connector, its identifier helpers, the connector test suite, and three
docs pages.
**Config boundary (R1).** Added `normalizeBigQueryDatasetId`
(`packages/cli/src/context/connections/bigquery-identifiers.ts`, charset
`[A-Za-z0-9_]`) next to the existing project/region validators. In
`connectors/bigquery/connector.ts`, a single `parseBigQueryDatasetEntry(entry,
defaultProject, connectionId)` parses one entry by splitting on `.`: zero dots →
bare dataset in `defaultProject`; one dot → `project.dataset` (each segment
validated; empty segment throws); two or more dots → throws. `resolveDatasetRefs`
resolves `env:`/`file:` references first, trims/filters empties, then parses each.
`bigQueryConnectionConfigFromConfig` calls it with the billing `project_id` as the
default, so the canonical pair list is produced once at the boundary.
`KtxBigQueryResolvedConnectionConfig.datasetIds` changed from `string[]` to the new
`BigQueryDatasetRef[]` (`{ project, dataset }`). All errors name
`connections.<id>.dataset_ids entry "<entry>"`.
**Client port (R2).** `KtxBigQueryClient.dataset` now takes
`(datasetId, projectId)`; `DefaultBigQueryClientFactory` forwards
`client.dataset(datasetId, { projectId })` (`@google-cloud/bigquery` `DatasetOptions.projectId`).
`getClient()` still constructs the client with the **billing** `project_id`, so
`createQueryJob` bills locally regardless of the dataset's project (R8, acceptance 3).
**Per-dataset introspection (R3–R7, R9).** Every introspection site reads the
resolved pair: `introspectDataset(ref, …)` resolves `dataset(ref.dataset, ref.project)`
and labels tables (and the introspection-failure warning, via `tryIntrospectObject`'s
`catalog.db.object`) with `ref.project`; `primaryKeys(ref)` builds dataset-qualified
`` `<project>.<dataset>.INFORMATION_SCHEMA…` `` SQL; `testConnection` validates each
dataset under its own project; `getTableRowCount`'s default resolves through the first
pair. `introspect` sets `scope.catalogs` to the distinct set of dataset projects and
keeps `metadata.project_id` = billing. `scope.datasets` / `metadata.datasets` use a
`qualifiedDatasetLabel` helper — bare in the billing project (so the single-project
snapshot is byte-for-byte unchanged), `project.dataset` otherwise (so two projects with
the same dataset name stay distinct, R6/acceptance 4).
**`listTables` (R5).** Split into `listTables` (parse override entries, group by
project) and `listTablesInProject(project, region, datasets?)`. With no override it
lists the billing project's region (unchanged); with an override it runs one
region-`INFORMATION_SCHEMA.TABLES` query per distinct project, filtered to that
project's bare datasets, and labels rows with that project. The existing single-region
test is unchanged (bare entries collapse to one billing-project query).
**Docs (R10).** Added a "Cross-project datasets" subsection to
`integrations/primary-sources.mdx` (qualified-entry example + the setup/region caveats),
plus pointers from `configuration/ktx-yaml.mdx` and `cli-reference/ktx-setup.mdx`.
**Tests.** Extended `test/connectors/bigquery/connector.test.ts`: parse-to-pairs and
malformed-entry rejection (`proj.ds.table`, `proj.`, `.ds`); a foreign-only connection
calls `dataset('austin_311', 'bigquery-public-data')`, labels tables
`catalog: 'bigquery-public-data'`, builds the client with the billing project, and keeps
`metadata.project_id` local; a mixed `['bigquery-public-data.austin_311', 'analytics']`
connection introspects both under their own projects; and `['proj_a.shared',
'proj_b.shared']` stays distinct. The internal `datasetIds`-shape assertion was updated
to the pair list; all pre-existing behavioral tests pass unchanged.
**Verification.** `pnpm --filter @kaelio/ktx run type-check`, the connector suite
(18 tests), `test/setup-databases.test.ts` + `bigquery-identifiers.test.ts`,
`pnpm run build`, `pnpm run dead-code` (Biome + Knip default + production),
`pnpm run link:dev` (`ktx-dev` → 0.12.0), and `pre-commit` on the changed files all
pass. Acceptance criteria 1–4 are exercised by unit tests with the fake client factory;
criteria 5–6 by unit tests; criterion 3 (cross-project query bills locally) is
structurally guaranteed (single billing client) and asserted via the `createClient`
project. End-to-end ingest against live `bigquery-public-data` was not run here (no live
credentials in this worktree); the `link:dev` binary is ready for the playground agent to
validate.
**No deviations from the spec design.** The only judgment call: `scope.datasets`
renders bare-in-billing / qualified-otherwise rather than always-qualified, chosen to
satisfy both the no-regression requirement (R8/acceptance 5) and the disambiguation
requirement (R6/acceptance 4) with one unambiguous, dot-delimited form.

471
spider2-specs/specs/19-durable-bounded-relationship-detection.md
View file

@ -1,471 +0,0 @@
# Durable, resumable, bounded relationship detection during ingest enrichment
> Refined spec. Intake draft: `todo/19-durable-bounded-relationship-detection.md`.
>
> **Scope: make the expensive part of ingest enrichment survive an interrupted
> relationship stage.** Today the paid LLM descriptions + embeddings only become
> durable and queryable after the slowest, most-killable, least-valuable stage
> (relationship detection) also finishes. This spec moves the persistence boundary
> to the cost boundary, makes stage resume work across runs, and bounds + observes
> the one open-ended stage — the durability companion to spec 16 (bounded query
> execution), which this spec composes with rather than replaces.
## Problem
Three compounding failure modes, all confirmed in the current code, share one root
cause: **the three enrichment stages are treated as a single atomic unit for
persistence, identity, and bounding, even though they differ radically in cost,
durability value, runtime, and likelihood of being killed.**
`runLocalScanEnrichment` (`context/scan/local-enrichment.ts:472`) runs three stages
in a fixed order through `runEnrichmentStage` (`:413`):
| stage | order | cost | durability value | runtime on a large schema | likely to be killed |
|-------|-------|------|------------------|---------------------------|---------------------|
| `descriptions` (`:524`) | 1st | high — one paid LLM call per table | high | minutes | low |
| `embeddings` (`:553`) | 2nd | medium | high | seconds–minutes | low |
| `relationships` (`:587`) | 3rd | low — best-effort joins | low | **minutes, silent** | **high** |
The slowest, most-killable, least-valuable stage runs **last**, and it gates the
durability of the two expensive stages held in memory before it.
### 1. Enrichment is lost if relationship detection is interrupted
The queryable artifact agents search and execute against is the `_schema` manifest
YAML (`semantic-layer/<connectionId>/_schema/*.yaml`). It is written **twice**:
- bare (native column comments only) early, at `local-scan.ts:473`
(`writeLocalScanManifestShards`), before enrichment runs; and
- rewritten **with AI descriptions + accepted joins** by
`writeLocalScanEnrichmentArtifacts` (`local-enrichment-artifacts.ts:310`), called
from `local-scan.ts:510` **after** `runLocalScanEnrichment` returns — i.e. after
all three stages.
So the descriptions and embeddings reach the queryable layer only via that single
terminal write. If the process is killed/crashes/times out **during** the
`relationships` stage, `runLocalScanEnrichment` never returns, the terminal write
never runs, and the in-memory descriptions + embeddings are discarded — the
`_schema` retains only the bare native comments from the `:473` write.
Empirically (intake draft): ingesting a 95-table BigQuery dataset produced full
descriptions + embeddings (progress reached "Building embeddings 17/17"), then the
relationship stage ran silently past a supervising deadline and was killed; the
persisted `_schema` had **0** AI descriptions. The most expensive work is the most
likely to be thrown away.
> A stage-state store (below) does save each completed stage's output to an
> internal SQLite cache as the stage finishes — so the descriptions are not lost to
> the *resume cache*. They are simply never **promoted** to the queryable `_schema`
> until the terminal write. The data survives somewhere the agent cannot query, and
> (per failure mode 2) cannot be reused on the next run either.
### 2. Re-running does not resume — it re-spends
`runEnrichmentStage` resolves a completed stage with
`findCompletedStage({ runId, stage, inputHash })` (`local-enrichment.ts:427`), and
the store keys on **`runId`**: `SqliteLocalScanEnrichmentStateStore` declares
`PRIMARY KEY (run_id, stage)` and filters lookups by `run_id`
(`sqlite-local-enrichment-state-store.ts:83,91–115`). `runId` is minted fresh per
ingest invocation (`record.runId`). The cache therefore only resolves *within* one
run; re-running an interrupted ingest gets a new `runId`, misses every cached
stage, and **recomputes descriptions + embeddings from scratch** — re-paying for
LLM work that already succeeded.
The store already computes and persists `inputHash` next to `runId` —
a stable `sha256` of `{ snapshot, mode, detectRelationships, providerIdentity,
relationshipSettings }` (`enrichment-state.ts:78`). The correct content key is
already on the row; the lookup just uses the volatile column. This is a keying
defect, not a missing capability.
### 3. Relationship detection is unobservable and unbounded
`discoverKtxRelationships` (`context/scan/relationship-discovery.ts:218`) profiles a
row sample of **every enabled table** (`profileKtxRelationshipSchema`,
`relationship-profiling.ts:320` — one sampled query per table at
`profileConcurrency`, default 4), validates candidate joins
(`relationship-validation.ts:237` — one coverage query per candidate), and detects
composite keys (`relationship-composite-candidates.ts:515` — per-table plus
cross-table queries). None of the controls the rest of the scan pipeline relies on
were ever wired into this stack:
- **No progress.** `discoverKtxRelationships` does not accept a progress port; the
caller can only emit start/end around it (`local-enrichment.ts:600,611` —
`update(0, 'Detecting relationships')` … `update(1, 'found N')`). Minutes of
silence between.
- **No honored cancellation.** `KtxScanContext.signal` exists on the contract
(`types.ts`) but **no sub-stage reads it**.
- **No time budget.** Validation has a *count* budget (`validationBudget`, default
`min(2 × tableCount, 1000)`); profiling and composite detection have none. On a
schema with hundreds–thousands of tables, profiling is O(tables) silent queries
with no internal stop condition.
A supervisor watching for liveness cannot tell a slow-but-working profile from a
true hang, and nothing inside the stage will voluntarily stop — so on a very large
schema it runs far past any reasonable deadline and is killed (which, via failure
mode 1, takes the descriptions with it).
## Generic use case (independent of any benchmark)
Any context layer that enriches a real warehouse with paid LLM work must make that
work durable the instant it is produced, resume it across process restarts without
re-paying, and bound the open-ended profiling stage so a large catalog cannot hang
ingest indefinitely. A data team ingesting a 500-table production warehouse over a
flaky connection, a rate-limited LLM budget, or a CI step with a wall-clock limit
hits all three failure modes regardless of any benchmark. This is general
durability and cost hygiene for the ingest pipeline; the benchmark only made it
acute at scale.
## Design decisions (resolved during refinement)
These resolve ambiguities the intake draft left open. They constrain the
implementer; the exact code is theirs (requirement-level, per the specs README).
### D1 — Checkpoint queryable artifacts at the cost boundary, before relationships
As soon as the last non-relationship stage completes — `embeddings` when an
embedding provider is configured, otherwise `descriptions` — persist the
descriptions + embeddings into the **queryable** `_schema` manifest (and the raw
`descriptions.json` / `embeddings.json` enrichment artifacts), **before** the
`relationships` stage runs. The relationship stage then writes its joins on top: the
manifest builder already re-reads and preserves existing descriptions and
manual/inferred joins on rewrite (`loadExistingManifestState`,
`local-enrichment-artifacts.ts:196`), so the second write is additive, not
destructive.
Net invariant: **the descriptions + embeddings are always durable and queryable the
moment they are computed**, even if relationship detection then fails, is
interrupted, is budget-truncated, or is skipped. A failed/partial/skipped
relationship stage degrades to "no joins" or "partial joins" — **never** to "no
descriptions." This is the inverse guarantee the current terminal-write ordering
violates.
The bare `:473` manifest write stays — it is the queryable schema for the
no-providers / enrichment-disabled path. The checkpoint is an additional write that
runs only when enrichment produced descriptions.
> Orientation (the implementer owns the seam): the lowest-coupling shape is a
> checkpoint hook — `runLocalScanEnrichment` invokes a caller-supplied callback once
> the last non-relationship stage completes, and `local-scan.ts` supplies a callback
> that calls the existing `writeLocalScanEnrichmentArtifacts` for the
> descriptions + embeddings + manifest only (no generated joins yet). The final
> write after the relationship stage proceeds as today. Relationship-specific
> artifacts (`relationships.json`, `relationship-profile.json`,
> `relationship-diagnostics.json`) are written by the final/relationship write, not
> the checkpoint, so the checkpoint never emits misleading empty relationship
> diagnostics.
>
> Rejected alternative: move all artifact writing inside `runLocalScanEnrichment`
> (inject the file store / project). That couples the enrichment module to
> persistence for no gain — the writer already lives in `local-scan.ts` and the
> checkpoint needs only a one-line hook, not a relocation.
### D2 — Resume by content identity, not by `runId`
Re-key completed-stage resolution on **`(connectionId, stage, inputHash)`**,
independent of `runId`, so a re-run with an unchanged schema and config resumes the
finished `descriptions` / `embeddings` stages from cache and re-runs only what
actually failed. `inputHash` is already the content fingerprint; `connectionId`
scopes it to the right source. When several rows share a content identity (one per
prior run), the most recent `updatedAt` wins.
`runId` stays on the stored row for diagnostics and for `listRunStages`, but leaves
the uniqueness/lookup key.
The state store is a **disposable local resume cache** (`.ktx` local state,
regenerable from a fresh ingest). Re-key it with **no migration bridge** — recreate
the table if its on-disk shape differs from the new `(connection_id, stage,
input_hash)` key, consistent with ktx's no-backward-compatibility policy. Losing the
old cache only means one ingest cannot resume; it never corrupts a queryable
artifact.
> Rejected alternative: include `syncId` or `mode` in the key. `mode` and the rest
> are already folded into `inputHash`; adding them again would only narrow the key
> and re-break cross-run resume when an incidental field differs.
### D3 — Make the relationship stage observable and bounded
Thread three things the rest of the pipeline already supports through
`discoverKtxRelationships` into profiling, validation, and composite detection:
- **Progress** through the existing progress port (the relationship phase is
already `progress?.startPhase(0.25)` at `local-enrichment.ts:586`): emit per-unit
liveness — "Profiling table K/N", "Validating candidate K/M", and the equivalent
for composite probing — so a supervisor can distinguish slow-but-working from
hung.
- **A flat wall-clock budget** for the whole relationship stage: a new
`scan.relationships.detectionBudgetMs`, a positive integer of milliseconds,
project-level, validated like the other `scan.relationships` fields, **default
600_000 (10 min), enforced by default.** Checked at unit boundaries (before each
table profile, each candidate validation, each composite probe). It sits **above**
spec 16's per-query deadline (default 30s): each individual query is already
bounded; this bounds the *sum* of them.
- **Honored cancellation:** where `KtxScanContext.signal` is available, the same
unit-boundary check honors it, so external cancellation stops the stage too.
On budget exhaustion or abort: stop scheduling new work, let in-flight queries
finish (each already bounded by spec 16), finalize with the relationships found so
far, and return a **partial** result — never an unbounded hang and never an
exception that would lose the checkpointed descriptions.
> Rejected alternative — per-table-scaled budget (N seconds × table count). It is a
> second formula to reason about and "more tables → more budget" partly re-opens the
> unbounded door this requirement closes. One flat, generous, project-level number
> matches how the other `scan.relationships` knobs are shaped and is enough for a
> best-effort stage whose partial output is durable and improvable (D4).
>
> Rejected alternative — a global `KTX_RELATIONSHIP_BUDGET_MS` env knob or a
> per-call override. One opinionated project-level default with a config override is
> the canonical ktx shape; no second runtime path.
### D4 — A budget-truncated partial is a successful, cached, completed stage
A graceful budget stop is **not** a failure. The relationship stage saves its
partial result like any completed stage (so a plain re-run resumes it for free, no
re-querying) and marks it `partial` with a reason in the relationship diagnostics
plus a recoverable scan warning. Because `detectionBudgetMs` lives in
`relationshipSettings ⊂ inputHash`, **raising the budget changes the content
identity and triggers a fresh, fuller run** — that is the only "try harder"
mechanism, with no extra flag or runtime path.
Distinguish the two stop kinds:
- **Process killed mid-stage** (crash / SIGKILL / supervisor): nothing is saved as
completed, so the next run recomputes the relationship stage (after resuming
descriptions/embeddings from cache via D2). This is the primary durability path.
- **Graceful budget/abort stop**: a partial *is* saved as completed-partial and
resumed cheaply on re-run, unless the budget is raised.
## Requirements
### 1. Checkpoint descriptions + embeddings before relationship detection
The descriptions and embeddings MUST be persisted into the durable, queryable
`_schema` manifest (and the raw enrichment artifacts) as soon as the last
non-relationship stage completes, before the `relationships` stage runs.
Relationship detection appends/merges its joins on completion. The expensive LLM +
embedding enrichment MUST be queryable even if the relationship stage subsequently
fails, is interrupted, is budget-truncated, or is skipped. A failed/partial/skipped
relationship stage MUST degrade to "no/partial joins," never to "no descriptions."
### 2. Stage resume resolves by content identity across runs
Completed-stage resolution MUST key on `(connectionId, stage, inputHash)`,
independent of `runId`, so re-running an interrupted ingest resumes the finished
`descriptions` / `embeddings` stages from cache and re-runs only what failed.
Re-running after an interruption MUST NOT re-issue LLM description or embedding
calls for stages that already completed. The resume cache MAY be recreated without a
migration bridge if its schema changes (it is disposable local state).
### 3. Relationship detection emits progress and honors a wall-clock budget
The relationship stage MUST emit per-unit progress through the existing progress
port (at minimum per-table during profiling and per-candidate during validation) so
liveness is observable. It MUST enforce a flat wall-clock budget
(`scan.relationships.detectionBudgetMs`, default 600_000 ms, project-level,
overridable, validated as a positive integer) checked at unit boundaries and layered
above spec 16's per-query deadline, and MUST honor `KtxScanContext.signal` where
available. On budget exhaustion or abort it MUST stop scheduling new work, finalize
with the relationships found so far, and return a partial result rather than running
unboundedly or throwing.
### 4. A budget-truncated relationship result is durable and marked partial
A graceful budget/abort stop MUST persist the partial relationship result as a
completed stage (so a plain re-run resumes it without re-querying) and MUST mark it
`partial` — in the relationship diagnostics artifact and as a recoverable scan
warning — so downstream consumers can see the joins are incomplete. Raising
`detectionBudgetMs` (which changes `inputHash`) MUST cause a fresh, fuller
relationship run; no separate flag is introduced for "redo." A process killed
mid-stage MUST NOT leave a completed record (so it recomputes on re-run).
### 5. No regression for small or uninterrupted ingests
A small or single-run ingest that is never interrupted MUST produce the same
artifacts and the same relationship output as today. The checkpoint write MUST be
idempotent with the final write (descriptions survive the join rewrite); the budget
default MUST be generous enough that normal and large-but-tractable schemas complete
relationship detection fully, hitting the budget only on pathological scale.
## Acceptance criteria
- **Durability across interruption:** interrupting an ingest **during** relationship
detection still leaves a queryable semantic layer carrying the table/column
descriptions + embeddings that were generated (verified: re-open the connection;
AI descriptions are present in `_schema`, not just native comments).
- **Resume does not re-spend:** re-running an interrupted ingest does **not**
regenerate descriptions/embeddings whose stage already completed (verified: no LLM
description calls and no embedding calls for the cached tables; only the failed
stage re-runs). Resolution is by `(connectionId, stage, inputHash)`, so the resume
survives a fresh `runId`.
- **Observable + bounded relationships:** a connection with hundreds of tables emits
relationship-stage progress (per-table profiling, per-candidate validation) and
completes within `detectionBudgetMs`; when the budget is hit, the stage stops
gracefully and persists the partial relationships found so far — without
discarding enrichment — marked `partial` in diagnostics and via a recoverable
warning.
- **Partial is cached and improvable:** re-running with an unchanged budget resumes
the partial relationship result from cache (no re-querying); raising
`detectionBudgetMs` triggers a fresh, fuller relationship run.
- **Budget validation:** `detectionBudgetMs` defaults to 600_000, honors a project
override, and rejects an invalid value (zero / negative / non-integer) as a clear
`ktx.yaml` config error.
- **No regression:** small/single-run ingests behave exactly as before — identical
artifacts and relationship output when nothing is interrupted; the checkpoint +
final writes leave descriptions intact alongside the generated joins.
## Non-goals
- **Bounding the descriptions stage's per-table LLM call.** Whether an individual
enrichment LLM call can wedge is a separate concern (already being addressed in the
working tree via a per-table enrichment timeout). This spec ensures whatever
descriptions *did* complete are durable; it does not own the per-call timeout.
- **Changing relationship-detection quality, thresholds, or the candidate/validation
algorithm.** The accept/review thresholds, scoring, and the existing
`validationBudget` count cap are unchanged; this spec adds durability,
cross-run resume, progress, and a time budget around them.
- **A per-connection or per-call relationship budget, or a global env override.**
One flat project-level `detectionBudgetMs`; no second runtime path (D3).
- **A new per-query timeout.** Spec 16 already bounds individual queries; this spec
composes above it and does not re-implement query-level deadlines.
- **Replacing the per-query deadline with the stage budget, or vice versa.** They
are independent and layered: a single query is bounded by spec 16; the stage's sum
is bounded by `detectionBudgetMs`.
- **A general checkpoint framework for every ingest stage.** The checkpoint is
specifically the descriptions+embeddings → queryable-manifest promotion before
relationships; it is not a generic per-stage artifact-flush abstraction.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns the
design.
- **Enrichment orchestration** — `context/scan/local-enrichment.ts`:
`runLocalScanEnrichment` (`:472`), the three `runEnrichmentStage` calls
(`descriptions` `:524`, `embeddings` `:553`, `relationships` `:587`),
`runEnrichmentStage` (`:413`) and its `findCompletedStage` lookup (`:427`). Add the
checkpoint hook after the last non-relationship stage; thread the progress port,
signal, and budget into the relationship stage.
- **Scan driver / write ordering** — `context/scan/local-scan.ts`: bare manifest
write (`:473`), enrichment call (`:492`, currently passing only
`{ runId, progress }` as `context` — wire `signal` through here too), terminal
`writeLocalScanEnrichmentArtifacts` (`:510`), and the enrichment-failure catch
(`:530`, which after D1 no longer loses descriptions). Supply the checkpoint
callback here.
- **Artifact writer** — `context/scan/local-enrichment-artifacts.ts`:
`writeLocalScanEnrichmentArtifacts` (`:310`), `writeLocalScanManifestShards`
(`:270`), and the description-preserving merge in `loadExistingManifestState`
(`:196`) — the basis for the additive checkpoint/final write.
- **Resume cache** — `context/scan/sqlite-local-enrichment-state-store.ts`:
`PRIMARY KEY (run_id, stage)` (`:83`), `findCompletedStage` (`:91`),
`saveCompletedStage` (`:117`). Re-key on `(connection_id, stage, input_hash)`,
pick latest `updated_at`, recreate the table if shape differs (disposable cache).
Lookup interface `KtxScanEnrichmentStageLookup` and `findCompletedStage`
in `context/scan/enrichment-state.ts` (`:10,46`); `computeKtxScanEnrichmentInputHash`
(`:78`).
- **Relationship stack (progress + budget + signal)** —
`context/scan/relationship-discovery.ts` (`discoverKtxRelationships` `:218`, accept
a progress port and budget/deadline + signal),
`context/scan/relationship-profiling.ts` (`profileKtxRelationshipSchema` `:320` —
per-table progress + budget check),
`context/scan/relationship-validation.ts` (`validateKtxRelationshipDiscoveryCandidates`
`:237` — per-candidate progress + budget check, alongside the existing
`validationBudget`),
`context/scan/relationship-composite-candidates.ts`
(`discoverKtxCompositeRelationships` `:515` — budget check).
- **Config** — `context/project/config.ts` `scan.relationships`
(`KtxScanRelationshipConfig`, `:171–213`): add `detectionBudgetMs` (positive
integer ms, default 600_000) to the zod schema and the default config builder.
- **Partial marker** — `context/scan/relationship-diagnostics.ts`
(`buildKtxRelationshipDiagnostics`, the profile/diagnostics artifact shape) carries
a `partial` flag + reason; add a recoverable warning code to the
`KtxScanWarningCode` union in `context/scan/types.ts` (e.g.
`relationship_detection_partial`).
- **Tests** — durability: a fixture ingest interrupted during the relationship stage
leaves AI descriptions in the queryable `_schema`. Resume: a second run with a
fresh `runId` and unchanged `inputHash` resolves the cached descriptions/embeddings
(assert no LLM/embedding calls) and re-runs only relationships. Budget: a schema
large enough (or a tiny `detectionBudgetMs` as the test seam) hits the budget,
emits per-unit progress, returns partial, persists it marked `partial`, and a
re-run resumes the partial; raising the budget re-runs. Resolver/config unit tests
for `detectionBudgetMs` (default / override / invalid). Regression: small
uninterrupted ingest yields identical artifacts and relationship output.
- After implementing, rebuild and re-link so the playground picks it up:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation, not a requirement)
The Spider 2.0-Lite BigQuery slice has datasets with hundreds–thousands of tables
(`ebi_chembl` 785, `fec` 486, `ga360` 366, …). Enriching them with claude-code
costs real, rate-limited LLM budget; losing that enrichment to a relationship-stage
interruption — and re-spending it on every retry — makes large-schema ingest
impractical, and an unbounded profiling stage runs past any supervising deadline and
is killed. This is a general durability/cost property of the ingest pipeline,
independent of the benchmark; the benchmark only made it acute at scale. Do not
encode any benchmark specifics in the implementation.
## Implementation notes
Implemented on branch `write-feature-spec-wiki` (ktx worktree `tallinn-v2`). All
four design decisions shipped; no deviations from the resolved design.
**D2 — resume by content identity** (`sqlite-local-enrichment-state-store.ts`,
`enrichment-state.ts`, `local-enrichment.ts`): the stage table is re-keyed to
`PRIMARY KEY (connection_id, stage, input_hash)`; `findCompletedStage` looks up by
`(connectionId, stage, inputHash)` ordered by `updated_at DESC` (most recent
content identity wins). `KtxScanEnrichmentStageLookup.runId` became `connectionId`;
`runId` stays on the row for diagnostics/`listRunStages`. The store drops and
recreates the table when the on-disk primary key differs (disposable cache, no
migration bridge), detected via `PRAGMA table_info`.
**D3 — observable + bounded relationship stage** (new
`relationship-detection-budget.ts`): a sticky `KtxRelationshipDetectionBudget`
(`check()`/`stopReason()`) built from `detectionBudgetMs` + `ctx.signal` + an
injectable `now`, plus `mapWithBudget` (a budget-aware concurrent map that
generalizes and replaces the old `mapWithConcurrency`). Threaded through
`discoverKtxRelationships` → profiling (per-table progress + budget stop),
validation (per-candidate progress + budget stop; budget-skipped candidates
degrade to the existing `validation_unattempted` review), and composite
detection (budget stops at PK-detection and coverage-probe boundaries).
`discoverKtxRelationships` now accepts `progress` and `now` and returns
`partial: { reason } | null`. The clock check fires only when work remains, so a
deadline elapsing after the last unit never marks a fully-processed stage partial.
**D1 — checkpoint before relationships** (`local-enrichment.ts`,
`local-enrichment-artifacts.ts`, `local-scan.ts`): `runLocalScanEnrichment` fires a
caller-supplied `onCheckpoint` once descriptions/embeddings complete and before
the relationship stage runs, gated on `shouldDetectRelationships` so the
no-relationship path keeps a single write. `local-scan.ts` supplies a callback
calling the new `writeLocalScanEnrichmentCheckpoint` (descriptions.json +
embeddings.json + manifest with descriptions and no generated joins — no
relationship artifacts, so no misleading empty diagnostics). The shared
description/embedding JSON writer was factored out so checkpoint and final writes
stay one implementation. `ctx.signal` is now threaded from `RunLocalScanOptions`
into the enrichment context (completing the existing `KtxScanContext.signal`
contract already read by the budget and the in-flight description timeout).
**D4 — partial is durable + marked** (`relationship-diagnostics.ts`,
`local-enrichment.ts`, `local-enrichment-artifacts.ts`): the diagnostics artifact
carries `partial` + `partialReason`; `runLocalScanEnrichment` pushes a recoverable
`relationship_detection_partial` warning (new `KtxScanWarningCode`) when truncated.
A graceful budget/abort stop returns normally, so the relationship stage saves as a
completed-partial record and resumes cheaply; a process killed mid-stage saves
nothing and recomputes. Raising `detectionBudgetMs` changes `inputHash`
(it lives in `relationshipSettings`), forcing a fresh, fuller run — the only
"try harder" mechanism, no extra flag.
**Config** (`config.ts`): `scan.relationships.detectionBudgetMs`, positive integer
ms, default `600_000`, validated like the other relationship fields. Documented in
`docs-site/content/docs/configuration/ktx-yaml.mdx`.
**Tests** (all green): budget unit tests (`relationship-detection-budget.test.ts`);
cross-run resume + table-recreate (`enrichment-state.test.ts`,
`local-enrichment.test.ts`); progress/budget/abort partial
(`relationship-discovery.test.ts`); partial persisted/resumed/re-run-on-raise +
checkpoint ordering + no-checkpoint-when-skipped (`local-enrichment.test.ts`);
end-to-end durability — a relationship-stage failure still leaves AI descriptions
in the queryable `_schema` (`local-scan.test.ts`); diagnostics partial flag
(`relationship-diagnostics.test.ts`); config default/override/invalid
(`config.test.ts`). `pnpm --filter @kaelio/ktx type-check`, `pnpm run dead-code`,
and `pnpm run build && pnpm run link:dev` all pass. (Pre-existing and unrelated:
three `analytics-skill-content.test.ts` markdown-structure assertions fail on this
branch from earlier analytics-skill commits — untouched here.)

533
spider2-specs/specs/20-resilient-enrichment-under-slow-llm.md
View file

@ -1,533 +0,0 @@
# Resilient enrichment under a slow/hung LLM backend
> Refined spec. Intake draft: `todo/20-resilient-enrichment-under-slow-llm.md`.
>
> **Scope: make the descriptions enrichment stage survive a hung LLM backend and
> an interrupted run.** Two compounding gaps live *inside* the per-table
> description-enrichment path: (1) the per-table LLM timeout fires in JS but does
> not terminate a wedged subprocess backend, so a hung table wedges the whole
> stage indefinitely; (2) descriptions are persisted only at full-stage
> completion, so any interruption discards every already-enriched table. This is
> the enrichment-stage analog of spec 16 (enforced query cancellation — a deadline
> that *stops the work*, not just abandons the promise) and spec 19 (move the
> durability boundary to the cost boundary so expensive LLM work is not lost). It
> composes with both rather than replacing them.
## Problem
Two compounding failure modes on the per-table description-enrichment path, both
confirmed in the current code and observed end-to-end together. Their union turned
a single hung table into an indefinite wedge *plus* total loss of an entire
stage's LLM work.
### 1. The per-table LLM timeout does not terminate the work
`KtxDescriptionGenerator.generateBatchedTableDescriptions`
(`context/scan/description-generation.ts`, the bounded call ~760–866) wraps the
per-table `this.llmRuntime.generateObject(...)` call in `retryAsync` with a fresh
`AbortSignal.timeout(KTX_ENRICH_LLM_TIMEOUT_MS)` per attempt (commit `01f63380`).
A fired timeout is surfaced as `KtxAbortedError` so it is **not** retried (one
wedge stays one timeout, not 3×). That is the correct policy — but the abort never
actually stops a subprocess backend, so the timeout is cosmetic.
The runtime is selected by the `backend` config field
(`context/llm/local-config.ts`, `KTX_LLM_BACKENDS =
['none','anthropic','vertex','gateway','claude-code','codex']`). Two backends spawn
a **child process the SDK owns** and to which ktx hands only an `AbortSignal`:
- **`codex`** (`@openai/codex-sdk`, via `context/llm/codex-runtime.ts` →
`codex-sdk-runner.ts`): the SDK runs `spawn(executable, args, { signal })`. Node's
`spawn` signal-option sends the child **SIGTERM** (not SIGKILL) on abort, and the
SDK consumes the child's stdout with `for await (const line of rl)`, re-throwing
the abort error **only after that loop ends**. A child wedged on a hung provider
socket survives SIGTERM → its stdout never closes → the readline loop never ends
→ the SDK never throws → ktx's `await generateObject` **never settles**, past the
per-attempt timeout, indefinitely. The child leaks (open provider connections,
~0% CPU).
- **`claude-code`** (`@anthropic-ai/claude-agent-sdk`, via
`context/llm/claude-code-runtime.ts`, `collectResult` ~275–322): on abort it calls
best-effort `queryResult.interrupt?.()` (errors swallowed) and only checks
`throwIfAborted` **between** streamed messages. A wedged child emits no message, so
the `for await (const message of queryResult)` loop blocks and the graceful
`interrupt()` may never land — the same hang class.
By contrast, **HTTP backends** (`anthropic`/`vertex`/`gateway`/`openai`, via
`context/llm/ai-sdk-runtime.ts`) pass `abortSignal` straight to the AI SDK's
`generateObject`, which cancels the underlying `fetch` natively — the await settles
promptly and there is no child to leak.
So ktx holds **no kill handle** on the subprocess backends, and SIGTERM is too
gentle for a wedged child. Spec 16's mechanism (ktx *itself* forks
`read-query-child` and `SIGKILL`s it) works precisely because ktx owns the fork —
which it does not here.
Observed (BigQuery ingest, codex backend, 2026-06-23): with
`KTX_ENRICH_LLM_TIMEOUT_MS=1800000` (30 min, an operator override), two of
`covid19_usa`'s 252-column tables hung; the stage sat at **268/285 for 41+
minutes** — well past the 30-min per-attempt timeout — with exactly two codex
children, each holding 3 ESTABLISHED connections at ~0% CPU, until killed by hand.
### 2. Descriptions are persisted only at full-stage completion
`generateDescriptions` (`context/scan/local-enrichment.ts` ~279–352) fans out
per-table work through `pLimit(DESCRIPTION_TABLE_CONCURRENCY)` (default 4) and
**accumulates every table's result in an in-memory `updates` array**, returned only
when the whole stage finishes. `runEnrichmentStage` (~413, ~421–474) then calls
`saveCompletedStage` (writing the whole-stage row to `local_scan_enrichment_stages`)
**after** `compute()` returns, and the spec-19 checkpoint write
(`writeLocalScanEnrichmentCheckpoint`, `local-enrichment-artifacts.ts` ~351–379,
fired by the `onCheckpoint` hook in `local-scan.ts`) also runs **only once the
descriptions stage completes**. There is no within-stage persistence: while the
stage runs, every enriched table's description lives only in memory.
So if the stage cannot complete — 2 of 285 tables hang (gap #1), or the process is
killed, or a supervising watchdog fires — **all** already-enriched tables are lost,
even though their (expensive, paid) LLM descriptions were finished. On the next run,
`findCompletedStage` finds no row, so the descriptions stage **recomputes from
scratch**.
Observed (same run): `covid19_usa` had **283/285** tables enriched in memory but
**0** rows in `local_scan_enrichment_stages` and **0** `ai:` descriptions on disk;
killing the wedged ingest discarded all 283, forcing a from-scratch re-ingest. The
cost of 2 pathological tables was 283 tables' worth of redone LLM calls.
Sharper still (re-ingest with a short, *enforced* timeout): even when the stage
**runs to the end** — the 2 hung tables hit their timeout and were skipped, so
**283/285** descriptions were generated and the ingest reported success (`Scan
completed` / `Ingest finished`, embeddings built, exit 0) — the descriptions were
**still persisted as 0** (0 `ai:` on disk, 0 stage rows). So the loss is **not**
only "discarded on kill": a stage that completes with *any* skipped/aborted table
threw away **every** successfully-generated description. The skip must be
**graceful** — a skipped table costs one missing description, not the entire stage's
output — which is the strongest argument for per-table incremental persistence: the
283 good descriptions should have been durable the moment each was produced.
The on-disk artifacts already carry everything needed to fix this *additively*: the
`_schema` manifest encodes per-table completion (a table with `descriptions.ai` is
AI-enriched), and rewrites preserve existing descriptions
(`mergeDescriptionsPreservingExternal`, `manifest.ts` ~96–115;
`loadExistingManifestState`, `local-enrichment-artifacts.ts` ~196–253 — the basis
spec 19 relies on). The durable record and the resume-skip set can be **derived from
the system's own on-disk state**, with no new cache schema.
## Generic use case (independent of any benchmark)
Anyone ingesting a large or wide schema with an LLM enrichment backend —
especially a **subprocess** backend, the common local/desktop setup — will
eventually hit a table whose description call hangs: a provider stall, a rate-limit
black-hole, a pathologically large prompt. Without an *enforced* timeout, one such
table wedges the entire ingest indefinitely and leaks the spawned child; without
*incremental* persistence, any interruption throws away all the per-table LLM work
already done — the dominant ingest cost. Both fixes make large-schema enrichment
**resilient and resumable**: a few bad tables degrade to a few skipped
descriptions, not a hung process and a from-scratch redo. This is core robustness
for a general-purpose ingestion product, wholly independent of any benchmark.
## Design decisions (resolved during refinement)
These resolve ambiguities the intake draft left open. They constrain the
implementer; the exact code is theirs (requirement-level, per the specs README).
### D1 — One bounded-call guarantee; enforcement follows the backend's nature
The canonical contract is a single guarantee for the per-table enrichment call:
**the in-flight work terminates and ktx's await settles within the per-table
deadline plus a small grace, on every backend.** How that guarantee is met follows
from a structural property of the configured backend — *does it own a subprocess?*
— not from a hand-maintained list of provider names:
- **Subprocess-backed (`codex`, `claude-code`):** the SDK's own abort is
insufficient (SIGTERM-only, and ktx has no kill handle), so ktx runs the call
behind a **boundary it can hard-kill** — a short-lived ktx-owned child process,
made a **process-group leader** (`detached`). The SDK's grandchild (the
`codex`/`claude` binary) inherits that group. On deadline (or `ctx.signal`), ktx
**tree-kills the whole group with SIGKILL** — reaping the wrapper *and* the
grandchild — and rejects promptly. This mirrors spec 16's child-process +
SIGKILL mechanism, extended by the critical step that **killing the immediate
child is not enough**: the grandchild would otherwise orphan to init and keep its
provider connections. Killing the group is the real fix.
- **HTTP-backed (`anthropic`/`vertex`/`gateway`/`openai`):** unchanged. The existing
in-process `abortSignal` → `fetch` cancellation already satisfies the contract —
the await settles promptly and there is no subprocess to leak. Routing these
through a subprocess would pay fork + IPC + credential-passing cost for no benefit.
> The branch on "subprocess-backed?" is behavior following from an input the backend
> declares about itself, not vendor enumeration — the same guarantee is reached two
> ways because the backends differ structurally. This matches the intake's own split
> ("subprocess SIGKILL for process-backed; request abort for HTTP-backed").
>
> Rejected alternative — a *settle-only race* (reject ktx's promise on the deadline
> regardless of the SDK, but leave the SDK's child running). It unwedges the stage
> but leaves the orphaned child holding provider connections — the exact leak the
> incident showed — so it fails the intake's "actually cancelled" requirement and
> compounds over a long ingest that hits several hung tables.
>
> Rejected alternative — a *persistent ktx subprocess pool* hosting the runtime,
> killed and respawned on timeout. Terminate-on-deadline destroys the worker, so a
> pool needs respawn + in-flight job-tracking for no benefit: the enrichment call is
> low-frequency relative to its own latency and already concurrency-bounded (4), so
> one short-lived child per call (spec 16's resolved choice) is simpler and as fast.
**Portability.** ktx supports Windows, where POSIX process groups and
`process.kill(-pgid, …)` do not exist. The tree-kill MUST be portable: a detached
process group + `kill(-pgid, 'SIGKILL')` on POSIX, and a tree-terminating
equivalent on Windows (e.g. `taskkill /pid <pid> /T /F` or a job object) so the
grandchild is reaped on every platform the subprocess backends run on.
### D2 — Default stays moderate and the retry/skip policy is unchanged
The per-table timeout default stays **120s** (`KTX_ENRICH_LLM_TIMEOUT_MS`), with the
existing per-attempt retry (`KTX_ENRICH_LLM_ATTEMPTS`, default 3) and the
no-retry-on-timeout policy. A hung table costs **at most one timeout**, then the
table is skipped with the existing `enrichment_timeout` warning and the stage
proceeds. The 30-min value in the incident was an operator stopgap chosen *because*
the timeout was cosmetic; once D1 makes the timeout actually terminate the work, a
long timeout is strictly worse for a hang (a hang costs the full timeout), so the
moderate default is the correct operating point. The retry loop stays in
`description-generation.ts`: each attempt runs through the bounded boundary (D1), so
a transient backend error retries while a timeout surfaces as `KtxAbortedError` and
does not.
> Not introducing a new `ktx.yaml` config field for the timeout. The existing env
> override is the tuning seam; adding a per-connection/per-call/global knob would
> multiply the runtime surface for no stated need (one opinionated default + the
> existing env override is the canonical ktx shape).
### D3 — Persist descriptions incrementally; derive the resume-skip set from on-disk state
During the descriptions fan-out, flush completed tables **per batch** (every N
tables / on a timer, at a cadence that bounds the at-risk window) to the durable
on-disk artifacts, reusing spec 19's additive write:
- the raw descriptions artifact (`descriptions.json`) is the **resume-skip source**;
- the `_schema` manifest is updated additively (`mergeDescriptionsPreservingExternal`
preserves prior `ai:`/`db:`/external keys) so finished descriptions are also
**queryable** the moment they are computed — the spec-19 invariant, one level
deeper. The implementer MAY bound manifest-rewrite cost on huge schemas by
rewriting only changed shards.
On resume, `generateDescriptions` reads the existing record, **skips any table
already enriched**, computes only the remainder, and returns the merged full set so
the embeddings stage, the checkpoint write, and the stage-store row all see a
complete result exactly as today.
**The skip is `inputHash`-gated**, preserving spec 19's recompute semantics. The
durable record is tagged with the descriptions stage's `inputHash`
(`computeKtxScanEnrichmentInputHash`). Resume reuses it to skip tables **only when
the current `inputHash` matches** — a genuine resume-after-interruption of the same
content identity. A changed `inputHash` (schema or enrichment settings changed)
ignores the prior record for skipping and recomputes the stage as today; the
manifest write stays additive regardless. The artifact's on-disk shape may gain the
`inputHash` tag with **no migration bridge** (ktx owns the artifact; a stale-shaped
record simply forces one non-incremental run), consistent with ktx's
no-backward-compatibility policy.
> The skip set is **derived from the artifacts ktx already writes**, not from a new
> per-table cache table. The manifest's `ai:` field already encodes "this table is
> enriched"; a parallel per-table SQLite record would be a second source of truth for
> the same fact and would drift. The whole-stage `local_scan_enrichment_stages` row is
> still written at stage completion (it remains the stage-level resume gate — a clean
> re-run skips the descriptions stage as today); the incremental record only matters
> when the stage did **not** complete — exactly the case where no row exists and
> `compute()` re-runs.
### D4 — A killed-mid-stage run is durable; resume is cheap
A process killed mid-stage (gap #1 wedge, SIGKILL, crash, supervisor) leaves the
per-batch-flushed tables durable on disk. The next run resumes the descriptions
stage (no completed `local_scan_enrichment_stages` row → `compute()` runs again),
but `generateDescriptions` now **re-issues LLM calls only for the unfinished
tables**. A failed/skipped table (timeout or exhausted retries) is left for the
remainder set and is retried on the next resume — never silently treated as done.
## Requirements
### 1. The per-table enrichment timeout is enforced for subprocess backends
When the per-table deadline fires (or `ctx.signal` aborts) on a subprocess-backed
backend (`codex`, `claude-code`), the in-flight LLM work — the spawned child **and
its descendants** — MUST be terminated (SIGKILL of the process group / tree), and
ktx's `generateObject` await MUST settle within the deadline plus a small bounded
grace. A hung table MUST cost at most ~one timeout of wall-clock, never unbounded.
The termination MUST be portable across the platforms the subprocess backends run on
(POSIX process-group kill and a Windows tree-kill equivalent). HTTP-backed backends
keep their existing native `abortSignal` → `fetch` cancellation; the guarantee is one
contract met two ways, branching on the backend's structural "owns a subprocess"
property, not on a list of provider names.
### 2. The timeout default and retry/skip policy are unchanged
The default per-table timeout stays moderate (current 120s, `KTX_ENRICH_LLM_TIMEOUT_MS`),
with the existing per-attempt retry (default 3, `KTX_ENRICH_LLM_ATTEMPTS`) and the
no-retry-on-timeout policy. On timeout, the table is skipped with the existing
`enrichment_timeout` recoverable warning and the stage proceeds. No new
per-connection / per-call / global timeout knob is added.
### 3. Descriptions are persisted incrementally during the stage
Enriched descriptions MUST be flushed to the durable on-disk artifacts **per batch**
(per-table or per-N-tables / on a timer) during the descriptions stage, at a cadence
that bounds the at-risk window to a small number of tables. The flush MUST be
idempotent and additive (never clobber a prior `ai:` description; preserve `db:` and
external keys via the existing merge). Finished tables MUST remain durable even if the
stage never completes — is wedged, killed, or interrupted. A failed/skipped
relationship/embedding stage or a killed descriptions stage MUST NOT lose the
descriptions already flushed.
### 4. Resume re-enriches only the unfinished tables
On a resumed ingest with an unchanged `inputHash`, the descriptions stage MUST
re-issue LLM description calls **only for tables not already enriched**, deriving the
already-enriched set from the on-disk artifacts (the `inputHash`-tagged durable
record / the manifest's `ai:` descriptions), and MUST return the merged full result
so downstream stages behave as on a fresh run. A changed `inputHash` (schema or
enrichment settings changed) MUST recompute the stage as today (spec 19's
inputHash-gated semantics preserved). The durable record MAY be recreated without a
migration bridge if its on-disk shape changes (it is regenerable local/artifact
state).
### 5. No regression for small or uninterrupted ingests
A small or single-run ingest that is never interrupted MUST produce the same
artifacts (descriptions, manifest, embeddings) as today. The incremental flush MUST
be idempotent with the spec-19 checkpoint and the terminal write (descriptions
survive the embeddings/relationship rewrites). The bounded-call boundary MUST NOT
change a normal successful enrichment's output, only how a wedged call is terminated.
### 6. A skipped table costs one description, never the stage's output
A descriptions stage that **completes** with one or more skipped/aborted tables MUST
persist every successfully-generated description (the durable record and the `ai:`
manifest entries) and MUST mark the stage completed (a `local_scan_enrichment_stages`
row, embeddings + downstream proceeding) — it MUST NOT discard the whole stage's
output because some tables were skipped. No single table's failure may reject the
per-table fan-out: a per-table failure degrades to one missing description (left for
the resume remainder), not a failed stage. A genuine `ctx.signal` cancellation is the
only thing that fails the stage (so it resumes), and even then the already-flushed
descriptions remain durable.
## Acceptance criteria
- **Enforced timeout (subprocess backend):** a subprocess-backed enrichment call
that hangs past the deadline is terminated within the deadline plus a small grace;
ktx's await settles, the spawned child **and a grandchild it spawned** both exit
(verified via the child's `exit`, not left spinning), and the table is skipped with
an `enrichment_timeout` warning. The stage advances rather than wedging. A
`ctx.signal` abort terminates the same way.
- **HTTP backend unaffected:** an HTTP-backed enrichment call still cancels promptly
on abort via the existing native path, with no subprocess involved.
- **Default + policy:** the default timeout is 120s and a timeout is not retried (one
wedge = one timeout); a transient error is still retried up to the attempt limit.
- **Graceful skip persists the rest:** a stage that completes with one table failing
(timeout, exhausted retries, or an unexpected throw) still writes the other N−1
descriptions to the durable record + `ai:` `_schema` and marks the stage completed
(a `local_scan_enrichment_stages` row exists); the failed table is a single `null`
description left for the resume remainder, not a discarded stage.
- **Incremental durability:** interrupting the descriptions stage after K of N tables
leaves those K durable on disk (raw artifact + `ai:` descriptions in `_schema`),
with no completed `local_scan_enrichment_stages` row.
- **Resume does not re-spend:** re-running the interrupted ingest (unchanged
`inputHash`, fresh `runId`) issues **no** LLM description calls for the K already-
enriched tables and enriches only the remaining N−K; the returned result is the
full merged set. A changed `inputHash` recomputes the stage.
- **No regression:** a small uninterrupted ingest yields identical artifacts and the
same descriptions/embeddings output as today; the incremental flush is idempotent
with the checkpoint and terminal writes.
## Non-goals
- **Incremental persistence of embeddings.** Embeddings are fast and already covered
by spec 19's stage-level cross-run resume; the dominant loss is descriptions. This
spec scopes incremental persistence to the `descriptions` stage.
- **Changing the timeout default, retry counts, or adding a timeout config knob.**
D2 keeps the moderate default and the single env tuning seam.
- **Routing HTTP backends through the subprocess boundary.** Their native abort
already meets the contract; a subprocess would add cost and a credential-passing
surface for no benefit.
- **A persistent subprocess pool.** One short-lived ktx child per subprocess-backed
call; no pool, no respawn/job-tracking (D1).
- **Re-implementing spec 16 (per-query deadline) or spec 19 (relationship-stage
budget, cost-boundary checkpoint, cross-run stage resume).** This spec composes
above them: spec 16 bounds individual queries, spec 19 makes whole stages durable
and resumable, and this spec hardens the per-table enrichment call's termination
and adds within-stage description durability.
- **A general per-stage incremental-flush framework.** The incremental flush is
specifically the descriptions stage; it is not a generic abstraction over every
enrichment stage.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns the
design.
- **Bounded per-table call (gap #1)** — `context/scan/description-generation.ts`,
`KtxDescriptionGenerator.generateBatchedTableDescriptions` (the bounded+retry block
~760–866; `enrichTimeoutMs` ~769, `enrichAttempts` ~770, `KtxAbortedError` on
timeout ~811, `enrichment_timeout`/`enrichment_failed` warnings ~858). The retry
loop stays here; each attempt runs through the kill boundary for subprocess
backends.
- **LLM runtime + backend selection** — `context/llm/runtime-port.ts`
(`KtxLlmRuntimePort.generateObject`, `abortSignal` on the input),
`context/llm/local-config.ts` (~127–163, selects `CodexKtxLlmRuntime` /
`ClaudeCodeKtxLlmRuntime` / `AiSdkKtxLlmRuntime`), `context/project/config.ts`
(`KTX_LLM_BACKENDS`). The "owns a subprocess" property should be declared by the
backend/runtime (e.g. on the runtime interface), not inferred from a name list.
- **Subprocess backends** — `context/llm/codex-runtime.ts` +
`context/llm/codex-sdk-runner.ts` (`CodexSdkCliRunner.runStreamed`, the SDK's
`spawn(executable, args, { signal })` is in `@openai/codex-sdk`),
`context/llm/claude-code-runtime.ts` (`collectResult` ~275–322, the `interrupt()`
abort path). These are what the kill boundary must wrap and tree-kill.
- **Reuse spec 16's mechanism (extended to group/tree kill)** —
`connectors/sqlite/read-query-child.ts` (the forked child shape) and
`connectors/sqlite/connector.ts` `runReadQueryOffProcess` (~292–350: `fork`,
deadline timer, `child.kill('SIGKILL')`, `settle()`, the `.js`-if-exists-else-`.ts`
child-URL resolver ~25–27, knip dynamic entry). Gap #1 differs by making the child a
process-group leader and killing the **group/tree** (the SDK grandchild), portably.
Abort helpers: `context/core/abort.ts` (`createAbortError`, `throwIfAborted`,
`linkAbortSignal`). Note the new child hosts an LLM runtime, so the implementer owns
passing the backend config/credentials to it (env/IPC) and serializing the
structured result back.
- **Incremental persistence (gap #2)** —
`context/scan/local-enrichment.ts` (`generateDescriptions` ~279–352: the per-table
`pLimit` fan-out and the in-memory `updates` accumulation; `runEnrichmentStage`
~413/~421–474 with `findCompletedStage` ~427 and `saveCompletedStage`; the
`onCheckpoint` hook ~598–612). Make `generateDescriptions` resume-aware: read the
existing record, skip already-enriched tables, flush per batch, return the merged
full set.
- **Artifact writer + additive merge** — `context/scan/local-enrichment-artifacts.ts`
(`writeLocalScanEnrichmentCheckpoint` ~351–379, `writeEnrichmentDescriptionArtifacts`
with `descriptions.json` ~316, `writeLocalScanManifestShards` ~270–308,
`loadExistingManifestState` ~196–253, `tableDescription`/`columnDescription`
~75–105); `context/scan/manifest.ts` (`mergeDescriptionsPreservingExternal` ~96–115,
`SCAN_MANAGED_DESCRIPTION_KEYS`). Factor a per-batch flush that reuses the additive
description/manifest write; tag the durable record with `inputHash`.
- **Stage store + input hash** —
`context/scan/sqlite-local-enrichment-state-store.ts` (`STAGES_TABLE =
'local_scan_enrichment_stages'`, PK `(connection_id, stage, input_hash)`,
`findCompletedStage`, `saveCompletedStage`),
`context/scan/enrichment-state.ts` (`computeKtxScanEnrichmentInputHash` ~78). The
whole-stage row stays; the `inputHash` is the gate for the resume-skip set.
- **Scan driver** — `context/scan/local-scan.ts` (the `onCheckpoint` wiring and the
terminal `writeLocalScanEnrichmentArtifacts`), and `KtxScanContext.signal`
(`context/scan/types.ts`) which the kill boundary must honor.
- **Tests** — gap #1: a fake subprocess-backed runtime whose child hangs (ignores
SIGTERM) is killed at a tiny test-seam deadline; assert the await settles within
deadline+grace, the child and a spawned grandchild both exit, and the table is
skipped with `enrichment_timeout`; assert an HTTP-backed abort still settles via the
native path. gap #2: interrupt the descriptions stage after K/N tables (a flush
seam), assert the K are durable (raw artifact + `ai:` in `_schema`) with no completed
stage row; a resume with matching `inputHash` issues no LLM calls for the K and
enriches only N−K; a changed `inputHash` recomputes; regression: a small
uninterrupted ingest yields identical artifacts.
- After implementing, rebuild and re-link so the playground picks it up:
`pnpm run build && pnpm run link:dev`.
## Benchmark context (motivation, not a requirement)
Surfaced during the Spider 2.0-Lite **BigQuery** ingest (2026-06-23, codex enrichment
backend). Re-enriching the giant public datasets, `covid19_usa` wedged at 268/285 for
41+ minutes on 2 hung 252-column tables; the 30-min per-table `AbortSignal` timeout
never killed the hung codex children, and because descriptions checkpoint only at
stage completion, the 283 already-enriched tables were unrecoverable — the operator
had to kill, cache-bust, and re-ingest the database from scratch (with a short timeout
as a stopgap). The benchmark merely exercised a large/wide multi-dataset ingest at
scale; the gaps and the fixes are generic production hygiene for any agent that
enriches a real warehouse with a subprocess LLM backend. Do not encode any benchmark
specifics in the implementation.
## Implementation notes
Implemented on branch `write-feature-spec-wiki`. Both gaps shipped; all acceptance
criteria are covered by tests. The full ktx test surface for the touched code is
green (the only failures in the whole suite are 3 pre-existing assertions in
`test/skills/analytics-skill-content.test.ts` about the analytics SKILL.md markdown
— an unrelated subsystem this change does not touch).
### Gap #1 — enforced timeout for subprocess backends
- **Structural property on the runtime, not a name list.** Added
`subprocessForkSpec(): SubprocessRuntimeForkSpec | null` to `KtxLlmRuntimePort`
(`context/llm/runtime-port.ts`). `CodexKtxLlmRuntime` / `ClaudeCodeKtxLlmRuntime`
return a serializable `{ backend, projectDir, modelSlots }`; `AiSdkKtxLlmRuntime`
(and the deterministic stub) return `null`. The per-table call branches on this,
never on a vendor list (D1).
- **Shared structured core.** Both subprocess runtimes gained
`generateStructuredJson(jsonSchema)` (returns the raw object; the caller
Zod-validates). Their existing `generateObject` was refactored to delegate to the
same streaming core, so structured generation has one implementation.
- **Kill boundary.** New `context/llm/subprocess-generate-object.ts`
(`runGenerateObjectInSubprocess`, `KtxSubprocessDeadlineError`) forks a ktx-owned
child (`subprocess-generate-object-child.ts`) **detached** (process-group leader);
the SDK's model binary inherits the group. On the deadline or `ctx.signal`, ktx
tree-kills the group with `SIGKILL` (`process.kill(-pid, …)` on POSIX,
`taskkill /pid <pid> /T /F` on Windows) and rejects promptly; on success the raw
output is Zod-validated. Credentials reach the child via inherited `process.env`
(the runtimes re-derive their allowlisted env), never over IPC.
- **Wiring.** `KtxDescriptionGenerator.generateBatchedTableDescriptions`
(`context/scan/description-generation.ts`) routes each retry attempt through the
boundary for subprocess backends and keeps the native `AbortSignal` → `fetch`
path for HTTP backends. A fired deadline maps to the existing
`KtxAbortedError`/`enrichment_timeout` no-retry policy (one wedge = one timeout);
default stays 120s (D2).
- **Tests.** `test/context/llm/subprocess-generate-object.test.ts` forks a real
fixture child that spawns a grandchild and ignores SIGTERM, and asserts the
deadline/abort tree-kills both (the grandchild PID is reaped) and the await
settles within deadline+grace; plus success / schema-failure / child-error paths.
`test/context/scan/description-generation.test.ts` adds the generator-level
timeout-skip and the "HTTP backend spawns no child" cases.
### Gap #2 — incremental descriptions persistence + resume
- **Durable record + resume store.** `createKtxScanDescriptionResumeStore`
(`context/scan/local-enrichment-artifacts.ts`) writes the descriptions-so-far to
a durable record (inputHash-tagged) and **only the manifest shards that gained a
table this batch** (new `onlyChangedTableNames` filter on
`writeLocalScanManifestShards`, additive merge preserved). `load(inputHash)`
returns the prior enriched set only on a matching inputHash (D3).
- **Resume-aware fan-out.** `generateDescriptions` (`context/scan/local-enrichment.ts`)
loads the prior record, skips already-enriched tables, enriches only the
remainder, flushes every `DESCRIPTION_FLUSH_EVERY` (10) completed tables (a single
in-flight flush; the final force-flush drains the tail), and returns the full
merged set (recovered + fresh + `null` for still-failed, so failures are retried,
D4). Wired through `local-scan.ts` (store constructed when not `--dry-run`).
- **Graceful-skip backstop (requirement 6).** The per-table worker wraps the call in
a try/catch: any non-cancellation failure degrades to one `null` description + an
`enrichment_failed` warning and the fan-out continues, so no single table can
reject `Promise.all` / abort the stage. This makes the "one skipped table costs one
description, not the stage's output" guarantee live at the stage boundary
(`generateBatchedTableDescriptions` already degrades its own failures; this is the
explicit backstop). A `ctx.signal` cancellation still propagates (the stage fails
and resumes), and the already-flushed descriptions stay durable. This closes the
field bug where a completed-with-skips stage persisted 0 descriptions / 0 stage rows.
- **Deviation from the spec's literal path (necessary correction).** The durable
record lives at a **stable, non-`syncId`** path
(`raw-sources/<connectionId>/live-database/enrichment-progress/descriptions.json`),
not the `syncId`-scoped `…/<syncId>/enrichment/descriptions.json` the spec named.
Reason: a from-scratch interruption (the incident's exact case — no prior
*completed* run) gets a **fresh `syncId`** on the next run
(`buildSyncId` in `context/ingest/local-stage-ingest.ts`), so a `syncId`-scoped
record would be unreachable on resume. The manifest is already at the stable
per-connection scope (`semantic-layer/<connectionId>/_schema/`), so this keeps the
resume source at the same stable scope. The `syncId`-scoped `enrichment/descriptions.json`
debug artifact written by the terminal/checkpoint writers is unchanged.
- **Tests.** `test/context/scan/description-resume.test.ts` drives
`runLocalScanEnrichment` against a real git-backed project: a fresh run flushes a
durable record + `ai:` manifest descriptions; a matching-`inputHash` resume issues
zero LLM calls and returns the full merged set; a partial record re-enriches only
the missing tables; a changed `inputHash` recomputes; the changed-shard filter
rewrites only the affected shard; and (requirement 6) a run where one table fails
still persists the other tables (durable record + `ai:`) and **completes the stage**
(a completed `local_scan_enrichment_stages` row), with the failed table left `null`
for resume.
### Incidental
- Fixed a stale assertion in `description-generation.test.ts` ("does not run
per-column fallback…" expected 1 call) to `3`, matching the retry policy added in
commit `01f63380` (D2 / acceptance: a transient error retries up to the attempt
limit). The HTTP path is unchanged; the assertion simply predated the retry.
- No new `ktx.yaml` config field or runtime knob was added (D2). The rate-limit
governor is not wired into the scan-enrichment path, so the kill-boundary child
loses no pacing.
- Rebuilt and re-linked (`pnpm run build && pnpm run link:dev`); the child compiles
to `dist/context/llm/subprocess-generate-object-child.js`.

567
spider2-specs/specs/21-selective-enrichment-stages.md
View file

@ -1,567 +0,0 @@
# Selective enrichment stages (`--stages`) + per-stage cache keys
> Refined spec. Intake draft: `todo/21-selective-enrichment-stages.md`.
>
> **Scope: make the three enrichment stages independently invalidatable and
> independently re-runnable.** Today one coarse cache key gates all three stages,
> so changing any one stage's inputs re-pays for every stage — most painfully the
> expensive per-table `descriptions`. And there is no CLI surface to re-run a
> chosen subset. This spec splits the key per stage (so a change invalidates only
> the stage it touched) and adds a `--stages` flag that force-re-runs a chosen
> subset while preserving the others. It is the operability follow-on to spec 19
> (durable, cross-run stage resume) and spec 20 (resilient, per-table-resumable
> descriptions); it composes with both rather than replacing them.
## Problem
Enrichment has three stages — **`descriptions`** (one paid LLM call per table),
**`embeddings`** (sentence-transformer vectors over the schema + descriptions),
**`relationships`** (FK/join detection, optionally LLM-proposed). After specs 19
and 20 these stages are durable and resumable, but they are still **coupled for
cache invalidation and unreachable for selective re-run**. Three facts make a
targeted re-run impossible without a full, expensive re-enrich.
### 1. One coarse cache key gates all three stages
`runLocalScanEnrichment` (`context/scan/local-enrichment.ts:611`) computes a single
`inputHash` from `{ snapshot, mode, detectRelationships, providerIdentity,
relationshipSettings }` and every stage reuses it — `descriptions` (~`:642`),
`embeddings` (~`:673`), `relationships` (~`:729`). `providerIdentity` itself
(`localScanProviderIdentity`, `local-scan.ts:241–255`) is one blob conflating the
description LLM identity, the embedding model/dimensions/batch size, **and** the
whole relationship config — and it redundantly re-encodes `mode` and
`relationships`, which the coarse hash already mixes in.
The consequence: flipping `scan.relationships.llmProposals`, switching the LLM
backend, or upgrading the embeddings model changes the **one** hash and so
invalidates **all three** stages. ktx then re-runs the expensive per-table
`descriptions` even though they did not conceptually change. The headline cost of
the system — paid LLM description calls — is thrown away on any unrelated
enrichment-config edit.
### 2. No CLI surface to select stages
The enrichment internals already support a relationships-only path
(`KtxScanMode` `'relationships'`, `types.ts:12` — `descriptions`/`embeddings` are
gated on `mode === 'enriched'` at `local-enrichment.ts:632`, while
`shouldDetectRelationships` admits `mode === 'relationships'` at `:624–626`). But
`ktx ingest` hardcodes `mode: 'enriched'` (`public-ingest.ts:973`) and exposes no
flag to select a subset (`ingest-commands.ts:26–49` — only `--no-query-history`
and friends). The relationships-only capability is built but unreachable, and there
is no way at all to ask for "descriptions only" or "embeddings only."
### 3. The foundation for "touch one stage, keep the rest" already exists
The per-stage store `local_scan_enrichment_stages` is keyed
`(connection_id, stage, input_hash)` (spec 19) and the descriptions write is
additive — `mergeDescriptionsPreservingExternal` (`manifest.ts`) and
`loadExistingManifestState` (`local-enrichment-artifacts.ts`) preserve prior `ai:`,
`db:`, and external description keys on rewrite; spec 20's per-table resume record
(`createKtxScanDescriptionResumeStore`, `local-enrichment-artifacts.ts:286`) already
re-issues LLM calls only for the still-failed tables. So "recompute one stage, leave
the others byte-for-byte" needs only two missing pieces: **per-stage key
granularity** and a **CLI surface** to select stages.
**Requirement:** let an operator re-run a chosen subset of enrichment stages on an
already-ingested connection, recomputing only those stages, preserving the others'
artifacts untouched, and **re-paying only for what genuinely changed** — never
re-running the costly `descriptions` because an unrelated stage's inputs moved.
## Generic use case (independent of any benchmark)
Any team running ktx in production maintains its semantic layer over time: they
improve the description prompt or switch the description LLM, upgrade the embeddings
model, or turn on LLM-proposed joins. Today each of those forces a **full re-enrich
of every connection** — re-running the expensive per-table descriptions even when
only embeddings or relationships changed. Two routine operations should be cheap and
targeted:
- **"Re-embed everything on the new model."** Swapping the embeddings model should
recompute only embeddings, leaving descriptions and joins on disk.
- **"Backfill joins now that `llmProposals` is on."** Enabling LLM-proposed
relationships should recompute only relationships.
And one operation needs an explicit trigger because no input changed:
- **"These descriptions came out thin — re-run them with a longer timeout."** A
connection whose description coverage is poor because tables timed out (same
snapshot, same LLM, so the hash is unchanged) should be re-runnable on demand,
cheaply retrying only the tables that failed.
This is core operability for a long-lived ingestion product and is wholly
independent of any benchmark.
## Design decisions (resolved during refinement)
These resolve ambiguities the intake draft left open. They constrain the
implementer; the exact code is theirs (requirement-level, per the specs README).
### D1 — Split the coarse hash into three per-stage input hashes
Replace the single `computeKtxScanEnrichmentInputHash` call with **per-stage** hash
computation, each keyed on only that stage's own inputs. Decompose the
`localScanProviderIdentity` blob into the slices each stage actually depends on:
- **`descriptions`** → `{ snapshot, llmIdentity }`, where `llmIdentity` is the
description-LLM identity (`llm.models.default`, `baseUrlConfigured`). **Not** the
embedding model/dimensions/batch size, **not** relationship settings.
- **`embeddings`** → `{ snapshot, embeddingIdentity, descriptionDigest }`, where
`embeddingIdentity` is `{ model, dimensions, batchSize }` and `descriptionDigest`
is a stable digest of the resolved description text the embeddings consume (the
same text `buildEmbeddings` → `buildKtxColumnEmbeddingText` feeds the model,
`local-enrichment.ts:466–486`, `embedding-text.ts:17–44`). This content-addresses
embeddings on their real upstream (D4).
- **`relationships`** → `{ snapshot, relationshipSettings (incl. `llmProposals` and
`detectionBudgetMs`), llmIdentity }`. **Not** the description content (decision X,
D5), **not** the embedding identity.
`mode` and `detectRelationships` drop out of the per-stage inputs: each stage
produces output under exactly one mode, so the stage name already scopes that, and
re-mixing `mode` only re-couples the keys. After the split, flipping `llmProposals`
invalidates only `relationships`; swapping the embeddings model invalidates only
`embeddings`; switching the description LLM invalidates only `descriptions`.
The per-stage hash becomes the key everywhere a single hash is used today: the
`local_scan_enrichment_stages` lookup/save in `runEnrichmentStage`, and the spec-20
descriptions resume record (`createKtxScanDescriptionResumeStore`), which is now
keyed on the **descriptions** stage's hash — so changing the embedding model no
longer busts the descriptions resume record, a strict improvement.
> **No migration bridge.** The stage store and the descriptions resume record are
> disposable local `.ktx` state (regenerable from a fresh ingest). The new per-stage
> keys simply miss the old coarse-keyed rows, forcing one full re-enrich on the next
> run after upgrade. Recreate/ignore stale-shaped records with no compatibility
> shim, consistent with specs 19/20 and ktx's no-backward-compatibility policy.
### D2 — `--stages <comma-list>` selects a subset; one gate, no new mode
Add `ktx ingest [connectionId] --stages <comma-list>`, a non-empty subset of
`descriptions,embeddings,relationships`. Plural because it takes a **set**:
`--stages relationships` and `--stages descriptions,embeddings` both read naturally,
and the plural signals "list expected." Flag absent = all three (today's behavior).
A Commander custom parser validates each name against the canonical stage registry
and parses into an ordered, de-duplicated set. **An unknown or empty stage name is a
hard `InvalidArgumentError`** — never silently ignored. The set threads CLI →
`runKtxPublicIngest` (`KtxScanArgs`) → `runLocalScan` → `runLocalScanEnrichment`.
Inside enrichment the run set is **`(mode/provider-eligible stages) ∩ (selected
stages)`** — a single gate. Each existing stage block additionally checks
membership in the selected set (`descriptions`/`embeddings` already gate on
`mode === 'enriched'` + providers; `relationships` on `shouldDetectRelationships`).
This adds **no** new `KtxScanMode` variant and **no** second parallel selection
path; `mode` keeps meaning "the connection's enrichment level," and `--stages` means
"which of those stages to (re)compute this run." A named stage that cannot run
because a prerequisite is absent (e.g. `--stages embeddings` with no embedding
provider configured) MUST fail or warn clearly, never silently no-op.
> Rejected alternative — repurpose `mode` (`--stages relationships` →
> `mode: 'relationships'`). It only expresses single-stage cases, leaves
> `descriptions,embeddings` with no mode, and creates two ways to say "relationships
> only." The explicit stage set is the one canonical selector.
### D3 — A named stage force-re-runs; per-table resume still avoids re-paying
Naming a stage in `--stages` carries the intent "recompute this," so a named stage
**re-enters its `compute()`, bypassing the spec-19 completed-row short-circuit** in
`runEnrichmentStage` (`local-enrichment.ts:538–547`). The spec-20 machinery still
applies **inside** `compute()`:
- `--stages descriptions` re-enters `generateDescriptions`, which loads the
per-table resume record and re-issues LLM calls **only for the still-null/failed
tables** (when the descriptions hash is unchanged) — the "fill thin coverage with
a longer `KTX_ENRICH_LLM_TIMEOUT_MS`" case, paying only for the gaps.
- A genuine input change (e.g. switching the LLM → a new descriptions hash)
invalidates the resume record and rebuilds the stage fully, as today.
Stages **not** named are skipped entirely — not run, not resumed — and their
on-disk artifacts are left exactly as they are (additive write; preserve-others is
already the behavior). The **no-flag default is unchanged**: all eligible stages
run, the completed-row short-circuit is respected (spec-19 cross-run resume).
Behavior follows from the input (did you explicitly name the stage?), not the call
path. A consequence to state plainly: `--stages descriptions,embeddings,relationships`
is **not** identical to passing no flag — naming all three is the explicit "force a
full enrichment recompute," whereas no flag is "ingest, resuming whatever is done."
### D4 — Downstream staleness: one real edge, content-addressed, surfaced not silent
The only hard dependency between stages is **`descriptions → embeddings`**
(embeddings embed the description text; `relationships` is decoupled, D5). Two
mechanisms keep it correct without a hardcoded dependency table:
- **Self-healing via content-addressing.** Because the embeddings hash includes
`descriptionDigest` (D1), re-running `descriptions` changes that digest, so a
later embeddings run (or a full ingest) sees a hash miss and recomputes — stale
embeddings can never silently persist across a future embeddings run. (Without
this, the embeddings hash would be unchanged after a description edit and a later
run would wrongly short-circuit on stale vectors.)
- **Surfaced immediately.** After a selective run, for each **unselected** stage that
has artifacts on disk, recompute its *current* per-stage hash from on-disk state
and compare it to the stored completed-row hash; if they differ, emit a
**recoverable `enrichment_stage_stale` warning** naming the stale stage and the
cascade command (e.g. `--stages descriptions,embeddings`). This is derived from the
system's own state — it also catches "you changed the embedding model in `ktx.yaml`
but only ran `--stages descriptions`."
The run **never silently leaves a stale-but-unflagged downstream**, and **never
silently auto-cascades** extra work — the operator is told and decides. Re-running
`descriptions` does **not** flag `relationships` stale (D5).
### D5 — Relationships are decoupled from description content, but still get it as context
`relationships` keys on `{ snapshot, relationshipSettings, llmIdentity }` and is
**not** invalidated or stale-flagged by a description change (decision X). Rationale:
relationships are the low-value, best-effort, expensive-to-probe stage (spec 19's
own framing); coupling them to description content would make every routine
description re-run also invalidate joins — re-opening the exact over-invalidation
this spec exists to close.
Independently, a `relationships`-only run (descriptions stage not running this
invocation) MUST **hydrate its working schema from the persisted on-disk enriched
`_schema`** (AI descriptions + embeddings) so `llmProposals` runs with full
description context, not raw column names. Today the relationship stage builds its
schema from the bare snapshot (db comments only — `local-enrichment.ts:621,688,740`
never merge the AI descriptions), so this also closes a latent gap: both the
full-run and the relationships-only paths MUST feed `llmProposals` the
best-available descriptions (fresh-this-run if `descriptions` ran, else on-disk) —
behavior from inputs, not path.
### D6 — Scope: enrichment stages only, composable with existing flags
`--stages` controls only the three enrichment stages. It is **orthogonal to and
composable with** the existing `--no-query-history` flag — a pure joins backfill
across everything is `ktx ingest --all --stages relationships --no-query-history`.
Schema introspection still runs (it is the hash substrate and the enrichment base,
and it is cheap — no LLM). The stage-name namespace is built as a **registry** so it
can later extend to the broader scan phases (schema / query-history / source /
memory) and subsume the inconsistent negative `--no-query-history` flag — but that
unification is **out of scope** here.
## Requirements
### 1. Per-stage input hashes
Each enrichment stage MUST key its cache lookup/save and (for `descriptions`) its
resume record on a hash of only that stage's own inputs, per D1
(`descriptions` ← snapshot + LLM identity; `embeddings` ← snapshot + embedding
identity + a digest of the embedded description text; `relationships` ← snapshot +
relationship settings + LLM identity). Changing one stage's inputs MUST invalidate
**only** that stage. The single coarse `computeKtxScanEnrichmentInputHash` over
`{ snapshot, mode, detectRelationships, providerIdentity, relationshipSettings }`
MUST be removed in favor of per-stage computation. The stage store and the
descriptions resume record MAY be recreated without a migration bridge (disposable
local state).
### 2. `--stages` flag with strict validation
`ktx ingest` MUST accept `--stages <comma-list>`, a non-empty subset of
`descriptions,embeddings,relationships`, defaulting (when absent) to all three. An
unknown or empty stage name MUST be a hard parse error (`InvalidArgumentError`),
never silently ignored. The selected set MUST thread through to enrichment and gate
which stage blocks run as `(mode/provider-eligible) ∩ (selected)` — one gate, no new
`KtxScanMode` variant, no second selection path. A selected stage whose prerequisite
is missing MUST fail or warn clearly, not silently no-op.
### 3. Selecting a stage force-re-runs it; unselected stages are preserved
A stage named in `--stages` MUST re-enter its `compute()`, bypassing the
completed-stage short-circuit, while still using the spec-20 per-table resume record
so `descriptions` re-issues LLM calls only for still-failed tables (unchanged hash)
and rebuilds fully on a changed hash. A stage **not** named MUST NOT run and MUST
leave its on-disk artifacts untouched. The no-flag default MUST preserve spec-19
cross-run resume (all eligible stages, completed-row short-circuit respected).
### 4. Downstream staleness is surfaced, never silent
After a selective run, the run MUST emit a recoverable `enrichment_stage_stale`
warning for every **unselected** stage whose current per-stage hash no longer
matches its stored completed-row hash (derived from on-disk state, naming the stage
and the cascade command). The embeddings hash MUST include a digest of the embedded
description text so a later embeddings run self-heals after a description change. The
run MUST NOT silently leave a stale-but-unflagged downstream and MUST NOT silently
auto-cascade. A description change MUST NOT stale-flag `relationships`.
### 5. Relationships run with description context
When the `relationships` stage runs without `descriptions` having run in the same
invocation, it MUST hydrate its working schema from the persisted on-disk enriched
`_schema` (AI descriptions + embeddings) so `llmProposals` has the same description
context as a full enriched run, not bare column names. The full-run and
relationships-only paths MUST feed `llmProposals` descriptions consistently.
### 6. No regression for normal ingests
A normal `ktx ingest` with no `--stages` flag MUST produce the same artifacts as
today (descriptions, embeddings, manifest, relationships) and MUST preserve spec-19
cross-run resume and spec-20 per-table description resume. The per-stage hash split
MUST NOT change a normal run's output, only which stages a *changed* input
invalidates.
## Acceptance criteria
- **Per-stage invalidation isolation:** flipping `scan.relationships.llmProposals`
re-runs only `relationships` (descriptions + embeddings resolve from cache, no LLM
description calls, no re-embedding); swapping the embeddings model re-runs only
`embeddings`; switching the description LLM re-runs only `descriptions`. Verified by
asserting no LLM description calls / no embed calls for the unaffected stages.
- **Flag parse + validation:** `--stages relationships` and
`--stages descriptions,embeddings` parse to the right set; `--stages foo`,
`--stages` (empty), and `--stages descriptions,foo` each fail with a clear
`InvalidArgumentError`.
- **Resume-aware force-rerun:** on a connection whose `descriptions` stage completed
with K failed/null tables (unchanged hash), `--stages descriptions` re-issues LLM
calls for exactly those K tables and leaves the already-good descriptions
untouched; the run completes and the K are now enriched. A changed descriptions
hash instead rebuilds all tables.
- **Preserve others:** after `--stages descriptions`, the on-disk `embeddings` and
`relationships` artifacts are byte-stable (unselected stages did not run).
- **Derived staleness warning:** after `--stages descriptions` changes the
descriptions, the run emits `enrichment_stage_stale` for `embeddings` (its
recomputed hash diverged) and does **not** emit it for `relationships` (decision
X); a subsequent `--stages embeddings` clears it.
- **Relationships context:** a `--stages relationships` run on an already-described
connection feeds the on-disk AI descriptions into `llmProposals` (verified: the
proposal prompt carries descriptions, not just column names).
- **No regression:** a normal uninterrupted `ktx ingest` (no flag) yields identical
artifacts and the same descriptions/embeddings/relationship output as today, with
spec-19/20 resume intact.
## Non-goals
- **Unifying `--stages` with the broader scan phases or `--no-query-history`.** The
namespace is built to extend later; this spec ships only the three enrichment
stages, composable with the existing query-history flag (D6).
- **A new `KtxScanMode` variant or a second stage-selection path.** One gate,
`(eligible) ∩ (selected)` (D2).
- **Coupling `relationships` to description content** (decision X, D5). Improving
descriptions does not invalidate or stale-flag joins.
- **Auto-cascading downstream re-runs.** Staleness is surfaced as a warning; the
operator chooses to cascade (D4).
- **Capturing prompt/code-level description-prompt changes in the hash.** The
descriptions hash keys on snapshot + LLM identity (config/model), not the prompt
text; a pure prompt improvement that does not change a hash input will not
force-rebuild already-good descriptions. Forcing that is out of scope — the
operator changes a real input or selects the stage with a changed config.
- **Re-implementing spec 19 (cross-run stage resume, completed-row store) or spec 20
(per-table description resume, enforced timeout).** This spec composes above them:
it splits the key those stages resume on and adds the CLI surface to select and
force-re-run stages.
- **A general per-phase incremental-flush framework.** The selection mechanism is the
three enrichment stages; it is not a generic abstraction over every ingest phase.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns the
design.
- **Coarse hash → per-stage hashes** — `context/scan/enrichment-state.ts`
(`computeKtxScanEnrichmentInputHash` `:78`, `ComputeKtxScanEnrichmentInputHashInput`
`:57`): replace with per-stage hash functions (or one function taking a per-stage
input slice). `context/scan/local-enrichment.ts` (`:611` single hash; the three
`runEnrichmentStage` calls at `descriptions` ~`:635`, `embeddings` ~`:666`,
`relationships` ~`:722`; `runEnrichmentStage` `:524` and its short-circuit
`:538–547`). The `descriptions` hash also feeds `generateDescriptions`'
`resumeStore.load(inputHash)` (`:345`).
- **Provider-identity decomposition** — `context/scan/local-scan.ts`
(`localScanProviderIdentity` `:241–255`, the enrichment call site `:498–537`):
split into `llmIdentity` / `embeddingIdentity`, drop the redundant `mode` /
`relationships` re-encoding, and pass each stage only its slice.
- **`descriptionDigest`** — `context/scan/local-enrichment.ts` (`buildEmbeddings`
`:457–486`) and `context/scan/embedding-text.ts` (`buildKtxColumnEmbeddingText`
`:17–44`): digest the resolved per-column/table description text that the embeddings
consume, and fold that digest into the embeddings hash.
- **CLI flag** — `commands/ingest-commands.ts` (`:26–49` option declarations,
`:51–104` action handler): add `--stages` with a custom parser that validates
against the canonical stage registry (`KTX_SCAN_ENRICHMENT_STAGES` in
`enrichment-state.ts:4`) and rejects unknown/empty names with `InvalidArgumentError`.
Thread through `public-ingest.ts` (`KtxScanArgs` build `:969–978`, `mode: 'enriched'`
`:973`) → `scan.ts` (`runKtxScan`) → `local-scan.ts` (`runLocalScan`) →
`runLocalScanEnrichment`.
- **Stage gating + force-rerun** — `context/scan/local-enrichment.ts`: gate each stage
block on membership in the selected set (`descriptions` `:632`, `embeddings`
`:663–665`, `relationships` `:720`); make a named stage bypass the completed-row
short-circuit in `runEnrichmentStage` while the inner `compute()` keeps the spec-20
per-table resume. `KtxLocalScanEnrichmentInput` (`:60–85`) gains the selected-stage
set.
- **Staleness detection + warning** — `context/scan/local-enrichment.ts` (after the
stage blocks): recompute each unselected stage's current hash from on-disk state,
compare to the stored completed-row hash, push a recoverable warning on mismatch.
Add `enrichment_stage_stale` to the `KtxScanWarningCode` union in
`context/scan/types.ts` (alongside `relationship_detection_partial`).
- **Relationships description context** — `context/scan/local-enrichment.ts`
(`schema` built at `:621`/`:688`, passed to `discoverKtxRelationships` `:736–746`):
hydrate `schema` with the best-available descriptions (fresh-this-run or loaded from
the on-disk `_schema` via `loadExistingManifestState`,
`local-enrichment-artifacts.ts`) before relationship detection.
- **Stage store + resume record** —
`context/scan/sqlite-local-enrichment-state-store.ts`
(`local_scan_enrichment_stages`, PK `(connection_id, stage, input_hash)`,
`findCompletedStage`, `saveCompletedStage`); `createKtxScanDescriptionResumeStore`
(`local-enrichment-artifacts.ts:286–332`, path `:265–267`, inputHash gate
`:305–307`) — both now keyed on the relevant per-stage hash. No migration bridge.
- **Config inputs** — `context/project/config.ts` (`scanRelationshipsSchema`
`:171–218` incl. `llmProposals` `:174` and `detectionBudgetMs`;
`scan.enrichment.embeddings` model/dimensions/batchSize; `llm.models.default`,
`llm.provider.gateway.base_url`): the sources of each per-stage identity slice.
- **Tests** — per-stage invalidation isolation (flip one input, assert only the
matching stage recomputes); `--stages` parse/validate (good subsets + unknown/empty
rejected); resume-aware force-rerun (`--stages descriptions` retries only the null
tables, leaves good ones, completes); preserve-others (unselected artifacts
byte-stable); derived staleness (`enrichment_stage_stale` fires for embeddings after
a descriptions change, not for relationships; cleared by a later `--stages
embeddings`); relationships-only run feeds on-disk descriptions to `llmProposals`;
regression — a normal no-flag ingest yields identical artifacts with spec-19/20
resume intact.
- After implementing, rebuild and re-link so the playground picks it up:
`pnpm run build && pnpm run link:dev`.
- **Docs:** add `--stages` to the `ktx ingest` CLI reference
(`docs-site/content/docs/cli-reference/`) and note the per-stage cache behavior
where enrichment/ingest is described.
## Benchmark context (motivation, not a requirement)
Surfaced during the Spider 2.0-Lite multi-backend ingestion (2026-06-24). A
level-aware audit found (a) a tail of BigQuery datasets with poor *column*-description
coverage (`google_dei` ~1%, `gnomAD`, `usfs_fia`, …) that want a **`descriptions`-only**
re-run with a longer timeout, and (b) a desire to **backfill joins** across all
already-ingested datasets after enabling `llmProposals` — without re-paying for
descriptions. Both were blocked by the coarse single `inputHash` (flipping
`llmProposals` or re-describing invalidated the whole enrichment) and the absence of a
stage-selective CLI flag. The benchmark merely exercised large-scale multi-backend
ingestion at scale; the gap and the fix are generic production operability. Do not
encode any benchmark specifics in the implementation.
## Implementation notes
Shipped on branch `write-feature-spec-wiki`. All seven requirements implemented;
all acceptance criteria covered by tests.
**What was built / where:**
- **Per-stage hashes (D1, Req 1).** `context/scan/enrichment-state.ts`: removed the
coarse `computeKtxScanEnrichmentInputHash` and added
`computeKtxDescriptionsStageHash` (snapshot + `llmIdentity`),
`computeKtxEmbeddingsStageHash` (snapshot + `embeddingIdentity` + `descriptionDigest`),
`computeKtxRelationshipsStageHash` (snapshot + `relationshipSettings` + `llmIdentity`),
plus `computeKtxScanDescriptionDigest` and the `KtxScanLlmIdentity` /
`KtxScanEmbeddingIdentity` types. `KTX_SCAN_ENRICHMENT_STAGES` is now exported as the
canonical registry. `local-scan.ts` `localScanProviderIdentity` was split into
`localScanLlmIdentity` + `localScanEmbeddingIdentity` (dropping the redundant
`mode`/`relationships` re-encoding). `mode`/`detectRelationships` dropped out of the
keys. No migration bridge — the stage store + descriptions resume record just miss the
old coarse-keyed rows.
- **`descriptionDigest` (D1/D4).** `local-enrichment.ts`: extracted
`buildKtxColumnEmbeddingTexts(snapshot, descriptions)`, shared by the embeddings stage
and the digest, so the embeddings hash content-addresses the exact text the model sees.
- **`--stages` flag (D2/D6, Req 2).** `commands/ingest-commands.ts`:
`parseEnrichmentStagesOption` (Commander parser) validates against the registry,
rejects unknown/empty with `InvalidArgumentError`, returns an ordered de-duplicated
set; threaded through `KtxPublicIngestArgs` → `context-build-view` → `KtxScanArgs` →
`RunLocalScanOptions` → `KtxLocalScanEnrichmentInput`. One gate
(`(eligible) ∩ (selected)`); no new `KtxScanMode`. A selected-but-ineligible stage
emits a new `enrichment_stage_skipped` warning (never a silent no-op).
- **Force-rerun (D3, Req 3).** `runEnrichmentStage` gained `forceRecompute`; a named
stage bypasses the spec-19 completed-row short-circuit while `generateDescriptions`
still consults the spec-20 per-table resume record (retries only failed tables on an
unchanged hash).
- **Descriptions hydration + `llmProposals` context (D5, Req 5).** `runLocalScanEnrichment`
resolves best-available descriptions (fresh-this-run, else on-disk via a lazy
`loadPriorDescriptions` thunk wired from `local-scan.ts` →
`loadOnDiskDescriptionUpdates` in `local-enrichment-artifacts.ts`). `snapshotToKtxEnrichedSchema`
now merges `ai` descriptions, and `relationship-llm-proposal.ts` `buildEvidencePacket`
now carries the resolved description text — closing the latent gap on **both** the
full-run and relationships-only paths.
- **Derived staleness (D4, Req 4).** `enrichment_stage_stale` warning code +
`findLatestCompletedStage` on the state store (interface + sqlite + test store). After a
selective run, each unselected stage with a completed row is compared against its
freshly recomputed hash; a mismatch warns and names the cascade command. Relationships
are never flagged by a description change (decoupled per D5).
- **Docs.** `docs-site/content/docs/cli-reference/ktx-ingest.mdx`: `--stages` flag row, a
"Selecting enrichment stages" section (per-stage cache, force-rerun, staleness), and
examples.
**Deviation from the spec — embeddings hydration is descriptions-only.** D5 states a
relationships-only run should hydrate "AI descriptions **and** embeddings" from the
on-disk `_schema`. Investigation found the `_schema` manifest shards store only
descriptions; embedding vectors are written to a **syncId-scoped** `enrichment/embeddings.json`
that no code reads back, and each run mints a fresh syncId — so there is no durable
per-connection embeddings artifact to hydrate from. A relationships-only run therefore
hydrates **descriptions** (required for, and verified against, the `llmProposals`
acceptance criterion) but **not** embeddings. Consequence: a `--stages relationships`
backfill gets deterministic + name-based + LLM-proposed candidates (the point of
`llmProposals`), but not the embedding-similarity candidates a full run would add.
Durable embeddings hydration (persist vectors at a stable per-connection path, or read
them from the vector index) is a clean follow-on and was left out of scope.
**Tests:** `enrichment-state.test.ts` (per-stage hash stability + isolation),
`commands/ingest-commands.test.ts` (parser good/bad subsets, threading, text-capture
guard), `local-enrichment.test.ts` (force-rerun bypasses short-circuit + preserves
others, naming all three forces a full recompute, per-stage invalidation isolation,
prerequisite warning, on-disk descriptions reach `llmProposals`, resume-aware forced
descriptions rerun, derived `enrichment_stage_stale` fires for embeddings/not
relationships and clears after re-embed). Full `pnpm --filter @kaelio/ktx run test`,
`type-check`, `dead-code`, and `build` pass. (One pre-existing unrelated failure in
`test/skills/analytics-skill-content.test.ts` — the analytics `SKILL.md` lacks a
`**Window functions**` heading the test expects — was present before this work and left
untouched.)
---
## ⚠️ Defect found in post-implementation validation (2026-06-24)
**`--stages` subset excluding `descriptions` WIPES existing on-disk descriptions.** Violates Req
"preserve-others / a selective run never deletes another stage's artifacts."
**Reproduction (deterministic):**
- `northwind` before: 110 `ai:` column/table descriptions, 0 join edges.
- `ktx-dev ingest northwind --stages relationships` → completes in ~35s, adds **22 join edges** ✅
but the rewritten `public.yaml` has **0 descriptions** (no `ai:`, no `db:`, columns bare). ❌
- A full `ktx-dev ingest northwind` (all stages) restores 110 descriptions + keeps the 22 joins.
**Likely root cause:** the relationships-only path rewrites the schema from the raw snapshot + only the
freshly-run stage. The implementation notes claim `snapshotToKtxEnrichedSchema` merges `ai` descriptions
and that descriptions are hydrated "fresh-this-run, else on-disk via `loadPriorDescriptions`" — but on the
**write path** of a subset run the prior descriptions are NOT merged into the emitted schema (they reach
the `llmProposals` evidence packet only). So the on-disk `_schema` loses them.
**Impact:** blocks the intended joins-everywhere backfill (`--stages relationships` across all dbs) and the
`--stages descriptions`-only re-runs — either would destroy the unselected stage's artifacts across every
db. Caught on a 1-db validation before any rollout.
**Acceptance fix:** after any `--stages` subset, the on-disk `_schema` must **retain all prior `ai:`/`db:`
descriptions** (and prior joins when descriptions-only) for stages not named — only the named stages'
artifacts change. Add a regression test that ingests a fully-enriched fixture, runs `--stages relationships`,
and asserts description count is unchanged while joins increase.
### ✅ Fixed (2026-06-24)
**Real root cause (deeper than the first diagnosis):** the wipe happened in **two** places, and the first
fix attempt only addressed one. `runLocalScan` (`context/scan/local-scan.ts`) writes the **structural**
manifest shard from the bare snapshot *before* enrichment runs; that write merges with the on-disk shard,
but the merge (`mergeDescriptionsPreservingExternal`, `live-database/manifest.ts`) treats `ai`/`db` as
**scan-managed** and overwrites them with whatever the run emits — and the structural write emits none. So a
subset run deleted the descriptions on the structural pre-write, *then* `runLocalScanEnrichment` read the
already-wiped shard via `loadPriorDescriptions` and had nothing to restore. (A unit-level enrichment test
passed because it never exercised the structural pre-write — a divergent-harness miss; the regression test
was rewritten to go through the full `runLocalScan` path.)
**What changed:**
- `runLocalScanEnrichment` (`local-enrichment.ts`) now returns the **best-available** descriptions
(`resolveDownstreamDescriptions()` — fresh-this-run if `descriptions` ran, else the on-disk ones) as
`descriptionUpdates`, instead of `[]` when the stage is skipped — so the enrichment write re-applies them.
- `runLocalScan` (`local-scan.ts`) now, on a subset run, **captures the prior on-disk descriptions before
the structural manifest write** and feeds them to both the structural write and enrichment — so the
structural pre-write preserves them too (robust even if relationship detection later fails).
- Joins were already preserved for `--stages descriptions` via the existing manual/inferred
`preservedJoins` path; verified by a symmetric test.
**Tests:** `local-scan.test.ts` — a full `runLocalScan` `--stages relationships` run preserves on-disk `ai`
descriptions while adding a join (RED without the fix, GREEN with it). `local-enrichment.test.ts` — the
enrichment-layer contract (`--stages relationships` preserves descriptions / `--stages descriptions`
preserves joins).
**Live validation (northwind, 15 tables):** `--stages relationships` BEFORE `ai:110 joins:22` → AFTER
`ai:110 joins:22` (descriptions intact; previously wiped to 0). `--stages descriptions` restored the
descriptions from the spec-20 resume record (`ai:0 → ai:110`) with **no** LLM calls while keeping `joins:22`.
Full `pnpm --filter @kaelio/ktx run test` (3089 passed), `type-check`, `dead-code`, and `build` pass.

463
spider2-specs/specs/22-resumable-and-fault-tolerant-source-ingest.md
View file

@ -1,463 +0,0 @@
# Resumable and fault-tolerant source ingest
> Refined spec. No intake draft — surfaced by a real user report, not the
> playground agent (see Motivation). Lives beside the analogous scan-durability
> specs 19/20.
>
> **Scope: make `ktx ingest` (the source-ingest work-unit pipeline behind dbt /
> Metabase / Notion) survive interruption and partial failure on large
> projects.** Two compounding gaps live on the source-ingest path: (1) an
> interrupted run restarts every work unit from scratch — there is no cross-run
> reuse of already-generated work-unit output, so a multi-day dbt ingest loses
> *all* progress to a single VPN/network blip; (2) the final integration gate is
> all-or-nothing — one artifact that cannot pass it (after LLM repair) discards
> the **entire** run with nothing committed. This is the source-ingest analog of
> spec 19 (move the durability boundary to the cost boundary so expensive LLM
> work is not lost) and spec 20 (a stage survives an interruption with per-item
> durability). It **reuses** the same content-keyed durability primitive those
> specs established rather than copying it.
## Problem
Two independent failure modes on the source-ingest work-unit (WU) pipeline,
both confirmed in the current code, both observed by a user on a ~2-day dbt
ingest. Their union makes large-project ingest brittle: any interruption is
total loss, and any single unfixable artifact at the end is total loss.
### 1. An interrupted run resumes nothing — every work unit re-runs
`IngestBundleRunner` (`context/ingest/ingest-bundle.runner.ts`) executes a run as
a sequence of stages: fetch → parse/extract into **work units** → run each WU as
an isolated agent loop in a child worktree (`runIsolatedWorkUnit` →
`executeWorkUnit`, `stages/stage-3-work-units.ts`) → integrate the successful WU
patches → reconcile → finalize → final gates → one atomic squash commit
(`squashMergeIntoMain`, ~2716). The WU stage is where the LLM cost lives: each WU
is an agent loop that reads its `rawFiles`/`dependencyPaths` and writes SL/wiki
artifacts, producing a git patch (`WorkUnitOutcome.patchPath` /
`patchTouchedPaths`, `stage-3-work-units.ts:31-46`).
The only persisted cross-run state is `SqliteBundleIngestStore`
(`context/ingest/sqlite-bundle-ingest-store.ts`): run metadata, the final report,
and provenance — all written at or near **run completion**. There is **no
checkpoint of completed WU output**. A run that dies mid-flight (the user's
VPN/network drop) leaves nothing reusable: the next `ktx ingest` re-fetches,
re-parses, and **re-executes every WU from scratch**, re-paying the entire LLM
cost. The store even keys `job_id` UNIQUE, so a re-run is a brand-new job with no
relationship to the interrupted one.
> Observed (user report, large dbt project): a run reached deep into its
> work-unit progress and was lost to a network blip; the follow-up run started
> over from zero. On a ~2-day ingest this is the difference between a 5-minute
> resume and a 2-day redo.
### 2. The final integration gate is all-or-nothing
After all surviving WUs are integrated, `validateFinalIngestArtifacts`
(`context/ingest/artifact-gates.ts:96`) runs the final gate. It checks, across
the *integrated* tree:
- **intrinsic source validity** — `validateTouchedSources` →
`validateWuTouchedSources` (`stages/validate-wu-sources.ts:124`) →
`validateSingleSource` (`context/sl/tools/sl-warehouse-validation.ts:56`),
which runs a **live warehouse dry-run** (`SELECT * FROM (sql) LIMIT 1`);
- **cross-artifact references** — dangling join targets
(`findJoinTargetErrors`, `validate-wu-sources.ts:89`), dangling `wiki→wiki`
refs (`validateWikiRefs` → `findMissingWikiRefs`), broken `wiki→sl_ref`s
(`validateWikiSlRefs`, `artifact-gates.ts:39`), and broken wiki body refs
(`findInvalidWikiBodyRefs`).
On any error it **`throw`s a single concatenated string** (`artifact-gates.ts:129`).
The runner catches it, runs the LLM repair `repairFinalGateFailure`
(`runner.ts:2595`, `maxAttempts: 2`), and if repair still fails, **re-throws**
(`runner.ts:2623`) → `markFailed` → the squash never runs → `commitSha: null`
(`runner.ts:2729`) → **the whole run is discarded, nothing committed.**
The crucial asymmetry: a WU that fails *on its own terms* never reaches this gate
— `executeWorkUnit` already validates each WU in isolation (`validateWikiRefs`
~143, `validateTouchedSources` ~150) and **soft-fails** it (`failWithReset`,
~155: the WU resets, is excluded from integration, and the run continues). So by
the time the final gate runs, intrinsic single-source failures are rare. The
gate fails predominantly on **cross-artifact dangling references**: WU-A's source
joins to a source WU-B was meant to create, but WU-B failed/was-excluded, so
A's join now points at nothing. Each WU passed *alone*; the break only appears
once the survivors are integrated — and that break currently nukes the run.
> Observed (user report): a run completed all task generation and then failed at
> the final integration gate on a **single model**; because the gate is
> all-or-nothing, that one failure discarded an ~18h run with nothing committed.
## Generic use case (independent of any benchmark)
Anyone ingesting a large warehouse/BI/dbt project with an LLM pipeline will hit
both failures. Large ingests run long enough that an interruption is a *when*,
not an *if* (laptop sleep, VPN reconnect, transient provider error, an operator
ctrl-C on an apparently-stuck run), and a large artifact set makes it
near-certain that *some* model lands a cross-reference its sibling didn't
produce. Without cross-run reuse, every interruption is a from-scratch redo of
the dominant (LLM) cost; without partial commit, one unfixable artifact throws
away every good one. Both fixes make large-project ingest **resilient and
resumable**: an interruption costs only the unfinished work, and a single bad
model costs only that model — not the run. This is core robustness for a
general-purpose ingestion product.
## Design decisions (resolved during refinement)
These resolve the design space explored during refinement. They constrain the
implementer; the exact code is theirs (requirement-level, per the specs README).
### D1 — Resume is automatic and content-keyed at the work-unit level
A successful WU's output is cached across runs, keyed by a **content hash of its
inputs**, with **no `--resume` flag**. Re-running the same `ktx ingest`
transparently replays any WU whose inputs are byte-identical to a cached success
and re-runs only the changed, failed, or missing WUs. The key is computed over:
the contents of the WU's `rawFiles` + `dependencyPaths` (the bytes the WU reads,
`types.ts:19-28`), the adapter/source identity, and a **version/prompt
fingerprint** (ktx version + the WU system/user prompt + model role). A changed
dbt model busts only that model's entry; everything unchanged replays for free.
> No flag, no config knob. Content-keying makes resume automatic; a flag would
> double the state space for no benefit. This is the same shape scan uses
> (`computeKtxScanEnrichmentInputHash`, spec 19), reached here for the WU
> pipeline.
### D2 — The cached unit is the successful WU's patch; replay verifies or recomputes
The cache stores a successful WU's **output artifacts**: its git patch
(`patchPath` content / `patchTouchedPaths`) plus the metadata integration needs
(`actions`, `touchedSlSources`, `slDisallowed`). On a cache hit, the runner
**replays the patch** into the session worktree — no agent loop, no LLM — exactly
where it would have integrated a freshly-run WU. If a cached patch **fails to
apply** (the surrounding tree drifted), the entry is discarded and the WU
**recomputes**. So a stale hit degrades to "recompute," never to a corrupt tree:
the cache can only make a run faster, never wrong.
### D3 — One durability primitive, shared by scan and ingest
Per the "one capability, one implementation" rule, the content-keyed store is
**extracted** into a shared primitive and **both** scan and ingest route through
it — not copied. Scan's `sqlite-local-enrichment-state-store.ts` (PK
`(connection_id, stage, input_hash)`, `findCompletedStage` / `saveCompletedStage`)
and its `inputHash` computation (`enrichment-state.ts`) are generalized to a
content-keyed result cache; scan is migrated onto the shared primitive **in the
same change** so no second copy exists even transiently. The ingest cache is a
new logical namespace (e.g. keyed `(connectionId, sourceKey, workUnitInputHash)`)
on that one store.
> Extract-and-share in one PR, not "build a copy for ingest now, unify later."
> A temporary fork is exactly the divergence the rule forbids; the one-time
> extraction cost is paid once and both paths benefit from every later fix.
### D4 — Only successes are cached; failures retry on the next run
A failed WU is **not** recorded as terminal — the next run retries it. WU
failures on this path are dominantly transient (network, provider stall, an LLM
slip), and the user's explicit ask is "resume and finish the rest," so a failure
must not be sticky. This deliberately differs from scan's stage store (which
caches failed stages and re-throws): there the failure is the stage's
deterministic verdict; here a WU failure is usually a blip to retry. Caching only
successes also keeps the invariant simple — a cache entry always means "this
exact input already produced this exact good output."
### D5 — The final gate becomes non-fatal: deterministic dangling-edge prune
Replace the gate's fatal `throw`-after-repair with a deterministic reconciliation
that always yields a committable, internally-consistent tree:
1. `validateFinalIngestArtifacts` is refactored to **return structured findings**
(the danglers it already computes internally — join targets, `wiki→wiki`,
`wiki→sl_ref`, wiki body refs — plus any intrinsic source failure) instead of
flattening them into a thrown string.
2. **Drop the rare self-invalid source first.** A source that fails its *own*
validation at the final gate (intrinsic — rare, since stage 3 already filters
these) is removed, establishing the surviving artifact set.
3. **Prune the dead edges in a single pass** over that surviving set. For each
dangling reference — whether it pointed at an absent sibling or at a
just-dropped source — **remove that reference from its owner** (drop the join
entry, remove the `wiki ref` / `sl_ref`, remove the broken body link), keeping
the owning artifact. Because nodes are dropped first (step 2) and pruning only
removes edges, pruning **cannot create a new dangling edge, so one pass
suffices; no fixpoint.**
4. Re-run the gate to **confirm** the remainder is clean (warehouse dry-runs are
cached per D6/D2, ref checks are in-memory, so this is cheap), then squash-commit
the remainder. If the confirm pass *still* fails, that is a real bug — fail the
run loudly rather than commit a dirty tree.
`repairFinalGateFailure` (the LLM repair, `runner.ts:2595` / `final-gate-repair.ts`)
is **removed**. The deterministic prune supersedes it for the referential class,
and the rare intrinsic case is handled by drop.
> **Prune the edge, do not cascade the node.** The rejected alternative drops the
> *referencing artifact* and, transitively, everything that referenced *it* — a
> node-quarantine fixpoint that cascades healthy artifacts and needs a closure
> search, a confirm loop, and an un-apply step. Pruning the dead edge keeps the
> dependent intact (minus one pointer that never resolved anyway), needs no
> fixpoint, and acts on findings the gate already produces.
>
> **Why remove the LLM repair rather than keep it as a pre-prune step.** Repair
> can occasionally *fix* a ref (e.g. correct a typo'd source name) where prune
> merely deletes it, preserving marginally more content. We drop it anyway:
> determinism beats an LLM round-trip with variance on the commit path, prune
> guarantees a commit where repair could only `throw`, and deleting it is a net
> maintenance reduction. The decision is reversible — repair could later run as a
> best-effort pass *before* prune — but the default is prune-only.
### D6 — Prune runs on the integrated tree, never poisons the cache (resume ∘ prune compose)
Pruning is applied to the **integrated session worktree** at gate time and is
**re-derived from the current survivor set on every run**. It MUST NOT mutate the
cached WU patches (D2). This makes resume and prune compose correctly and
**self-heal**:
- Run 1: WU-A (joins to B) succeeds and is cached *with its join intact*; WU-B
fails; the gate prunes A's join-to-B from the integrated tree and commits A
without it.
- Run 2 (after the root cause is fixed): A's input is unchanged → A **replays
from cache with its join restored**; B now succeeds and exists; the gate finds
no dangler and commits both, fully linked.
So a ref pruned because of a sibling's failure costs nothing permanent: fixing
the sibling and re-running restores the link for free. The cache stores
intent (the WU's real output); prune is a per-run consistency projection over
whatever survived.
### D7 — Pruning is faithful and never silent
A pruned reference was, by definition, non-functional (its target was absent), so
removing it loses nothing executable — and removing dangling SL joins is already
the established fix for the SL engine's eager orphan-join rejection. Every prune
and every drop MUST be **recorded in the run report and a trace event** naming
the artifact, the removed reference, and the absent target. The report status
MUST reflect partial completion (extend the existing `failedWorkUnits`
mechanism, `IngestBundleResult`, `types.ts:204-213`, with the pruned-refs /
dropped-sources detail) so a partial run is visibly partial, never a silent
"success."
### D8 — Cache state is regenerable; no migration bridge
The WU cache is regenerable local state under `.ktx/`. Its on-disk/SQLite shape
may change with **no migration bridge** — a stale-shaped or absent cache simply
forces a full (non-resumed) run, exactly today's behavior. Consistent with ktx's
no-backward-compatibility policy; the cache is an optimization, never a source of
truth.
## Requirements
1. **Cross-run WU resume, automatic and content-keyed.** A successful WU's output
MUST be cached keyed by a content hash over its input bytes
(`rawFiles` + `dependencyPaths`), the adapter/source identity, and a
version/prompt fingerprint (ktx version + WU prompt + model role). Re-running
`ktx ingest` MUST replay cached successes without an agent loop / LLM call and
re-run only changed, failed, or missing WUs. No `--resume` flag and no config
knob is added.
2. **Replay verifies or recomputes.** On a cache hit the runner MUST replay the
stored patch into the session worktree; if the patch does not apply cleanly the
entry MUST be discarded and the WU recomputed. A cache hit MUST NOT be able to
produce a tree different from what a fresh run of that WU would have integrated.
3. **Only successes are cached.** A failed WU MUST NOT be recorded as terminal; it
MUST be retried on the next run.
4. **Conservative invalidation.** The input hash MUST change when the ktx version,
the WU prompt, or the model role changes (bias toward recompute). Under-keying
(stale reuse) is a correctness bug; over-keying (an unnecessary recompute) is
acceptable.
5. **The final gate is non-fatal.** A final-gate failure MUST NOT discard the run.
`validateFinalIngestArtifacts` MUST return structured findings; the runner MUST
deterministically **prune** every dangling reference from its owning artifact
and **drop** any source that fails its own validation, then commit the
remaining internally-consistent tree.
6. **Single-pass prune, dependents survive.** Pruning MUST remove dead *edges*, not
cascade-drop owning artifacts; it MUST complete in a single pass (no fixpoint)
because edge removal cannot create new dangling edges. A dependent that loses
one dangling ref MUST otherwise be committed intact.
7. **Prune composes with resume.** Pruning MUST operate on the integrated tree and
MUST NOT mutate cached WU patches. A reference pruned in one run because its
target was absent MUST be restored automatically on a later run once the target
exists (resume replays the owner's intact patch).
8. **Confirm before commit.** After pruning/dropping, the gate MUST be re-run on
the remainder and MUST pass before the squash; if it still fails the run MUST
fail loudly rather than commit a dirty tree.
9. **`repairFinalGateFailure` is removed.** The LLM final-gate repair path and its
obsolete tests/branches MUST be deleted (no dormant compatibility path).
10. **Every prune/drop is reported.** Each pruned reference and dropped source MUST
be recorded in the run report and a trace event (artifact, removed ref, absent
target). A run that pruned or dropped anything MUST report as partial, never as
an unqualified success.
11. **One shared durability primitive.** The content-keyed store MUST be a single
implementation used by both scan and ingest; scan MUST be migrated onto it in
the same change. No second copy may exist, even transiently.
12. **No regression for clean runs.** A small, uninterrupted run whose every WU
passes and whose final gate is clean MUST produce byte-identical artifacts and
the same `commitSha`/report shape (modulo new, empty pruned/dropped fields) as
today.
## Acceptance criteria
- **Resume skips completed work:** interrupt an ingest after K of N WUs have
succeeded; re-run the same command (unchanged inputs); the run issues **zero**
agent loops / LLM calls for the K cached WUs, runs only the remaining N−K, and
produces the same final artifacts as an uninterrupted run.
- **Changed model busts only its entry:** edit one dbt model between runs; the
re-run re-executes **only** the WU(s) whose input bytes changed and replays the
rest from cache.
- **Stale patch self-corrects:** a cached patch that no longer applies (forced
drift in a test) causes that WU to recompute, not a corrupt tree or a crash.
- **Failures retry:** a WU that fails in run 1 (transient error) is **not** cached;
run 2 retries it and, on success, integrates it.
- **One bad model no longer nukes the run:** a run where WU-B fails so WU-A's join
to B dangles **commits** — A is committed with the dangling join **pruned**, the
report lists the pruned ref, and `commitSha` is non-null (contrast: today this
throws and commits nothing).
- **No cascade:** in that scenario A (and any other artifact that only referenced
B) is committed intact except for the single pruned reference; nothing healthy
is dropped.
- **Self-heal:** fix B's root cause and re-run; A replays from cache with its join
intact, B succeeds, and the final tree commits both fully linked with no prune.
- **Intrinsic drop:** a source that fails its own warehouse dry-run at the final
gate (forced) is dropped, refs to it are pruned, and the rest commits; the drop
is reported.
- **Repair is gone:** `repairFinalGateFailure` and its tests no longer exist; the
gate path has no LLM call.
- **One store:** scan and ingest both resume through the same content-keyed
primitive (one implementation; scan's behavior is unchanged by the migration —
spec 19/20 acceptance still passes).
- **Clean-run regression:** a small uninterrupted all-passing ingest yields
identical artifacts, `commitSha`, and report (empty pruned/dropped fields) to
today.
## Non-goals
- **Resuming the cross-WU stages.** Reconciliation, finalization, and the final
gate re-run every time; their inputs depend on the full survivor set and their
cost is small relative to WU generation. Only WU generation is cached.
- **A `--resume` flag or any timeout/cache config knob.** Content-keying makes
resume automatic (D1); one opinionated default is the canonical ktx shape.
- **Caching failed WUs as terminal.** Failures retry (D4).
- **Node-cascade quarantine of the final gate.** Prune edges, do not drop
dependents (D5). No closure search, confirm-loop-over-nodes, or un-apply step.
- **Tolerating dangling references (warn instead of remove).** Unsafe — the SL
engine eagerly rejects orphan joins — so dead edges must be removed, not kept.
- **Keeping the LLM final-gate repair.** Removed (D5/req 9).
- **A general per-stage resume framework beyond the shared content-keyed store.**
The store is the one shared primitive (D3); this spec does not abstract every
ingest stage into a resumable framework.
- **Re-implementing spec 19/20 (scan durability).** This spec composes the same
primitive onto the source-ingest WU pipeline.
## Implementation orientation
Line numbers drift; treat these as anchors, not addresses. The implementer owns
the design.
- **Run flow + the all-or-nothing seam** — `context/ingest/ingest-bundle.runner.ts`:
WU run + integration of successful patches (~1600–1900), the final-gate block
(~2549–2587, `runFinalArtifactGates`), the repair-then-rethrow that must be
replaced by prune (~2588–2644; the fatal `throw` ~2623), and the atomic squash
(~2701–2729; `commitSha: null` when nothing is touched ~2729). The prune step
slots between the gate findings and the squash, operating on `sessionWorktree`.
- **Work units & cacheable output** — `context/ingest/types.ts` (`WorkUnit`
~19–28: `rawFiles`/`peerFileIndex`/`dependencyPaths`; `IngestBundleResult`
~204–213: extend with pruned/dropped detail);
`context/ingest/stages/stage-3-work-units.ts` (`executeWorkUnit`; the per-WU
validation + `failWithReset` ~134–157 that already soft-fails a WU;
`WorkUnitOutcome` ~31–46 with `patchPath`/`patchTouchedPaths`/`actions`/
`touchedSlSources` — the cache payload). The cache lookup/replay wraps the
per-WU execution; only the agent-loop branch is skipped on a hit.
- **The gate (make it return findings)** — `context/ingest/artifact-gates.ts`
(`validateFinalIngestArtifacts` ~96; the internal per-artifact danglers from
`validateWikiSlRefs` ~39, `validateWikiRefs` ~74, `findInvalidWikiBodyRefs`;
the concatenated `throw` ~129 to replace with a structured return);
`context/ingest/stages/validate-wu-sources.ts` (`validateWuTouchedSources` ~124;
`findJoinTargetErrors` ~89 already returns missing join targets per source —
the join-edge danglers to prune); `context/sl/tools/sl-warehouse-validation.ts`
(`validateSingleSource` ~56 — the intrinsic warehouse dry-run; its failures are
the drop set, not the prune set).
- **Per-ref-type pruners (pair 1:1 with the validators)** — join: remove the
offending `joins[]` entry from the source YAML; `wiki refs`/`sl_refs`: remove
the entry from page frontmatter (`context/wiki/wiki-ref-validation.ts`
`findMissingWikiRefs`); wiki body refs: remove the broken link token
(`context/ingest/wiki-body-refs.ts` `findInvalidWikiBodyRefs`). Each pruner is
deterministic and edits the integrated worktree only.
- **Remove the LLM repair** — `context/ingest/final-gate-repair.ts`
(`repairFinalGateFailure`) and the `constrained-repair.ts` usage for
`final_artifact_gate`; delete the call site (~2595) and its tests.
- **Durability primitive to extract & share** —
`context/scan/sqlite-local-enrichment-state-store.ts` (`local_scan_enrichment_stages`,
PK `(connection_id, stage, input_hash)`, `findCompletedStage`/`saveCompletedStage`),
`context/scan/enrichment-state.ts` (`computeKtxScanEnrichmentInputHash` ~78), and
the resume wrapper `runEnrichmentStage` (`context/scan/local-enrichment.ts`).
Generalize to a content-keyed result cache; migrate scan onto it; add the ingest
namespace. The existing ingest store
`context/ingest/sqlite-bundle-ingest-store.ts` (`SqliteBundleIngestStore`) is
where ingest-side persistence lives — the WU cache sits alongside it under
`.ktx/`.
- **Tests** — resume: run an ingest against a real git-backed project with a fake
agent runner, interrupt after K WUs, assert the re-run issues no agent loops for
the K and the same artifacts result; changed-input bust; stale-patch recompute;
failed-WU retry. Prune: a fixture where one WU fails so a sibling's join/wiki
ref dangles → assert the run commits the sibling with the ref pruned, reports the
prune, and `commitSha` is non-null; assert no cascade; assert self-heal on a
follow-up run; assert intrinsic drop. Migration: spec 19/20 scan acceptance still
green on the shared primitive. Regression: a small uninterrupted all-passing
ingest is byte-identical to today.
- After implementing, rebuild and re-link so the playground picks it up:
`pnpm run build && pnpm run link:dev`.
## Motivation (the real report, not a benchmark)
A user ingesting a fairly large dbt project (~2-day run) hit both gaps together.
First, an interruption — a VPN drop / network blip — lost all progress because
ingest cannot resume; they had to restart from scratch. Second, on a later run
that completed all task generation, a **single model** failed the final
integration gate, and because the gate is all-or-nothing the one failure
discarded an ~18h run with nothing committed. Their ask: "some form of resume or
checkpoint (or at least reusing the patches that were already generated), and a
way to skip or quarantine a single failing model instead of failing the entire
run." This spec delivers both — resume via the content-keyed WU cache, and
partial commit via deterministic dangling-edge pruning. Unlike specs 19/20 this
gap was surfaced by a real user on a real warehouse, not by the benchmark; the
fix is generic production hygiene for any large ingest.
## Implementation notes
Shipped on branch `write-feature-spec-wiki` (squash-merge target). All 12
requirements and every acceptance criterion are covered by committed code and
tests; the full `@kaelio/ktx` package suite is green.
What was built and where:
- **Shared content-keyed durability primitive** — `context/cache/content-result-cache.ts`
+ `sqlite-content-result-cache.ts` (`SqliteContentResultCache`, `local_content_results`).
Scan was migrated onto it in the same change (`context/scan/sqlite-local-enrichment-state-store.ts`
is now a thin adapter; the old `local_scan_enrichment_stages` table is dropped),
so no second copy exists (D3 / req 11).
- **Content-keyed WU cache + replay** — `context/ingest/work-unit-cache.ts`
(`computeIngestWorkUnitInputHash` over raw/dependency bytes + source identity +
CLI version + prompt fingerprint + model role; success-only `saveSuccessfulWorkUnitCache`).
Replay/recompute and stale-recompute state refresh wrap the WU loop in
`ingest-bundle.runner.ts` (D1/D2/D4 / reqs 1–4).
- **Non-fatal final gate** — `artifact-gates.ts` `validateFinalIngestArtifacts`
returns structured findings; `context/ingest/final-gate-prune.ts` deterministically
drops self-invalid sources and prunes dangling edges in a single pass, then a
confirm gate runs before squash (D5/D6 / reqs 5–8). `finalGatePrunedReferences`
/ `finalGateDroppedSources` are recorded in the report + trace and surface as a
`partial` outcome (D7 / req 10). `repairFinalGateFailure` and its tests are
deleted (req 9).
Deviations / decisions worth noting (all preserve spec intent):
- **Cache stores artifact content snapshots (payload schema v2), not just a raw
git patch.** Replay materializes the owner's artifacts against the *current*
base, so a ref pruned in one run because a sibling failed is restored for free
on a later run once the sibling exists — without re-running the owner's agent
loop (D2/D6 / req 7 self-heal). A drifted/stale snapshot degrades to recompute.
- **Final-gate prune/drop resolves sources through the canonical
`resolveSlSourceFile` resolver**, not a derived `semantic-layer/<conn>/<name>.yaml`
path, so it works for uppercase / hash-derived source filenames (not only
lowercase demo names).
- **`executeWorkUnit` defers pruneable cross-artifact findings** (missing join
target / wiki ref / sl_ref) to the final gate instead of soft-failing the WU;
only intrinsic `source_validation` failures remain fatal at the WU level. This
is what lets a sibling-failed WU's owner survive to be pruned rather than be
excluded upstream (reqs 5–7, "no cascade").
- The raw report record keeps `status: 'completed'`; partial completion is derived
by `ingestReportOutcome` from the populated prune/drop fields.

66
spider2-specs/todo/03-multi-connection-routing-in-analytics-skill.md
View file

@ -1,66 +0,0 @@
# Multi-connection routing guidance in the ktx-analytics skill
## Problem
The agent-facing `ktx-analytics` skill (installed into agent environments via
the ktx skills/install mechanism, see `.ktx/agents/install-manifest.json` in
projects) describes the query workflow — wiki_search → sl_read_source →
sl_query / sql_execution — but assumes the connection is obvious. In a
multi-connection project nothing tells the agent to *first decide which
connection the question is about*, and several tools silently require it:
- `sql_execution`, `sl_read_source`, `entity_details`: `connectionId`
**required**;
- `sl_query`, `discover_data`, `dictionary_search`: optional, but
auto-inference only works with exactly one connection
(`local-query.ts` `resolveLocalConnectionId` ~29-38 — throws with zero or
multiple connections).
An agent that skips routing either errors out or, worse, queries the wrong
database when names overlap.
## Generic use case
Any ktx project with more than one connection — the common shape for a data
org (warehouse + product DB + events DB). Routing is the first step of every
question, and the skill should encode it so individual agents don't have to
rediscover it.
## Requirements
1. **Add an explicit routing step (step 0) to the skill's workflow:**
- Call `connection_list` to see what exists.
- Match the question's domain to a connection using connection ids/names,
`discover_data` hits, and wiki context — not guesswork.
- If genuinely ambiguous after discovery, ask the user rather than pick.
2. **Thread the resolved `connectionId` everywhere:** all subsequent
`sl_query`, `sql_execution`, `sl_read_source`, `entity_details`,
`dictionary_search`, `discover_data` calls, and `wiki_search` once spec 01
lands (search scoped to the resolved connection plus unscoped pages).
3. **Single-connection projects stay frictionless:** the skill should say
routing is trivial when `connection_list` returns one entry — don't add a
mandatory ceremony step for the common simple case.
4. **Capture routing knowledge:** when the agent learns a non-obvious
question-domain → connection mapping, the skill should encourage
`memory_ingest` so the mapping becomes wiki knowledge for next time.
This is a docs/prompt change in the skill content (plus any skill-install
plumbing if the skill is versioned); no engine changes required.
## Acceptance criteria
- In a fixture project with ≥2 connections, an agent following the skill
resolves the correct connection before its first data query, and no tool
call fails with "connectionId is required".
- In a single-connection project the skill-driven flow is unchanged (no
extra mandatory steps).
- Skill text nowhere assumes a default/implicit connection.
## Benchmark context (motivation only)
Spider 2.0-Lite local subset = 30 SQLite connections in one project; every
one of the 135 questions targets exactly one of them. Connection ids are set
to the benchmark's database names, so with this skill guidance routing is
mechanical (`connection_list` + name match) and needs no benchmark-specific
instructions — which is the point: the harness gives the agent only the
question text.

51
spider2-specs/todo/04-offline-schema-docs-adapter.md
View file

@ -1,51 +0,0 @@
# Offline schema-documentation ingest adapter
> **Priority: LOW / backlog.** Explicitly **not** needed for the Spider
> 2.0-Lite benchmark — we verified the benchmark's offline schema files
> (DDL dumps + sample-row JSONs) are a strict subset of what the live SQLite
> scan already captures (DDL, types, PKs, sample values, cardinality
> profiling). Implement specs 01-03 first; pick this up only if a real
> use case shows up.
## Problem
The ingest pipeline's schema knowledge comes from live database scans
(`live-database` adapter) or BI-tool adapters (metabase, looker, dbt…).
There is no adapter for **offline schema documentation**: files describing
tables/columns that exist outside the database — column-description
spreadsheets, data dictionaries, DDL exports with comments, hand-maintained
schema docs.
## Generic use case
Teams whose richest schema documentation lives outside `information_schema`:
a wiki export of column meanings, a governance tool's CSV data dictionary,
DDL files with COMMENT clauses the production scan can't see, or
environments where ktx has no live access at all and must build the semantic
layer from documentation alone.
## Requirements (sketch — refine when picked up)
1. A new ingest adapter (peer of `metabase`/`dbt` in
`context/ingest/adapters/`) consuming a configured local path of schema
docs per connection.
2. Input formats to start: DDL files (`.sql`/`.csv` of CREATE statements)
and tabular column dictionaries (CSV/JSON: table, column, description,
…). Extensible to other formats.
3. Output: **enrichment, not duplication** — merge descriptions/metadata
into the manifest-backed semantic-layer sources and dictionary for the
matching connection. Where a live scan exists, offline docs fill gaps
(descriptions, enum meanings, deprecation notes) and flag drift
(documented column missing from live schema and vice versa) rather than
creating parallel wiki pages that duplicate schema info.
4. Works without live database access (documentation-only bootstrap of a
connection's semantic layer), clearly marked as unverified-against-live.
## Acceptance criteria (sketch)
- Given a connection with a live scan plus an offline column dictionary,
semantic-layer sources carry the documented descriptions, and drift
between doc and live schema is reported.
- Given a connection with docs only (no live access), `sl list`/`sl read`
expose manifest sources built from the docs.
- No wiki pages are created that merely restate table/column lists.

59
spider2-specs/todo/05-composite-key-join-detection.md
View file

@ -1,59 +0,0 @@
# Composite-key (multi-column) join detection
> Priority: MEDIUM. Found empirically during the first Spider2-lite sqlite
> smoke test (2026-06-13): relationship detection emitted **zero joins** for a
> database whose fact tables are linked only by composite keys. Agents still
> answered correctly by inferring the join from shared `grain`, so this didn't
> cost benchmark points — but it forces inference that explicit joins would
> remove, and the gap is generic.
## Problem
Relationship detection appears to emit only single-column joins. For the IPL
sqlite database, every table came back with `joins=0`, even though its fact
tables are connected by a 4-column composite key
(`match_id, over_id, ball_id, innings_no`) shared across `ball_by_ball`,
`batsman_scored`, `extra_runs`, and `wicket_taken`. The semantic layer did
correctly record that shared key as each table's `grain`, which is why agents
could recover the relationship — but no `joins:` entries were produced for the
fact-to-fact links.
## Generic use case
Event/fact tables keyed by composite business keys are common: ledger lines
(`account_id, period, line_no`), telemetry (`device_id, ts, metric`), sports
ball-by-ball, EAV/log schemas. Whenever there are no single-column FKs but a
multi-column key recurs across tables, ktx should detect and document the join
so agents (and `sl_query`) don't have to infer it.
## Requirements
1. Relationship detection considers **multi-column** join candidates, not just
single-column ones. A strong signal already exists in ktx: when two tables
share an identical (or subset/superset) declared `grain`, that grain is a
prime composite-join candidate.
2. Emitted joins carry the full composite condition, e.g.
`on: a.match_id = b.match_id AND a.over_id = b.over_id AND a.ball_id = b.ball_id AND a.innings_no = b.innings_no`,
with a sensible `relationship` cardinality.
3. The existing validation/threshold machinery
(`scan.relationships.acceptThreshold` etc.) applies to composite candidates
too; profile-based validation should check join selectivity on the full key.
4. No regression for single-column joins; don't explode combinatorially —
bound candidate generation (e.g. only consider shared-grain keys and
declared/!inferred PK overlaps, cap column count).
5. `sl_query` can compile a join across a composite-key relationship.
## Acceptance criteria
- For a fixture with two tables sharing a 3- or 4-column grain and no
single-column FK, ingest emits a composite join between them with the full
multi-column `on` condition.
- `sl read <source>` shows the composite join; `sl_query` can traverse it.
- Single-column join detection is unchanged on existing fixtures.
## Benchmark context (motivation only)
IPL (and similar ball-by-ball/event schemas in the Spider2-lite local set)
have no single-column FKs; their joins are entirely composite. Explicit
composite joins would let the agent rely on documented relationships instead
of inferring them from grain.

89
spider2-specs/todo/13-canonical-authoritative-source-measures.md
View file

@ -1,89 +0,0 @@
# Canonical / authoritative-source measures in the semantic layer
## Problem
Many schemas contain an **authoritative table** that already encodes a metric's
business rules — an official standings/leaderboard table, a general-ledger or
period-end balance table, a materialized summary/snapshot — alongside the **raw
transactional** rows the metric *could* be re-derived from. Re-deriving the metric
from the raw rows frequently diverges from the canonical definition, because the
authoritative table bakes in rules the raw data doesn't expose (drop-scores,
penalties, adjustments, reconciliations, as-of snapshots).
Today ktx's semantic layer doesn't distinguish "authoritative summary" tables from
raw fact tables, so the analytics skill has no signal that one source is canonical
for a metric — and the agent often re-derives from raw rows and gets a defensible-
but-different number.
## Generic use case (independent of any benchmark)
- "Championship points per competitor this season" — a sports schema may hold both
raw per-event results AND an official standings table that applies drop-scores
and penalties. The standings table is the canonical source; summing raw results
is wrong.
- "Account balance as of month end" — prefer a ledger/balance-snapshot table over
re-summing every transaction (which may miss adjustments).
- "Monthly recognized revenue" — prefer a finance summary table over re-deriving
from line items.
In each case a real analyst should be steered to the authoritative source.
## Requirements
1. **Detect candidate authoritative tables during ingest.** Heuristics only —
e.g. tables whose name/role suggests a summary (`*standings*`, `*balance*`,
`*summary*`, `*snapshot*`, `*ledger*`), tables that are a coarser-grained
aggregation of another table, or tables documented as authoritative in provided
docs/wiki. Surface them as such in the semantic layer.
2. **Represent the metric as an SL measure backed by the authoritative table.**
Where a canonical source exists, define the measure over it so a query for that
metric resolves to the authoritative source by default. (The analytics skill
already prefers SL measures over raw SQL — spec 07/skill rule — so this plugs
into existing behavior.)
3. **Keep raw re-derivation available** as a non-default alternative; the measure
documents which source it uses and why, so the choice is transparent and
overridable.
## Fairness boundary (HARD — this spec is fairness-sensitive)
The choice of authoritative source MUST be driven by **schema/structure or provided
documentation** — the table exists, is structured as a summary, or is documented as
authoritative. It must **NEVER** be driven by observing which interpretation matches
a benchmark gold answer. Concretely:
- ✅ Fair: "a table named/structured as official standings exists and aggregates the
raw results → treat it as the canonical points source."
- ❌ Forbidden: "for question X, use table T because that's what reproduces the gold
result." That is per-instance gold-tuning (cheating) and must not appear in ktx,
the ingest heuristics, or any mapping.
If a metric is genuinely underspecified and only the gold answer disambiguates the
intended source, it is **not fairly fixable** — leave it. Whether this feature helps
any specific benchmark instance is therefore *conditional* on a real schema/doc basis
existing; do not manufacture one.
## Leak-safety (hard constraint)
No benchmark table names, queries, gold values, or instance-specific mappings
anywhere in the spec, the heuristics, or tests. Examples must be synthetic/generic.
## Acceptance criteria
- Ingest can flag candidate authoritative/summary tables via generic heuristics
(name/role/aggregation/doc signals), with no benchmark-specific rules.
- The semantic layer can express a measure as backed by a designated authoritative
source; the skill resolves the metric to it by default; raw re-derivation remains
available and the choice is documented.
- Tests use synthetic schemas only; no gold-derived mappings exist anywhere.
## Benchmark context (motivation only)
Some SQLite-subset metric questions are underspecified between a raw-derivation and
an authoritative-table interpretation (e.g. season points from raw results vs an
official standings table). This is the roadmap's "canonical semantic-layer measures
from schema + provided docs" item. It is fair ONLY where schema/docs support one
source; the gold-only cases are explicitly out of scope (fixing them would require
tuning to gold). Larger than the spec 09–12 skill-content tweaks: this touches
ingest + the semantic-layer model.

57
spider2-specs/todo/17-lifecycle-event-metrics.md
View file

@ -1,57 +0,0 @@
# 17 — Lifecycle-event metrics in the semantic layer
**Status:** draft (intake). Requirement-level; the implementer refines into `specs/17-*.md`.
## Problem / requirement
Many entities carry **several lifecycle timestamps** for the same record — an order has
`placed/purchased`, `approved`, `shipped/carrier-handoff`, `delivered`, and `estimated-delivery`
times; a ticket has `opened`, `assigned`, `resolved`, `closed`; a payment has `initiated`,
`authorized`, `settled`. When an analyst asks for a count/volume/rate of records **in a named
completed state, by period** ("delivered orders by month", "resolved tickets per week", "settled
payments by day"), the correct time anchor is the timestamp of *that named event*, not the
record-creation timestamp.
Today ktx ingests these timestamps as **peer date dimensions** with good column descriptions, but it
does **not model the lifecycle event itself** — so nothing in the semantic layer tells a solver (or a
human) that "delivered orders over time" should be anchored to the delivery timestamp. The choice is
left to per-query reasoning, which is exactly where it goes wrong. (A companion analytics-skill rule
now nudges the *solver* — ktx commit `226341cf` — but the durable, reusable home for this is the
**model**, so any consumer of the semantic layer gets it for free.)
**Requirement:** during enrichment/ingestion, when a source has a state/status column plus one or more
lifecycle timestamps whose names/descriptions map to that state's values, infer **lifecycle-event
metrics** — e.g. a `delivered_orders` metric defined as `COUNT(*)` filtered to the delivered state with
its **default time dimension** set to the matching event timestamp (`order_delivered_customer_date`),
distinct from the creation-anchored `orders` metric. Keep the inference conservative and
source-traceable (column names + enriched descriptions only); never invent a state/timestamp pairing
that the schema/descriptions don't independently support.
## Sketch (implementer to refine)
- Detect (state column, lifecycle-timestamp) pairs from column names + enrichment descriptions
(e.g. status value `delivered` ↔ `*_delivered_*_date`; `resolved` ↔ `resolved_at`).
- Emit a metric per detected completed state: filter = the state predicate, grain = record,
`defaultTimeDimension` = the matching event timestamp.
- Surface these via `discover_data` / `entity_details` so "delivered orders over time" retrieves the
delivery-anchored metric rather than a bare row count over the creation date.
- Gate behind the existing `enrichment.mode: llm` path; respect the conservative-inference bar
(precision over recall — a wrong pairing is worse than none).
## Generic use case (independent of the benchmark)
Any operational/transactional schema (e-commerce orders, support tickets, payments, claims, shipments)
has this multi-timestamp lifecycle shape. An analyst asking "how many X were <completed-state> last
month" almost always means *entered that state* last month. Encoding the event→timestamp mapping in the
model makes every downstream question (BI tool, ad-hoc SQL, an LLM agent) pick the right anchor without
re-deriving it, and prevents the silent "grouped by when they started" error.
## Benchmark context (motivation only — not a benchmark-specific rule)
Surfaced by the `spider2-autofix` loop, round r1: Spider 2.0-Lite `Brazilian_E_Commerce` cases local028
("delivered orders for each month") and local031 ("highest monthly delivered orders volume") both failed
because the solver bucketed delivered orders by `order_purchase_timestamp` instead of
`order_delivered_customer_date`. The trace showed the solver had both columns and even compared both
date bases for local031 before choosing purchase. A skill-text rule flipped both cases this round; this
spec is the **model-layer** form of the same fix, which would make the right anchor the default for any
solver and any lifecycle schema.