apunkt/ktx
1
0
Fork 0
mirror of https://github.com/Kaelio/ktx.git synced 2026-06-10 08:05:14 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge remote-tracking branch 'origin/main' into fix-sl-query-source-column-type

Browse source 
# Conflicts:
#	packages/context/skills/metabase_ingest/SKILL.md
#	packages/context/skills/sl_capture/SKILL.md
This commit is contained in:
Andrey Avtomonov 2026-05-15 01:43:02 +02:00
commit cd49d5d4ae
168 changed files with 3567 additions and 1621 deletions
Download patch file Download diff file Expand all files Collapse all files

6
packages/context/prompts/memory_agent_backfill.md
View file

@ -3,7 +3,7 @@ You are backfilling knowledge from a historical chat transcript or archived SQL
</role>
<stance>
Moderately conservative. Historical content is not directly steering current work, so spurious captures will surface in future chats and annoy users. But genuine patterns are worth saving — these backfills exist because the content is known to contain value.
Moderately conservative. Historical content is not directly steering current work, so spurious captures will surface in future chats and annoy users. But genuine patterns are worth saving - these backfills exist because the content is known to contain value.
Capture only when the signal is unambiguous: a metric definition stated plainly, a reusable SQL pattern, a documented correction, a durable business rule. Skip casual chatter and ambiguous interpretations.
</stance>
@ -12,10 +12,10 @@ Capture only when the signal is unambiguous: a metric definition stated plainly,
1. Read the wiki and SL indexes to avoid creating duplicates.
2. If the content has wiki-style signal, load the `wiki_capture` skill and follow its workflow.
3. If the content has SL-style signal, load the `sl` skill and follow its Part 3 workflow.
4. Prefer updating existing entries over creating new ones — backfills often duplicate existing knowledge.
4. Prefer updating existing entries over creating new ones - backfills often duplicate existing knowledge.
5. When done, exit the loop.
</workflow>
<scope>
Wiki writes follow the session's scope selection (USER for user-scoped enabled, GLOBAL otherwise). The `wiki_write` tool picks automatically — focus on capture judgment.
Wiki writes follow the session's scope selection (USER for user-scoped enabled, GLOBAL otherwise). The `wiki_write` tool picks automatically - focus on capture judgment.
</scope>

8
packages/context/prompts/memory_agent_bundle_ingest_reconcile.md
View file

@ -3,12 +3,12 @@ You are the reconciliation agent for a multi-file ingest bundle. Stage 3 WorkUni
</role>
<stance>
Parsimonious. Stage 3 WUs already loaded `ingest_triage` and handled conflicts they saw. Your sweep is the safety net for contradictions that are only visible when you can see the whole job at once — e.g. two WUs that each looked clean in isolation but collectively form a near-duplicate cluster. Do not redo work Stage 3 already did.
Parsimonious. Stage 3 WUs already loaded `ingest_triage` and handled conflicts they saw. Your sweep is the safety net for contradictions that are only visible when you can see the whole job at once - e.g. two WUs that each looked clean in isolation but collectively form a near-duplicate cluster. Do not redo work Stage 3 already did.
</stance>
<workflow>
1. Load `ingest_triage`, then `sl_capture` + `wiki_capture`.
2. Call `stage_list()` for the full index of this job's writes. If it is empty AND you have no evictions, exit — the runner short-circuits this case but the skill still teaches you to bail fast.
2. Call `stage_list()` for the full index of this job's writes. If it is empty AND you have no evictions, exit - the runner short-circuits this case but the skill still teaches you to bail fast.
3. If the system prompt includes `<canonical_pins>`, apply those pins before flagging a same-name or near-duplicate conflict. A pinned `canonicalArtifactKey` keeps the contested name when it is present in the Stage Index; competing variants keep or receive disambiguated names.
4. Sweep both exact-key conflicts and near-duplicate writes. Compare WUs that wrote overlapping SL source names, overlapping wiki keys, the same `tables:` or `sl_refs:` action details, or obviously equivalent topic titles under different wiki keys. Call `stage_diff` to see the actual difference, and use `wiki_read`/`sl_read_source` when two different keys appear to describe the same table, metric, or source-of-truth mapping. If they're the same content, leave one canonical artifact and record the duplicate as subsumed. If they differ per `ingest_triage` rules, apply the correct resolution (rename + capture; election of canonical; silent replace for expression-only re-ingest change; or pinned canonical), then call `emit_conflict_resolution` with the artifact key and decision.
5. For any `wiki_write`, `wiki_remove`, `sl_write_source`, or `sl_edit_source` call you make during reconciliation, include `rawPaths` with only the raw paths that directly caused that reconciliation action.
@ -24,6 +24,6 @@ Wiki keys must be flat slugs, not directory paths. If a Stage 3 page used a path
</scope>
<do_not>
- Do not overwrite a Stage 3 WU's resolution that already matches `ingest_triage` output — that's churn.
- Do not treat two SL sources with the same logical meaning but legitimately different domains (e.g. `finance.revenue` and `marketing.revenue`) as a conflict — that's by design.
- Do not overwrite a Stage 3 WU's resolution that already matches `ingest_triage` output - that's churn.
- Do not treat two SL sources with the same logical meaning but legitimately different domains (e.g. `finance.revenue` and `marketing.revenue`) as a conflict - that's by design.
</do_not>

8
packages/context/prompts/memory_agent_external_ingest.md
View file

@ -10,19 +10,19 @@ A single artifact typically produces multiple actions: one SL source per table/v
<workflow>
1. Review the wiki and SL indexes in the prompt. Prefer updating existing entries over creating duplicates.
2. Load the `sl` skill for SL-writes and `wiki_capture` for wiki-writes. Both skills describe schema, decision rules, and editing patterns — follow them.
2. Load the `sl` skill for SL-writes and `wiki_capture` for wiki-writes. Both skills describe schema, decision rules, and editing patterns - follow them.
3. For each distinct element in the artifact (table/view, measure, dimension group, derived column, computed filter, business rule, alias): decide whether it belongs in the SL, in the wiki, or both.
4. Write SL sources first (so they have stable names), then wiki pages that reference them via `sl_refs`.
5. When the artifact mixes data definitions with business rules, capture BOTH — one in each store, linked.
5. When the artifact mixes data definitions with business rules, capture BOTH - one in each store, linked.
6. When you're done, exit the loop without calling any more tools.
</workflow>
<scope>
All wiki writes go to the GLOBAL scope — they will be visible to every user of this KTX project. Phrase wiki pages as objective business knowledge, not personal preference. The `wiki_write` tool handles scope selection automatically for external ingest.
All wiki writes go to the GLOBAL scope - they will be visible to every user of this KTX project. Phrase wiki pages as objective business knowledge, not personal preference. The `wiki_write` tool handles scope selection automatically for external ingest.
</scope>
<do_not>
- Do not fabricate measures, joins, or rules that aren't in the artifact.
- Do not invent column names. If a type is unclear, omit it rather than guess.
- Do not mirror presentation hints (LookML `link:`, `map_layer_name:`, HTML formatting) into SL — those belong in wiki if anywhere.
- Do not mirror presentation hints (LookML `link:`, `map_layer_name:`, HTML formatting) into SL - those belong in wiki if anywhere.
</do_not>

4
packages/context/prompts/memory_agent_research.md
View file

@ -1,5 +1,5 @@
<role>
You capture durable knowledge from an analytics assistant's chat turn. The user just asked a question, the assistant answered, and you are running after the turn to decide what — if anything — is worth saving for future chats.
You capture durable knowledge from an analytics assistant's chat turn. The user just asked a question, the assistant answered, and you are running after the turn to decide what - if anything - is worth saving for future chats.
</role>
<criteria>
@ -21,7 +21,7 @@ Skip:
2. Identify durable knowledge OR reusable data patterns in the turn.
3. If the turn has wiki-style signal (preferences, definitions, conventions), load the `wiki_capture` skill and follow its workflow.
4. If the turn has SL-style signal (reusable metric aggregations, new joins, derived dimensions), load the `sl` skill and follow its Part 3 (capture) workflow.
5. A single turn can produce BOTH a wiki page and an SL source — load both skills and author the edge once on the wiki via `sl_refs: [source_name]`. The reverse edge (wiki pages that cite the SL source) is derived by the reconciler; do not set `knowledge_refs:` on the SL side.
5. A single turn can produce BOTH a wiki page and an SL source - load both skills and author the edge once on the wiki via `sl_refs: [source_name]`. The reverse edge (wiki pages that cite the SL source) is derived by the reconciler; do not set `knowledge_refs:` on the SL side.
6. When you're done, exit the loop without calling any more tools. Do NOT emit a final text summary.
</workflow>

4
packages/context/prompts/skills/page_triage_classifier.md
View file

@ -12,11 +12,11 @@ Reusable templates and scripts are durable knowledge regardless of subject matte
Analytics evidence (BI tools like Looker, Metabase, Tableau) is durable knowledge of *how the organization defines its metrics and segments*. The `signals.objectType` tells you what you are looking at:
- `looker_explore` (or any explore-like analytics surface) -> `full` by default. Explores enumerate dimensions, measures, and joins — these are the canonical schema-of-the-business and warrant the full WorkUnit agent so each measure can become a candidate. Skip only if the excerpt is empty or contains zero measures and zero descriptive text.
- `looker_explore` (or any explore-like analytics surface) -> `full` by default. Explores enumerate dimensions, measures, and joins - these are the canonical schema-of-the-business and warrant the full WorkUnit agent so each measure can become a candidate. Skip only if the excerpt is empty or contains zero measures and zero descriptive text.
- `looker_dashboard` (or any named dashboard with tile queries, filters, calculated fields) -> `full` when it has multiple tiles or named metrics, `light` when one or two tiles with trivial fields, `skip` only when usage hints make it clear it is unused (e.g. `queryCount30d` and `uniqueUsers30d` are both zero) AND there are no calculated fields, filters, or named tiles worth extracting.
- `looker_look` (or any saved query) -> `light` when the query is a simple field listing, `full` when it has custom calculations, non-trivial filters, or aggregation expressions, `skip` only when usage is zero AND the query is a default field listing.
Treat dashboard/Look filter values, saved aggregations, calculated fields, and named tiles as candidate metric/segment definitions — they are durable. Do **not** mark BI evidence as `skip` solely because it is "configuration" or "tied to a data model"; that is exactly the durable knowledge we want to capture.
Treat dashboard/Look filter values, saved aggregations, calculated fields, and named tiles as candidate metric/segment definitions - they are durable. Do **not** mark BI evidence as `skip` solely because it is "configuration" or "tied to a data model"; that is exactly the durable knowledge we want to capture.
Examples:

6
packages/context/skills/dbt_ingest/SKILL.md
View file

@ -6,13 +6,13 @@ callers: [memory_agent]
# dbt → KTX (bundle ingest)
Use this skill for **uploaded** dbt projects (`dbt_project.yml` at stage root, `models/**`, `sources/**`, `schema.yml`). There is **no** `fetch()` in v1 — scheduled `dbt parse` / `manifest.json` pulls are out of scope; host-provided dbt sync may still backfill structured test metadata into `_schema` on the next sync.
Use this skill for **uploaded** dbt projects (`dbt_project.yml` at stage root, `models/**`, `sources/**`, `schema.yml`). There is **no** `fetch()` in v1 - scheduled `dbt parse` / `manifest.json` pulls are out of scope; host-provided dbt sync may still backfill structured test metadata into `_schema` on the next sync.
## Mapping (models / sources → SL)
| dbt | KTX | Notes |
|-----|--------|--------|
| `models:` entry with `columns:` | **Overlay** on the manifest table with the same name (after `discover_data` / `entity_details`) | One SL source per physical table; model name may differ from DB name — resolve with `read_raw_file` + warehouse context. |
| `models:` entry with `columns:` | **Overlay** on the manifest table with the same name (after `discover_data` / `entity_details`) | One SL source per physical table; model name may differ from DB name - resolve with `read_raw_file` + warehouse context. |
| `sources:` → `tables:` | Same as models; use `identifier` when present instead of logical `name`. | Schema + name must match how the connection sees tables. |
| Column `description` | `column_overrides[].descriptions.user` on the overlay | Do not overwrite `dbt` description keys from sync. |
| `data_tests: not_null` / `unique` | Short hint in column `descriptions` or notes: “dbt: not null”, “dbt: unique” | Full structured metadata lands in manifest via **sync**; the skill keeps bundle-time SL text useful for the agent. |
@ -73,4 +73,4 @@ If the same bundle also has MetricFlow `semantic_models:` / `metrics:`, the **`m
- Do not invent column names, grain keys, or measure expressions from dbt model names, descriptions, tests, or common naming patterns.
- Do not write computed `columns:`, `column_overrides:`, `grain:`, or `measures:` for a dbt model unless those exact column names are confirmed by dbt YAML columns or warehouse schema discovery.
- Do not invent joins from `relationships` tests if the target model/table is not found in SL or the warehouse.
- Do not read `peerFileIndex` paths — use `read_raw_file` only on `rawFiles` and `dependencyPaths` from the WorkUnit.
- Do not read `peerFileIndex` paths - use `read_raw_file` only on `rawFiles` and `dependencyPaths` from the WorkUnit.

12
packages/context/skills/ingest_triage/SKILL.md
View file

@ -4,7 +4,7 @@ description: Classify and resolve conflicts detected during bundle ingest (struc
callers: [memory_agent]
---
# Ingest Triage — conflict classification and resolution
# Ingest Triage - conflict classification and resolution
This skill is loaded in two contexts:
- By a Stage 3 WorkUnit agent when `sl_discover` reveals that a prior WU (or a prior sync) already wrote something that overlaps with what the current WU is about to write.
@ -15,12 +15,12 @@ Apply the rules below before every write that could collide with an existing art
## Decision tree
1. **Is this the same artifact I'm producing now, or a different one with the same name?**
Read both. If names match and content matches (modulo whitespace): no conflict — skip the write, the prior one stands.
Read both. If names match and content matches (modulo whitespace): no conflict - skip the write, the prior one stands.
2. **If content differs, is it an expression-only change (e.g. a different `sql:` body for the same measure name, same grain, same columns)?**
Re-ingest change (expression-only): silently replace via `sl_edit_source`. No flag.
3. **If the difference is structural — grain, columns, filter, join shape — is the current bundle the re-ingest of a previously-ingested bundle (i.e. `priorProvenance` has a row for this raw file and artifact)?**
3. **If the difference is structural - grain, columns, filter, join shape - is the current bundle the re-ingest of a previously-ingested bundle (i.e. `priorProvenance` has a row for this raw file and artifact)?**
Re-ingest change (semantic break): replace + flag. Record in the IngestReport's `conflicts_resolved` list with `flagged_for_human: true`.
4. **If there's no prior-sync row (both are from THIS job), check for same-ingest contradictions:**
@ -37,11 +37,11 @@ Apply the rules below before every write that could collide with an existing art
## Why same-ingest vs re-ingest differs
Within ONE bundle there's no user signal telling us which duplicate wins — we capture all variants and flag. Across bundles, re-uploading IS the signal that the new state is intended — we replace silently for expression changes and flag for semantic breaks.
Within ONE bundle there's no user signal telling us which duplicate wins - we capture all variants and flag. Across bundles, re-uploading IS the signal that the new state is intended - we replace silently for expression changes and flag for semantic breaks.
## Naming disambiguation hints
When you rename to disambiguate, prefer domain suffixes that match the containing view/table/collection name: `customers.churn_risk_score` → `customers.churn_risk_engagement_based` (if the `customer_churn` view computes it from engagement); `billing.churn_risk_score` → `billing.churn_risk_billing_based`. Avoid numeric suffixes (`churn_risk_1`, `churn_risk_2`) — they disclose nothing.
When you rename to disambiguate, prefer domain suffixes that match the containing view/table/collection name: `customers.churn_risk_score` → `customers.churn_risk_engagement_based` (if the `customer_churn` view computes it from engagement); `billing.churn_risk_score` → `billing.churn_risk_billing_based`. Avoid numeric suffixes (`churn_risk_1`, `churn_risk_2`) - they disclose nothing.
## Applying canonical pins
@ -62,7 +62,7 @@ When you perform rename + capture, also write one page named `<canonical-concept
- One paragraph per variant: what it computes, where it came from (raw file + line range), when to use it.
- A closing "Choosing between these" paragraph if the variants are legitimately domain-specific.
Do not attempt to rank variants or pick a "best" — that's user-override territory.
Do not attempt to rank variants or pick a "best" - that's user-override territory.
## Silence rules

16
packages/context/skills/lookml_ingest/SKILL.md
View file

@ -17,7 +17,7 @@ LookML views map to SL sources, `measure:` to measures, `explore: { join: }` to
| `view: X { sql_always_where: <p> }` | **Standalone** with `sql: SELECT * FROM <base> WHERE <p>` | Enforcement, not opt-in |
| `explore: { join: Y { sql_on: …; relationship: … } }` | `joins:` entry `{ to: Y, on: "<local> = Y.<col>", relationship: … }` | On the overlay or standalone |
| `conditionally_filter` / `always_filter` | `segments: [{ name, expr }]` | Callers reference by name |
| Manifest entry | `_schema/*.yaml` | **Never edit** — auto-imported |
| Manifest entry | `_schema/*.yaml` | **Never edit** - auto-imported |
Type map: `date`/`datetime`/`timestamp` → `time`; `yesno` → `boolean`; `number` → `number`; `string` → `string`. Ignore `drill_fields:` (UI only).
@ -92,14 +92,14 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
`sql_execution({connectionName: "warehouse", sql: "SELECT 1 FROM analytics.orders LIMIT 0"})`.
Replace `warehouse`, `analytics`, and `orders` with the verified connection,
schema or dataset, and table from the WorkUnit evidence.
3. Use only those names in `sql:`, `columns:`, and `grain:`. Map each `dimension_group` to ONE `{ name: <physical_col>, type: time, role: time }` entry — never one per timeframe.
3. Use only those names in `sql:`, `columns:`, and `grain:`. Map each `dimension_group` to ONE `{ name: <physical_col>, type: time, role: time }` entry - never one per timeframe.
| LookML input | KTX `columns:` entry |
|---|---|
| `dimension_group: month { type: time; timeframes: [month]; sql: ${TABLE}.month_date ;; }` | `{ name: month_date, type: time, role: time }` |
| `dimension_group: date { type: time; timeframes: [raw, date, week, month]; sql: ${TABLE}.date ;; }` | `{ name: date, type: time, role: time }` — single entry, NOT `date_raw`/`date_date`/`date_week` |
| `dimension_group: date { type: time; timeframes: [raw, date, week, month]; sql: ${TABLE}.date ;; }` | `{ name: date, type: time, role: time }` - single entry, NOT `date_raw`/`date_date`/`date_week` |
**After every `sl_write_source`**: call `sl_validate`. It runs `SELECT * FROM (<your sql:>) LIMIT 0` against the connection. If a column name was invented, the warehouse's `Unrecognized name: …` error comes back verbatim. Treat that as a hard failure — re-read the real columns with `sl_discover` and rewrite.
**After every `sl_write_source`**: call `sl_validate`. It runs `SELECT * FROM (<your sql:>) LIMIT 0` against the connection. If a column name was invented, the warehouse's `Unrecognized name: …` error comes back verbatim. Treat that as a hard failure - re-read the real columns with `sl_discover` and rewrite.
## Provenance markers
@ -110,13 +110,13 @@ When a wiki mixes LookML source prose with `sl_discover` output, tag sections:
Customers fan out many-to-one into `accounts` via `account_id`.
<!-- /from -->
<!-- from: bq_schema -->
`customers.admin_user_id` is nullable — orphan rows exist.
`customers.admin_user_id` is nullable - orphan rows exist.
<!-- /from -->
```
Invisible in most renderers; lets a future pass audit provenance.
## Example 1 — overlay (thin wrapper)
## Example 1 - overlay (thin wrapper)
LookML (excerpt):
@ -155,7 +155,7 @@ joins:
relationship: many_to_one
```
## Example 2 — standalone from `derived_table`
## Example 2 - standalone from `derived_table`
```lookml
view: lab_results {
@ -189,7 +189,7 @@ measures:
- { name: avg_delta, expr: "avg(delta)" }
```
## Example 3 — standalone with `sql_always_where`
## Example 3 - standalone with `sql_always_where`
```lookml
view: rpt_daily_braze_email {

34
packages/context/skills/metabase_ingest/SKILL.md
View file

@ -84,7 +84,7 @@ For each card:
4. Decide:
- Simple aggregation on a table that already has a source → `sl_edit_source` to add a measure.
- Join between tables that should be linked in the SL graph → `sl_edit_source` to add a join.
- Complex derived SQL (CTEs, multi-layer aggregation, scoring models) → `sl_write_source` with `source_type: sql`. When the SQL projects/filters from a single manifest-backed base table, set `inherits_columns_from: <manifest_key>` so columns inherit type and description from the manifest — see `sl_capture` skill for the slim form. Use `sl_discover` to discover the manifest key from the table reference in the SQL (it accepts `MARTS.CONSIGNMENTS`, `ANALYTICS.MARTS.CONSIGNMENTS`, or `CONSIGNMENTS`).
- Complex derived SQL (CTEs, multi-layer aggregation, scoring models) → `sl_write_source` with `source_type: sql`. When the SQL projects/filters from a single manifest-backed base table, set `inherits_columns_from: <manifest_key>` so columns inherit type and description from the manifest - see `sl_capture` skill for the slim form. Use `sl_discover` to discover the manifest key from the table reference in the SQL (it accepts `MARTS.CONSIGNMENTS`, `ANALYTICS.MARTS.CONSIGNMENTS`, or `CONSIGNMENTS`).
- New base table not yet in the semantic layer → `sl_write_source` with `source_type: table`.
- Trivial query (`SELECT *`, simple `COUNT(*)` with no business logic) → do nothing; the runner will record this card as `action_type='skipped'`.
- Duplicate of an existing measure → same as trivial; do nothing for this card.
@ -102,7 +102,7 @@ Overlay shape: `name:` plus any of `measures:`, `segments:`, `descriptions:`, `j
**Join discovery:** When your card's SQL references warehouse tables (e.g. in `FROM` or `JOIN` clauses), call `sl_discover({ query: '<table>' })` before writing. The matching manifest entry's `name` is the value you use in `joins: [- to: <name>]` only when the card output exposes a local key that matches the target source grain (for example `account_id = mart_account_segments.account_id`). Do not declare a KTX join just because the card SQL joins that table internally. If the output only exposes display fields such as `account_name`, keep the SQL source self-contained or project the key before adding the join. Use `many_to_one` for FK-to-dimension joins, `one_to_many` for the reverse.
**Hard rule on join columns (prevents broken joins):** For every join you declare, the local column on the left of `on:` MUST be (a) present in your source's projected output and (b) a key/ID column, never a display value. If the natural FK isn't in your SELECT, add it to SELECT before declaring the join. Joining `account_name = mart_account_segments.account_id` is always wrong — names are not identifiers and the equality produces zero matches. The validator rejects this with a "display value to identifier" error; the tool will refuse to save it. Add `account_id` to your SELECT and join on `account_id = mart_account_segments.account_id`, or omit the join entirely.
**Hard rule on join columns (prevents broken joins):** For every join you declare, the local column on the left of `on:` MUST be (a) present in your source's projected output and (b) a key/ID column, never a display value. If the natural FK isn't in your SELECT, add it to SELECT before declaring the join. Joining `account_name = mart_account_segments.account_id` is always wrong - names are not identifiers and the equality produces zero matches. The validator rejects this with a "display value to identifier" error; the tool will refuse to save it. Add `account_id` to your SELECT and join on `account_id = mart_account_segments.account_id`, or omit the join entirely.
## priorProvenance
@ -114,7 +114,7 @@ If the WU prompt includes a `priorProvenance` section for a card, it tells you w
## Deduplication
Before writing, scan all cards in this WU for near-duplicate groups — cards whose `resolvedSql` shares the same CTEs, base tables, joins, and aggregation structure but differs only in:
Before writing, scan all cards in this WU for near-duplicate groups - cards whose `resolvedSql` shares the same CTEs, base tables, joins, and aggregation structure but differs only in:
- Trailing filters (e.g. `date_trunc(week, date)` vs `date_trunc(month, date)`).
- Minor `WHERE` clause variations.
- Column aliases or output column subsets.
@ -124,7 +124,7 @@ When you find a group of near-duplicates:
1. Create ONE generalized source from the most comprehensive card in the group.
2. Strip card-specific trailing filters from the SQL so the source covers all variants (e.g. keep daily grain instead of filtering to week/month).
3. If each card had a distinct measure or filter, add them as separate measures on the single source.
4. For all cards except the canonical one, do nothing — they'll be recorded as `action_type='skipped'` automatically by the runner.
4. For all cards except the canonical one, do nothing - they'll be recorded as `action_type='skipped'` automatically by the runner.
Do NOT merge cards with fundamentally different business logic, even if they share CTEs.
@ -132,7 +132,7 @@ Do NOT merge cards with fundamentally different business logic, even if they sha
When a card's `resolvedSql` contains `GROUP BY` with aggregation functions (`SUM`, `COUNT`, `AVG`, …):
1. **Detect**: simple aggregation on base tables/joins — `SELECT` with `GROUP BY`, no complex CTEs or window functions.
1. **Detect**: simple aggregation on base tables/joins - `SELECT` with `GROUP BY`, no complex CTEs or window functions.
2. **Decompose**: strip the `GROUP BY` and aggregation functions. Keep `FROM`, `JOIN`, and `WHERE` intact.
3. **Expose row-level columns**: include the grouped-by columns AND the raw columns being aggregated (e.g. `money_out` instead of `SUM(money_out) AS total_money_out`).
4. **Define aggregations as measures**: convert each aggregation into a KSL measure (e.g. `sum(money_out)`).
@ -144,17 +144,17 @@ Exception: keep the pre-aggregated SQL when the query involves multi-CTE pipelin
Every card carries a `resolvedSql` field. Check the staged card's `resolutionStatus` first:
- `resolutionStatus: "resolved"` — `{{#N}}` references are inlined and `[[ ... ]]` optional clauses have been dropped locally. If the resolved SQL contains no other parameters the SQL is executable as-is. If the card had **required** (non-bracketed) `{{ var }}` placeholders, the SQL is prefixed with a placeholder-warning comment block listing every dummy substitution Metabase made — see "Step A" below.
- `resolutionStatus: "fallback"` — Metabase failed to resolve. The SQL still contains `{{#N}}`, `{{#N-name}} alias`, `{{ var }}`, and `[[ ... ]]` syntax. Do the translation steps below before writing a source.
- `resolutionStatus: "resolved"` - `{{#N}}` references are inlined and `[[ ... ]]` optional clauses have been dropped locally. If the resolved SQL contains no other parameters the SQL is executable as-is. If the card had **required** (non-bracketed) `{{ var }}` placeholders, the SQL is prefixed with a placeholder-warning comment block listing every dummy substitution Metabase made - see "Step A" below.
- `resolutionStatus: "fallback"` - Metabase failed to resolve. The SQL still contains `{{#N}}`, `{{#N-name}} alias`, `{{ var }}`, and `[[ ... ]]` syntax. Do the translation steps below before writing a source.
### Step A — Handle dummy-substituted placeholders (resolved cards only)
### Step A - Handle dummy-substituted placeholders (resolved cards only)
When a card has a required `{{ var }}` outside any `[[ ]]` block, the resolver substitutes a **dummy value** purely so Metabase's parser will accept the query. The resulting SQL is prefixed with a comment like:
```sql
-- PLACEHOLDER_WARNING: this SQL was extracted from a Metabase card with
-- unbound template parameters. The placeholders below were substituted with DUMMY
-- values to satisfy Metabase's parser — they DO NOT represent intended filters.
-- values to satisfy Metabase's parser - they DO NOT represent intended filters.
-- Drop the corresponding clauses (or expose them as runtime SL filters) before
-- persisting this SQL as a semantic-layer source.
-- {{ auction_end }} (type=dimension, widget=date/all-options) → '2020-01-01~2020-12-31'
@ -165,7 +165,7 @@ WHERE start_date >= '2020-01-01' AND start_date < '2021-01-01' AND status = 'pla
For each listed placeholder: locate the WHERE clause(s) in the SQL that reference the dummy literal and **drop them**, then strip the warning comment. SL chat-time filters compose narrowing predicates dynamically, so the source should represent the unfiltered dataset.
For `fallback` cards, dropping is simpler — the SQL still has the `[[ ... ]]` brackets and `{{ var }}` placeholders intact:
For `fallback` cards, dropping is simpler - the SQL still has the `[[ ... ]]` brackets and `{{ var }}` placeholders intact:
```sql
-- before:
@ -177,18 +177,18 @@ WHERE 1=1
WHERE 1=1
```
### Step B — Inline `{{#N}}` references (fallback cards only)
### Step B - Inline `{{#N}}` references (fallback cards only)
Resolved cards already have `{{#N}}` inlined for you. For `fallback` cards, each `{{#N}}` (or `{{#N-some-slug}}`) in the SQL refers to another card's `resolvedSql`. The referenced card is in the WU's `rawFiles` or `dependencyPaths`. Read it with `read_raw_file`, then inline its SQL.
If the reference has an alias (`from {{#5996-listing-interactions}} tb`), the **outer** SQL probably uses that alias (`select tb.* ...`, `tb.column_name`, etc.). When you inline, you must EITHER:
1. **Pick a single base table inside the inlined SQL and rename its alias to the outer alias.** Useful when the inlined card is `SELECT * FROM listings JOIN ...` — set the LISTINGS alias to `tb` and `tb.*` keeps working in the outer query.
1. **Pick a single base table inside the inlined SQL and rename its alias to the outer alias.** Useful when the inlined card is `SELECT * FROM listings JOIN ...` - set the LISTINGS alias to `tb` and `tb.*` keeps working in the outer query.
2. **Replace the outer alias references with explicit columns from the inlined SQL.** Useful when the inlined card has multiple JOINs and `tb.*` is ambiguous.
Never leave the outer alias dangling: after inlining, **grep your SQL for the outer alias name and rewrite or remove every reference**. A leftover `tb.*` with no `tb` table is the most common failure mode here.
### Step C — Inlining cleanup checklist
### Step C - Inlining cleanup checklist
After Steps A and B, your SQL must:
@ -209,11 +209,11 @@ For `source_type: sql`:
- If `sl_discover` resolves the table, it is not outside the manifest. Do not write an `unmapped-table-*` fallback for resolved `orbit_raw`, `mart`, or other manifest-backed sources just because they appear inside card SQL.
- If `sl_discover` cannot resolve a referenced table at all, write a single-line `wiki_write` with key `unmapped-table-<table_name>` and `rawPaths: ["cards/<id>.json"]` so the gap is documented, then call `emit_unmapped_fallback` with the staged card path as `rawPath`, `reason: "missing_target_table"`, `tableRef: "<table_name>"`, and `fallback: "wiki_only"`. Do not use this fallback if `sl_discover` resolved the table/source.
Joins on manifest-backed names compose: the manifest's joins are inherited automatically, and any overlay `joins:` are merged on top (deduped by `to` + `on`). Use `disable_joins: ["<on-clause>"]` in the overlay to suppress a specific manifest join. If `sl_discover` shows a manifest-backed source with `Joins: 0` and the warehouse FK metadata is genuinely absent, declaring application-level joins via the overlay is fair game — bootstrap with `sl_write_source` (overlay shape above), then refine via `sl_edit_source`.
Joins on manifest-backed names compose: the manifest's joins are inherited automatically, and any overlay `joins:` are merged on top (deduped by `to` + `on`). Use `disable_joins: ["<on-clause>"]` in the overlay to suppress a specific manifest join. If `sl_discover` shows a manifest-backed source with `Joins: 0` and the warehouse FK metadata is genuinely absent, declaring application-level joins via the overlay is fair game - bootstrap with `sl_write_source` (overlay shape above), then refine via `sl_edit_source`.
## Cross-card references (`{{#N}}`)
Resolved cards (`resolutionStatus: "resolved"`) have these inlined for you. Unresolved cards (`resolutionStatus: "fallback"`) need manual handling — see "SQL translation from raw native to KSL" above.
Resolved cards (`resolutionStatus: "resolved"`) have these inlined for you. Unresolved cards (`resolutionStatus: "fallback"`) need manual handling - see "SQL translation from raw native to KSL" above.
## Provenance markers
@ -237,7 +237,7 @@ Source definitions must follow ktx-sl YAML conventions:
- `columns`: all columns with correct types (`string`, `number`, `time`, `boolean`).
- Time columns: mark with `role: time`.
- `joins`: use correct `relationship` types (`many_to_one` for FK→PK, `one_to_many` for reverse).
- `joins.on`: `local_column = TARGET_SOURCE.target_column` — the right side MUST include the target source name.
- `joins.on`: `local_column = TARGET_SOURCE.target_column` - the right side MUST include the target source name.
- `measures.expr`: aggregation expression (e.g. `"sum(amount)"`); optional `filter` for business rules; required `description`.
Measure naming: descriptive `snake_case` (e.g. `total_revenue`, `avg_order_value`).
@ -250,4 +250,4 @@ Measure naming: descriptive `snake_case` (e.g. `total_revenue`, `avg_order_value
- If two measures differ only by a filter (e.g. `revenue` vs `paid_revenue`), they are distinct.
- Use the card's `name` + `description` to write meaningful measure descriptions.
- When multiple cards in a WU are near-duplicates, create ONE generalized source; the runner will skip the rest automatically.
- Process every card in the WU — don't stop early.
- Process every card in the WU - don't stop early.

26
packages/context/skills/metricflow_ingest/SKILL.md
View file

@ -15,7 +15,7 @@ A MetricFlow `semantic_model` maps to an SL source; MetricFlow `measures` map to
| `semantic_model: X { model: ref('t') }` with measures + dimensions | **Overlay** at `<connId>/X.yaml` with `measures`, computed-only `columns`, `column_overrides`, `joins` | The `model:` ref resolves to a manifest table. |
| `semantic_model: X { model: source('s','t') }` | **Overlay** at `<connId>/X.yaml` over table `t`. | Same shape; `source()` still resolves to a physical table. |
| `semantic_model: X { model: <literal> }` with no manifest entry | **Standalone** with explicit `sql:`, `grain:`, `columns:` | Happens when the dbt manifest isn't available. |
| `semantic_model: Y { extends: X }` | **Merge** Y's measures/dimensions/entities into X's overlay, or write a single overlay named for the most-derived child (Y) containing both X's and Y's primitives | Do not emit a second overlay for X — flatten. |
| `semantic_model: Y { extends: X }` | **Merge** Y's measures/dimensions/entities into X's overlay, or write a single overlay named for the most-derived child (Y) containing both X's and Y's primitives | Do not emit a second overlay for X - flatten. |
| `measures: [{ name, agg, expr }]` | `measures: [{ name, expr: "<agg>(<expr>)" }]` | Aggregation inlined. `agg: count_distinct` → `count(distinct ...)`. |
| `entities: [{ name, type: primary }]` | `grain: [<entity_name-or-expr>]` on the overlay/standalone | Primary/unique entities drive grain. |
| `entities: [{ name, type: foreign }]` | `joins:` entry joining to the primary-entity's semantic_model | Only when a matching primary is discoverable. |
@ -24,10 +24,10 @@ A MetricFlow `semantic_model` maps to an SL source; MetricFlow `measures` map to
| `metrics: [{ type: derived, type_params: { expr, metrics } }]` | **Derived measure** on whichever source owns the referenced measures, with `expr:` referencing measure names | If the metric spans models, still write it once on the source owning the "primary" measure (the one the agent judges most central). Mention the cross-model chain in the description. |
| `metrics: [{ type: ratio, type_params: { numerator, denominator } }]` | Same as derived; `expr: "numerator / NULLIF(denominator, 0)"` if no explicit expr | Safe-division by default. |
| `metrics: [{ type: cumulative, type_params: { window, grain_to_date } }]` | **Standalone** source with a window-function SQL; reference the resulting column as a normal measure | KTX SL has no first-class cumulative primitive (spec Non-goals). |
| `metrics: [{ type: conversion }]` | **Flag for human** — do NOT write. Emit a wiki note describing the intended semantics. | No KTX equivalent in v1. |
| `metrics: [{ type: conversion }]` | **Flag for human** - do NOT write. Emit a wiki note describing the intended semantics. | No KTX equivalent in v1. |
| Metric not mappable | Wiki page `<metric_name>-definition.md` with the full YAML body quoted | Capture the intent even if we can't emit SL. |
Type map: MetricFlow `time` to KTX `time`; `categorical` to `string`; `number` to `number`; `boolean` to `boolean`. Follow `expr` over `name` when both differ — `expr` is the physical column.
Type map: MetricFlow `time` to KTX `time`; `categorical` to `string`; `number` to `number`; `boolean` to `boolean`. Follow `expr` over `name` when both differ - `expr` is the physical column.
Verify each MetricFlow model source table with entity_details before producing
the corresponding sl_write_source.
@ -67,7 +67,7 @@ Within one WorkUnit, multiple semantic_models linked by `extends:` are guarantee
1. Start with the most-derived child (the one that no other semantic_model extends).
2. Walk the `extends:` chain upward, accumulating measures, dimensions, entities.
3. Write ONE overlay/standalone, named for the most-derived child's SL-appropriate name (not the base).
4. Parents that lack their own distinctive content should NOT get a separate overlay. If a parent has unique measures a child doesn't inherit, consider whether the base is used elsewhere — if yes, write both; if no, still one overlay.
4. Parents that lack their own distinctive content should NOT get a separate overlay. If a parent has unique measures a child doesn't inherit, consider whether the base is used elsewhere - if yes, write both; if no, still one overlay.
5. Measure/dimension name collisions: child wins, but note the overridden parent in the overlay's description or in a sibling wiki page.
The spec's worked example has `orders`, `orders_ext` (extends orders), and `metrics/orders_final.yml` (defines `revenue` referencing both). The right output is ONE overlay named `orders_ext` (or `orders` if the team's naming favors the base) containing `order_count`, `gross_amount`, `refund_amount`, and a derived `revenue` measure. Provenance tags point to all three source files.
@ -88,9 +88,9 @@ call `sql_execution` with the same warehouse connection name, for example:
`sql:` must be sourced from raw files, `entity_details`, or a successful SQL
probe.
After every `sl_write_source`, call `sl_validate`. The warehouse will reject invented columns with `Unrecognized name: <name>` — treat as a hard failure and re-read the schema.
After every `sl_write_source`, call `sl_validate`. The warehouse will reject invented columns with `Unrecognized name: <name>` - treat as a hard failure and re-read the schema.
## Cumulative metrics — sql-standalone fallback
## Cumulative metrics - sql-standalone fallback
KTX SL has no first-class `window:` or `grain_to_date:` primitive in v1 (spec Non-goals). Translate a MetricFlow cumulative metric to a standalone SL source with a window-function SQL:
@ -125,7 +125,7 @@ measures:
Pick the time column based on the semantic_model's `defaults.agg_time_dimension` (e.g. `ordered_at`). If the MetricFlow config omits it, probe the base table for time-typed columns and choose the most obvious. After writing the standalone SQL source, call `emit_unmapped_fallback` with `rawPath` set to the MetricFlow file path, `reason: "cumulative_metric_unsupported"`, and `fallback: "sql_standalone"`.
## Conversion metrics — flag for human
## Conversion metrics - flag for human
```yaml
metrics:
@ -159,7 +159,7 @@ name: orders_ext
Line ranges (`#L<start>-<end>`) point to the exact YAML span within the file (the `semantic_models:` entry for its own `name`). Use `read_raw_span` to identify those ranges before writing.
## Example 1 — single semantic_model to overlay
## Example 1 - single semantic_model to overlay
```yaml
# MetricFlow:
@ -185,7 +185,7 @@ measures:
grain: [order_id]
```
## Example 2 — extends chain → one flattened overlay
## Example 2 - extends chain → one flattened overlay
```yaml
# MetricFlow:
@ -232,7 +232,7 @@ measures:
grain: [order_id]
```
## Example 3 — derived metric spanning two semantic_models
## Example 3 - derived metric spanning two semantic_models
```yaml
# models/sales.yml
@ -256,7 +256,7 @@ metrics:
metrics: [{name: revenue}, {name: cost}]
```
Because the WorkUnit bundles all three files (cross-component union via the metric), write the derived measure on ONE of the two sources — pick the source whose domain "owns" the metric (here, `sales` — margin is inherently a sales metric). Cross-source references aren't native in KTX SL; treat the metric's operands as already-resolvable in the target source's query context OR emit a standalone SQL that joins the two tables:
Because the WorkUnit bundles all three files (cross-component union via the metric), write the derived measure on ONE of the two sources - pick the source whose domain "owns" the metric (here, `sales` - margin is inherently a sales metric). Cross-source references aren't native in KTX SL; treat the metric's operands as already-resolvable in the target source's query context OR emit a standalone SQL that joins the two tables:
```yaml
# <connId>/sales.yaml
@ -269,7 +269,7 @@ measures:
```
```yaml
# <connId>/margin.yaml — standalone because it spans two tables
# <connId>/margin.yaml - standalone because it spans two tables
# <!-- from: .../models/sales.yml#L1-8 -->
# <!-- from: .../models/costs.yml#L1-8 -->
# <!-- from: .../metrics/margin.yml#L1-8 -->
@ -292,7 +292,7 @@ measures:
Also write a wiki page at `wiki/global/margin-metric.md` explaining the cross-source origin.
## Example 4 — filtered metric creates a new measure
## Example 4 - filtered metric creates a new measure
```yaml
metrics:

2
packages/context/skills/notion_synthesize/SKILL.md
View file

@ -67,7 +67,7 @@ Search existing wiki pages for the same `tables:` or `sl_refs:` frontmatter and
- Do not create SL sources under the Notion connection just because a page mentions a warehouse, dbt, Looker, or Metabase object. Use the mapped warehouse/source connection after discovery, or emit an unmapped fallback and write wiki-only.
- Distinguish fallback reasons precisely: if a non-Notion warehouse/dbt connection exists but `sl_discover` cannot find the named table/source, use `no_physical_table`; reserve `no_connection_mapping` for cases where there is no plausible non-Notion target connection at all.
- If `sl_discover` resolves the table/source, do not call `emit_unmapped_fallback` for that table. Use the resolved source for `sl_refs`, overlay edits, or wiki-only documentation.
- When calling `emit_unmapped_fallback`, pass the table or source identifier as `tableRef` (e.g. `tableRef: "<schema>.<table>"`) — the tool generates the canonical detail string from the reason code and `tableRef`. Use the optional `clarification` field only to add context that does not contradict the reason. Do not restate the reason in `clarification`.
- When calling `emit_unmapped_fallback`, pass the table or source identifier as `tableRef` (e.g. `tableRef: "<schema>.<table>"`) - the tool generates the canonical detail string from the reason code and `tableRef`. Use the optional `clarification` field only to add context that does not contradict the reason. Do not restate the reason in `clarification`.
## Identifier Verification Protocol

40
packages/context/skills/sl/SKILL.md
View file

@ -1,6 +1,6 @@
---
name: sl
description: KTX's semantic layer — a structured catalog of sources (tables/views), measures, joins, and segments expressed as YAML. Covers the schema and how to query it via `sl_query`. Use when the task involves querying pre-defined metrics (ARR, churn, retention, LTV, MAU) or reading SL source YAML to understand the catalog. Capture is handled by the `sl_capture` skill (memory-agent only).
description: KTX's semantic layer - a structured catalog of sources (tables/views), measures, joins, and segments expressed as YAML. Covers the schema and how to query it via `sl_query`. Use when the task involves querying pre-defined metrics (ARR, churn, retention, LTV, MAU) or reading SL source YAML to understand the catalog. Capture is handled by the `sl_capture` skill (memory-agent only).
---
# Semantic Layer
@ -8,10 +8,10 @@ description: KTX's semantic layer — a structured catalog of sources (tables/vi
KTX's semantic layer (SL) is a structured catalog. Each **source** represents a table, a SQL view, or an overlay that enriches a manifest-backed table with measures, computed columns, joins, and named segments. The catalog is the single source of truth for reusable business metrics.
This skill covers two parts:
- **Part 1** — Schema reference (what an SL source looks like).
- **Part 2** — Querying via `sl_query`.
- **Part 1** - Schema reference (what an SL source looks like).
- **Part 2** - Querying via `sl_query`.
Capture (when and how to add new patterns to the SL) is a separate concern handled by the memory-agent — see the `sl_capture` skill if you are running in capture mode. The research agent **reads** and **queries** the SL via the tools described here; it does not write to it.
Capture (when and how to add new patterns to the SL) is a separate concern handled by the memory-agent - see the `sl_capture` skill if you are running in capture mode. The research agent **reads** and **queries** the SL via the tools described here; it does not write to it.
For capture-time identifier verification, load `sl_capture`. Synthesis writer
skills must verify warehouse identifiers with `discover_data`,
@ -19,7 +19,7 @@ skills must verify warehouse identifiers with `discover_data`,
---
## Part 1 — Schema reference
## Part 1 - Schema reference
An SL source is a YAML file at `semantic-layer/<connectionId>/<source_name>.yaml`. There are three flavors:
@ -34,7 +34,7 @@ descriptions:
measures:
- name: total_revenue
expr: sum(amount)
description: Total order revenue — filter by status or region at query time
description: Total order revenue - filter by status or region at query time
columns: # computed dimensions only
- name: is_large_order
type: boolean
@ -53,7 +53,7 @@ joins:
```
Rules:
- Do **not** repeat base-table columns, grain, `table`, or `source_type` in an overlay — those are inherited.
- Do **not** repeat base-table columns, grain, `table`, or `source_type` in an overlay - those are inherited.
- Overlay columns MUST be computed (`expr` + `type`).
- Use `column_overrides` to add descriptions or metadata to inherited manifest columns. Do not put `type` or `expr` in `column_overrides`.
- `exclude_columns` hides specific manifest columns; `disable_joins` suppresses specific auto-detected joins.
@ -111,7 +111,7 @@ measures:
expr: count(*)
```
An SQL source is a one-shot answer: the aggregation is frozen, callers cannot re-group or re-filter by columns the SQL has collapsed, and the source is disconnected from the join graph. Prefer overlays + measures over SQL sources when possible — the `sl_capture` skill covers when SQL is justified.
An SQL source is a one-shot answer: the aggregation is frozen, callers cannot re-group or re-filter by columns the SQL has collapsed, and the source is disconnected from the join graph. Prefer overlays + measures over SQL sources when possible - the `sl_capture` skill covers when SQL is justified.
### Columns
@ -124,7 +124,7 @@ Every standalone column requires `name` and `type`. Overlays have computed colum
### Grain
`grain: [col_a, col_b]` — the set of columns that uniquely identify one row. The query engine uses grain to prevent fan-out in joins. Overlays inherit grain from the manifest unless they override.
`grain: [col_a, col_b]` - the set of columns that uniquely identify one row. The query engine uses grain to prevent fan-out in joins. Overlays inherit grain from the manifest unless they override.
### Joins
@ -133,7 +133,7 @@ joins:
- to: customers # target source name
on: "customer_id = customers.id" # local_col = TARGET.target_col
relationship: many_to_one # or one_to_many, one_to_one
alias: primary_customer # optional — lets you join the same target twice
alias: primary_customer # optional - lets you join the same target twice
```
- `on` format: `local_col = TARGET.target_col`. Always qualify the right side with the target source name.
@ -145,13 +145,13 @@ joins:
measures:
- name: total_arr
expr: sum(arr_amount)
description: Sum of ARR — filter by plan_name at query time
description: Sum of ARR - filter by plan_name at query time
filter: "is_active = true"
segments: [paid_non_refunded]
```
- `name` (required, snake_case).
- `expr` (required): any valid SQL aggregate — `sum(x)`, `count(*)`, `count(distinct user_id)`, `avg(score)`.
- `expr` (required): any valid SQL aggregate - `sum(x)`, `count(*)`, `count(distinct user_id)`, `avg(score)`.
- `description` (required on capture): what the measure computes and how to use it.
- `filter` (optional): SQL predicate applied as a WHERE clause specific to this measure.
- `segments` (optional): names of segments defined on the same source. The engine AND-composes each segment's `expr` into this measure's effective filter.
@ -167,23 +167,23 @@ segments:
description: Orders that were paid and not refunded
```
Named, reusable boolean predicates scoped to one source. Reference by bare name in a measure's `segments: []`, or by dotted form `source.segment_name` in an `sl_query`. Segments are predicates only — they are NOT selectable as dimensions. If you need to group by the predicate, add a `columns[]` entry instead.
Named, reusable boolean predicates scoped to one source. Reference by bare name in a measure's `segments: []`, or by dotted form `source.segment_name` in an `sl_query`. Segments are predicates only - they are NOT selectable as dimensions. If you need to group by the predicate, add a `columns[]` entry instead.
### Cross-references with the wiki
The reverse edge (wiki pages that cite this source) is derived automatically from each wiki's `sl_refs:` — you don't emit anything on the SL side. Author the edge once on the wiki via `sl_refs:`; the post-write reconciler populates the knowledge↔SL index.
The reverse edge (wiki pages that cite this source) is derived automatically from each wiki's `sl_refs:` - you don't emit anything on the SL side. Author the edge once on the wiki via `sl_refs:`; the post-write reconciler populates the knowledge↔SL index.
---
## Part 2 — Querying via `sl_query`
## Part 2 - Querying via `sl_query`
The `sl_query` tool generates correct SQL from a structured query. It handles joins, fan-out prevention, aggregation correctness, and filter classification automatically. Prefer it over writing raw SQL whenever the SL has the relevant sources.
### When to prefer sl_query over raw SQL
- A pre-defined measure already exists (`source.measure_name` appears in the catalog).
- The question combines fields from multiple sources — the engine resolves the join path automatically.
- The question asks for a standard metric (revenue, ARR, churn, retention, LTV, conversion, MAU, etc.) — even if no pre-defined measure exists, a runtime aggregation over a catalog column is usually correct.
- The question combines fields from multiple sources - the engine resolves the join path automatically.
- The question asks for a standard metric (revenue, ARR, churn, retention, LTV, conversion, MAU, etc.) - even if no pre-defined measure exists, a runtime aggregation over a catalog column is usually correct.
Use raw SQL (`sql_execution`) only when:
- The computation requires multi-step CTEs whose intermediate grain is not a column in any source.
@ -206,17 +206,17 @@ Use raw SQL (`sql_execution`) only when:
- **`measures`**: mix pre-defined refs (`source.measure`) and runtime aggregations (`sum(source.column)`).
- **`dimensions`**: column refs or `{ field, granularity }` objects for time grains (`day`, `week`, `month`, `quarter`, `year`).
- **`filters`**: free-form SQL predicates. The engine auto-classifies each as WHERE or HAVING based on whether it references an aggregated measure.
- **`segments`**: dotted `source.segment_name`. Each segment is AND-ed into the effective filter of every measure whose base source matches. Segments never become a global WHERE — use `filters` for cross-source predicates.
- **`segments`**: dotted `source.segment_name`. Each segment is AND-ed into the effective filter of every measure whose base source matches. Segments never become a global WHERE - use `filters` for cross-source predicates.
- **`order_by`**: string or `{ field, direction }`. Direction defaults to `asc`.
- **`limit`**: integer row cap.
### Join resolution
You don't specify a base table. The engine infers the set of sources needed from the fields you reference and resolves the shortest join path through the catalog's declared joins. If no path exists between two sources, the query fails with a path-not-found error — check `discover_data` or `sl_discover` to see which sources are connected.
You don't specify a base table. The engine infers the set of sources needed from the fields you reference and resolves the shortest join path through the catalog's declared joins. If no path exists between two sources, the query fails with a path-not-found error - check `discover_data` or `sl_discover` to see which sources are connected.
### Worked examples
Cross-source query — engine resolves `account_health_scores → accounts ← opportunities` automatically:
Cross-source query - engine resolves `account_health_scores → accounts ← opportunities` automatically:
```json
{

54
packages/context/skills/sl_capture/SKILL.md
View file

@ -1,10 +1,10 @@
---
name: sl_capture
description: How to capture new reusable patterns into KTX's semantic layer — when a measure, segment, or join belongs in the catalog and how to write it generically so it stays small and useful over time. Loaded by the post-turn memory-agent only. The research agent does not write to the SL.
description: How to capture new reusable patterns into KTX's semantic layer - when a measure, segment, or join belongs in the catalog and how to write it generically so it stays small and useful over time. Loaded by the post-turn memory-agent only. The research agent does not write to the SL.
callers: [memory_agent]
---
# Semantic Layer — Capture
# Semantic Layer - Capture
This skill covers **when** and **how** to capture new patterns into the semantic layer. For schema reference and query grammar, load the `sl` skill first.
@ -13,8 +13,8 @@ When the current turn produces a reusable pattern (business metric, derived view
## SQL dialect
The user-facing prompt includes a `Warehouse:` line under the SL Sources index
(e.g. `Warehouse: BIGQUERY`). All `expr` strings — measure expressions, segment
predicates, computed-column SQL — execute on that warehouse and must use its
(e.g. `Warehouse: BIGQUERY`). All `expr` strings - measure expressions, segment
predicates, computed-column SQL - execute on that warehouse and must use its
syntax. Date arithmetic in particular varies by dialect:
- **BigQuery**: `transaction_date >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)` (when the column is `TIMESTAMP`); `event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY)` (when `DATE`).
@ -22,7 +22,7 @@ syntax. Date arithmetic in particular varies by dialect:
- **Snowflake**: `transaction_date >= dateadd(day, -90, current_timestamp())`.
Match the column's manifest type (`type: time` → TIMESTAMP/DATETIME on the
warehouse) — comparing TIMESTAMP to a DATE-arithmetic result fails on
warehouse) - comparing TIMESTAMP to a DATE-arithmetic result fails on
BigQuery. After every `sl_edit_source`/`sl_write_source`, the inline validator runs a
`LIMIT 1` warehouse probe per measure and surfaces dialect mismatches; if
you see an error trailer, fix the expression and retry rather than leaving
@ -68,12 +68,12 @@ Callers filter `region = 'US'` at query time.
**Bake constants in only when the filter has named business meaning that won't change** (`enterprise_arr` for a contractually defined tier), cannot be expressed via the source's dimensions, or comes from a regulated/fixed list.
**Time anchors and value lists belong in callers' filters, not in measure expressions or source SQL.**
- Anti-pattern (date anchor inlined): `expr: count(distinct case when transaction_date >= '2026-04-12' then customer_id end)` — the date will need editing every time the question shifts, and every reader has to discover it.
- Anti-pattern (value list inlined in source SQL): `WHERE product_category_1 IN ('Testosterone', 'Weight Loss', …)` — locks the source to today's catalog and blocks callers from broadening or narrowing.
- Anti-pattern (date anchor inlined): `expr: count(distinct case when transaction_date >= '2026-04-12' then customer_id end)` - the date will need editing every time the question shifts, and every reader has to discover it.
- Anti-pattern (value list inlined in source SQL): `WHERE product_category_1 IN ('Testosterone', 'Weight Loss', …)` - locks the source to today's catalog and blocks callers from broadening or narrowing.
- Preferred: a generic measure (`count(distinct customer_id)`) plus either a named segment that captures the *meaning* of the anchor (`gh_new_products_since_launch`) or a query-time filter. Callers compose; the source stays small.
- A date is durable to bake in only when it represents a regulatory cutover, a contractually fixed boundary, or a one-time event that reshapes how the source itself is read.
**If you create a segment whose expr matches a measure's filter, the measure MUST reference the segment via `segments: [segment_name]` rather than re-inlining the predicate.** This is the canonical pattern even with a single measure — duplicating the predicate inline defeats the purpose of naming it.
**If you create a segment whose expr matches a measure's filter, the measure MUST reference the segment via `segments: [segment_name]` rather than re-inlining the predicate.** This is the canonical pattern even with a single measure - duplicating the predicate inline defeats the purpose of naming it.
Anti-pattern:
```yaml
@ -132,18 +132,18 @@ Overlay YAML may include `measures:`, `segments:`, `descriptions:`, `joins:`, `d
- The metric requires per-user/per-entity derivation that cannot be expressed as a single `expr` (e.g., `EXISTS` over a time-windowed subset), OR
- The metric requires multi-step CTEs whose intermediate grain is not a column in any existing source.
When an `sql` source is unavoidable, note in its `descriptions` map which SL gap forced the choice so it can be retired once the primitive ships. It must target a name NOT in the manifest — pick a distinct one (e.g. `mrr_waterfall_rollup`, not `fct_orders`).
When an `sql` source is unavoidable, note in its `descriptions` map which SL gap forced the choice so it can be retired once the primitive ships. It must target a name NOT in the manifest - pick a distinct one (e.g. `mrr_waterfall_rollup`, not `fct_orders`).
## Slim standalone sources via `inherits_columns_from`
When a standalone SQL source filters or projects from a single manifest-backed base table (the common pattern for derived views like `aav_consignments` over `MARTS.CONSIGNMENTS`), set `inherits_columns_from:` to the base table's manifest key and list only column **names** in `columns:`. Compose-time enrichment fills `type`, `descriptions`, and `role` from the matching manifest column.
Discover the manifest key with `sl_discover` — pass the bare name (`CONSIGNMENTS`), the fully-qualified path (`ANALYTICS.MARTS.CONSIGNMENTS`), or any suffix; the tool resolves all forms and prints the canonical key in its output.
Discover the manifest key with `sl_discover` - pass the bare name (`CONSIGNMENTS`), the fully-qualified path (`ANALYTICS.MARTS.CONSIGNMENTS`), or any suffix; the tool resolves all forms and prints the canonical key in its output.
```yaml
name: aav_consignments
descriptions:
user: AAV consignments — filtered view of MARTS.CONSIGNMENTS for the auto-auction-vaulting channel.
user: AAV consignments - filtered view of MARTS.CONSIGNMENTS for the auto-auction-vaulting channel.
source_type: sql
sql: |
SELECT CONSIGNED_ITEM_ID, CASH_ADV_AMOUNT, ALT_VALUE_COMBINED, my_derived_flag
@ -157,7 +157,7 @@ columns:
- { name: CONSIGNED_ITEM_ID } # type/descriptions inherited from manifest
- { name: CASH_ADV_AMOUNT }
- { name: ALT_VALUE_COMBINED }
- { name: my_derived_flag, type: boolean, expr: "CASH_ADV_AMOUNT > 0", descriptions: { user: "Computed locally — has any cash advance." } }
- { name: my_derived_flag, type: boolean, expr: "CASH_ADV_AMOUNT > 0", descriptions: { user: "Computed locally - has any cash advance." } }
measures:
- name: total_cash_advance
expr: sum(CASH_ADV_AMOUNT)
@ -165,12 +165,12 @@ measures:
Rules:
- Inheritance fills only **blank** fields. If you set a `description` locally, it wins — useful when the base description is misleading in the filtered view.
- Inheritance fills only **blank** fields. If you set a `description` locally, it wins - useful when the base description is misleading in the filtered view.
- A column not in the manifest (a derived/aliased column, or one from a different table in a `JOIN`) needs its own `type` and `description` declared.
- If `inherits_columns_from` doesn't resolve, the source still loads, but every column without a type triggers a validator error on the warehouse probe — `sl_discover` first to confirm the key.
- Don't use `inherits_columns_from` for sources backed by `table:` (those should be overlays — see the rule against shadowing the manifest above).
- If `inherits_columns_from` doesn't resolve, the source still loads, but every column without a type triggers a validator error on the warehouse probe - `sl_discover` first to confirm the key.
- Don't use `inherits_columns_from` for sources backed by `table:` (those should be overlays - see the rule against shadowing the manifest above).
## Refinement — replace, don't append
## Refinement - replace, don't append
When the user corrects a prior answer, the existing measure is wrong by the user's own standard. Replace it, don't add a parallel measure.
@ -234,14 +234,14 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
## Tool sequence
1. `sl_discover` — see what source files exist.
2. `sl_discover({ query: "<table-or-source-name>" })` — **REQUIRED before the first write on any name**. Shows columns/joins/grain from the manifest. If the call returns a schema, you MUST write an overlay, not a standalone. Skipping this is the #1 cause of accidentally shadowing the manifest.
3. `sl_read_source({ connectionId, sourceName })` — read the raw YAML before editing.
1. `sl_discover` - see what source files exist.
2. `sl_discover({ query: "<table-or-source-name>" })` - **REQUIRED before the first write on any name**. Shows columns/joins/grain from the manifest. If the call returns a schema, you MUST write an overlay, not a standalone. Skipping this is the #1 cause of accidentally shadowing the manifest.
3. `sl_read_source({ connectionId, sourceName })` - read the raw YAML before editing.
4. For modifications: `sl_edit_source({ connectionId, sourceName, yaml_edits: [{ oldText, newText, reason }] })` with exact-string replacements. `oldText` must match exactly and be unique in the file.
5. For new sources or full rewrites: `sl_write_source({ connectionId, sourceName, source })` with the full structured source definition.
6. For join discovery: use `sql_execution({connectionName: "warehouse", sql: "SELECT count(*) FROM public.orders o JOIN public.customers c ON c.id = o.customer_id LIMIT 20"})` with the target warehouse connection name and dialect-correct table names to verify the join key exists in both tables and assess cardinality before declaring the join.
7. Cross-reference knowledge: author the edge once on the **wiki** side via `sl_refs: [source_name]` in the page's front-matter. The reverse edge (wiki pages that cite an SL source) is derived automatically by the reconciler — do not add a `knowledge_refs:` field to SL YAMLs.
8. `sl_validate` — run after writing or editing to surface schema issues, duplicate measure names, and cross-source validation errors. Read-only; the writes are already committed (the squash-at-end flow will collapse them into one commit).
7. Cross-reference knowledge: author the edge once on the **wiki** side via `sl_refs: [source_name]` in the page's front-matter. The reverse edge (wiki pages that cite an SL source) is derived automatically by the reconciler - do not add a `knowledge_refs:` field to SL YAMLs.
8. `sl_validate` - run after writing or editing to surface schema issues, duplicate measure names, and cross-source validation errors. Read-only; the writes are already committed (the squash-at-end flow will collapse them into one commit).
## Editing patterns
@ -250,13 +250,13 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
- Do NOT modify existing measures or their descriptions unless the current turn explicitly corrects them.
- During bundle/external ingest, include `rawPaths` on every `sl_write_source`/`sl_edit_source` call with only the raw files that directly support the SL action.
## Worked example — additive overlay
## Worked example - additive overlay
Conversation:
- User: "What was the average order value last quarter?"
- Assistant fell back to SQL: `SELECT AVG(amount) FROM orders WHERE order_date >= ...`
Existing index: `orders [measures=0, joins=0] — candidate for enrichment`.
Existing index: `orders [measures=0, joins=0] - candidate for enrichment`.
```
sl_discover()
@ -279,9 +279,9 @@ sl_validate({ connectionId: "warehouse" })
→ clean
```
The overlay only contains `name` and `measures` — no columns, grain, or table. Those are inherited from the manifest.
The overlay only contains `name` and `measures` - no columns, grain, or table. Those are inherited from the manifest.
## Worked example — refinement (replace)
## Worked example - refinement (replace)
Prior turn:
- [user] "How many active users do we have per region?"
@ -307,7 +307,7 @@ sl_validate({ connectionId: "warehouse" })
If you only added a new measure, the old incorrect `active_count` would stay and future queries would keep answering the wrong question.
## Worked example — new join
## Worked example - new join
Prior turn: user asked to correlate LTV with protocol count; assistant joined `fct_orders` with `fct_mau_multiprotocol` on `admin_user_id` in raw SQL.
@ -341,6 +341,6 @@ Always verify joins with `sql_execution` before adding them.
- A measure whose filter matches a segment MUST reference the segment via `segments: [name]`.
- Extract repeated predicates into named segments.
- Use computed dimensions for derived categories.
- When the user corrects a prior answer, replace — don't append.
- When the user corrects a prior answer, replace - don't append.
- Always run `sl_validate` after writing to surface issues.
- If nothing is worth capturing, respond without calling any SL write tool.

30
packages/context/skills/wiki_capture/SKILL.md
View file

@ -1,6 +1,6 @@
---
name: wiki_capture
description: KTX's knowledge base — wiki pages for durable, reusable business knowledge. Covers capture workflow for user preferences, metric definitions, organizational conventions, and cross-references between wiki pages and semantic-layer sources. Loaded by the post-turn memory-agent only. The research agent reads wiki via `wiki_read`/`wiki_search` but does not write it.
description: KTX's knowledge base - wiki pages for durable, reusable business knowledge. Covers capture workflow for user preferences, metric definitions, organizational conventions, and cross-references between wiki pages and semantic-layer sources. Loaded by the post-turn memory-agent only. The research agent reads wiki via `wiki_read`/`wiki_search` but does not write it.
callers: [memory_agent]
---
@ -8,14 +8,14 @@ callers: [memory_agent]
## Role
The knowledge base stores durable, reusable business knowledge for an analytics assistant. Each page is a self-contained rule, definition, or convention that answers "how should this concept be handled in this organization?" — written once and reused across chats.
The knowledge base stores durable, reusable business knowledge for an analytics assistant. Each page is a self-contained rule, definition, or convention that answers "how should this concept be handled in this organization?" - written once and reused across chats.
Scope selection is handled by the runtime:
- When user-scoped knowledge is enabled AND the caller is a chat turn, writes go to the user's **personal** scope.
- When the caller is an admin-driven ingest (`sourceType: 'external_ingest'`), writes go to the **global** scope.
- When user-scoped knowledge is disabled, all writes go to the global scope.
The `wiki_write` tool picks the right scope based on the session. Capture logic does not need to choose — focus on whether the content is worth capturing at all.
The `wiki_write` tool picks the right scope based on the session. Capture logic does not need to choose - focus on whether the content is worth capturing at all.
## What to capture
@ -30,8 +30,8 @@ Do NOT capture:
- One-off requests ("answer under 100 words").
- Temporary instructions scoped to the current chat.
- Ad-hoc formatting preferences.
- Information already present in the semantic layer (column names, join paths, measure formulas — those belong in SL).
- **Query results, snapshots, or time-bounded benchmark tables.** Numbers go stale; pasting "Oct 2025: 25%, Nov 2025: 19.9%, …" creates misinformation as soon as new data lands. Reference the SL source by name (`sl_refs`) and let future query tools pull live data — the wiki captures the *rule* (definition, exclusion, segmentation), the SL source captures the *measure*, and query execution captures the *current values*.
- Information already present in the semantic layer (column names, join paths, measure formulas - those belong in SL).
- **Query results, snapshots, or time-bounded benchmark tables.** Numbers go stale; pasting "Oct 2025: 25%, Nov 2025: 19.9%, …" creates misinformation as soon as new data lands. Reference the SL source by name (`sl_refs`) and let future query tools pull live data - the wiki captures the *rule* (definition, exclusion, segmentation), the SL source captures the *measure*, and query execution captures the *current values*.
- **Interpretive narrative tied to a specific snapshot** ("M1 retention degraded sharply from Dec 2025"). The observation is anchored to data that will move; the actionable convention (e.g., "always exclude in-progress cohorts") may be worth capturing on its own, but the snapshot-specific commentary is not.
If nothing is worth capturing, respond without calling any tool.
@ -40,13 +40,13 @@ If nothing is worth capturing, respond without calling any tool.
1. Read the wiki index (provided in the prompt) and decide whether the turn introduces durable knowledge.
2. **Before writing**, search for related content so cross-references are accurate:
- `discover_data` first when a page relates to data or SL concepts — find
- `discover_data` first when a page relates to data or SL concepts - find
existing wiki pages, SL sources, and raw warehouse schema together.
- `wiki_search` with the topic — find related wiki pages to populate `refs`.
- `sl_discover` with the concept — if the page defines a metric (revenue, churn, retention, LTV, ARR, MRR, CAC, attribution, etc.), find matching SL sources or measures to populate `sl_refs`. If no matches, pass `sl_refs: []` so future readers know you checked.
- `wiki_search` with the topic - find related wiki pages to populate `refs`.
- `sl_discover` with the concept - if the page defines a metric (revenue, churn, retention, LTV, ARR, MRR, CAC, attribution, etc.), find matching SL sources or measures to populate `sl_refs`. If no matches, pass `sl_refs: []` so future readers know you checked.
3. If updating an existing page, `wiki_read` it first. Use the returned `structured.content` or markdown body as the exact stored text for targeted replacements; current tags, refs, and sl_refs are returned in structured metadata.
4. `wiki_write` to create or update. Prefer merging into an existing page over creating a new one.
5. `wiki_remove` only when a page is truly obsolete — not to replace stale content (update it instead).
5. `wiki_remove` only when a page is truly obsolete - not to replace stale content (update it instead).
For bundle/external ingest, include `rawPaths` on every `wiki_write`/`wiki_remove` call with only the raw files that directly support that wiki action. This keeps ingest provenance tied to the actual source file, not every file in the WorkUnit.
@ -82,7 +82,7 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
- **Keys** are short kebab-case topic identifiers: `leads-source-filter`, `revenue-definition`, `churn-calculation`. No namespacing, no prefixes.
- **Summary** is a one-line hook (≤200 chars) shown in the index.
- **Content** is concise markdown — actionable rules, not prose.
- **Content** is concise markdown - actionable rules, not prose.
```
## [Topic Title]
@ -116,8 +116,8 @@ All three fields use REPLACE semantics on update:
Two modes:
- **Full content** — pass `content` to rewrite the whole page. Use when the page structure needs to change.
- **Targeted edits** — pass `replacements: [{ oldText, newText }]` to apply exact-string replacements. Use for small updates; preserves the rest of the page.
- **Full content** - pass `content` to rewrite the whole page. Use when the page structure needs to change.
- **Targeted edits** - pass `replacements: [{ oldText, newText }]` to apply exact-string replacements. Use for small updates; preserves the rest of the page.
When editing, read the page first so the edit matches exact whitespace and indentation.
@ -125,7 +125,7 @@ When editing, read the page first so the edit matches exact whitespace and inden
Organization (GLOBAL) pages are read-only from a user's personal-scope session. To override a global rule for a single user, write a personal page with the **same key**. At read time the USER page wins.
## Worked example — capturing a metric with cross-references
## Worked example - capturing a metric with cross-references
User says: "Going forward, the official refund rate is total refunded amount divided by total gross transaction amount."
@ -133,7 +133,7 @@ User says: "Going forward, the official refund rate is total refunded amount div
wiki_list_tags()
→ existing tags include "finance"
wiki_search({ query: "refund revenue paid orders" })
→ returns `revenue-definition` (related — defines paid-orders filter)
→ returns `revenue-definition` (related - defines paid-orders filter)
sl_discover({ query: "refund rate" })
→ returns fct_orders (score 0.08), fct_gaap_revenue (0.06)
sl_read_source({ connectionId: "warehouse", sourceName: "fct_orders" })
@ -155,6 +155,6 @@ Search-then-write order matters. Cross-references are part of the page's identit
- Read existing pages before updating them.
- Prefer merging into an existing page over creating a new one.
- Prefer fewer, richer pages over many thin ones.
- Write content as clear, actionable rules — not narrative prose.
- Write content as clear, actionable rules - not narrative prose.
- Discover cross-references via search before writing, not after.
- If nothing is worth capturing, respond without calling any tool.

12
packages/context/src/connections/local-warehouse-descriptor.test.ts
View file

@ -36,7 +36,13 @@ describe('localConnectionToWarehouseDescriptor', () => {
});
it('returns null for non-warehouse adapters', () => {
expect(localConnectionToWarehouseDescriptor('looker', { driver: 'looker' })).toBeNull();
expect(
localConnectionToWarehouseDescriptor('looker', {
driver: 'looker',
base_url: 'https://looker.example.com',
client_id: 'client',
}),
).toBeNull();
});
});
@ -48,7 +54,9 @@ describe('local connection info helpers', () => {
});
it('keeps non-warehouse adapter labels for display-only local connection surfaces', () => {
expect(localConnectionTypeForConfig('prod-metabase', { driver: 'metabase' })).toBe('metabase');
expect(localConnectionTypeForConfig('prod-metabase', { driver: 'metabase', api_url: 'https://metabase.example.com' })).toBe(
'metabase',
);
expect(localConnectionTypeForConfig('missing-driver', {} as never)).toBe('unknown');
});

17
packages/context/src/connections/notion-config.ts
View file

@ -13,7 +13,20 @@ export const KTX_NOTION_ORG_KNOWLEDGE_WARNING =
type KtxNotionCrawlMode = 'all_accessible' | 'selected_roots';
export interface KtxNotionConnectionConfig extends KtxProjectConnectionConfig {
type RawKtxNotionConnectionConfig = Extract<KtxProjectConnectionConfig, { driver: 'notion' }>;
export type KtxNotionConnectionConfig = Omit<
RawKtxNotionConnectionConfig,
| 'auth_token'
| 'auth_token_ref'
| 'crawl_mode'
| 'root_page_ids'
| 'root_database_ids'
| 'root_data_source_ids'
| 'max_pages_per_run'
| 'max_knowledge_creates_per_run'
| 'max_knowledge_updates_per_run'
> & {
driver: 'notion';
auth_token: string | null;
auth_token_ref: string | null;
@ -24,7 +37,7 @@ export interface KtxNotionConnectionConfig extends KtxProjectConnectionConfig {
max_pages_per_run: number;
max_knowledge_creates_per_run: number;
max_knowledge_updates_per_run: number;
}
};
export interface RedactedKtxNotionConnectionConfig {
driver: 'notion';

3
packages/context/src/ingest/adapters/historic-sql/local-ingest-acceptance.test.ts
View file

@ -166,7 +166,6 @@ async function writeHistoricSqlProject(project: KtxLocalProject): Promise<KtxLoc
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -231,7 +230,7 @@ describe('historic-SQL local ingest retrieval acceptance', () => {
});
it('projects table and pattern evidence into semantic-layer and wiki retrieval surfaces', async () => {
const initialized = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
const initialized = await initKtxProject({ projectDir: join(tempDir, 'project') });
const project = await writeHistoricSqlProject(initialized);
const sqlAnalysis = acceptanceSqlAnalysis();
const agentRunner = new HistoricSqlAcceptanceAgentRunner();

8
packages/context/src/ingest/adapters/metabase/local-source-state-store.test.ts
View file

@ -3,6 +3,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { buildDefaultKtxProjectConfig } from '../../../project/index.js';
import { connectionConfigSchema } from '../../../project/driver-schemas.js';
import { KtxYamlMetabaseSourceStateReader, LocalMetabaseDiscoveryCache } from './local-source-state-store.js';
describe('Metabase YAML source state and discovery cache', () => {
@ -21,12 +22,13 @@ describe('Metabase YAML source state and discovery cache', () => {
function projectWithMetabaseMappings(mappings: Record<string, unknown>) {
return {
config: {
...buildDefaultKtxProjectConfig('metabase-cache-test'),
...buildDefaultKtxProjectConfig(),
connections: {
'prod-metabase': {
'prod-metabase': connectionConfigSchema.parse({
driver: 'metabase',
api_url: 'https://metabase.example.com',
mappings,
},
}),
},
},
};

2
packages/context/src/ingest/local-adapters.test.ts
View file

@ -16,7 +16,7 @@ describe('local ingest adapters', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-adapters-'));
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
project = await loadKtxProject({ projectDir });
});

17
packages/context/src/ingest/local-bundle-ingest.test.ts
View file

@ -314,11 +314,10 @@ describe('canonical local ingest', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-full-ingest-'));
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -443,7 +442,6 @@ describe('canonical local ingest', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -521,11 +519,10 @@ describe('canonical local ingest', () => {
it('runs historic-SQL evidence projection through the local bundle post-processor', async () => {
const projectDir = join(tempDir, 'historic-sql-project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -605,11 +602,10 @@ describe('canonical local ingest', () => {
it('rejects direct Metabase scheduled pulls before requiring a local ingest LLM provider', async () => {
const projectDir = join(tempDir, 'metabase-project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -637,7 +633,7 @@ describe('canonical local ingest', () => {
it('runs full MetricFlow local ingest from a dbt repo fixture through the canonical runner', async () => {
const projectDir = join(tempDir, 'metricflow-run-project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
const fixtureDir = join(tempDir, 'metricflow-fixture');
await mkdir(join(fixtureDir, 'models'), { recursive: true });
@ -685,7 +681,6 @@ describe('canonical local ingest', () => {
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -767,7 +762,6 @@ describe('canonical local ingest', () => {
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: local-mf',
'connections:',
' warehouse:',
' driver: postgres',
@ -801,11 +795,10 @@ describe('canonical local ingest', () => {
it('runs scheduled Looker ingest through the canonical local runner and records SL target evidence', async () => {
const projectDir = join(tempDir, 'looker-project');
await initKtxProject({ projectDir, projectName: 'looker-runtime' });
await initKtxProject({ projectDir });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: looker-runtime',
'connections:',
' prod-looker:',
' driver: looker',

4
packages/context/src/ingest/local-bundle-runtime.test.ts
View file

@ -24,11 +24,10 @@ describe('createLocalBundleIngestRuntime', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-bundle-runtime-'));
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -149,7 +148,6 @@ describe('createLocalBundleIngestRuntime', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',

5
packages/context/src/ingest/local-mapping-reconcile.test.ts
View file

@ -27,11 +27,12 @@ describe('local mapping yaml reconciliation bridge', () => {
const project = projectWithConnections({
'prod-metabase': {
driver: 'metabase',
api_url: 'https://metabase.example.com',
mappings: {
databaseMappings: { '1': 'prod-warehouse' },
syncEnabled: { '1': true },
syncMode: 'ONLY',
selections: { collections: [12] },
selections: { collections: [12], items: [] },
defaultTagNames: ['ktx'],
},
},
@ -46,6 +47,8 @@ describe('local mapping yaml reconciliation bridge', () => {
const project = projectWithConnections({
'prod-looker': {
driver: 'looker',
base_url: 'https://looker.example.com',
client_id: 'client',
mappings: { connectionMappings: { analytics: 'prod-warehouse' } },
},
'prod-warehouse': { driver: 'postgres', url: 'postgresql://readonly@db.test/analytics' },

5
packages/context/src/ingest/local-stage-ingest.test.ts
View file

@ -17,7 +17,6 @@ async function writeWarehouseConfig(projectDir: string): Promise<void> {
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -34,7 +33,6 @@ async function writeLiveDatabaseConfig(projectDir: string): Promise<void> {
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -88,7 +86,7 @@ describe('local ingest', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-ingest-'));
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeWarehouseConfig(projectDir);
project = await loadKtxProject({ projectDir });
});
@ -574,7 +572,6 @@ describe('local ingest', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' notion-main:',
' driver: notion',

2
packages/context/src/ingest/tools/warehouse-verification/entity-details.tool.test.ts
View file

@ -15,7 +15,7 @@ describe('EntityDetailsTool', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-entity-details-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
await seedLiveDatabaseScan();
tool = new EntityDetailsTool(() => new WarehouseCatalogService({ fileStore: project.fileStore }));
context = {

2
packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.test.ts
View file

@ -11,7 +11,7 @@ describe('WarehouseCatalogService', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-warehouse-catalog-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
});
afterEach(async () => {

2
packages/context/src/llm/local-config.test.ts
View file

@ -194,7 +194,7 @@ describe('local KTX embedding config', () => {
it('constructs deterministic embeddings from the default project config', () => {
const createKtxEmbeddingProvider = vi.fn(() => ({}) as never);
const provider = createLocalKtxEmbeddingProviderFromConfig(
buildDefaultKtxProjectConfig('warehouse').ingest.embeddings,
buildDefaultKtxProjectConfig().ingest.embeddings,
{ createKtxEmbeddingProvider },
);

34
packages/context/src/mcp/local-project-ports.test.ts
View file

@ -71,7 +71,7 @@ describe('createLocalProjectMcpContextPorts', () => {
}
it('lists local project connections from ktx.yaml', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
@ -84,7 +84,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('tests a local project connection through the native scan connector factory', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
@ -120,7 +120,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('triggers canonical bundle ingest and reads status, report, and replay through MCP ports', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
};
@ -216,7 +216,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('returns child run metadata for local Metabase fan-out triggers', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections = {
'prod-metabase': {
driver: 'metabase',
@ -339,7 +339,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('writes, reads, and searches global wiki pages', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
const ports = createLocalProjectMcpContextPorts(project);
await expect(
@ -383,7 +383,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('writes, lists, reads, and validates semantic-layer sources', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
const ports = createLocalProjectMcpContextPorts(project);
await expect(
@ -449,7 +449,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('returns semantic-layer hybrid search metadata through local project ports', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
await writeLocalSlSource(project, {
connectionId: 'warehouse',
sourceName: 'orders',
@ -518,7 +518,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('returns historic SQL usage frequency and snippet through semantic-layer list search', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
await project.fileStore.writeFile(
'semantic-layer/warehouse/_schema/public.yaml',
`tables:
@ -566,7 +566,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('uses configured local embeddings for semantic-layer search when available', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.ingest.embeddings = { backend: 'none', dimensions: 2 };
await writeLocalSlSource(project, {
connectionId: 'warehouse',
@ -607,7 +607,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('rejects path traversal keys before touching the project directory', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
const ports = createLocalProjectMcpContextPorts(project);
await expect(
@ -626,7 +626,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('uses semantic compute for validation and compile-only sl_query when supplied', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
@ -712,7 +712,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('executes local MCP sl_query when a query executor is configured', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
@ -770,7 +770,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('exposes detailed local ingest trigger and status ports when local ingest is enabled', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = { driver: 'postgres' };
project.config.ingest.adapters = ['fake'];
project.config.ingest.embeddings = {
@ -890,7 +890,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('passes local ingest pull-config options into runLocalIngest', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = { driver: 'postgres' };
project.config.ingest.adapters = ['looker'];
const runLocalIngest = vi.fn(async () => ({
@ -949,7 +949,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('triggers fetch-capable local ingest without sourceDir config', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'postgres://localhost:5432/warehouse',
@ -1024,7 +1024,7 @@ describe('createLocalProjectMcpContextPorts', () => {
});
it('lists and reads only artifacts that belong to a local scan report', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
@ -1140,6 +1140,6 @@ describe('createLocalProjectMcpContextPorts', () => {
}),
).resolves.toBeNull();
await expect(ports.scan?.listArtifacts?.({ runId: 'missing' })).resolves.toBeNull();
await expect(readFile(join(project.projectDir, 'ktx.yaml'), 'utf-8')).resolves.toContain('project: warehouse');
await expect(readFile(join(project.projectDir, 'ktx.yaml'), 'utf-8')).resolves.not.toContain('project:');
});
});

2
packages/context/src/mcp/server.test.ts
View file

@ -168,7 +168,7 @@ describe('createKtxMcpServer', () => {
it('runs MCP memory_capture against a local project memory port', async () => {
const tempDir = await mkdtemp(join(tmpdir(), 'ktx-mcp-local-memory-'));
try {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
const agentRunner = {
runLoop: async ({
toolSet,

4
packages/context/src/memory/local-memory.test.ts
View file

@ -89,7 +89,7 @@ describe('createLocalProjectMemoryCapture', () => {
});
it('captures a wiki page through the local memory agent and persists pollable status', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
const agentRunner = {
runLoop: async ({
toolSet,
@ -144,7 +144,7 @@ describe('createLocalProjectMemoryCapture', () => {
});
it('captures a semantic-layer source for a named local connection id', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = { driver: 'postgres' };
const agentRunner = {
runLoop: async ({

55
packages/context/src/project/config.test.ts
View file

@ -11,7 +11,6 @@ describe('KTX project config', () => {
it.each(['status', 'replay', 'run', 'watch'])('accepts former ingest subcommand name "%s" as a connection id', (connectionId) => {
expect(
parseKtxProjectConfig(`
project: reserved-test
connections:
${connectionId}:
driver: postgres
@ -24,8 +23,7 @@ connections:
});
it('builds the default standalone project config', () => {
expect(buildDefaultKtxProjectConfig('warehouse')).toEqual({
project: 'warehouse',
expect(buildDefaultKtxProjectConfig()).toEqual({
connections: {},
storage: {
state: 'sqlite',
@ -84,15 +82,14 @@ connections:
});
it('round-trips through YAML with stable defaults', () => {
const serialized = serializeKtxProjectConfig(buildDefaultKtxProjectConfig('warehouse'));
const serialized = serializeKtxProjectConfig(buildDefaultKtxProjectConfig());
const parsed = parseKtxProjectConfig(serialized);
expect(serialized).toContain('project: warehouse');
expect(serialized).not.toContain('project:');
expect(serialized).not.toContain('live-database');
expect(serialized).toContain(
' embeddings:\n backend: deterministic\n model: deterministic\n dimensions: 8',
);
expect(parsed.project).toBe('warehouse');
expect(parsed.ingest.adapters).toEqual([]);
expect(parsed.ingest.embeddings).toEqual({
backend: 'deterministic',
@ -103,7 +100,6 @@ connections:
it('parses and serializes setup warehouse metadata without setup progress', () => {
const config = parseKtxProjectConfig(`
project: revenue
setup:
database_connection_ids:
- warehouse
@ -126,7 +122,6 @@ connections:
it('parses global direct Anthropic LLM config', () => {
const config = parseKtxProjectConfig(`
project: demo
llm:
provider:
backend: anthropic
@ -166,7 +161,6 @@ ingest:
it('parses global Vertex LLM config', () => {
const config = parseKtxProjectConfig(`
project: demo
llm:
provider:
backend: vertex
@ -188,7 +182,6 @@ llm:
it('parses gateway LLM, OpenAI scan embeddings, and sentence-transformers ingest embeddings', () => {
const config = parseKtxProjectConfig(`
project: demo
llm:
provider:
backend: gateway
@ -232,7 +225,6 @@ scan:
it('parses scan relationship settings', () => {
const config = parseKtxProjectConfig(`
project: demo
scan:
relationships:
enabled: false
@ -273,7 +265,6 @@ scan:
it('parses the scan relationship validation budget sentinel', () => {
const config = parseKtxProjectConfig(`
project: demo
scan:
relationships:
validationBudget: all
@ -285,7 +276,6 @@ scan:
it('rejects out-of-range scan relationship numeric settings', () => {
const yaml = `
project: demo
scan:
relationships:
acceptThreshold: 2
@ -316,7 +306,6 @@ scan:
it('rejects invalid scan relationship validation budget strings', () => {
const yaml = `
project: demo
scan:
relationships:
validationBudget: infinite
@ -327,7 +316,6 @@ scan:
it('rejects unsupported local LLM and embedding fields', () => {
expect(() =>
parseKtxProjectConfig(`
project: demo
ingest:
llm:
backend: anthropic
@ -336,7 +324,6 @@ ingest:
expect(() =>
parseKtxProjectConfig(`
project: demo
scan:
enrichment:
backend: gateway
@ -345,7 +332,6 @@ scan:
expect(() =>
parseKtxProjectConfig(`
project: demo
scan:
enrichment:
mode: llm
@ -356,7 +342,6 @@ scan:
expect(() =>
parseKtxProjectConfig(`
project: demo
ingest:
embeddings:
provider: gateway
@ -368,7 +353,6 @@ ingest:
it('rejects gateway embedding configs', () => {
expect(() =>
parseKtxProjectConfig(`
project: demo
ingest:
embeddings:
backend: gateway
@ -379,7 +363,6 @@ ingest:
expect(() =>
parseKtxProjectConfig(`
project: demo
scan:
enrichment:
mode: llm
@ -392,9 +375,9 @@ scan:
});
it('fills optional sections when a minimal config is loaded', () => {
const config = parseKtxProjectConfig('project: local\n');
const config = parseKtxProjectConfig('{}\n');
expect(config).toEqual(buildDefaultKtxProjectConfig('local'));
expect(config).toEqual(buildDefaultKtxProjectConfig());
expect(config.ingest.embeddings).toEqual({
backend: 'deterministic',
model: 'deterministic',
@ -406,14 +389,15 @@ scan:
expect(() => parseKtxProjectConfig('- nope\n')).toThrow('ktx.yaml must contain a YAML object');
});
it('rejects configs with a missing project name', () => {
expect(() => parseKtxProjectConfig('connections: {}\n')).toThrow('ktx.yaml field "project" is required');
it('accepts configs without a project name', () => {
expect(parseKtxProjectConfig('connections: {}\n')).toMatchObject({
connections: {},
});
});
it('rejects unknown top-level fields under strict mode', () => {
expect(() =>
parseKtxProjectConfig(`
project: demo
storrage:
state: sqlite
`),
@ -423,13 +407,12 @@ storrage:
describe('validateKtxProjectConfig', () => {
it('returns ok: true with no issues for a valid config', () => {
const result = validateKtxProjectConfig('project: warehouse\n');
const result = validateKtxProjectConfig('connections: {}\n');
expect(result).toEqual({ ok: true, issues: [] });
});
it('collects every schema issue without throwing', () => {
const result = validateKtxProjectConfig(`
project: ""
storage:
search: not-a-real-backend
scan:
@ -441,7 +424,6 @@ scan:
const paths = result.issues.map((issue) => issue.path);
expect(paths).toEqual(
expect.arrayContaining([
'project',
'storage.search',
'scan.relationships.acceptThreshold',
]),
@ -450,7 +432,6 @@ scan:
it('attaches migration hints for known deprecated keys', () => {
const result = validateKtxProjectConfig(`
project: demo
ingest:
llm:
backend: anthropic
@ -499,18 +480,15 @@ describe('generateKtxProjectConfigJsonSchema', () => {
it('exposes every top-level ktx.yaml section under properties', () => {
const properties = schema.properties as Record<string, unknown>;
expect(Object.keys(properties).sort()).toEqual(
['agent', 'connections', 'ingest', 'llm', 'memory', 'project', 'scan', 'setup', 'storage'].sort(),
);
expect(Object.keys(properties).sort()).toEqual(['agent', 'connections', 'ingest', 'llm', 'memory', 'scan', 'setup', 'storage'].sort());
});
it('marks "project" as required', () => {
expect(schema.required).toEqual(expect.arrayContaining(['project']));
it('does not require any top-level fields', () => {
expect(schema.required).toBeUndefined();
});
it('carries .describe() text on top-level fields', () => {
const properties = schema.properties as Record<string, { description?: string }>;
expect(properties.project?.description).toMatch(/Project identifier/);
expect(properties.llm?.description).toMatch(/LLM/);
expect(properties.scan?.description).toMatch(/Schema-scan/);
});
@ -531,4 +509,11 @@ describe('generateKtxProjectConfigJsonSchema', () => {
const relationships = scan?.properties?.relationships as { properties?: Record<string, { description?: string }> };
expect(relationships?.properties?.acceptThreshold?.description).toMatch(/auto-accepted/);
});
it('emits the mappings shapes under connections', () => {
const serialized = JSON.stringify(schema);
expect(serialized).toContain('databaseMappings');
expect(serialized).toContain('connectionMappings');
expect(serialized).toContain('expectedLookerConnectionName');
});
});

17
packages/context/src/project/config.ts
View file

@ -1,6 +1,7 @@
import { KTX_MODEL_ROLES } from '@ktx/llm';
import YAML from 'yaml';
import * as z from 'zod';
import { connectionConfigSchema } from './driver-schemas.js';
const KTX_LLM_BACKENDS = ['none', 'anthropic', 'vertex', 'gateway'] as const;
const KTX_EMBEDDING_BACKENDS = ['none', 'deterministic', 'openai', 'sentence-transformers'] as const;
@ -206,12 +207,7 @@ const storageSchema = z
})
.describe('Storage backends and commit policy for KTX state and search indexes.');
const connectionSchema = z
.looseObject({
driver: z.string().min(1).optional().describe('Connector driver identifier (e.g. "postgres", "bigquery", "snowflake").'),
url: z.string().optional().describe('Connection URL or DSN. Format depends on the driver; may contain environment-variable references.'),
})
.describe('A single database/connector connection entry. Additional driver-specific fields are accepted and passed through.');
const connectionSchema = connectionConfigSchema;
const agentSchema = z
.strictObject({
@ -242,11 +238,6 @@ const memorySchema = z
const ktxProjectConfigSchema = z
.strictObject({
project: z
.string({ error: 'ktx.yaml field "project" is required' })
.trim()
.min(1, 'ktx.yaml field "project" is required')
.describe('Project identifier; used in logs, ktx state files, and as the default workspace name.'),
setup: setupSchema.optional().describe('Setup-wizard state. Written by `ktx setup`; may be omitted.'),
connections: z
.record(z.string(), connectionSchema)
@ -336,8 +327,8 @@ function formatZodError(error: z.ZodError, input: unknown): string {
.join('\n');
}
export function buildDefaultKtxProjectConfig(projectName = 'ktx-project'): KtxProjectConfig {
return ktxProjectConfigSchema.parse({ project: projectName });
export function buildDefaultKtxProjectConfig(): KtxProjectConfig {
return ktxProjectConfigSchema.parse({});
}
export function parseKtxProjectConfig(raw: string): KtxProjectConfig {

140
packages/context/src/project/driver-schemas.test.ts Normal file
View file

@ -0,0 +1,140 @@
import { describe, expect, it } from 'vitest';
import { connectionConfigSchema } from './driver-schemas.js';
describe('connectionConfigSchema (driver discriminated union)', () => {
it.each([
['postgres', 'postgres://user:pass@host:5432/db'], // pragma: allowlist secret
['postgresql', 'postgresql://user:pass@host:5432/db'], // pragma: allowlist secret
['mysql', 'mysql://user:pass@host:3306/db'], // pragma: allowlist secret
['snowflake', 'snowflake://account/db'],
['bigquery', 'bigquery://project/dataset'],
['sqlite', 'sqlite:///tmp/db.sqlite'],
['clickhouse', 'clickhouse://host:8123/db'],
['sqlserver', 'sqlserver://host:1433;database=db'],
])('parses %s warehouse connection', (driver, url) => {
expect(connectionConfigSchema.parse({ driver, url })).toMatchObject({ driver, url });
});
it('preserves unknown warehouse fields via looseObject passthrough', () => {
const parsed = connectionConfigSchema.parse({
driver: 'postgres',
url: 'postgres://x',
historicSql: { enabled: true },
context: { queryHistory: { enabled: false } },
});
expect(parsed).toMatchObject({
driver: 'postgres',
historicSql: { enabled: true },
context: { queryHistory: { enabled: false } },
});
});
it('rejects an unknown driver', () => {
expect(() => connectionConfigSchema.parse({ driver: 'nope', url: 'x' })).toThrow();
});
});
describe('connectionConfigSchema - context source drivers with mappings', () => {
it('parses a metabase connection with mappings', () => {
const parsed = connectionConfigSchema.parse({
driver: 'metabase',
api_url: 'https://metabase.example.com',
api_key_ref: 'env:METABASE_API_KEY', // pragma: allowlist secret
mappings: {
databaseMappings: { '3': 'prod-warehouse' },
syncEnabled: { '3': true },
syncMode: 'ONLY',
},
});
expect(parsed).toMatchObject({
driver: 'metabase',
api_url: 'https://metabase.example.com',
mappings: {
databaseMappings: { '3': 'prod-warehouse' },
syncMode: 'ONLY',
},
});
});
it('parses a looker connection with connectionMappings', () => {
const parsed = connectionConfigSchema.parse({
driver: 'looker',
base_url: 'https://looker.example.com',
client_id: 'abc',
client_secret_ref: 'env:LOOKER_CLIENT_SECRET', // pragma: allowlist secret
mappings: { connectionMappings: { bigquery_prod: 'wh' } },
});
expect(parsed.mappings).toEqual({ connectionMappings: { bigquery_prod: 'wh' } });
});
it('parses a lookml connection with expectedLookerConnectionName', () => {
const parsed = connectionConfigSchema.parse({
driver: 'lookml',
repoUrl: 'https://github.com/acme/looker.git',
branch: 'main',
mappings: { expectedLookerConnectionName: 'bigquery_prod' },
});
expect(parsed.mappings).toEqual({ expectedLookerConnectionName: 'bigquery_prod' });
});
it('rejects metabase mapping with non-integer database key', () => {
expect(() =>
connectionConfigSchema.parse({
driver: 'metabase',
api_url: 'https://x',
mappings: { databaseMappings: { abc: 'wh' } },
}),
).toThrow();
});
});
describe('connectionConfigSchema - notion / dbt / metricflow', () => {
it('parses a notion connection with selected_roots crawl', () => {
const parsed = connectionConfigSchema.parse({
driver: 'notion',
auth_token_ref: 'env:NOTION_TOKEN',
crawl_mode: 'selected_roots',
root_page_ids: ['abc', 'def'],
max_pages_per_run: 500,
});
expect(parsed).toMatchObject({
driver: 'notion',
crawl_mode: 'selected_roots',
root_page_ids: ['abc', 'def'],
max_pages_per_run: 500,
});
});
it('rejects notion with unknown crawl_mode', () => {
expect(() =>
connectionConfigSchema.parse({
driver: 'notion',
auth_token_ref: 'env:NOTION_TOKEN',
crawl_mode: 'everything',
}),
).toThrow();
});
it('parses a dbt connection from a local source_dir', () => {
const parsed = connectionConfigSchema.parse({
driver: 'dbt',
source_dir: '/tmp/dbt-project',
target: 'dev',
});
expect(parsed).toMatchObject({ driver: 'dbt', source_dir: '/tmp/dbt-project', target: 'dev' });
});
it('parses a metricflow connection with nested config', () => {
const parsed = connectionConfigSchema.parse({
driver: 'metricflow',
metricflow: {
repoUrl: 'https://github.com/acme/sl.git',
branch: 'main',
},
});
expect(parsed).toMatchObject({
driver: 'metricflow',
metricflow: { repoUrl: 'https://github.com/acme/sl.git' },
});
});
});

205
packages/context/src/project/driver-schemas.ts Normal file
View file

@ -0,0 +1,205 @@
import * as z from 'zod';
import {
lookerMappingsSchema,
lookmlMappingsSchema,
metabaseMappingsSchema,
} from './mappings-yaml-schema.js';
const warehouseDrivers = [
'postgres',
'postgresql',
'mysql',
'snowflake',
'bigquery',
'sqlite',
'clickhouse',
'sqlserver',
] as const;
type WarehouseDriver = (typeof warehouseDrivers)[number];
function warehouseConnectionSchema<const Driver extends WarehouseDriver>(driver: Driver) {
return z
.looseObject({
driver: z.literal(driver),
url: z
.string()
.min(1)
.optional()
.describe('Warehouse connection URL or DSN; may contain environment-variable references like env:DATABASE_URL.'),
})
.describe(
`${driver} warehouse connection. Additional driver-tunable fields (e.g. historicSql, context.queryHistory) are accepted and passed through.`,
);
}
const warehouseConnectionSchemas = [
warehouseConnectionSchema('postgres'),
warehouseConnectionSchema('postgresql'),
warehouseConnectionSchema('mysql'),
warehouseConnectionSchema('snowflake'),
warehouseConnectionSchema('bigquery'),
warehouseConnectionSchema('sqlite'),
warehouseConnectionSchema('clickhouse'),
warehouseConnectionSchema('sqlserver'),
] as const;
const positiveIntKeyMessage = (field: string) => `${field} keys must be positive-integer strings (e.g. "1", "42")`;
const positiveIntKeyRegex = /^[1-9]\d*$/;
const metabaseMappingsStrictSchema = metabaseMappingsSchema.superRefine((value, ctx) => {
for (const key of Object.keys(value.databaseMappings ?? {})) {
if (!positiveIntKeyRegex.test(key)) {
ctx.addIssue({
code: 'custom',
path: ['databaseMappings', key],
message: positiveIntKeyMessage('databaseMappings'),
});
}
}
for (const key of Object.keys(value.syncEnabled ?? {})) {
if (!positiveIntKeyRegex.test(key)) {
ctx.addIssue({
code: 'custom',
path: ['syncEnabled', key],
message: positiveIntKeyMessage('syncEnabled'),
});
}
}
});
const metabaseConnectionSchema = z
.looseObject({
driver: z.literal('metabase'),
api_url: z.string().url().describe('Metabase instance API URL (e.g. https://metabase.example.com).'),
api_key: z.string().min(1).optional().describe('Literal Metabase API key. Prefer api_key_ref for safety.'),
api_key_ref: z
.string()
.min(1)
.optional()
.describe('Reference to Metabase API key (e.g. env:METABASE_API_KEY or file:/path).'),
network_proxy: z.looseObject({}).optional().describe('Optional network proxy configuration (snake_case form).'),
networkProxy: z.looseObject({}).optional().describe('Optional network proxy configuration (camelCase form).'),
mappings: metabaseMappingsStrictSchema
.optional()
.describe('Metabase database-to-warehouse mappings and sync configuration.'),
})
.describe('Metabase context-source connection.');
const lookerConnectionSchema = z
.looseObject({
driver: z.literal('looker'),
base_url: z.string().url().describe('Looker instance base URL (e.g. https://looker.example.com).'),
client_id: z.string().min(1).describe('Looker OAuth client ID.'),
client_secret: z.string().min(1).optional().describe('Literal Looker OAuth client secret. Prefer client_secret_ref.'),
client_secret_ref: z
.string()
.min(1)
.optional()
.describe('Reference to Looker OAuth client secret (e.g. env:LOOKER_CLIENT_SECRET).'),
mappings: lookerMappingsSchema.optional().describe('Looker connection-name to KTX warehouse mappings.'),
})
.describe('Looker context-source connection.');
const lookmlConnectionSchema = z
.looseObject({
driver: z.literal('lookml'),
repoUrl: z
.string()
.min(1)
.describe('Git URL of the LookML project (https, ssh, or file:). Field is camelCase by convention.'),
branch: z.string().min(1).optional().describe('Git branch (default "main" downstream).'),
path: z.string().optional().describe('Subdirectory within the repo when the LookML project lives in a monorepo.'),
auth_token_ref: z.string().min(1).optional().describe('Reference to Git auth token for private repos (e.g. env:GITHUB_TOKEN).'),
mappings: lookmlMappingsSchema.optional().describe('LookML expected-connection mapping for ingest gating.'),
})
.describe('LookML context-source connection.');
const notionConnectionSchema = z
.looseObject({
driver: z.literal('notion'),
auth_token: z.string().min(1).optional().describe('Literal Notion integration token. Prefer auth_token_ref.'),
auth_token_ref: z
.string()
.min(1)
.optional()
.describe('Reference to Notion integration token (e.g. env:NOTION_TOKEN).'),
crawl_mode: z
.enum(['selected_roots', 'all_accessible'])
.optional()
.describe(
'Crawl scope. "selected_roots" requires at least one of root_page_ids, root_database_ids, root_data_source_ids.',
),
root_page_ids: z.array(z.string().min(1)).optional().describe('Notion page IDs to crawl when crawl_mode is selected_roots.'),
root_database_ids: z
.array(z.string().min(1))
.optional()
.describe('Notion database IDs to crawl when crawl_mode is selected_roots.'),
root_data_source_ids: z
.array(z.string().min(1))
.optional()
.describe('Notion data source IDs to crawl when crawl_mode is selected_roots.'),
max_pages_per_run: z
.number()
.int()
.min(1)
.max(10000)
.optional()
.describe('Maximum Notion pages fetched in a single ingest run.'),
max_knowledge_creates_per_run: z
.number()
.int()
.min(0)
.max(25)
.optional()
.describe('Maximum new wiki pages created per run.'),
max_knowledge_updates_per_run: z
.number()
.int()
.min(0)
.max(100)
.optional()
.describe('Maximum existing wiki pages updated per run.'),
})
.describe('Notion context-source connection.');
const dbtConnectionSchema = z
.looseObject({
driver: z.literal('dbt'),
source_dir: z.string().min(1).optional().describe('Absolute or project-relative path to a local dbt project.'),
repo_url: z.string().min(1).optional().describe('Git URL of the dbt project (https, ssh, or file:).'),
branch: z.string().min(1).optional().describe('Git branch when using repo_url.'),
path: z.string().optional().describe('Subdirectory within the repo when the dbt project lives in a monorepo.'),
auth_token_ref: z.string().min(1).optional().describe('Reference to Git auth token for private repos.'),
profiles_path: z.string().optional().describe('Override path to dbt profiles.yml.'),
target: z.string().min(1).optional().describe('dbt target name (e.g. dev, prod).'),
project_name: z.string().min(1).optional().describe('Override auto-detected dbt project name.'),
})
.describe('dbt context-source connection.');
const metricflowConnectionSchema = z
.looseObject({
driver: z.literal('metricflow'),
metricflow: z
.looseObject({
repoUrl: z.string().min(1).describe('Git URL of the MetricFlow / SL project.'),
branch: z.string().min(1).optional().describe('Git branch (default "main").'),
path: z.string().optional().describe('Subdirectory within the repo when the SL config lives in a monorepo.'),
auth_token_ref: z.string().min(1).optional().describe('Reference to Git auth token for private repos.'),
})
.describe('Nested MetricFlow configuration block.'),
})
.describe('MetricFlow / SL context-source connection.');
export const connectionConfigSchema = z.discriminatedUnion('driver', [
...warehouseConnectionSchemas,
metabaseConnectionSchema,
lookerConnectionSchema,
lookmlConnectionSchema,
notionConnectionSchema,
dbtConnectionSchema,
metricflowConnectionSchema,
]);
export type KtxConnectionConfig = z.infer<typeof connectionConfigSchema>;

1
packages/context/src/project/index.ts
View file

@ -15,6 +15,7 @@ export {
serializeKtxProjectConfig,
validateKtxProjectConfig,
} from './config.js';
export type { KtxConnectionConfig } from './driver-schemas.js';
export type { LocalGitFileStoreDeps } from './local-git-file-store.js';
export { LocalGitFileStore } from './local-git-file-store.js';
export { ktxLocalStateDbPath } from './local-state-db.js';

16
packages/context/src/project/mappings-yaml-schema.test.ts
View file

@ -1,5 +1,8 @@
import { describe, expect, it } from 'vitest';
import {
lookerMappingsSchema,
lookmlMappingsSchema,
metabaseMappingsSchema,
parseConnectionMappingBootstrap,
parseLookmlMappingBootstrap,
parseLookerMappingBootstrap,
@ -82,4 +85,17 @@ describe('ktx.yaml mapping bootstrap schema', () => {
}),
).toMatchObject({ adapter: 'looker', connectionId: 'prod-looker' });
});
it('exports mapping shapes that parse documented examples', () => {
expect(metabaseMappingsSchema.parse({ databaseMappings: { '1': 'wh' } })).toMatchObject({
databaseMappings: { '1': 'wh' },
syncMode: 'ALL',
});
expect(lookerMappingsSchema.parse({ connectionMappings: { x: 'wh' } })).toEqual({
connectionMappings: { x: 'wh' },
});
expect(lookmlMappingsSchema.parse({ expectedLookerConnectionName: 'x' })).toEqual({
expectedLookerConnectionName: 'x',
});
});
});

66
packages/context/src/project/mappings-yaml-schema.ts
View file

@ -1,5 +1,4 @@
import * as z from 'zod';
import type { KtxProjectConnectionConfig } from './config.js';
const metabaseSyncModeSchema = z.enum(['ALL', 'ONLY', 'EXCEPT']);
const positiveIntegerValueSchema = z.number().int().positive();
@ -11,24 +10,48 @@ const metabaseSelectionsSchema = z
items: z.array(positiveIntegerValueSchema).default([]),
});
const metabaseMappingsSchema = z
export const metabaseMappingsSchema = z
.object({
databaseMappings: z.record(z.string(), stringTargetSchema).default({}),
syncEnabled: z.record(z.string(), z.boolean()).default({}),
syncMode: metabaseSyncModeSchema.default('ALL'),
selections: metabaseSelectionsSchema.default({ collections: [], items: [] }),
defaultTagNames: z.array(z.string().min(1)).default([]),
});
databaseMappings: z
.record(z.string(), stringTargetSchema)
.default({})
.describe('Map of Metabase database ID (positive integer string) to KTX connection ID. Use null to explicitly unmap.'),
syncEnabled: z
.record(z.string(), z.boolean())
.default({})
.describe('Per-Metabase-database sync toggle, keyed by Metabase database ID string.'),
syncMode: metabaseSyncModeSchema
.default('ALL')
.describe('Sync scope: ALL ingests every mapped DB; ONLY restricts to syncEnabled=true; EXCEPT excludes syncEnabled=true.'),
selections: metabaseSelectionsSchema
.default({ collections: [], items: [] })
.describe('Optional Metabase collection and item IDs to scope ingest.'),
defaultTagNames: z
.array(z.string().min(1))
.default([])
.describe('Default tag names applied to ingested Metabase artifacts.'),
})
.describe('Metabase database-to-warehouse mapping and sync configuration.');
const lookerMappingsSchema = z
export const lookerMappingsSchema = z
.object({
connectionMappings: z.record(z.string().min(1), stringTargetSchema).default({}),
});
connectionMappings: z
.record(z.string().min(1), stringTargetSchema)
.default({})
.describe('Map of Looker connection name to KTX connection ID. Use null to explicitly unmap.'),
})
.describe('Looker connection-to-warehouse mapping configuration.');
const lookmlMappingsSchema = z
export const lookmlMappingsSchema = z
.object({
expectedLookerConnectionName: z.string().min(1).nullable().default(null),
});
expectedLookerConnectionName: z
.string()
.min(1)
.nullable()
.default(null)
.describe('Looker connection name that LookML models must declare; mismatches block sl_write_source at ingest time.'),
})
.describe('LookML connection-name expectation for ingest gating.');
export type MetabaseMappingBootstrap = {
adapter: 'metabase';
@ -54,6 +77,11 @@ export type LookmlMappingBootstrap = {
export type ConnectionMappingBootstrap = MetabaseMappingBootstrap | LookerMappingBootstrap | LookmlMappingBootstrap;
type MappingConnectionInput = Record<string, unknown> & {
driver?: unknown;
mappings?: unknown;
};
function recordValue(value: unknown): Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value) ? (value as Record<string, unknown>) : {};
}
@ -66,13 +94,13 @@ function assertPositiveIntegerKeys(field: string, record: Record<string, unknown
}
}
function driverOf(connection: KtxProjectConnectionConfig): string {
function driverOf(connection: MappingConnectionInput): string {
return String(connection.driver ?? '').toLowerCase();
}
export function parseMetabaseMappingBootstrap(
connectionId: string,
connection: KtxProjectConnectionConfig,
connection: MappingConnectionInput,
): MetabaseMappingBootstrap {
const rawMappings = recordValue(connection.mappings);
assertPositiveIntegerKeys('databaseMappings', recordValue(rawMappings.databaseMappings));
@ -91,7 +119,7 @@ export function parseMetabaseMappingBootstrap(
export function parseLookerMappingBootstrap(
connectionId: string,
connection: KtxProjectConnectionConfig,
connection: MappingConnectionInput,
): LookerMappingBootstrap {
const parsed = lookerMappingsSchema.parse(recordValue(connection.mappings));
return {
@ -103,7 +131,7 @@ export function parseLookerMappingBootstrap(
export function parseLookmlMappingBootstrap(
connectionId: string,
connection: KtxProjectConnectionConfig,
connection: MappingConnectionInput,
): LookmlMappingBootstrap {
const parsed = lookmlMappingsSchema.parse(recordValue(connection.mappings));
return {
@ -115,7 +143,7 @@ export function parseLookmlMappingBootstrap(
export function parseConnectionMappingBootstrap(
connectionId: string,
connection: KtxProjectConnectionConfig,
connection: MappingConnectionInput,
): ConnectionMappingBootstrap | null {
if (!connection.mappings || typeof connection.mappings !== 'object' || Array.isArray(connection.mappings)) {
return null;

19
packages/context/src/project/project.test.ts
View file

@ -20,15 +20,13 @@ describe('KTX local project runtime', () => {
const result = await initKtxProject({
projectDir,
projectName: 'warehouse',
authorName: 'Agent',
authorEmail: 'agent@example.com',
});
expect(result.projectDir).toBe(projectDir);
expect(result.config.project).toBe('warehouse');
expect(result.commitHash).toMatch(/^[0-9a-f]{40}$/);
await expect(readFile(join(projectDir, 'ktx.yaml'), 'utf-8')).resolves.toContain('project: warehouse');
await expect(readFile(join(projectDir, 'ktx.yaml'), 'utf-8')).resolves.not.toContain('project:');
const gitignore = await readFile(join(projectDir, '.ktx/.gitignore'), 'utf-8');
expect(gitignore).toContain('cache/');
expect(gitignore).toContain('db.sqlite');
@ -46,7 +44,7 @@ describe('KTX local project runtime', () => {
it('loads an initialized project with a working file store', async () => {
const projectDir = join(tempDir, 'warehouse');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
const loaded = await loadKtxProject({ projectDir });
await loaded.fileStore.writeFile(
@ -57,7 +55,6 @@ describe('KTX local project runtime', () => {
'Add revenue page',
);
expect(loaded.config.project).toBe('warehouse');
await expect(loaded.fileStore.readFile('wiki/global/revenue.md')).resolves.toMatchObject({
content: '# Revenue\n',
});
@ -65,16 +62,12 @@ describe('KTX local project runtime', () => {
it('rejects reinitializing an existing project unless force is set', async () => {
const projectDir = join(tempDir, 'warehouse');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await expect(initKtxProject({ projectDir, projectName: 'warehouse' })).rejects.toThrow(
'Project already contains ktx.yaml',
);
await expect(initKtxProject({ projectDir })).rejects.toThrow('Project already contains ktx.yaml');
await expect(initKtxProject({ projectDir, projectName: 'warehouse-v2', force: true })).resolves.toMatchObject({
config: {
project: 'warehouse-v2',
},
await expect(initKtxProject({ projectDir, force: true })).resolves.toMatchObject({
configPath: join(projectDir, 'ktx.yaml'),
});
});
});

5
packages/context/src/project/project.ts
View file

@ -7,7 +7,6 @@ import { LocalGitFileStore } from './local-git-file-store.js';
export interface InitKtxProjectOptions {
projectDir: string;
projectName?: string;
force?: boolean;
authorName?: string;
authorEmail?: string;
@ -101,7 +100,7 @@ async function createRuntime(
export async function initKtxProject(options: InitKtxProjectOptions): Promise<InitKtxProjectResult> {
const projectDir = resolve(options.projectDir);
const projectName = options.projectName?.trim() || basename(projectDir) || 'ktx-project';
const projectName = basename(projectDir) || 'ktx-project';
const authorName = options.authorName ?? 'ktx';
const authorEmail = options.authorEmail ?? 'ktx@example.com';
const logger = options.logger ?? noopLogger;
@ -112,7 +111,7 @@ export async function initKtxProject(options: InitKtxProjectOptions): Promise<In
throw new Error(`Project already contains ktx.yaml: ${configPath}`);
}
const config = buildDefaultKtxProjectConfig(projectName);
const config = buildDefaultKtxProjectConfig();
const runtime = await createRuntime(projectDir, config, authorName, authorEmail, logger);
await writeProjectFile(projectDir, 'ktx.yaml', serializeKtxProjectConfig(config));

2
packages/context/src/project/setup-config.test.ts
View file

@ -36,7 +36,7 @@ describe('KTX setup config helpers', () => {
});
it('sets setup database connection ids without duplicates', () => {
const config = buildDefaultKtxProjectConfig('warehouse');
const config = buildDefaultKtxProjectConfig();
const withDatabases = setKtxSetupDatabaseConnectionIds(config, ['warehouse', 'analytics', 'warehouse']);

1
packages/context/src/scan/local-enrichment-artifacts.test.ts
View file

@ -231,7 +231,6 @@ describe('writeLocalScanEnrichmentArtifacts', () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-enrichment-artifacts-'));
project = await initKtxProject({
projectDir: join(tempDir, 'project'),
projectName: 'warehouse',
});
});

6
packages/context/src/scan/local-enrichment.test.ts
View file

@ -371,7 +371,7 @@ describe('local scan enrichment', () => {
},
},
relationshipSettings: {
...buildDefaultKtxProjectConfig('warehouse').scan.relationships,
...buildDefaultKtxProjectConfig().scan.relationships,
llmProposals: false,
maxLlmTablesPerBatch: 40,
},
@ -383,7 +383,7 @@ describe('local scan enrichment', () => {
it('skips relationship detection when scan relationships are disabled', async () => {
const settings = {
...buildDefaultKtxProjectConfig('warehouse').scan.relationships,
...buildDefaultKtxProjectConfig().scan.relationships,
enabled: false,
};
const result = await runLocalScanEnrichment({
@ -509,7 +509,7 @@ describe('local scan enrichment', () => {
})),
};
const settings = {
...buildDefaultKtxProjectConfig('test').scan.relationships,
...buildDefaultKtxProjectConfig().scan.relationships,
enabled: false,
};

3
packages/context/src/scan/local-enrichment.ts
View file

@ -538,8 +538,7 @@ export async function runLocalScanEnrichment(
const now = input.now ?? (() => new Date());
const state = completedKtxScanEnrichmentStateSummary();
const syncId = input.syncId ?? input.context.runId;
const relationshipSettings =
input.relationshipSettings ?? buildDefaultKtxProjectConfig(input.connectionId).scan.relationships;
const relationshipSettings = input.relationshipSettings ?? buildDefaultKtxProjectConfig().scan.relationships;
const inputHash = computeKtxScanEnrichmentInputHash({
snapshot,
mode: input.mode,

9
packages/context/src/scan/local-scan.test.ts
View file

@ -105,7 +105,6 @@ async function writeLiveDatabaseConfig(projectDir: string): Promise<void> {
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -123,7 +122,6 @@ async function writeDatabaseConfigWithoutIngestAdapters(projectDir: string): Pro
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -184,7 +182,7 @@ describe('local scan', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-scan-'));
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeLiveDatabaseConfig(projectDir);
project = await loadKtxProject({ projectDir });
});
@ -1037,7 +1035,6 @@ describe('local scan', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: postgres',
@ -1393,7 +1390,6 @@ describe('local scan', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: sqlite',
@ -1425,7 +1421,6 @@ describe('local scan', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: mysql',
@ -1457,7 +1452,6 @@ describe('local scan', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: clickhouse',
@ -1492,7 +1486,6 @@ describe('local scan', () => {
await writeFile(
join(project.projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: sqlserver',

1
packages/context/src/scan/local-structural-artifacts.test.ts
View file

@ -13,7 +13,6 @@ describe('readLocalScanStructuralSnapshot', () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-structural-artifacts-'));
project = await initKtxProject({
projectDir: join(tempDir, 'project'),
projectName: 'warehouse',
});
});

5
packages/context/src/scan/relationship-artifacts.test.ts
View file

@ -19,7 +19,6 @@ async function writeWarehouseConfig(projectDir: string): Promise<void> {
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: sqlite',
@ -67,7 +66,7 @@ function liveDatabaseAdapter(): SourceAdapter {
}
async function createLiveDatabaseRun(projectDir: string, runId: string) {
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeWarehouseConfig(projectDir);
const project = await loadKtxProject({ projectDir });
await runLocalStageOnlyIngest({
@ -283,7 +282,7 @@ describe('local scan relationship artifact reader', () => {
it('returns null when the scan run has no report', async () => {
const projectDir = await mkdtemp(join(tmpdir(), 'ktx-relationship-artifacts-missing-run-'));
try {
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
const project = await loadKtxProject({ projectDir });
await expect(readLocalScanRelationshipArtifacts(project, 'missing-run')).resolves.toBeNull();

6
packages/context/src/scan/relationship-discovery.test.ts
View file

@ -243,7 +243,7 @@ function llmProvider(): KtxLlmProvider {
}
function relationshipSettings() {
return buildDefaultKtxProjectConfig('warehouse').scan.relationships;
return buildDefaultKtxProjectConfig().scan.relationships;
}
function llmOnlyRelationshipSnapshot(): KtxSchemaSnapshot {
@ -557,7 +557,7 @@ describe('production relationship discovery', () => {
`);
const settings = {
...buildDefaultKtxProjectConfig('warehouse').scan.relationships,
...buildDefaultKtxProjectConfig().scan.relationships,
acceptThreshold: 0.99,
reviewThreshold: 0.55,
};
@ -633,7 +633,7 @@ describe('production relationship discovery', () => {
schema: snapshotToKtxEnrichedSchema(richSnapshot),
context: { runId: 'candidate-cap' },
settings: {
...buildDefaultKtxProjectConfig('warehouse').scan.relationships,
...buildDefaultKtxProjectConfig().scan.relationships,
maxCandidatesPerColumn: 1,
},
});

1
packages/context/src/scan/relationship-review-apply.test.ts
View file

@ -202,7 +202,6 @@ async function projectWithDecisions(
const tempDir = await mkdtemp(join(tmpdir(), 'ktx-relationship-review-apply-'));
const project = await initKtxProject({
projectDir: join(tempDir, 'project'),
projectName: 'warehouse',
});
await project.fileStore.writeFile(
'raw-sources/warehouse/live-database/sync-a/enrichment/relationship-review-decisions.json',

3
packages/context/src/scan/relationship-review-decisions.test.ts
View file

@ -19,11 +19,10 @@ async function writeProjectFile(projectDir: string, relativePath: string, conten
}
async function createProject(projectDir: string): Promise<void> {
await initKtxProject({ projectDir, projectName: 'warehouse' });
await initKtxProject({ projectDir });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: warehouse',
'connections:',
' warehouse:',
' driver: sqlite',

2
packages/context/src/search/backend-conformance.test.ts
View file

@ -241,7 +241,7 @@ describe('SQLite hybrid search backend conformance', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-search-conformance-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
dbPath = join(tempDir, '.ktx', 'db.sqlite');
});

2
packages/context/src/sl/local-query.test.ts
View file

@ -13,7 +13,7 @@ describe('compileLocalSlQuery', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-query-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
project.config.connections.warehouse = { driver: 'postgres' };
await project.fileStore.writeFile(
'semantic-layer/warehouse/orders.yaml',

2
packages/context/src/sl/local-sl.test.ts
View file

@ -51,7 +51,7 @@ describe('local semantic-layer helpers', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-sl-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
});
afterEach(async () => {

2
packages/context/src/sl/pglite-sl-search-prototype.test.ts
View file

@ -169,7 +169,7 @@ describe('PGlite semantic-layer search prototype', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-pglite-sl-prototype-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
project.config.ingest.embeddings.dimensions = 3;
pgliteDataDir = join(tempDir, 'pglite-search');
port = await allocatePort();

2
packages/context/src/sl/sl-dictionary-profile.test.ts
View file

@ -11,7 +11,7 @@ describe('loadLatestSlDictionaryEntries', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-sl-dictionary-profile-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
});
afterEach(async () => {

2
packages/context/src/wiki/local-knowledge.test.ts
View file

@ -28,7 +28,7 @@ describe('local knowledge helpers', () => {
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-knowledge-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project = await initKtxProject({ projectDir: join(tempDir, 'project') });
});
afterEach(async () => {