apunkt/ktx
1
0
Fork 0
mirror of https://github.com/Kaelio/ktx.git synced 2026-06-07 07:55:13 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge remote-tracking branch 'origin/main' into improve-setup-cli-flow

Browse source 
# Conflicts:
#	packages/cli/src/setup-databases.ts
#	packages/cli/src/setup-sources.ts
This commit is contained in:
Andrey Avtomonov 2026-05-13 16:55:03 +02:00
commit 4dde0932a2
183 changed files with 1256 additions and 986 deletions
Download patch file Download diff file Expand all files Collapse all files

6
README.md
View file

@ -19,7 +19,7 @@ reviewable project files that agents can use while planning, querying, and
updating analytics work.
A KTX project is a directory of plain files — YAML semantic sources, Markdown
knowledge pages, and SQLite state — that you commit to git and review in PRs,
wiki pages, and SQLite state — that you commit to git and review in PRs,
just like dbt models.
## Who KTX is for
@ -105,7 +105,7 @@ my-project/
│ ├── orders.yaml # Semantic source definitions
│ ├── customers.yaml
│ └── order_items.yaml
├── knowledge/
├── wiki/
│ ├── global/
│ │ ├── revenue.md # Business definitions and rules
│ │ └── segment-classification.md
@ -118,7 +118,7 @@ my-project/
└── db.sqlite # Local state (git-ignored)
```
Semantic sources and knowledge pages are committed to git. The `.ktx/` directory
Semantic sources and wiki pages are committed to git. The `.ktx/` directory
holds ephemeral state and is git-ignored — delete it and KTX rebuilds on the
next run.

65
docs-site/content/docs/cli-reference/ktx-wiki.mdx
View file

@ -1,9 +1,9 @@
---
title: "ktx wiki"
description: "List or search knowledge pages."
description: "List, read, search, or write wiki pages."
---
Manage knowledge pages in your KTX project. Knowledge pages are Markdown documents that capture business definitions, rules, and gotchas. Agents search them for context when answering questions about your data.
Manage wiki pages in your KTX project. Wiki pages are Markdown documents that capture business definitions, rules, and gotchas. Agents search them for context when answering questions about your data.
## Command signature
@ -16,7 +16,9 @@ ktx wiki <subcommand> [options]
| Subcommand | Description |
|-----------|-------------|
| `list` | List local wiki pages |
| `read <key>` | Read one local wiki page |
| `search <query>` | Search local wiki pages |
| `write <key>` | Write one local wiki page |
## Options
@ -27,6 +29,13 @@ ktx wiki <subcommand> [options]
| `--json` | Print JSON output | `false` |
| `--user-id <id>` | Local user id | `local` |
### `wiki read`
| Flag | Description | Default |
|------|-------------|---------|
| `--json` | Print JSON output | `false` |
| `--user-id <id>` | Local user id | `local` |
### `wiki search`
| Flag | Description | Default |
@ -35,6 +44,18 @@ ktx wiki <subcommand> [options]
| `--user-id <id>` | Local user id | `local` |
| `--limit <number>` | Maximum search results | — |
### `wiki write`
| Flag | Description | Default |
|------|-------------|---------|
| `--user-id <id>` | Local user id | `local` |
| `--scope <scope>` | Scope: `global` or `user` | `global` |
| `--summary <summary>` | Wiki page summary (required) | — |
| `--content <content>` | Wiki page content (required) | — |
| `--tag <tag>` | Wiki tag; repeatable | — |
| `--ref <ref>` | Wiki ref; repeatable | — |
| `--sl-ref <ref>` | Semantic-layer ref; repeatable | — |
## Examples
```bash
@ -44,16 +65,48 @@ ktx wiki list
# List all wiki pages as JSON
ktx wiki list --json
# Read a specific wiki page
ktx wiki read revenue-definitions
# Read a specific wiki page as JSON
ktx wiki read revenue-definitions --json
# Search wiki pages
ktx wiki search "monthly recurring revenue"
# Search wiki pages as JSON
ktx wiki search "monthly recurring revenue" --json --limit 10
# Write a global wiki page
ktx wiki write revenue-definitions \
--summary "Canonical revenue metric definitions" \
--content "## MRR\nMonthly Recurring Revenue is calculated as..."
# Write a user-scoped wiki page
ktx wiki write my-notes \
--scope user \
--summary "Personal analysis notes" \
--content "Things to check when revenue numbers look off..."
# Write a page with tags and references
ktx wiki write churn-rules \
--summary "Churn calculation business rules" \
--content "A customer is considered churned when..." \
--tag finance \
--tag retention \
--sl-ref customers \
--sl-ref subscriptions
# Write a page with external references
ktx wiki write data-freshness \
--summary "Data pipeline SLAs and freshness guarantees" \
--content "The orders table refreshes every 15 minutes..." \
--ref "https://wiki.example.com/data-pipelines"
```
## Output
Wiki commands print local knowledge pages and search results.
Wiki commands print local wiki pages and search results. Agents should search first, then read the most relevant page by key.
```json
{
@ -74,5 +127,7 @@ Wiki commands print local knowledge pages and search results.
| Error | Cause | Recovery |
|-------|-------|----------|
| Search returns no results | The query terms do not match summaries, tags, or content | Retry with business synonyms or run ingest to capture more context |
| A page is missing | The page has not been created by ingest or memory capture yet | Run ingest, then search again with `ktx wiki search` |
| Search returns no results | The query terms do not match summaries, tags, or content | Retry with business synonyms, then create a page if the knowledge is missing |
| Read fails for a key | The page key is wrong or scoped to a different user | Run `ktx wiki list` or search again to get the exact key |
| Write fails due to missing fields | `--summary` or `--content` was omitted | Pass both fields, and keep the summary short enough for search results |
| Agent writes duplicate pages | It did not search existing pages first | Always run `ktx wiki search` before `ktx wiki write` |

16
docs-site/content/docs/concepts/context-as-code.mdx
View file

@ -7,9 +7,9 @@ description: Treat analytics context like code — version it, review it, merge
dbt proved that analytics transformations belong in version control. Before dbt, SQL lived in BI tools, scheduling systems, and spreadsheets — scattered, unreviewed, impossible to audit. "Analytics as code" changed that: put your models in git, review them in PRs, deploy them by merging.
KTX applies the same principle to analytics context. Metric definitions, business rules, join relationships, knowledge pages — these are artifacts that determine whether an agent produces correct results. They change over time. They need review. They need history. They need to be treated like code.
KTX applies the same principle to analytics context. Metric definitions, business rules, join relationships, wiki pages — these are artifacts that determine whether an agent produces correct results. They change over time. They need review. They need history. They need to be treated like code.
A KTX project is a git repository. Semantic sources are YAML files. Knowledge pages are Markdown files. Changes are commits. Updates are pull requests. Deployment is a merge. The entire lifecycle of your analytics context follows the same workflow your team already uses for dbt models, application code, and infrastructure.
A KTX project is a git repository. Semantic sources are YAML files. Wiki pages are Markdown files. Changes are commits. Updates are pull requests. Deployment is a merge. The entire lifecycle of your analytics context follows the same workflow your team already uses for dbt models, application code, and infrastructure.
## Auto-ingestion
@ -19,9 +19,9 @@ An ingestion run works like this:
1. **Adapters extract metadata.** Each configured source — dbt, LookML, Metabase, MetricFlow, Notion, or your live database — provides structured metadata about models, metrics, dimensions, questions, and documentation.
2. **The LLM agent reconciles.** KTX doesn't blindly overwrite existing context. An LLM agent compares incoming metadata against your current semantic sources and knowledge pages. It decides what to create, what to update, and what to leave alone. If your dbt project added a new model, the agent writes a new semantic source. If a Metabase question references a metric you've already defined, the agent skips the duplicate.
2. **The LLM agent reconciles.** KTX doesn't blindly overwrite existing context. An LLM agent compares incoming metadata against your current semantic sources and wiki pages. It decides what to create, what to update, and what to leave alone. If your dbt project added a new model, the agent writes a new semantic source. If a Metabase question references a metric you've already defined, the agent skips the duplicate.
3. **Files are written.** New and updated YAML sources and Markdown knowledge pages are written to the project directory. Every decision is recorded in the session transcript.
3. **Files are written.** New and updated YAML sources and Markdown wiki pages are written to the project directory. Every decision is recorded in the session transcript.
This reconciliation step is what separates auto-ingestion from a simple sync. A naive import would overwrite your hand-tuned metric definitions every time dbt's manifest changes. KTX's agent-driven approach merges intelligently: it respects your edits, fills gaps, and flags conflicts for human review.
@ -43,7 +43,7 @@ dbt / Looker / Metabase / Notion
|
| + 3 new sources
| ~ 2 updated joins
| + 1 knowledge page
| + 1 wiki page
v
open PR
|
@ -57,7 +57,7 @@ dbt / Looker / Metabase / Notion
agents see updated context
```
A typical branch shows a semantic diff: "this ingest added 3 new sources from dbt, updated 2 join definitions based on schema changes, and created 1 knowledge page from a Notion doc." Analytics engineers review the diff, verify that the new sources look correct, and merge.
A typical branch shows a semantic diff: "this ingest added 3 new sources from dbt, updated 2 join definitions based on schema changes, and created 1 wiki page from a Notion doc." Analytics engineers review the diff, verify that the new sources look correct, and merge.
Teams usually run this on demand while setting up a source, then schedule it once the source is stable. A cron job or CI schedule can run `ktx ingest run --connection-id <id> --adapter <adapter> --no-input` overnight on an ingest branch so the latest dbt manifests, BI metadata, and documentation updates are ready for review each morning.
@ -69,9 +69,9 @@ This workflow gives you the same review guarantees you have for dbt models. No s
Context improves over time through two feedback channels.
**Analyst corrections.** When an analytics engineer spots something wrong — a measure formula that doesn't match the business definition, a join that should be `many_to_one` instead of `one_to_many`, a knowledge page that's out of date — they edit the YAML or Markdown directly and commit. These corrections become part of the project's git history, and the next ingestion run respects them. If you manually fix a measure definition, KTX won't overwrite it on the next ingest.
**Analyst corrections.** When an analytics engineer spots something wrong — a measure formula that doesn't match the business definition, a join that should be `many_to_one` instead of `one_to_many`, a wiki page that's out of date — they edit the YAML or Markdown directly and commit. These corrections become part of the project's git history, and the next ingestion run respects them. If you manually fix a measure definition, KTX won't overwrite it on the next ingest.
**Agent feedback.** When an agent queries the semantic layer and gets unexpected results — a query that returns no rows because of a bad filter, a join path that produces duplicated results — it can flag the issue. These signals feed back into the context: knowledge pages can note known data quality issues, and source definitions can be tightened with better filters, join paths, or grain declarations.
**Agent feedback.** When an agent queries the semantic layer and gets unexpected results — a query that returns no rows because of a bad filter, a join path that produces duplicated results — it can flag the issue. These signals feed back into the context: wiki pages can note known data quality issues, and source definitions can be tightened with better filters, join paths, or grain declarations.
Each of these channels makes the next ingestion cycle better. Analyst corrections teach the system what your team considers authoritative. Agent feedback surfaces gaps in coverage. Context is not a static artifact — it's a living system that converges toward accuracy with every iteration.

14
docs-site/content/docs/concepts/the-context-layer.mdx
View file

@ -30,7 +30,7 @@ A context layer is the infrastructure that gives agents the business knowledge t
KTX organizes context into four pillars:
- Semantic sources
- Knowledge pages
- Wiki pages
- Scan artifacts
- Provenance
@ -67,7 +67,7 @@ measures:
expr: count(id)
```
**Knowledge pages** are Markdown documents that capture business definitions, rules, and operating context — the kind of context that doesn't fit in a schema definition. Pages have structured frontmatter (summary, tags, semantic layer references) and free-form content. Agents search them when they need to understand why a metric works a certain way, not just how to compute it.
**Wiki pages** are Markdown documents that capture business definitions, rules, and operating context — the kind of context that doesn't fit in a schema definition. Pages have structured frontmatter (summary, tags, semantic layer references) and free-form content. Agents search them when they need to understand why a metric works a certain way, not just how to compute it.
```markdown
---
@ -97,13 +97,13 @@ Together, these four pillars give agents enough context to produce analytics art
## How KTX compares
KTX is a context layer with an agent-native semantic layer at its core. MetricFlow, Cube, and Malloy model metrics, dimensions, joins, and generated SQL. KTX covers that semantic-layer work, then adds the context agents need to use and maintain it: knowledge pages, schema scans, provenance, ingestion, validation, and agent-facing CLI commands.
KTX is a context layer with an agent-native semantic layer at its core. MetricFlow, Cube, and Malloy model metrics, dimensions, joins, and generated SQL. KTX covers that semantic-layer work, then adds the context agents need to use and maintain it: wiki pages, schema scans, provenance, ingestion, validation, and agent-facing CLI commands.
The workflow is the difference. Traditional semantic layers are powerful, but they are usually built and maintained through manual modeling work, product-specific runtimes, or language-specific workflows. They are not agent-native by default, which makes them harder for agents to inspect, edit, validate, and review in a tight loop. KTX is designed for agents that need to read context, change semantic files, inspect generated SQL, and leave a reviewable git diff.
| | KTX semantic layer | MetricFlow | Cube | Malloy |
|---|---|---|---|---|
| **Model surface** | Plain YAML sources plus Markdown knowledge pages | YAML semantic models and metrics in a dbt project | YAML or JavaScript cubes, views, access policies, and pre-aggregations | `.malloy` models, query pipelines, notebooks, and annotations |
| **Model surface** | Plain YAML sources plus Markdown wiki pages | YAML semantic models and metrics in a dbt project | YAML or JavaScript cubes, views, access policies, and pre-aggregations | `.malloy` models, query pipelines, notebooks, and annotations |
| **What it models** | Sources, columns, measures, segments, joins, grain, filters, default time dimensions, and context references | Semantic models, entities, dimensions, measures, metrics, time grains, and metric types | Cubes, views, measures, dimensions, segments, joins, hierarchies, policies, and rollups | Sources, joins, dimensions, measures, calculations, nested results, and query pipelines |
| **Agent edit loop** | First-class. Agents can patch small files, save imperfect drafts, run validation, query through the CLI, inspect SQL, and refine in the same workflow | Possible, but the interface is a dbt/metric workflow rather than an agent context workflow | Possible through code-first models and platform APIs, but changes are tied to runtime deployment and governance concerns | Possible, but agents must operate in Malloy's language and compiler model |
| **Fan-out safety** | Explicit `grain` plus relationship metadata. KTX detects `one_to_many` fan-out, identifies chasm traps, pre-aggregates independent fact measures into CTEs, and rejects unsafe filters | Dataflow query planning for metric requests, multi-hop joins, metric time, and metric types | Runtime planner, modeled joins, primary keys, views, multi-fact views, and pre-aggregations | Symmetric aggregates and path-based aggregation in the language |
@ -111,7 +111,7 @@ The workflow is the difference. Traditional semantic layers are powerful, but th
| **Context around semantics** | Built in: wiki pages, scan artifacts, relationship inference, ingest transcripts, replay, and agent-facing CLI commands | Primarily metric and dbt project context | Descriptions and `meta.ai_context` inside the semantic model, plus platform agent features | Annotations/tags can carry metadata; surrounding context depends on the application |
| **Best fit** | Agents maintaining analytics code, metrics, joins, SQL, docs, and semantic definitions | Teams standardizing metrics inside dbt workflows | Production semantic APIs, BI integrations, access control, caching, and concurrency | Expressive modeling and exploratory analysis above SQL |
If you do not have a semantic layer, KTX can build an agent-native one from your database schema and enrich it with generated descriptions and knowledge pages. If you already use MetricFlow or LookML, KTX ingests from those tools and merges their context into KTX's files. You can keep your existing BI or metric-serving system while using KTX as the semantic and contextual surface agents work against.
If you do not have a semantic layer, KTX can build an agent-native one from your database schema and enrich it with generated descriptions and wiki pages. If you already use MetricFlow or LookML, KTX ingests from those tools and merges their context into KTX's files. You can keep your existing BI or metric-serving system while using KTX as the semantic and contextual surface agents work against.
## The plain-files philosophy
@ -125,7 +125,7 @@ my-project/
│ ├── orders.yaml # Semantic source definitions
│ ├── customers.yaml
│ └── order_items.yaml
├── knowledge/
├── wiki/
│ ├── global/
│ │ ├── revenue.md # Business definitions and rules
│ │ └── segment-classification.md
@ -140,7 +140,7 @@ my-project/
└── cache/ # Runtime cache (git-ignored)
```
Semantic sources and knowledge pages are committed to git. The SQLite database holds ephemeral state — scan results, embedding indexes, session logs — and is git-ignored. If you delete it, KTX rebuilds it on the next run.
Semantic sources and wiki pages are committed to git. The SQLite database holds ephemeral state — scan results, embedding indexes, session logs — and is git-ignored. If you delete it, KTX rebuilds it on the next run.
This means your analytics context travels with your code. You can fork it, branch it, review it in a PR, and merge it with the same tools you use for dbt models. There's no sync problem between a remote server and your local state. There's no migration to run. The files are the source of truth.

2
docs-site/content/docs/getting-started/introduction.mdx
View file

@ -88,5 +88,5 @@ Works with PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, and SQL Server.
| Set up a new KTX project | [Quickstart](/docs/getting-started/quickstart) |
| Explain what problem KTX solves | [The Context Layer](/docs/concepts/the-context-layer) |
| Scan a database and ingest metadata | [Building Context](/docs/guides/building-context) |
| Edit semantic sources or knowledge pages | [Writing Context](/docs/guides/writing-context) |
| Edit semantic sources or wiki pages | [Writing Context](/docs/guides/writing-context) |
| Look up exact command flags | [CLI Reference](/docs/cli-reference/ktx-setup) |

8
docs-site/content/docs/getting-started/quickstart.mdx
View file

@ -146,7 +146,7 @@ This is where KTX does the heavy lifting. It runs an enriched scan of your datab
│ ○ Leave context unbuilt and exit setup
```
The build scans each primary source with LLM enrichment, detects table relationships, and runs ingestion agents that reconcile metadata from your context sources into semantic-layer YAML files and knowledge pages.
The build scans each primary source with LLM enrichment, detects table relationships, and runs ingestion agents that reconcile metadata from your context sources into semantic-layer YAML files and wiki pages.
For a small database (under 50 tables), this takes a few minutes. Larger warehouses can take longer. You can press <kbd>d</kbd> to detach and let it run in the background:
@ -209,8 +209,8 @@ KTX writes project state as plain files so agents can inspect and edit changes i
| `ktx.yaml` | `ktx setup` | Main project configuration: connections, LLM settings, embeddings, and context sources |
| `.ktx/secrets/*` | `ktx setup` when file-backed secrets are selected | Local secret files referenced from `ktx.yaml`; do not commit these |
| `semantic-layer/<connection-id>/*.yaml` | context build, ingestion, or direct file edits | Semantic source definitions agents use for SQL generation |
| `knowledge/global/*.md` | ingestion, memory capture, or direct file edits | Shared business context and metric definitions |
| `knowledge/user/<user-id>/*.md` | memory capture or direct file edits | User-scoped notes for one agent/user context |
| `wiki/global/*.md` | ingestion, memory capture, `ktx wiki write --scope global`, or direct file edits | Shared business context and metric definitions |
| `wiki/user/<user-id>/*.md` | memory capture, `ktx wiki write --scope user`, or direct file edits | User-scoped notes for one agent/user context |
| `.claude/skills/ktx/SKILL.md`, `.agents/skills/ktx/SKILL.md` | CLI-mode agent integration setup | Agent instructions for calling public `ktx` commands |
## Verify it worked
@ -247,6 +247,6 @@ Agent integration ready: yes (claude-code:project)
## Next steps
- **Build more context** — learn about [scanning](/docs/guides/building-context), relationship detection, and ingestion workflows in the Building Context guide.
- **Refine your semantic layer** — the [Writing Context](/docs/guides/writing-context) guide covers source YAML, measures, joins, and knowledge pages.
- **Refine your semantic layer** — the [Writing Context](/docs/guides/writing-context) guide covers source YAML, measures, joins, and wiki pages.
- **Understand the architecture** — read [The Context Layer](/docs/concepts/the-context-layer) to learn why a context layer is more than a semantic layer.
- **Connect more agents** — see the [Agent Clients](/docs/integrations/agent-clients) integration page for per-tool setup details.

8
docs-site/content/docs/guides/building-context.mdx
View file

@ -53,7 +53,7 @@ Relationship scans run with `ktx scan <connection-id> --mode relationships`. Thi
## Ingestion
Ingestion pulls semantic context from your existing analytics tools — dbt projects, Looker models, Metabase questions, and more — and writes it into your KTX project as semantic sources and knowledge pages.
Ingestion pulls semantic context from your existing analytics tools — dbt projects, Looker models, Metabase questions, and more — and writes it into your KTX project as semantic sources and wiki pages.
### How it works
@ -61,7 +61,7 @@ Each ingest run follows this flow:
1. An **adapter** extracts metadata from your tool (dbt manifest, LookML files, Metabase API, etc.)
2. An **LLM agent** reconciles the extracted metadata with your existing context — it merges intelligently rather than overwriting
3. **Semantic sources** (YAML) and **knowledge pages** (Markdown) are written to your project directory
3. **Semantic sources** (YAML) and **wiki pages** (Markdown) are written to your project directory
### Running an ingest
@ -113,7 +113,7 @@ See [Context Sources](/docs/integrations/context-sources) for adapter-specific s
### What gets generated
A typical dbt ingest produces semantic sources and knowledge pages in your project:
A typical dbt ingest produces semantic sources and wiki pages in your project:
**Semantic source** (`semantic-layer/my-postgres/orders.yaml`):
@ -149,7 +149,7 @@ joins:
relationship: many_to_one
```
**Knowledge page** (`knowledge/global/order-status-definitions.md`):
**Wiki page** (`wiki/global/order-status-definitions.md`):
```markdown
---

6
docs-site/content/docs/guides/serving-agents.mdx
View file

@ -36,10 +36,10 @@ ktx sl query --json \
--max-rows 100
```
**Knowledge:**
**Wiki:**
```bash
# Search knowledge pages
# Search wiki pages
ktx wiki search "revenue recognition" --json --limit 10
```
@ -56,4 +56,4 @@ configuration. For manual setup or per-tool details, see the
[Agent Clients](/docs/integrations/agent-clients) integration page.
After configuration, the agent can immediately call KTX commands to list
sources, search knowledge, and query your semantic layer.
sources, search wiki pages, and query your semantic layer.

27
docs-site/content/docs/guides/writing-context.mdx
View file

@ -1,9 +1,9 @@
---
title: Writing Context
description: Write and refine semantic sources and knowledge pages.
description: Write and refine semantic sources and wiki pages.
---
After building context through scanning and ingestion, you'll want to refine it — edit semantic sources to match your business logic, add knowledge pages that capture tribal knowledge, and query your data through the semantic layer to verify everything works.
After building context through scanning and ingestion, you'll want to refine it — edit semantic sources to match your business logic, add wiki pages that capture tribal knowledge, and query your data through the semantic layer to verify everything works.
## Agent workflow summary
@ -218,20 +218,20 @@ The query planner is grain-aware — it understands the cardinality of joins and
If validation fails, fix the YAML before asking an agent to use the source. Common validation failures are missing columns, invalid join targets, and measure expressions that reference fields outside the source.
## Knowledge Pages
## Wiki Pages
Knowledge pages are Markdown files that capture business context — definitions, rules, gotchas, and anything an agent needs to understand beyond what the schema tells it.
Wiki pages are Markdown files that capture business context — definitions, rules, gotchas, and anything an agent needs to understand beyond what the schema tells it.
### What they are
When an agent asks "what counts as an active user?" or "why do revenue numbers differ between the dashboard and the SQL query?", the answer isn't in the schema. It's tribal knowledge that lives in Slack threads, Notion pages, or someone's head. Knowledge pages make that context searchable and available to agents.
When an agent asks "what counts as an active user?" or "why do revenue numbers differ between the dashboard and the SQL query?", the answer isn't in the schema. It's tribal knowledge that lives in Slack threads, Notion pages, or someone's head. Wiki pages make that context searchable and available to agents.
### Organization
Knowledge pages are organized by scope:
Wiki pages are organized by scope:
```
knowledge/
wiki/
├── global/ # Cross-cutting definitions
│ ├── order-status-definitions.md
│ ├── revenue-recognition-rules.md
@ -247,10 +247,11 @@ knowledge/
### Editing pages
Create and edit knowledge pages directly as Markdown files in the `knowledge/`
directory. Ingest and memory capture also create these pages automatically.
Create and edit wiki pages directly as Markdown files in the `wiki/`
directory, or with `ktx wiki write`. Ingest and memory capture also create
these pages automatically.
Knowledge page fields:
Wiki page fields:
| Field | Required | Description |
|-------|----------|-------------|
@ -279,7 +280,7 @@ Search uses both full-text matching and semantic similarity — it finds relevan
### Workflow: add searchable business context
1. Search first: `ktx wiki search "order status definitions"`.
2. If no page already covers the rule, create or edit a Markdown file under `knowledge/global/`.
2. If no page already covers the rule, create or edit a Markdown file under `wiki/global/`.
3. Include concise frontmatter; agents see the summary before loading full content.
4. Add `tags` values for the business area and `sl_refs` values for related semantic sources.
5. Search again with the user's likely wording to confirm the page is discoverable.
@ -290,6 +291,6 @@ Search uses both full-text matching and semantic similarity — it finds relevan
|------------------|--------------|----------|
| `ktx sl validate` reports a missing column | YAML references a column that is absent from the scanned table | Run a fresh scan or update the YAML to match the warehouse schema |
| Query compilation double-counts a measure | Join relationship or grain is missing or wrong | Add `grain` and explicit `relationship` values, then validate and recompile |
| Agent cannot find a metric | Measure name or description does not match business terminology | Add a measure description and a knowledge page with common synonyms |
| Knowledge search misses a page | Summary and tags do not include likely user wording | Rewrite the summary and add relevant tags, then search again |
| Agent cannot find a metric | Measure name or description does not match business terminology | Add a measure description and a wiki page with common synonyms |
| Wiki search misses a page | Summary and tags do not include likely user wording | Rewrite the summary and add relevant tags, then search again |
| Semantic-layer changes are hard to review | The YAML edit is too large or unfocused | Split the change into smaller source-file edits, then review the git diff |

4
docs-site/content/docs/integrations/agent-clients.mdx
View file

@ -124,7 +124,9 @@ All supported agent clients call the same KTX CLI commands:
| Command | Description |
|---------|-------------|
| `ktx status --json` | Return project setup and context readiness |
| `ktx wiki search <query> --json` | Search knowledge pages |
| `ktx wiki search <query> --json` | Search wiki pages |
| `ktx wiki read <key> --json` | Read a wiki page |
| `ktx wiki write <key>` | Write or update a wiki page |
| `ktx sl list --json` | List semantic-layer sources |
| `ktx sl search <query> --json` | Search semantic-layer sources |
| `ktx sl validate <source> --connection-id <id>` | Validate semantic source definitions |

14
docs-site/content/docs/integrations/context-sources.mdx
View file

@ -15,7 +15,7 @@ Agents should configure and ingest context sources in this order:
2. Store tokens as `env:NAME` or `file:/path/to/secret`.
3. Run `ktx ingest run --connection-id <connectionId> --adapter <adapter>` for one source or `ktx ingest run --connection-id <id> --adapter <adapter>`.
4. Check progress with `ktx ingest status --json`.
5. Review generated `semantic-layer/` YAML and `knowledge/` Markdown files in git.
5. Review generated `semantic-layer/` YAML and `wiki/` Markdown files in git.
6. Validate changed semantic sources with `ktx sl validate`.
## Shared source fields
@ -233,7 +233,7 @@ Generate an API key in Metabase: **Admin > Settings > Authentication > API Keys*
### What gets ingested
- Semantic sources generated from SQL queries in questions
- Knowledge pages for dashboards (purpose, key metrics, relationships)
- Wiki pages for dashboards (purpose, key metrics, relationships)
- Work units per dashboard and per question
### Warehouse mapping
@ -290,7 +290,7 @@ Generate API credentials in Looker: **Admin > Users > Edit > API Keys**.
### What gets ingested
- Semantic sources from explore field definitions
- Knowledge pages for dashboards (purpose, audience, key metrics)
- Wiki pages for dashboards (purpose, audience, key metrics)
- Triage signals for automated content classification
- Work units per explore and per dashboard
@ -310,11 +310,11 @@ Find Looker connection names in **Admin > Database > Connections**.
## Notion
Ingests pages and databases from a Notion workspace as knowledge pages. Useful for capturing business definitions, data dictionaries, and team documentation that agents need for context.
Ingests pages and databases from a Notion workspace as wiki pages. Useful for capturing business definitions, data dictionaries, and team documentation that agents need for context.
### What it provides
- Knowledge pages synthesized from Notion content
- Wiki pages synthesized from Notion content
- Page hierarchy and relationships
- Database schemas (when Notion databases describe data sources)
- Semantic clustering for organized ingestion
@ -364,7 +364,7 @@ Create an integration at [notion.so/my-integrations](https://www.notion.so/my-in
### What gets ingested
- Knowledge pages synthesized from Notion content (not raw copies)
- Wiki pages synthesized from Notion content (not raw copies)
- Domain context extracted and organized by topic
- Triage signals for classifying page relevance
- Work units clustered by semantic similarity for efficient processing
@ -381,6 +381,6 @@ Create an integration at [notion.so/my-integrations](https://www.notion.so/my-in
|------------------|--------------|----------|
| Adapter cannot read source files | `source_dir`, `repo_url`, `branch`, or `path` is wrong | Verify the path locally or clone the repo manually with the same credentials |
| Private repo/API authentication fails | Token env var or secret file is missing | Export the env var or update `auth_token_ref` to a readable file |
| Ingest creates duplicate context | Existing source names or knowledge pages do not match imported terminology | Review the diff, rename duplicates, and add knowledge pages with canonical names |
| Ingest creates duplicate context | Existing source names or wiki pages do not match imported terminology | Review the diff, rename duplicates, and add wiki pages with canonical names |
| Notion ingest skips pages | Integration lacks access or root ids are missing | Share pages with the Notion integration and set `root_page_ids` or use `all_accessible` carefully |
| Generated semantic sources fail validation | Tool metadata does not match the live warehouse schema | Map BI/source databases to primary warehouse connections and rerun validation |

8
docs-site/lib/llm-docs.ts
View file

@ -47,7 +47,7 @@ export function buildLlmsTxt() {
> Agent-native context layer for analytics engineering and database agents.
KTX provides semantic-layer files, warehouse scans, knowledge pages, provenance, and agent-facing tools that help coding agents answer analytics questions without inventing metrics or joins.
KTX provides semantic-layer files, warehouse scans, wiki pages, provenance, and agent-facing tools that help coding agents answer analytics questions without inventing metrics or joins.
## Agent Entry Points
@ -60,7 +60,7 @@ ${link("/docs/ai-resources/agent-instructions", "Agent Instructions", "Suggested
${link("/docs/getting-started/introduction", "Introduction", "What KTX is and who it is for")}
${link("/docs/getting-started/quickstart", "Quickstart", "Set up KTX and build your first context")}
${link("/docs/guides/writing-context", "Writing Context", "Write semantic sources and knowledge pages")}
${link("/docs/guides/writing-context", "Writing Context", "Write semantic sources and wiki pages")}
## Machine-Readable Documentation
@ -68,13 +68,13 @@ ${link("/docs/guides/writing-context", "Writing Context", "Write semantic source
- [Markdown access guide](${absoluteUrl("/docs/ai-resources/markdown-access.md")}): How to fetch llms.txt, llms-full.txt, and per-page Markdown
- [Quickstart markdown](${absoluteUrl("/docs/getting-started/quickstart.md")}): Human setup walkthrough
- [Semantic-layer CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-sl.md")}): Semantic-layer commands and JSON output
- [Wiki CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-wiki.md")}): Knowledge page commands and JSON output
- [Wiki CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-wiki.md")}): Wiki page commands and JSON output
## CLI Reference
${link("/docs/cli-reference/ktx-setup", "ktx setup", "Interactive project setup")}
${link("/docs/cli-reference/ktx-sl", "ktx sl", "Semantic-layer commands")}
${link("/docs/cli-reference/ktx-wiki", "ktx wiki", "Knowledge page commands")}
${link("/docs/cli-reference/ktx-wiki", "ktx wiki", "Wiki page commands")}
${link("/docs/cli-reference/ktx-connection", "ktx connection", "Connection management commands")}
## Integrations

2
examples/local-warehouse/ktx.yaml
View file

@ -19,7 +19,7 @@ agent:
max_iterations: 20
default_toolset:
- sl_query
- knowledge_search
- wiki_search
- sl_read_source
memory:
auto_commit: true

3
examples/local-warehouse/semantic-layer/warehouse/orders.yaml
View file

@ -1,6 +1,7 @@
name: orders
table: public.orders
description: Orders placed through the storefront.
descriptions:
user: Orders placed through the storefront.
grain:
- id
columns:

0
examples/local-warehouse/knowledge/global/revenue.md → examples/local-warehouse/wiki/global/revenue.md
View file

32
packages/cli/assets/demo/orbit/links/provenance.json
View file

@ -2,7 +2,7 @@
{
"id": "link-001",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/arr-contract-first.md",
"artifactKey": "wiki/global/arr-contract-first.md",
"sourceKind": "warehouse",
"sourcePath": "contracts",
"relationship": "describes",
@ -11,7 +11,7 @@
{
"id": "link-002",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/arr-contract-first.md",
"artifactKey": "wiki/global/arr-contract-first.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/arr-and-contract-reporting-notes.md",
"relationship": "derived_from",
@ -20,7 +20,7 @@
{
"id": "link-003",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/revenue-gross-to-net.md",
"artifactKey": "wiki/global/revenue-gross-to-net.md",
"sourceKind": "warehouse",
"sourcePath": "invoices",
"relationship": "describes",
@ -29,7 +29,7 @@
{
"id": "link-004",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/revenue-gross-to-net.md",
"artifactKey": "wiki/global/revenue-gross-to-net.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/revenue-reporting-policy.md",
"relationship": "derived_from",
@ -38,7 +38,7 @@
{
"id": "link-005",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/discount-expiration.md",
"artifactKey": "wiki/global/discount-expiration.md",
"sourceKind": "warehouse",
"sourcePath": "arr_movements",
"relationship": "describes",
@ -47,7 +47,7 @@
{
"id": "link-006",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/nrr-retention.md",
"artifactKey": "wiki/global/nrr-retention.md",
"sourceKind": "warehouse",
"sourcePath": "arr_movements",
"relationship": "describes",
@ -56,7 +56,7 @@
{
"id": "link-007",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/nrr-retention.md",
"artifactKey": "wiki/global/nrr-retention.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/retention-and-nrr-definition-notes.md",
"relationship": "derived_from",
@ -65,7 +65,7 @@
{
"id": "link-008",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/nrr-retention.md",
"artifactKey": "wiki/global/nrr-retention.md",
"sourceKind": "bi",
"sourcePath": "raw-sources/bi/account_retention.view.lkml",
"relationship": "derived_from",
@ -74,7 +74,7 @@
{
"id": "link-009",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/segment-classification.md",
"artifactKey": "wiki/global/segment-classification.md",
"sourceKind": "warehouse",
"sourcePath": "plans",
"relationship": "describes",
@ -83,7 +83,7 @@
{
"id": "link-010",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/segment-classification.md",
"artifactKey": "wiki/global/segment-classification.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/sales-ops-segmentation-guide.md",
"relationship": "derived_from",
@ -92,7 +92,7 @@
{
"id": "link-011",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/activation-policy.md",
"artifactKey": "wiki/global/activation-policy.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/activation-policy-decision-record.md",
"relationship": "derived_from",
@ -101,7 +101,7 @@
{
"id": "link-012",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/procurement-workflows.md",
"artifactKey": "wiki/global/procurement-workflows.md",
"sourceKind": "warehouse",
"sourcePath": "purchase_requests",
"relationship": "describes",
@ -110,7 +110,7 @@
{
"id": "link-013",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/customer-health-scoring.md",
"artifactKey": "wiki/global/customer-health-scoring.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/customer-health-playbook.md",
"relationship": "derived_from",
@ -119,7 +119,7 @@
{
"id": "link-014",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/customer-health-scoring.md",
"artifactKey": "wiki/global/customer-health-scoring.md",
"sourceKind": "warehouse",
"sourcePath": "support_tickets",
"relationship": "describes",
@ -128,7 +128,7 @@
{
"id": "link-015",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/support-escalation.md",
"artifactKey": "wiki/global/support-escalation.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/support-escalation-runbook.md",
"relationship": "derived_from",
@ -137,7 +137,7 @@
{
"id": "link-016",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/internal-test-exclusion.md",
"artifactKey": "wiki/global/internal-test-exclusion.md",
"sourceKind": "notion",
"sourcePath": "raw-sources/notion/analyst-onboarding.md",
"relationship": "derived_from",

2
packages/cli/assets/demo/orbit/manifest.json
View file

@ -47,7 +47,7 @@
"sourceCount": 46
},
"knowledge": {
"path": "knowledge/global",
"path": "wiki/global",
"pageCount": 28
},
"links": {

82
packages/cli/assets/demo/orbit/replay.memory-flow.v1.json
View file

@ -71,7 +71,7 @@
"type": "work_unit_started",
"unitKey": "revenue-and-contracts",
"skills": [
"knowledge_capture",
"wiki_capture",
"sl_capture"
],
"stepBudget": 40
@ -81,21 +81,21 @@
"unitKey": "revenue-and-contracts",
"target": "wiki",
"action": "created",
"key": "knowledge/global/arr-contract-first.md"
"key": "wiki/global/arr-contract-first.md"
},
{
"type": "candidate_action",
"unitKey": "revenue-and-contracts",
"target": "wiki",
"action": "created",
"key": "knowledge/global/revenue-gross-to-net.md"
"key": "wiki/global/revenue-gross-to-net.md"
},
{
"type": "candidate_action",
"unitKey": "revenue-and-contracts",
"target": "wiki",
"action": "created",
"key": "knowledge/global/discount-expiration.md"
"key": "wiki/global/discount-expiration.md"
},
{
"type": "candidate_action",
@ -127,7 +127,7 @@
"type": "work_unit_started",
"unitKey": "retention-and-segments",
"skills": [
"knowledge_capture",
"wiki_capture",
"sl_capture"
],
"stepBudget": 40
@ -137,14 +137,14 @@
"unitKey": "retention-and-segments",
"target": "wiki",
"action": "created",
"key": "knowledge/global/nrr-retention.md"
"key": "wiki/global/nrr-retention.md"
},
{
"type": "candidate_action",
"unitKey": "retention-and-segments",
"target": "wiki",
"action": "created",
"key": "knowledge/global/segment-classification.md"
"key": "wiki/global/segment-classification.md"
},
{
"type": "candidate_action",
@ -162,7 +162,7 @@
"type": "work_unit_started",
"unitKey": "procurement-and-activation",
"skills": [
"knowledge_capture",
"wiki_capture",
"sl_capture"
],
"stepBudget": 40
@ -172,14 +172,14 @@
"unitKey": "procurement-and-activation",
"target": "wiki",
"action": "created",
"key": "knowledge/global/activation-policy.md"
"key": "wiki/global/activation-policy.md"
},
{
"type": "candidate_action",
"unitKey": "procurement-and-activation",
"target": "wiki",
"action": "created",
"key": "knowledge/global/procurement-workflows.md"
"key": "wiki/global/procurement-workflows.md"
},
{
"type": "candidate_action",
@ -197,7 +197,7 @@
"type": "work_unit_started",
"unitKey": "support-and-health",
"skills": [
"knowledge_capture",
"wiki_capture",
"sl_capture"
],
"stepBudget": 40
@ -207,14 +207,14 @@
"unitKey": "support-and-health",
"target": "wiki",
"action": "created",
"key": "knowledge/global/customer-health-scoring.md"
"key": "wiki/global/customer-health-scoring.md"
},
{
"type": "candidate_action",
"unitKey": "support-and-health",
"target": "wiki",
"action": "created",
"key": "knowledge/global/support-escalation.md"
"key": "wiki/global/support-escalation.md"
},
{
"type": "candidate_action",
@ -232,7 +232,7 @@
"type": "work_unit_started",
"unitKey": "governance-and-exclusions",
"skills": [
"knowledge_capture"
"wiki_capture"
],
"stepBudget": 40
},
@ -241,7 +241,7 @@
"unitKey": "governance-and-exclusions",
"target": "wiki",
"action": "created",
"key": "knowledge/global/internal-test-exclusion.md"
"key": "wiki/global/internal-test-exclusion.md"
},
{
"type": "work_unit_finished",
@ -321,7 +321,7 @@
"unitKey": "revenue-and-contracts",
"target": "wiki",
"action": "created",
"key": "knowledge/global/arr-contract-first.md",
"key": "wiki/global/arr-contract-first.md",
"summary": "ARR follows contract precedence with cancellation and discount caveats.",
"rawFiles": [
"contracts",
@ -334,7 +334,7 @@
"unitKey": "revenue-and-contracts",
"target": "wiki",
"action": "created",
"key": "knowledge/global/revenue-gross-to-net.md",
"key": "wiki/global/revenue-gross-to-net.md",
"summary": "Invoice, refund, and revenue dashboard evidence reconcile gross to net revenue.",
"rawFiles": [
"invoices",
@ -346,7 +346,7 @@
"unitKey": "revenue-and-contracts",
"target": "wiki",
"action": "created",
"key": "knowledge/global/discount-expiration.md",
"key": "wiki/global/discount-expiration.md",
"summary": "Discount expiration is separated from organic contraction for retention reporting.",
"rawFiles": [
"contracts",
@ -394,7 +394,7 @@
"unitKey": "retention-and-segments",
"target": "wiki",
"action": "created",
"key": "knowledge/global/nrr-retention.md",
"key": "wiki/global/nrr-retention.md",
"summary": "NRR uses parent-account rollups and quarterly ARR movement windows.",
"rawFiles": [
"accounts",
@ -407,7 +407,7 @@
"unitKey": "retention-and-segments",
"target": "wiki",
"action": "created",
"key": "knowledge/global/segment-classification.md",
"key": "wiki/global/segment-classification.md",
"summary": "Segment labels come from plan mapping and sales-ops policy notes.",
"rawFiles": [
"accounts",
@ -432,7 +432,7 @@
"unitKey": "procurement-and-activation",
"target": "wiki",
"action": "created",
"key": "knowledge/global/activation-policy.md",
"key": "wiki/global/activation-policy.md",
"summary": "Activation policy changed on January 15, 2026 and is encoded for agents.",
"rawFiles": [
"purchase_requests",
@ -445,7 +445,7 @@
"unitKey": "procurement-and-activation",
"target": "wiki",
"action": "created",
"key": "knowledge/global/procurement-workflows.md",
"key": "wiki/global/procurement-workflows.md",
"summary": "Procurement requester activity and approval events explain product usage.",
"rawFiles": [
"purchase_requests",
@ -468,7 +468,7 @@
"unitKey": "support-and-health",
"target": "wiki",
"action": "created",
"key": "knowledge/global/customer-health-scoring.md",
"key": "wiki/global/customer-health-scoring.md",
"summary": "Customer health combines support severity, ARR exposure, and product usage.",
"rawFiles": [
"support_tickets",
@ -480,7 +480,7 @@
"unitKey": "support-and-health",
"target": "wiki",
"action": "created",
"key": "knowledge/global/support-escalation.md",
"key": "wiki/global/support-escalation.md",
"summary": "Escalation tiers map ticket severity to SLA expectations.",
"rawFiles": [
"support_tickets",
@ -503,7 +503,7 @@
"unitKey": "governance-and-exclusions",
"target": "wiki",
"action": "created",
"key": "knowledge/global/internal-test-exclusion.md",
"key": "wiki/global/internal-test-exclusion.md",
"summary": "Canonical metrics exclude internal and test accounts across source families.",
"rawFiles": [
"raw-sources/notion/analyst-onboarding.md"
@ -515,97 +515,97 @@
{
"rawPath": "contracts",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/arr-contract-first.md",
"artifactKey": "wiki/global/arr-contract-first.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/arr-and-contract-reporting-notes.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/arr-contract-first.md",
"artifactKey": "wiki/global/arr-contract-first.md",
"actionType": "wiki_written"
},
{
"rawPath": "invoices",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/revenue-gross-to-net.md",
"artifactKey": "wiki/global/revenue-gross-to-net.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/revenue-reporting-policy.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/revenue-gross-to-net.md",
"artifactKey": "wiki/global/revenue-gross-to-net.md",
"actionType": "wiki_written"
},
{
"rawPath": "arr_movements",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/discount-expiration.md",
"artifactKey": "wiki/global/discount-expiration.md",
"actionType": "wiki_written"
},
{
"rawPath": "arr_movements",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/nrr-retention.md",
"artifactKey": "wiki/global/nrr-retention.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/retention-and-nrr-definition-notes.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/nrr-retention.md",
"artifactKey": "wiki/global/nrr-retention.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/bi/account_retention.view.lkml",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/nrr-retention.md",
"artifactKey": "wiki/global/nrr-retention.md",
"actionType": "wiki_written"
},
{
"rawPath": "plans",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/segment-classification.md",
"artifactKey": "wiki/global/segment-classification.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/sales-ops-segmentation-guide.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/segment-classification.md",
"artifactKey": "wiki/global/segment-classification.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/activation-policy-decision-record.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/activation-policy.md",
"artifactKey": "wiki/global/activation-policy.md",
"actionType": "wiki_written"
},
{
"rawPath": "purchase_requests",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/procurement-workflows.md",
"artifactKey": "wiki/global/procurement-workflows.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/customer-health-playbook.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/customer-health-scoring.md",
"artifactKey": "wiki/global/customer-health-scoring.md",
"actionType": "wiki_written"
},
{
"rawPath": "support_tickets",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/customer-health-scoring.md",
"artifactKey": "wiki/global/customer-health-scoring.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/support-escalation-runbook.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/support-escalation.md",
"artifactKey": "wiki/global/support-escalation.md",
"actionType": "wiki_written"
},
{
"rawPath": "raw-sources/notion/analyst-onboarding.md",
"artifactKind": "wiki",
"artifactKey": "knowledge/global/internal-test-exclusion.md",
"artifactKey": "wiki/global/internal-test-exclusion.md",
"actionType": "wiki_written"
},
{

0
packages/cli/assets/demo/orbit/knowledge/global/.gitkeep → packages/cli/assets/demo/orbit/wiki/global/.gitkeep
View file

0
packages/cli/assets/demo/orbit/knowledge/global/customer-communication-policy.md → packages/cli/assets/demo/orbit/wiki/global/customer-communication-policy.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/new-hire-onboarding-policy.md → packages/cli/assets/demo/orbit/wiki/global/new-hire-onboarding-policy.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-activation-kpi-glossary.md → packages/cli/assets/demo/orbit/wiki/global/orbit-activation-kpi-glossary.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-activation-policy-change-jan-2026.md → packages/cli/assets/demo/orbit/wiki/global/orbit-activation-policy-change-jan-2026.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-arr-contract-first-definition.md → packages/cli/assets/demo/orbit/wiki/global/orbit-arr-contract-first-definition.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-company-overview.md → packages/cli/assets/demo/orbit/wiki/global/orbit-company-overview.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-customer-health-risk-definition.md → packages/cli/assets/demo/orbit/wiki/global/orbit-customer-health-risk-definition.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-customer-stakeholder-needs.md → packages/cli/assets/demo/orbit/wiki/global/orbit-customer-stakeholder-needs.md
View file

2
packages/cli/assets/demo/orbit/knowledge/global/orbit-customers-source.md → packages/cli/assets/demo/orbit/wiki/global/orbit-customers-source.md
View file

@ -57,4 +57,4 @@ Always join through `customer.id`. Do not join on `email`.
- **Join key:** Always use `customer.id`, never `email`.
- **Timezone:** `created_at` and `last_seen_at` are UTC. Confirm whether a question expects UTC or a local business day before filtering.
- **Paying vs. all:** `free` customers must be excluded from paying-customer follow-ups. Use `paying_customer_count`, not `customer_count`.
- **plan_tier values:** `free`, `pro`, `enterprise`. Note: `pro_plus` is a legacy alias for `growth` in the account/contract layer (see `orbit-plan-segment-normalization`), but `plan_tier` on this table uses `pro` not `pro_plus`.
- **plan_tier values:** `free`, `pro`, `enterprise`. Note: use the canonical plan names from the account/contract layer (see `orbit-plan-segment-normalization`); `plan_tier` on this table uses `pro` rather than `growth`.

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-dbt-exposures.md → packages/cli/assets/demo/orbit/wiki/global/orbit-dbt-exposures.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-dbt-project-overview.md → packages/cli/assets/demo/orbit/wiki/global/orbit-dbt-project-overview.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-how-we-work.md → packages/cli/assets/demo/orbit/wiki/global/orbit-how-we-work.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-known-product-gaps.md → packages/cli/assets/demo/orbit/wiki/global/orbit-known-product-gaps.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-mart-account-activity.md → packages/cli/assets/demo/orbit/wiki/global/orbit-mart-account-activity.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-mart-account-segments.md → packages/cli/assets/demo/orbit/wiki/global/orbit-mart-account-segments.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-mart-arr-daily.md → packages/cli/assets/demo/orbit/wiki/global/orbit-mart-arr-daily.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-mart-nrr-quarterly.md → packages/cli/assets/demo/orbit/wiki/global/orbit-mart-nrr-quarterly.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-mart-procurement-activity.md → packages/cli/assets/demo/orbit/wiki/global/orbit-mart-procurement-activity.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-mart-retention-movement-breakout.md → packages/cli/assets/demo/orbit/wiki/global/orbit-mart-retention-movement-breakout.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-mart-revenue-daily.md → packages/cli/assets/demo/orbit/wiki/global/orbit-mart-revenue-daily.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-metabase-sql-library-patterns.md → packages/cli/assets/demo/orbit/wiki/global/orbit-metabase-sql-library-patterns.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-nrr-discount-expiration-treatment.md → packages/cli/assets/demo/orbit/wiki/global/orbit-nrr-discount-expiration-treatment.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-plan-segment-normalization.md → packages/cli/assets/demo/orbit/wiki/global/orbit-plan-segment-normalization.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-procurement-qualifying-actions.md → packages/cli/assets/demo/orbit/wiki/global/orbit-procurement-qualifying-actions.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-product-design-principles.md → packages/cli/assets/demo/orbit/wiki/global/orbit-product-design-principles.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-product-review-checklist.md → packages/cli/assets/demo/orbit/wiki/global/orbit-product-review-checklist.md
View file

0
packages/cli/assets/demo/orbit/knowledge/global/orbit-revenue-gross-to-net-reconciliation.md → packages/cli/assets/demo/orbit/wiki/global/orbit-revenue-gross-to-net-reconciliation.md
View file

2
packages/cli/assets/demo/orbit/knowledge/global/sales-ops-cs-handoff-process.md → packages/cli/assets/demo/orbit/wiki/global/sales-ops-cs-handoff-process.md
View file

@ -27,7 +27,7 @@ Sales Ops must complete the handoff **before the first implementation call**. Cu
| Field | Notes |
|---|---|
| Current plan | Starter / Growth / Enterprise — use canonical plan name, not legacy aliases |
| Current plan | Starter / Growth / Enterprise — use canonical plan name |
| Account segment | self_serve / commercial / enterprise (see `orbit-plan-segment-normalization`) |
| Contract shape | Term, ARR, any discounts or custom terms |
| Renewal contact | Named person on the customer side responsible for renewal |

88
packages/cli/scripts/build-demo-assets.mjs
View file

@ -229,39 +229,39 @@ const knowledgePages = [
];
const provenanceLinks = [
['wiki', 'knowledge/global/arr-contract-first.md', 'warehouse', 'contracts', 'describes', 1],
['wiki', 'wiki/global/arr-contract-first.md', 'warehouse', 'contracts', 'describes', 1],
[
'wiki',
'knowledge/global/arr-contract-first.md',
'wiki/global/arr-contract-first.md',
'notion',
'raw-sources/notion/arr-and-contract-reporting-notes.md',
'derived_from',
0.95,
],
['wiki', 'knowledge/global/revenue-gross-to-net.md', 'warehouse', 'invoices', 'describes', 1],
['wiki', 'wiki/global/revenue-gross-to-net.md', 'warehouse', 'invoices', 'describes', 1],
[
'wiki',
'knowledge/global/revenue-gross-to-net.md',
'wiki/global/revenue-gross-to-net.md',
'notion',
'raw-sources/notion/revenue-reporting-policy.md',
'derived_from',
0.95,
],
['wiki', 'knowledge/global/discount-expiration.md', 'warehouse', 'arr_movements', 'describes', 1],
['wiki', 'knowledge/global/nrr-retention.md', 'warehouse', 'arr_movements', 'describes', 1],
['wiki', 'wiki/global/discount-expiration.md', 'warehouse', 'arr_movements', 'describes', 1],
['wiki', 'wiki/global/nrr-retention.md', 'warehouse', 'arr_movements', 'describes', 1],
[
'wiki',
'knowledge/global/nrr-retention.md',
'wiki/global/nrr-retention.md',
'notion',
'raw-sources/notion/retention-and-nrr-definition-notes.md',
'derived_from',
0.95,
],
['wiki', 'knowledge/global/nrr-retention.md', 'bi', 'raw-sources/bi/account_retention.view.lkml', 'derived_from', 0.85],
['wiki', 'knowledge/global/segment-classification.md', 'warehouse', 'plans', 'describes', 1],
['wiki', 'wiki/global/nrr-retention.md', 'bi', 'raw-sources/bi/account_retention.view.lkml', 'derived_from', 0.85],
['wiki', 'wiki/global/segment-classification.md', 'warehouse', 'plans', 'describes', 1],
[
'wiki',
'knowledge/global/segment-classification.md',
'wiki/global/segment-classification.md',
'notion',
'raw-sources/notion/sales-ops-segmentation-guide.md',
'derived_from',
@ -269,25 +269,25 @@ const provenanceLinks = [
],
[
'wiki',
'knowledge/global/activation-policy.md',
'wiki/global/activation-policy.md',
'notion',
'raw-sources/notion/activation-policy-decision-record.md',
'derived_from',
0.95,
],
['wiki', 'knowledge/global/procurement-workflows.md', 'warehouse', 'purchase_requests', 'describes', 1],
['wiki', 'wiki/global/procurement-workflows.md', 'warehouse', 'purchase_requests', 'describes', 1],
[
'wiki',
'knowledge/global/customer-health-scoring.md',
'wiki/global/customer-health-scoring.md',
'notion',
'raw-sources/notion/customer-health-playbook.md',
'derived_from',
0.9,
],
['wiki', 'knowledge/global/customer-health-scoring.md', 'warehouse', 'support_tickets', 'describes', 1],
['wiki', 'wiki/global/customer-health-scoring.md', 'warehouse', 'support_tickets', 'describes', 1],
[
'wiki',
'knowledge/global/support-escalation.md',
'wiki/global/support-escalation.md',
'notion',
'raw-sources/notion/support-escalation-runbook.md',
'derived_from',
@ -295,7 +295,7 @@ const provenanceLinks = [
],
[
'wiki',
'knowledge/global/internal-test-exclusion.md',
'wiki/global/internal-test-exclusion.md',
'notion',
'raw-sources/notion/analyst-onboarding.md',
'derived_from',
@ -490,7 +490,7 @@ function buildActions() {
unitKey: 'revenue-and-contracts',
target: 'wiki',
action: 'created',
key: 'knowledge/global/arr-contract-first.md',
key: 'wiki/global/arr-contract-first.md',
summary: 'ARR follows contract precedence with cancellation and discount caveats.',
rawFiles: ['contracts', 'arr_movements', 'raw-sources/notion/arr-and-contract-reporting-notes.md'],
status: 'success',
@ -499,7 +499,7 @@ function buildActions() {
unitKey: 'revenue-and-contracts',
target: 'wiki',
action: 'created',
key: 'knowledge/global/revenue-gross-to-net.md',
key: 'wiki/global/revenue-gross-to-net.md',
summary: 'Invoice, refund, and revenue dashboard evidence reconcile gross to net revenue.',
rawFiles: ['invoices', 'raw-sources/bi/revenue_exec.dashboard.lookml'],
status: 'success',
@ -508,7 +508,7 @@ function buildActions() {
unitKey: 'revenue-and-contracts',
target: 'wiki',
action: 'created',
key: 'knowledge/global/discount-expiration.md',
key: 'wiki/global/discount-expiration.md',
summary: 'Discount expiration is separated from organic contraction for retention reporting.',
rawFiles: ['contracts', 'arr_movements'],
status: 'success',
@ -544,7 +544,7 @@ function buildActions() {
unitKey: 'retention-and-segments',
target: 'wiki',
action: 'created',
key: 'knowledge/global/nrr-retention.md',
key: 'wiki/global/nrr-retention.md',
summary: 'NRR uses parent-account rollups and quarterly ARR movement windows.',
rawFiles: ['accounts', 'arr_movements', 'raw-sources/notion/retention-and-nrr-definition-notes.md'],
status: 'success',
@ -553,7 +553,7 @@ function buildActions() {
unitKey: 'retention-and-segments',
target: 'wiki',
action: 'created',
key: 'knowledge/global/segment-classification.md',
key: 'wiki/global/segment-classification.md',
summary: 'Segment labels come from plan mapping and sales-ops policy notes.',
rawFiles: ['accounts', 'plans', 'raw-sources/notion/sales-ops-segmentation-guide.md'],
status: 'success',
@ -571,7 +571,7 @@ function buildActions() {
unitKey: 'procurement-and-activation',
target: 'wiki',
action: 'created',
key: 'knowledge/global/activation-policy.md',
key: 'wiki/global/activation-policy.md',
summary: 'Activation policy changed on January 15, 2026 and is encoded for agents.',
rawFiles: ['purchase_requests', 'users', 'raw-sources/notion/activation-policy-decision-record.md'],
status: 'success',
@ -580,7 +580,7 @@ function buildActions() {
unitKey: 'procurement-and-activation',
target: 'wiki',
action: 'created',
key: 'knowledge/global/procurement-workflows.md',
key: 'wiki/global/procurement-workflows.md',
summary: 'Procurement requester activity and approval events explain product usage.',
rawFiles: ['purchase_requests', 'raw-sources/bi/procurement_activity.view.lkml'],
status: 'success',
@ -598,7 +598,7 @@ function buildActions() {
unitKey: 'support-and-health',
target: 'wiki',
action: 'created',
key: 'knowledge/global/customer-health-scoring.md',
key: 'wiki/global/customer-health-scoring.md',
summary: 'Customer health combines support severity, ARR exposure, and product usage.',
rawFiles: ['support_tickets', 'raw-sources/notion/customer-health-playbook.md'],
status: 'success',
@ -607,7 +607,7 @@ function buildActions() {
unitKey: 'support-and-health',
target: 'wiki',
action: 'created',
key: 'knowledge/global/support-escalation.md',
key: 'wiki/global/support-escalation.md',
summary: 'Escalation tiers map ticket severity to SLA expectations.',
rawFiles: ['support_tickets', 'raw-sources/notion/support-escalation-runbook.md'],
status: 'success',
@ -625,7 +625,7 @@ function buildActions() {
unitKey: 'governance-and-exclusions',
target: 'wiki',
action: 'created',
key: 'knowledge/global/internal-test-exclusion.md',
key: 'wiki/global/internal-test-exclusion.md',
summary: 'Canonical metrics exclude internal and test accounts across source families.',
rawFiles: ['raw-sources/notion/analyst-onboarding.md'],
status: 'success',
@ -665,27 +665,27 @@ function buildReplay(provenance, transcripts) {
{ type: 'raw_snapshot_written', syncId: 'demo-seeded-sync', rawFileCount: 29 },
{ type: 'diff_computed', added: 29, modified: 0, deleted: 0, unchanged: 0 },
{ type: 'chunks_planned', chunkCount: 5, workUnitCount: 5, evictionCount: 0 },
{ type: 'work_unit_started', unitKey: 'revenue-and-contracts', skills: ['knowledge_capture', 'sl_capture'], stepBudget: 40 },
{ type: 'work_unit_started', unitKey: 'revenue-and-contracts', skills: ['wiki_capture', 'sl_capture'], stepBudget: 40 },
{
type: 'candidate_action',
unitKey: 'revenue-and-contracts',
target: 'wiki',
action: 'created',
key: 'knowledge/global/arr-contract-first.md',
key: 'wiki/global/arr-contract-first.md',
},
{
type: 'candidate_action',
unitKey: 'revenue-and-contracts',
target: 'wiki',
action: 'created',
key: 'knowledge/global/revenue-gross-to-net.md',
key: 'wiki/global/revenue-gross-to-net.md',
},
{
type: 'candidate_action',
unitKey: 'revenue-and-contracts',
target: 'wiki',
action: 'created',
key: 'knowledge/global/discount-expiration.md',
key: 'wiki/global/discount-expiration.md',
},
{
type: 'candidate_action',
@ -709,20 +709,20 @@ function buildReplay(provenance, transcripts) {
key: 'orbit_demo.arr_movements',
},
{ type: 'work_unit_finished', unitKey: 'revenue-and-contracts', status: 'success' },
{ type: 'work_unit_started', unitKey: 'retention-and-segments', skills: ['knowledge_capture', 'sl_capture'], stepBudget: 40 },
{ type: 'work_unit_started', unitKey: 'retention-and-segments', skills: ['wiki_capture', 'sl_capture'], stepBudget: 40 },
{
type: 'candidate_action',
unitKey: 'retention-and-segments',
target: 'wiki',
action: 'created',
key: 'knowledge/global/nrr-retention.md',
key: 'wiki/global/nrr-retention.md',
},
{
type: 'candidate_action',
unitKey: 'retention-and-segments',
target: 'wiki',
action: 'created',
key: 'knowledge/global/segment-classification.md',
key: 'wiki/global/segment-classification.md',
},
{
type: 'candidate_action',
@ -735,7 +735,7 @@ function buildReplay(provenance, transcripts) {
{
type: 'work_unit_started',
unitKey: 'procurement-and-activation',
skills: ['knowledge_capture', 'sl_capture'],
skills: ['wiki_capture', 'sl_capture'],
stepBudget: 40,
},
{
@ -743,14 +743,14 @@ function buildReplay(provenance, transcripts) {
unitKey: 'procurement-and-activation',
target: 'wiki',
action: 'created',
key: 'knowledge/global/activation-policy.md',
key: 'wiki/global/activation-policy.md',
},
{
type: 'candidate_action',
unitKey: 'procurement-and-activation',
target: 'wiki',
action: 'created',
key: 'knowledge/global/procurement-workflows.md',
key: 'wiki/global/procurement-workflows.md',
},
{
type: 'candidate_action',
@ -760,20 +760,20 @@ function buildReplay(provenance, transcripts) {
key: 'orbit_demo.purchase_requests',
},
{ type: 'work_unit_finished', unitKey: 'procurement-and-activation', status: 'success' },
{ type: 'work_unit_started', unitKey: 'support-and-health', skills: ['knowledge_capture', 'sl_capture'], stepBudget: 40 },
{ type: 'work_unit_started', unitKey: 'support-and-health', skills: ['wiki_capture', 'sl_capture'], stepBudget: 40 },
{
type: 'candidate_action',
unitKey: 'support-and-health',
target: 'wiki',
action: 'created',
key: 'knowledge/global/customer-health-scoring.md',
key: 'wiki/global/customer-health-scoring.md',
},
{
type: 'candidate_action',
unitKey: 'support-and-health',
target: 'wiki',
action: 'created',
key: 'knowledge/global/support-escalation.md',
key: 'wiki/global/support-escalation.md',
},
{
type: 'candidate_action',
@ -783,13 +783,13 @@ function buildReplay(provenance, transcripts) {
key: 'orbit_demo.support_tickets',
},
{ type: 'work_unit_finished', unitKey: 'support-and-health', status: 'success' },
{ type: 'work_unit_started', unitKey: 'governance-and-exclusions', skills: ['knowledge_capture'], stepBudget: 40 },
{ type: 'work_unit_started', unitKey: 'governance-and-exclusions', skills: ['wiki_capture'], stepBudget: 40 },
{
type: 'candidate_action',
unitKey: 'governance-and-exclusions',
target: 'wiki',
action: 'created',
key: 'knowledge/global/internal-test-exclusion.md',
key: 'wiki/global/internal-test-exclusion.md',
},
{ type: 'work_unit_finished', unitKey: 'governance-and-exclusions', status: 'success' },
{ type: 'reconciliation_finished', conflictCount: 0, fallbackCount: 0 },
@ -835,7 +835,7 @@ function buildReplay(provenance, transcripts) {
async function writeGeneratedContext(rowCounts) {
for (const page of knowledgePages) {
await writeText(join('knowledge/global', page.file), renderKnowledgePage(page));
await writeText(join('wiki/global', page.file), renderKnowledgePage(page));
}
for (const table of semanticLayerTables) {
@ -908,7 +908,7 @@ async function writeGeneratedContext(rowCounts) {
},
generated: {
semanticLayer: { path: 'semantic-layer/orbit_demo', sourceCount: 6 },
knowledge: { path: 'knowledge/global', pageCount: 10 },
knowledge: { path: 'wiki/global', pageCount: 10 },
links: { path: 'links', linkCount: provenanceLinks.length },
},
});
@ -930,7 +930,7 @@ for (const relativeDir of [
'raw-sources/bi',
'raw-sources/notion',
'semantic-layer/orbit_demo',
'knowledge/global',
'wiki/global',
'links',
'reports',
]) {

2
packages/cli/src/cli-program.ts
View file

@ -216,7 +216,7 @@ export function resolveCommandProjectDirOverride(command: CommandWithGlobalOptio
function createBaseProgram(info: KtxCliPackageInfo, io: KtxCliIo): Command {
return new Command()
.name('ktx')
.description('Standalone KTX developer CLI')
.description('KTX data agent context layer CLI')
.option('--project-dir <path>', 'KTX project directory (default: KTX_PROJECT_DIR, nearest ktx.yaml, or cwd)')
.option('--debug', 'Enable diagnostic logging to stderr')
.version(`${info.name} ${info.version}`, '-v, --version', 'Show CLI version')

13
packages/cli/src/command-schemas.ts
View file

@ -3,6 +3,19 @@ import { z } from 'zod';
const projectDirSchema = z.string().min(1);
const stringArraySchema = z.array(z.string());
export const wikiWriteCommandSchema = z.object({
command: z.literal('write'),
projectDir: projectDirSchema,
key: z.string().min(1),
scope: z.enum(['GLOBAL', 'USER']),
userId: z.string().min(1),
summary: z.string().min(1),
content: z.string().min(1),
tags: stringArraySchema,
refs: stringArraySchema,
slRefs: stringArraySchema,
});
const orderBySchema = z.union([
z.string().min(1),
z.object({

49
packages/cli/src/commands/knowledge-commands.ts
View file

@ -1,9 +1,11 @@
import { type Command } from '@commander-js/extra-typings';
import { type Command, Option } from '@commander-js/extra-typings';
import {
collectOption,
type KtxCliCommandContext,
parsePositiveIntegerOption,
resolveCommandProjectDir,
} from '../cli-program.js';
import { wikiWriteCommandSchema } from '../command-schemas.js';
import type { KtxKnowledgeArgs } from '../knowledge.js';
import { profileMark } from '../startup-profile.js';
@ -17,7 +19,7 @@ async function runKnowledgeArgs(context: KtxCliCommandContext, args: KtxKnowledg
export function registerWikiCommands(program: Command, context: KtxCliCommandContext): void {
const wiki = program
.command('wiki')
.description('List or search local wiki pages')
.description('List, read, search, or write local wiki pages')
.showHelpAfterError()
.addHelpText(
'after',
@ -38,6 +40,22 @@ export function registerWikiCommands(program: Command, context: KtxCliCommandCon
});
});
wiki
.command('read')
.description('Read one local wiki page')
.argument('<key>', 'Wiki page key')
.option('--json', 'Print JSON output', false)
.option('--user-id <id>', 'Local user id', 'local')
.action(async (key: string, options: { userId: string; json?: boolean }, command) => {
await runKnowledgeArgs(context, {
command: 'read',
projectDir: resolveCommandProjectDir(command),
key,
userId: options.userId,
json: options.json,
});
});
wiki
.command('search')
.description('Search local wiki pages')
@ -55,4 +73,31 @@ export function registerWikiCommands(program: Command, context: KtxCliCommandCon
...(options.limit !== undefined ? { limit: options.limit } : {}),
});
});
wiki
.command('write')
.description('Write one local wiki page')
.argument('<key>', 'Wiki page key')
.option('--user-id <id>', 'Local user id', 'local')
.addOption(new Option('--scope <scope>', 'global or user').choices(['global', 'user']).default('global'))
.requiredOption('--summary <summary>', 'Wiki summary')
.requiredOption('--content <content>', 'Wiki content')
.option('--tag <tag>', 'Wiki tag; repeatable', collectOption, [])
.option('--ref <ref>', 'Wiki ref; repeatable', collectOption, [])
.option('--sl-ref <ref>', 'Semantic-layer ref; repeatable', collectOption, [])
.action(async (key: string, options, command) => {
const args = wikiWriteCommandSchema.parse({
command: 'write',
projectDir: resolveCommandProjectDir(command),
key,
scope: options.scope === 'user' ? 'USER' : 'GLOBAL',
userId: options.userId,
summary: options.summary,
content: options.content,
tags: options.tag,
refs: options.ref,
slRefs: options.slRef,
});
await runKnowledgeArgs(context, args);
});
}

12
packages/cli/src/commands/setup-commands.ts
View file

@ -121,7 +121,6 @@ function shouldShowSetupEntryMenu(
disableHistoricSql?: boolean;
historicSqlWindowDays?: number;
historicSqlMinExecutions?: number;
historicSqlMinCalls?: number;
historicSqlServiceAccountPattern?: string[];
historicSqlRedactionPattern?: string[];
skipDatabases?: boolean;
@ -194,7 +193,6 @@ function shouldShowSetupEntryMenu(
'disableHistoricSql',
'historicSqlWindowDays',
'historicSqlMinExecutions',
'historicSqlMinCalls',
'skipDatabases',
'source',
'sourceConnectionId',
@ -283,11 +281,6 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
.option('--disable-historic-sql', 'Disable Historic SQL for the selected database', false)
.option('--historic-sql-window-days <number>', 'Historic SQL query-history window', positiveInteger)
.option('--historic-sql-min-executions <number>', 'Minimum Historic SQL executions for a template', positiveInteger)
.option(
'--historic-sql-min-calls <number>',
'Alias for --historic-sql-min-executions',
positiveInteger,
)
.option(
'--historic-sql-service-account-pattern <pattern>',
'Historic SQL service-account regex; repeatable',
@ -379,7 +372,6 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
const mode = options.new ? 'new' : options.existing ? 'existing' : 'auto';
const resolvedAgentScope = options.global ? 'global' : options.agentScope;
const historicSqlMinExecutions = options.historicSqlMinExecutions ?? options.historicSqlMinCalls;
await runSetupArgs(context, {
command: 'run',
projectDir: resolveCommandProjectDir(command),
@ -410,7 +402,9 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
...(options.enableHistoricSql ? { enableHistoricSql: true } : {}),
...(options.disableHistoricSql ? { disableHistoricSql: true } : {}),
...(options.historicSqlWindowDays !== undefined ? { historicSqlWindowDays: options.historicSqlWindowDays } : {}),
...(historicSqlMinExecutions !== undefined ? { historicSqlMinExecutions } : {}),
...(options.historicSqlMinExecutions !== undefined
? { historicSqlMinExecutions: options.historicSqlMinExecutions }
: {}),
...(options.historicSqlServiceAccountPattern.length > 0
? { historicSqlServiceAccountPatterns: options.historicSqlServiceAccountPattern }
: {}),

6
packages/cli/src/demo-assets.test.ts
View file

@ -95,7 +95,7 @@ describe('demo assets', () => {
await expect(access(packagedDemoAssetPath('semantic-layer/dbt-main/mart_arr_daily.yaml'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('semantic-layer/postgres-warehouse/mart_account_activity.yaml'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('knowledge/global/orbit-company-overview.md'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('wiki/global/orbit-company-overview.md'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('links/provenance.json'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('reports/seeded-demo-report.json'))).resolves.toBeUndefined();
});
@ -108,7 +108,7 @@ describe('demo assets', () => {
await expect(access(join(projectDir, 'state.sqlite'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'reports'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'semantic-layer'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'knowledge'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'wiki'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'replays', 'replay.memory-flow.v1.json'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'raw-sources'))).resolves.toBeUndefined();
await expect(access(join(projectDir, '_schema'))).rejects.toMatchObject({ code: 'ENOENT' });
@ -129,7 +129,7 @@ describe('demo assets', () => {
await ensureSeededDemoProject({ projectDir, force: false });
await expect(access(join(projectDir, 'semantic-layer', 'dbt-main', 'mart_arr_daily.yaml'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'knowledge', 'global', 'orbit-company-overview.md'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'wiki', 'global', 'orbit-company-overview.md'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'links', 'provenance.json'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'reports', 'seeded-demo-report.json'))).resolves.toBeUndefined();
});

6
packages/cli/src/demo-assets.ts
View file

@ -29,7 +29,7 @@ const REQUIRED_SEEDED_ASSET_PATHS = [
DEMO_REPLAY_FILE,
join('semantic-layer', 'dbt-main', 'mart_arr_daily.yaml'),
join('semantic-layer', 'postgres-warehouse', 'mart_account_activity.yaml'),
join('knowledge', 'global', 'orbit-company-overview.md'),
join('wiki', 'global', 'orbit-company-overview.md'),
] as const;
function assetDir(): string {
@ -131,7 +131,7 @@ export async function ensureDemoProject(options: EnsureDemoProjectOptions): Prom
}
await mkdir(projectDir, { recursive: true });
for (const relativeDir of ['reports', 'semantic-layer', 'knowledge', 'replays', 'raw-sources', 'links']) {
for (const relativeDir of ['reports', 'semantic-layer', 'wiki', 'replays', 'raw-sources', 'links']) {
await mkdir(join(projectDir, relativeDir), { recursive: true });
}
@ -157,7 +157,7 @@ async function copySeededAssetDirectories(projectDir: string): Promise<void> {
await Promise.all([
copyDirIfExists(join(src, 'semantic-layer'), join(dest, 'semantic-layer')),
copyDirIfExists(join(src, 'knowledge'), join(dest, 'knowledge')),
copyDirIfExists(join(src, 'wiki'), join(dest, 'wiki')),
copyDirIfExists(join(src, 'raw-sources'), join(dest, 'raw-sources')),
copyDirIfExists(join(src, 'links'), join(dest, 'links')),
copyDirIfExists(join(src, 'reports'), join(dest, 'reports')),

72
packages/cli/src/index.test.ts
View file

@ -123,12 +123,12 @@ describe('runKtxCli', () => {
await expect(runKtxCli(['--help'], testIo.io)).resolves.toBe(0);
expect(testIo.stdout()).toContain('Usage: ktx [options] [command]');
expect(testIo.stdout()).toContain('KTX data agent context layer CLI');
for (const command of ['setup', 'connection', 'ingest', 'wiki', 'sl', 'status', 'scan']) {
expect(testIo.stdout()).toContain(`${command}`);
}
for (const removed of ['demo', 'init', 'connect', 'ask', 'knowledge', 'agent', 'completion', 'serve']) {
expect(testIo.stdout()).not.toContain(`${removed} [`);
expect(testIo.stdout()).not.toContain(`${removed} `);
expect(testIo.stdout()).not.toMatch(new RegExp(`^\\s+${removed}(?:\\s|\\[|$)`, 'm'));
}
expect(testIo.stdout()).toContain('--project-dir <path>');
expect(testIo.stdout()).toContain('KTX_PROJECT_DIR');
@ -139,22 +139,78 @@ describe('runKtxCli', () => {
expect(testIo.stderr()).toBe('');
});
it('rejects removed public wiki and sl read/write commands', async () => {
const sl = vi.fn(async () => 0);
it('routes public wiki read and write commands', async () => {
const knowledge = vi.fn(async () => 0);
const readIo = makeIo();
await expect(runKtxCli(['--project-dir', tempDir, 'wiki', 'read', 'revenue', '--json'], readIo.io, { knowledge }))
.resolves.toBe(0);
expect(knowledge).toHaveBeenCalledWith(
{
command: 'read',
projectDir: tempDir,
key: 'revenue',
userId: 'local',
json: true,
},
readIo.io,
);
const writeIo = makeIo();
await expect(
runKtxCli(
[
'--project-dir',
tempDir,
'wiki',
'write',
'revenue',
'--scope',
'user',
'--summary',
'Revenue',
'--content',
'Revenue.',
'--tag',
'finance',
'--ref',
'https://example.com/revenue',
'--sl-ref',
'orders',
],
writeIo.io,
{ knowledge },
),
).resolves.toBe(0);
expect(knowledge).toHaveBeenLastCalledWith(
{
command: 'write',
projectDir: tempDir,
key: 'revenue',
scope: 'USER',
userId: 'local',
summary: 'Revenue',
content: 'Revenue.',
tags: ['finance'],
refs: ['https://example.com/revenue'],
slRefs: ['orders'],
},
writeIo.io,
);
});
it('rejects removed public sl read/write commands', async () => {
const sl = vi.fn(async () => 0);
for (const argv of [
['--project-dir', tempDir, 'wiki', 'read', 'revenue'],
['--project-dir', tempDir, 'wiki', 'write', 'revenue', '--summary', 'Revenue', '--content', 'Revenue.'],
['--project-dir', tempDir, 'sl', 'read', 'orders', '--connection-id', 'warehouse'],
['--project-dir', tempDir, 'sl', 'write', 'orders', '--connection-id', 'warehouse', '--yaml', 'name: orders'],
]) {
const io = makeIo();
await expect(runKtxCli(argv, io.io, { knowledge, sl })).resolves.toBe(1);
await expect(runKtxCli(argv, io.io, { sl })).resolves.toBe(1);
expect(io.stderr()).toMatch(/unknown command|error:/);
}
expect(knowledge).not.toHaveBeenCalled();
expect(sl).not.toHaveBeenCalled();
});

18
packages/cli/src/ingest.test-utils.ts
View file

@ -159,7 +159,7 @@ export function bundleReportSnapshot(): IngestReportSnapshot {
rawFiles: ['cards/1.json', 'cards/2.json'],
status: 'success',
actions: [
{ target: 'wiki', type: 'created', key: 'knowledge/global/revenue.md', detail: 'Revenue overview' },
{ target: 'wiki', type: 'created', key: 'wiki/global/revenue.md', detail: 'Revenue overview' },
{ target: 'sl', type: 'updated', key: 'warehouse.orders', detail: 'Added order amount measure' },
],
touchedSlSources: [{ connectionId: 'warehouse', sourceName: 'warehouse.orders' }],
@ -178,7 +178,7 @@ export function bundleReportSnapshot(): IngestReportSnapshot {
{
rawPath: 'cards/1.json',
artifactKind: 'wiki',
artifactKey: 'knowledge/global/revenue.md',
artifactKey: 'wiki/global/revenue.md',
actionType: 'wiki_written',
},
{
@ -194,7 +194,7 @@ export function bundleReportSnapshot(): IngestReportSnapshot {
path: 'tool-transcripts/cards.jsonl',
toolCallCount: 4,
errorCount: 0,
toolNames: ['ingest_triage', 'knowledge_capture', 'sl_capture'],
toolNames: ['ingest_triage', 'wiki_capture', 'sl_capture'],
},
],
},
@ -376,7 +376,7 @@ const SYNC_MODE_METABASE_CARDS: MetabaseCard[] = [
collection_id: 12,
archived: false,
result_metadata: [],
dataset_query: { type: 'native', database: 1, native: { query: 'select 101 as id' } },
dataset_query: { type: 'native', database: 1, stages: [{ 'lib/type': 'mbql.stage/native', native: 'select 101 as id' }] },
parameters: [],
dashboard_count: 0,
},
@ -390,7 +390,7 @@ const SYNC_MODE_METABASE_CARDS: MetabaseCard[] = [
collection_id: 12,
archived: false,
result_metadata: [],
dataset_query: { type: 'native', database: 1, native: { query: 'select 102 as id' } },
dataset_query: { type: 'native', database: 1, stages: [{ 'lib/type': 'mbql.stage/native', native: 'select 102 as id' }] },
parameters: [],
dashboard_count: 0,
},
@ -404,7 +404,7 @@ const SYNC_MODE_METABASE_CARDS: MetabaseCard[] = [
collection_id: 13,
archived: false,
result_metadata: [],
dataset_query: { type: 'native', database: 1, native: { query: 'select 103 as id' } },
dataset_query: { type: 'native', database: 1, stages: [{ 'lib/type': 'mbql.stage/native', native: 'select 103 as id' }] },
parameters: [],
dashboard_count: 0,
},
@ -454,11 +454,11 @@ function createSyncModeMetabaseClient(): MetabaseRuntimeClient {
},
getAllCards: async () => SYNC_MODE_METABASE_CARDS.map(metabaseCardSummary),
convertMbqlToNative: async () => ({ query: 'select 1' }),
getNativeSql: (card) => card.dataset_query?.native?.query ?? null,
getNativeSql: (card) => card.dataset_query?.stages?.[0]?.native ?? null,
getTemplateTags: () => ({}),
getCardSql: async (card) => card.dataset_query?.native?.query ?? null,
getCardSql: async (card) => card.dataset_query?.stages?.[0]?.native ?? null,
getResolvedSql: async (card) => ({
resolvedSql: card.dataset_query?.native?.query ?? `select ${card.id} as id`,
resolvedSql: card.dataset_query?.stages?.[0]?.native ?? `select ${card.id} as id`,
templateTags: [],
resolutionStatus: 'resolved',
}),

3
packages/cli/src/ingest.test.ts
View file

@ -846,7 +846,6 @@ describe('runKtxIngest', () => {
patternPagesWritten: 30,
stalePatternPagesMarked: 2,
archivedPatternPages: 3,
legacyPagesDeleted: 4,
},
errors: [],
warnings: [],
@ -880,7 +879,7 @@ describe('runKtxIngest', () => {
expect(io.stderr()).toBe('');
expect(io.stdout()).toContain('Adapter: historic-sql\n');
expect(io.stdout()).toContain('Saved memory: 39 wiki, 57 SL\n');
expect(io.stdout()).toContain('Saved memory: 35 wiki, 57 SL\n');
});
it('returns a non-zero code when local ingest reports failed work units', async () => {

162
packages/cli/src/knowledge.test.ts
View file

@ -3,7 +3,6 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { initKtxProject } from '@ktx/context/project';
import type { KtxEmbeddingPort } from '@ktx/context';
import { type LocalKnowledgeScope, writeLocalKnowledgePage } from '@ktx/context/wiki';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { runKtxKnowledge } from './knowledge.js';
@ -41,29 +40,6 @@ class FakeEmbeddingPort implements KtxEmbeddingPort {
}
}
async function seedKnowledgePage(input: {
projectDir: string;
key: string;
summary: string;
content: string;
scope?: LocalKnowledgeScope;
tags?: string[];
refs?: string[];
slRefs?: string[];
}): Promise<void> {
const project = await initKtxProject({ projectDir: input.projectDir, projectName: 'warehouse' });
await writeLocalKnowledgePage(project, {
key: input.key,
scope: input.scope ?? 'GLOBAL',
userId: 'local',
summary: input.summary,
content: input.content,
tags: input.tags ?? [],
refs: input.refs ?? [],
slRefs: input.slRefs ?? [],
});
}
describe('runKtxKnowledge', () => {
let tempDir: string;
@ -75,16 +51,36 @@ describe('runKtxKnowledge', () => {
await rm(tempDir, { recursive: true, force: true });
});
it('lists and searches knowledge pages', async () => {
it('writes, reads, lists, and searches wiki pages', async () => {
const projectDir = join(tempDir, 'project');
await seedKnowledgePage({
projectDir,
key: 'metrics-revenue',
summary: 'Revenue',
content: 'Revenue is paid order value.',
tags: ['finance'],
slRefs: ['orders'],
});
await initKtxProject({ projectDir, projectName: 'warehouse' });
const writeIo = makeIo();
await expect(
runKtxKnowledge(
{
command: 'write',
projectDir,
key: 'metrics-revenue',
scope: 'GLOBAL',
userId: 'local',
summary: 'Revenue',
content: 'Revenue is paid order value.',
tags: ['finance'],
refs: [],
slRefs: ['orders'],
},
writeIo.io,
),
).resolves.toBe(0);
expect(writeIo.stdout()).toContain('Wrote wiki/global/metrics-revenue.md');
const readIo = makeIo();
await expect(
runKtxKnowledge({ command: 'read', projectDir, key: 'metrics-revenue', userId: 'local' }, readIo.io),
).resolves.toBe(0);
expect(readIo.stdout()).toContain('# metrics-revenue');
expect(readIo.stdout()).toContain('Revenue is paid order value.');
const listIo = makeIo();
await expect(runKtxKnowledge({ command: 'list', projectDir, userId: 'local' }, listIo.io)).resolves.toBe(0);
@ -97,16 +93,27 @@ describe('runKtxKnowledge', () => {
expect(searchIo.stdout()).toContain('metrics-revenue');
});
it('prints wiki list and search as public JSON envelopes', async () => {
it('prints wiki list, search, and read as public JSON envelopes', async () => {
const projectDir = join(tempDir, 'project');
await seedKnowledgePage({
projectDir,
key: 'metrics-revenue',
summary: 'Revenue',
content: 'Revenue is paid order value.',
tags: ['finance'],
slRefs: ['orders'],
});
await initKtxProject({ projectDir, projectName: 'warehouse' });
await expect(
runKtxKnowledge(
{
command: 'write',
projectDir,
key: 'metrics-revenue',
scope: 'GLOBAL',
userId: 'local',
summary: 'Revenue',
content: 'Revenue is paid order value.',
tags: ['finance'],
refs: [],
slRefs: ['orders'],
},
makeIo().io,
),
).resolves.toBe(0);
const listIo = makeIo();
await expect(runKtxKnowledge({ command: 'list', projectDir, userId: 'local', json: true }, listIo.io)).resolves.toBe(
@ -130,6 +137,48 @@ describe('runKtxKnowledge', () => {
data: { items: [expect.objectContaining({ key: 'metrics-revenue', summary: 'Revenue' })] },
meta: { command: 'wiki search' },
});
const readIo = makeIo();
await expect(
runKtxKnowledge({ command: 'read', projectDir, key: 'metrics-revenue', userId: 'local', json: true }, readIo.io),
).resolves.toBe(0);
expect(JSON.parse(readIo.stdout())).toMatchObject({
kind: 'wiki.page',
data: {
key: 'metrics-revenue',
summary: 'Revenue',
content: 'Revenue is paid order value.',
},
});
});
it('rejects slash-delimited write keys with a flat-key suggestion', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
const writeIo = makeIo();
await expect(
runKtxKnowledge(
{
command: 'write',
projectDir,
key: 'orbit/company-overview',
scope: 'GLOBAL',
userId: 'local',
summary: 'Orbit',
content: 'Orbit overview.',
tags: [],
refs: [],
slRefs: [],
},
writeIo.io,
),
).resolves.toBe(1);
expect(writeIo.stderr()).toContain(
'Invalid wiki key "orbit/company-overview". Wiki keys must be flat; use "orbit-company-overview".',
);
expect(writeIo.stdout()).toBe('');
});
it('explains empty search results for a project without wiki pages', async () => {
@ -143,19 +192,30 @@ describe('runKtxKnowledge', () => {
expect(searchIo.stdout()).toBe('');
expect(searchIo.stderr()).toContain('No local wiki pages found');
expect(searchIo.stderr()).toContain('Run ingest');
expect(searchIo.stderr()).not.toContain('ktx wiki write');
expect(searchIo.stderr()).toContain('ktx wiki write');
});
it('uses configured embeddings for semantic wiki search', async () => {
const projectDir = join(tempDir, 'semantic-project');
await seedKnowledgePage({
projectDir,
key: 'active-contract-arr-open-tickets',
summary: 'Active Contract ARR Ranked by Open Support Ticket Count',
content: 'Accounts ranked by annual recurring contract value and support ticket load.',
tags: ['historic-sql'],
});
await initKtxProject({ projectDir, projectName: 'warehouse' });
await expect(
runKtxKnowledge(
{
command: 'write',
projectDir,
key: 'active-contract-arr-open-tickets',
scope: 'GLOBAL',
userId: 'local',
summary: 'Active Contract ARR Ranked by Open Support Ticket Count',
content: 'Accounts ranked by annual recurring contract value and support ticket load.',
tags: ['historic-sql'],
refs: [],
slRefs: [],
},
makeIo().io,
),
).resolves.toBe(0);
const searchIo = makeIo();
await expect(

59
packages/cli/src/knowledge.ts
View file

@ -4,12 +4,31 @@ import {
type KtxEmbeddingPort,
} from '@ktx/context';
import { loadKtxProject } from '@ktx/context/project';
import { listLocalKnowledgePages, searchLocalKnowledgePages } from '@ktx/context/wiki';
import {
type LocalKnowledgeScope,
listLocalKnowledgePages,
readLocalKnowledgePage,
searchLocalKnowledgePages,
writeLocalKnowledgePage,
} from '@ktx/context/wiki';
import { writeJsonResult } from './io/print-list.js';
export type KtxKnowledgeArgs =
| { command: 'list'; projectDir: string; userId: string; json?: boolean }
| { command: 'search'; projectDir: string; query: string; userId: string; json?: boolean; limit?: number };
| { command: 'read'; projectDir: string; key: string; userId: string; json?: boolean }
| { command: 'search'; projectDir: string; query: string; userId: string; json?: boolean; limit?: number }
| {
command: 'write';
projectDir: string;
key: string;
scope: LocalKnowledgeScope;
userId: string;
summary: string;
content: string;
tags: string[];
refs: string[];
slRefs: string[];
};
interface KtxKnowledgeIo {
stdout: { write(chunk: string): void };
@ -56,6 +75,25 @@ export async function runKtxKnowledge(
}
return 0;
}
if (args.command === 'read') {
const page = await readLocalKnowledgePage(project, { key: args.key, userId: args.userId });
if (!page) {
throw new Error(`Wiki page "${args.key}" was not found`);
}
if (args.json) {
writeJsonResult(io, {
kind: 'wiki.page',
data: page,
meta: { command: 'wiki read' },
});
return 0;
}
io.stdout.write(`# ${page.key}\n\n`);
io.stdout.write(`Scope: ${page.scope}\n`);
io.stdout.write(`Summary: ${page.summary}\n\n`);
io.stdout.write(`${page.content}\n`);
return 0;
}
if (args.command === 'search') {
const results = await searchLocalKnowledgePages(project, {
query: args.query,
@ -75,7 +113,7 @@ export async function runKtxKnowledge(
const pages = await listLocalKnowledgePages(project, { userId: args.userId });
if (pages.length === 0) {
io.stderr.write(
`No local wiki pages found in ${project.projectDir}. Run ingest to capture wiki context, then retry the search.\n`,
`No local wiki pages found in ${project.projectDir}. Create one with \`ktx wiki write <key> --summary <summary> --content <content>\` or run ingest.\n`,
);
} else {
io.stderr.write(
@ -89,8 +127,19 @@ export async function runKtxKnowledge(
}
return 0;
}
const _exhaustive: never = args;
throw new Error(`Unsupported wiki command: ${JSON.stringify(_exhaustive)}`);
const write = await writeLocalKnowledgePage(project, {
key: args.key,
scope: args.scope,
userId: args.userId,
summary: args.summary,
content: args.content,
tags: args.tags,
refs: args.refs,
slRefs: args.slRefs,
});
io.stdout.write(`Wrote ${write.path}\n`);
return 0;
} catch (error) {
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
return 1;

2
packages/cli/src/local-adapters.ts
View file

@ -1,4 +1,3 @@
import { join } from 'node:path';
import {
createBigQueryLiveDatabaseIntrospection,
isKtxBigQueryConnectionConfig,
@ -298,7 +297,6 @@ function historicSqlOptionsForLocalRun(project: KtxLocalProject, options: KtxCli
const base = {
sqlAnalysis: ktxCliHistoricSqlAnalysis(options),
postgresBaselineRootDir: join(project.projectDir, '.ktx/cache/historic-sql'),
};
if (dialect === 'postgres') {

9
packages/cli/src/local-scan-connectors.test.ts
View file

@ -62,10 +62,7 @@ describe('createKtxCliScanConnector', () => {
expect(connector.driver).toBe('sqlite');
});
it.each([
['maxBytesBilled', ' maxBytesBilled: 123456789', 123456789],
['max_bytes_billed', ' max_bytes_billed: "987654321"', '987654321'],
])('passes BigQuery %s from standalone config', async (_label, byteCapLine, expectedMaxBytesBilled) => {
it('passes BigQuery max_bytes_billed from standalone config', async () => {
await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
await writeFile(
join(tempDir, 'ktx.yaml'),
@ -76,7 +73,7 @@ describe('createKtxCliScanConnector', () => {
' driver: bigquery',
' dataset_id: analytics',
' readonly: true',
byteCapLine,
' max_bytes_billed: "987654321"',
'',
].join('\n'),
'utf-8',
@ -90,7 +87,7 @@ describe('createKtxCliScanConnector', () => {
expect(bigQueryMock.constructorInputs).toEqual([
expect.objectContaining({
connectionId: 'warehouse',
maxBytesBilled: expectedMaxBytesBilled,
maxBytesBilled: '987654321',
}),
]);
});

2
packages/cli/src/local-scan-connectors.ts
View file

@ -6,7 +6,7 @@ const SUPPORTED_DRIVERS = 'sqlite, postgres, mysql, clickhouse, sqlserver, bigqu
function bigQueryMaxBytesBilled(
connection: KtxLocalProject['config']['connections'][string],
): number | string | undefined {
const raw = connection.maxBytesBilled ?? connection.max_bytes_billed;
const raw = connection.max_bytes_billed;
if (typeof raw === 'number') {
return Number.isFinite(raw) && raw > 0 ? raw : undefined;
}

4
packages/cli/src/memory-flow-hud.tsx
View file

@ -76,7 +76,7 @@ function tableName(key: string): string {
function humanizeInsight(key: string, target: 'sl' | 'wiki', summary: string | undefined): string {
if (summary) return summary;
const name = target === 'sl' ? tableName(key) : topicName(key);
return target === 'sl' ? `Query definition: ${name}` : `Knowledge page: ${name}`;
return target === 'sl' ? `Query definition: ${name}` : `Wiki page: ${name}`;
}
const INTERNAL_DEMO_CONNECTION_ID = 'orbit_demo';
@ -453,7 +453,7 @@ function CompletionSummary(props: {
)}
{wiki > 0 && (
<Text color={props.theme.complete}>
{' '}📝 {wiki} knowledge page{wiki === 1 ? '' : 's'} — so agents understand your business context
{' '}📝 {wiki} wiki page{wiki === 1 ? '' : 's'} — so agents understand your business context
</Text>
)}
</>

4
packages/cli/src/memory-flow-interactive.test.ts
View file

@ -46,9 +46,9 @@ function replay(): MemoryFlowReplayInput {
{ type: 'raw_snapshot_written', syncId: 'sync-1', rawFileCount: 2 },
{ type: 'diff_computed', added: 1, modified: 1, deleted: 0, unchanged: 0 },
{ type: 'chunks_planned', chunkCount: 2, workUnitCount: 2, evictionCount: 0 },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['knowledge_capture'], stepBudget: 4 },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['wiki_capture'], stepBudget: 4 },
{ type: 'work_unit_finished', unitKey: 'orders', status: 'success' },
{ type: 'work_unit_started', unitKey: 'customers', skills: ['knowledge_capture'], stepBudget: 4 },
{ type: 'work_unit_started', unitKey: 'customers', skills: ['wiki_capture'], stepBudget: 4 },
{ type: 'work_unit_finished', unitKey: 'customers', status: 'failed', reason: 'validation reset' },
{ type: 'reconciliation_finished', conflictCount: 0, fallbackCount: 1 },
{ type: 'saved', commitSha: 'abc12345', wikiCount: 1, slCount: 1 },

12
packages/cli/src/memory-flow-tui.test.tsx
View file

@ -23,10 +23,10 @@ function replayInput(): MemoryFlowReplayInput {
],
details: {
actions: [
{ unitKey: 'orders', target: 'wiki', action: 'created', key: 'knowledge/orders.md', summary: 'order lifecycle', rawFiles: ['orders'], status: 'success' },
{ unitKey: 'orders', target: 'wiki', action: 'created', key: 'wiki/orders.md', summary: 'order lifecycle', rawFiles: ['orders'], status: 'success' },
{ unitKey: 'customers', target: 'sl', action: 'updated', key: 'orbit_demo.customers', summary: 'customer metrics', rawFiles: ['customers'], status: 'success' },
],
provenance: [{ rawPath: 'orders', artifactKind: 'wiki', artifactKey: 'knowledge/orders.md', actionType: 'wiki_written' }],
provenance: [{ rawPath: 'orders', artifactKind: 'wiki', artifactKey: 'wiki/orders.md', actionType: 'wiki_written' }],
transcripts: [{ unitKey: 'orders', path: '/tmp/t.jsonl', toolCallCount: 2, errorCount: 0, toolNames: ['read_raw_span', 'wiki_write'] }],
},
events: [
@ -35,8 +35,8 @@ function replayInput(): MemoryFlowReplayInput {
{ type: 'raw_snapshot_written', syncId: 'sync-1', rawFileCount: 2 },
{ type: 'diff_computed', added: 1, modified: 1, deleted: 0, unchanged: 0 },
{ type: 'chunks_planned', chunkCount: 2, workUnitCount: 2, evictionCount: 0 },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['knowledge_capture'], stepBudget: 40 },
{ type: 'candidate_action', unitKey: 'orders', target: 'wiki', action: 'created', key: 'knowledge/orders.md' },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['wiki_capture'], stepBudget: 40 },
{ type: 'candidate_action', unitKey: 'orders', target: 'wiki', action: 'created', key: 'wiki/orders.md' },
{ type: 'work_unit_finished', unitKey: 'orders', status: 'success' },
{ type: 'work_unit_started', unitKey: 'customers', skills: ['sl_capture'], stepBudget: 40 },
{ type: 'candidate_action', unitKey: 'customers', target: 'sl', action: 'updated', key: 'orbit_demo.customers' },
@ -220,7 +220,7 @@ describe('MemoryFlowTuiApp', () => {
{ type: 'source_acquired', adapter: 'live-database', trigger: 'manual_resync', fileCount: 1 },
{ type: 'diff_computed', added: 1, modified: 0, deleted: 0, unchanged: 0 },
{ type: 'chunks_planned', chunkCount: 1, workUnitCount: 1, evictionCount: 0 },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['knowledge_capture'], stepBudget: 40 },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['wiki_capture'], stepBudget: 40 },
],
plannedWorkUnits: [{ unitKey: 'orders', rawFiles: ['orders'], peerFileCount: 0, dependencyCount: 1 }],
};
@ -240,7 +240,7 @@ describe('MemoryFlowTuiApp', () => {
{ type: 'source_acquired', adapter: 'dbt-descriptions', trigger: 'manual_resync', fileCount: 3 },
{ type: 'diff_computed', added: 11, modified: 0, deleted: 0, unchanged: 0 },
{ type: 'chunks_planned', chunkCount: 1, workUnitCount: 1, evictionCount: 0 },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['knowledge_capture'], stepBudget: 40 },
{ type: 'work_unit_started', unitKey: 'orders', skills: ['wiki_capture'], stepBudget: 40 },
],
plannedWorkUnits: [{ unitKey: 'orders', rawFiles: ['orders'], peerFileCount: 0, dependencyCount: 1 }],
};

25
packages/cli/src/next-steps.test.ts
View file

@ -6,8 +6,6 @@ import {
formatSetupNextStepLines,
} from './next-steps.js';
const command = (...parts: string[]) => parts.join(' ');
describe('KTX demo next steps', () => {
it('uses supported context-build commands before agent usage', () => {
expect(KTX_CONTEXT_BUILD_COMMANDS).toEqual([
@ -57,29 +55,6 @@ describe('KTX demo next steps', () => {
expect(rendered).not.toContain('Optional MCP:');
});
it('does not advertise removed Commander migration commands', () => {
const rendered = formatNextStepLines().join('\n');
expect(rendered).toContain('ktx status --json');
expect(rendered).not.toContain('ktx agent');
expect(rendered).toContain('ktx sl list');
expect(rendered).toContain('ktx wiki list');
for (const removed of [
command('ktx', 'ask'),
command('ktx', 'mcp'),
command('ktx', 'connect'),
command('ktx', 'knowledge'),
command('dev', 'model'),
command('dev', 'knowledge'),
command('ktx', 'ingest', 'run'),
command('ktx', 'ingest', 'replay'),
command('ktx', 'serve', '--mcp', 'stdio', '--user-id', 'local'),
]) {
expect(rendered).not.toContain(removed);
}
});
it('keeps setup next steps focused on building context when the build is not ready', () => {
const rendered = formatSetupNextStepLines({
setupReady: true,

2
packages/cli/src/public-ingest.ts
View file

@ -95,7 +95,7 @@ function normalizedDriver(connection: KtxProjectConnectionConfig): string {
}
function sourceDirForConnection(connection: KtxProjectConnectionConfig): string | undefined {
const value = connection.source_dir ?? connection.sourceDir;
const value = connection.source_dir;
return typeof value === 'string' && value.trim().length > 0 ? value.trim() : undefined;
}

8
packages/cli/src/setup-context.test.ts
View file

@ -277,9 +277,9 @@ describe('setup context build state', () => {
it('marks context complete without prompting when initial source ingest already made agent context', async () => {
await writeReadyProject(tempDir);
await mkdir(join(tempDir, 'semantic-layer', 'dbt-main'), { recursive: true });
await mkdir(join(tempDir, 'knowledge', 'global'), { recursive: true });
await mkdir(join(tempDir, 'wiki', 'global'), { recursive: true });
await writeFile(join(tempDir, 'semantic-layer', 'dbt-main', 'mart_revenue_daily.yaml'), 'name: mart_revenue_daily\n');
await writeFile(join(tempDir, 'knowledge', 'global', 'metrics.md'), '# Metrics\n');
await writeFile(join(tempDir, 'wiki', 'global', 'metrics.md'), '# Metrics\n');
await writeReadyEnrichedScanReport(tempDir);
const io = makeIo();
const runContextBuildMock = vi.fn(async () => ({ exitCode: 0, detached: false }));
@ -352,8 +352,8 @@ describe('setup context build state', () => {
await writeFile(join(tempDir, 'semantic-layer', 'warehouse', '_schema', 'public.yaml'), 'tables: {}\n');
const io = makeIo();
const runContextBuildMock = vi.fn(async () => {
await mkdir(join(tempDir, 'knowledge', 'global'), { recursive: true });
await writeFile(join(tempDir, 'knowledge', 'global', 'metrics.md'), '# Metrics\n');
await mkdir(join(tempDir, 'wiki', 'global'), { recursive: true });
await writeFile(join(tempDir, 'wiki', 'global', 'metrics.md'), '# Metrics\n');
await writeReadyEnrichedScanReport(tempDir);
return { exitCode: 0, detached: false };
});

2
packages/cli/src/setup-context.ts
View file

@ -433,7 +433,7 @@ async function defaultVerifyContextReady(projectDir: string): Promise<KtxSetupCo
ignoredDirectoryNames: new Set(['_schema']),
},
);
const wikiReady = await hasFileWithExtension(join(projectDir, 'knowledge'), new Set(['.md']));
const wikiReady = await hasFileWithExtension(join(projectDir, 'wiki'), new Set(['.md']));
const contextSourceReady =
targets.contextSourceConnectionIds.length === 0 || semanticLayerContextReady || wikiReady;
const ready = primarySourceScans.ready && contextSourceReady;

62
packages/cli/src/setup-databases.test.ts
View file

@ -64,8 +64,6 @@ function textInputPrompt(message: string): string {
return `${title}\n│\n│ ${bodyLines.join('\n│ ')}\n│ Press Escape to go back.\n│`;
}
const legacyHistoricSqlServiceAccountPatternsKey = ['serviceAccount', 'UserPatterns'].join('');
describe('setup databases step', () => {
let tempDir: string;
@ -1255,6 +1253,7 @@ describe('setup databases step', () => {
io.io,
{
testConnection: vi.fn(async () => 0),
rebuildNativeSqlite: vi.fn(async () => 1),
scanConnection: vi.fn(async (_projectDir: string, _connectionId: string, commandIo: KtxCliIo) => {
commandIo.stderr.write(
[
@ -1280,6 +1279,60 @@ describe('setup databases step', () => {
expect(io.stderr()).not.toMatch(/^Native SQLite is built for a different Node.js ABI\./m);
});
it('rebuilds native SQLite once and retries setup scanning after a Node ABI mismatch', async () => {
const io = makeIo();
const scanConnection = vi.fn(async (_projectDir: string, _connectionId: string, commandIo: KtxCliIo) => {
if (scanConnection.mock.calls.length === 1) {
commandIo.stderr.write(
[
"The module '/workspace/node_modules/better-sqlite3/build/Release/better_sqlite3.node'",
'was compiled against a different Node.js version using',
'NODE_MODULE_VERSION 147. This version of Node.js requires',
'NODE_MODULE_VERSION 137. Please try re-compiling or re-installing',
'the module (for instance, using `npm rebuild` or `npm install`).',
'',
].join('\n'),
);
return 1;
}
commandIo.stdout.write('What changed\n');
commandIo.stdout.write(' Semantic layer comparison found 0 changes across 56 tables\n');
commandIo.stdout.write(' New tables: 0\n');
commandIo.stdout.write(' Changed tables: 0\n');
commandIo.stdout.write(' Removed tables: 0\n');
commandIo.stdout.write(' Unchanged tables: 56\n');
return 0;
});
const rebuildNativeSqlite = vi.fn(async () => 0);
const result = await runKtxSetupDatabasesStep(
{
projectDir: tempDir,
inputMode: 'disabled',
databaseDrivers: ['postgres'],
databaseConnectionId: 'warehouse',
databaseUrl: 'env:DATABASE_URL',
databaseSchemas: [],
skipDatabases: false,
},
io.io,
{
testConnection: vi.fn(async () => 0),
scanConnection,
rebuildNativeSqlite,
},
);
expect(result.status).toBe('ready');
expect(rebuildNativeSqlite).toHaveBeenCalledOnce();
expect(rebuildNativeSqlite).toHaveBeenCalledWith(expect.anything());
expect(scanConnection).toHaveBeenCalledTimes(2);
expect(io.stderr()).toContain('Native SQLite is built for a different Node.js ABI.');
expect(io.stderr()).toContain('Rebuilding Native SQLite with pnpm run native:rebuild…');
expect(io.stdout()).toContain('◇ Scan complete for warehouse');
});
it('writes Historic SQL config for supported Snowflake databases after validation succeeds', async () => {
const io = makeIo();
const result = await runKtxSetupDatabasesStep(
@ -1325,7 +1378,6 @@ describe('setup databases step', () => {
redactionPatterns: ['(?i)secret'],
},
});
expect(config.connections.snowflake.historicSql).not.toHaveProperty(legacyHistoricSqlServiceAccountPatternsKey);
expect(config.ingest.adapters).toContain('historic-sql');
});
@ -1373,10 +1425,8 @@ describe('setup databases step', () => {
},
},
});
expect(config.connections.warehouse.historicSql).not.toHaveProperty('minCalls');
expect(config.connections.warehouse.historicSql).not.toHaveProperty('windowDays');
expect(config.connections.warehouse.historicSql).not.toHaveProperty('redactionPatterns');
expect(config.connections.warehouse.historicSql).not.toHaveProperty(legacyHistoricSqlServiceAccountPatternsKey);
expect(config.ingest.adapters).toContain('historic-sql');
expect(config.ingest.workUnits.maxConcurrency).toBe(6);
expect(io.stdout()).toContain('Historic SQL probe...');
@ -1430,7 +1480,6 @@ describe('setup databases step', () => {
redactionPatterns: [],
},
});
expect(config.connections.analytics.historicSql).not.toHaveProperty(legacyHistoricSqlServiceAccountPatternsKey);
expect(config.ingest.adapters).toContain('historic-sql');
});
@ -1480,7 +1529,6 @@ describe('setup databases step', () => {
},
},
});
expect(config.connections.warehouse.historicSql).not.toHaveProperty(legacyHistoricSqlServiceAccountPatternsKey);
});
it('prints a non-blocking Postgres Historic SQL probe failure after connection test succeeds', async () => {

121
packages/cli/src/setup-databases.ts
View file

@ -1,4 +1,8 @@
import { writeFile } from 'node:fs/promises';
import { execFile as execFileCallback } from 'node:child_process';
import { readFile, writeFile } from 'node:fs/promises';
import { delimiter, dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { promisify } from 'node:util';
import type { HistoricSqlDialect } from '@ktx/context/ingest';
import {
type KtxProjectConnectionConfig,
@ -19,6 +23,7 @@ import {
} from './setup-prompts.js';
const HISTORIC_SQL_WORK_UNIT_MAX_CONCURRENCY = 6;
const execFileAsync = promisify(execFileCallback);
export type KtxSetupDatabaseDriver =
| 'sqlite'
@ -41,7 +46,6 @@ export interface KtxSetupDatabasesArgs {
disableHistoricSql?: boolean;
historicSqlWindowDays?: number;
historicSqlMinExecutions?: number;
historicSqlMinCalls?: number;
historicSqlServiceAccountPatterns?: string[];
historicSqlRedactionPatterns?: string[];
skipDatabases: boolean;
@ -84,6 +88,7 @@ export interface KtxSetupDatabasesDeps {
prompts?: KtxSetupDatabasesPromptAdapter;
testConnection?: (projectDir: string, connectionId: string, io: KtxCliIo) => Promise<number>;
scanConnection?: (projectDir: string, connectionId: string, io: KtxCliIo) => Promise<number>;
rebuildNativeSqlite?: (io: KtxCliIo) => Promise<number>;
listSchemas?: (projectDir: string, connectionId: string) => Promise<string[]>;
listTables?: (projectDir: string, connectionId: string) => Promise<KtxTableListEntry[]>;
historicSqlProbe?: KtxSetupHistoricSqlProbe;
@ -819,14 +824,13 @@ async function maybeApplyHistoricSqlConfig(input: {
dialect,
filters: historicSqlFiltersForSetup(input.args.historicSqlServiceAccountPatterns),
};
delete common[['serviceAccount', 'UserPatterns'].join('')];
if (dialect === 'postgres') {
return {
...input.connection,
historicSql: {
...common,
minExecutions: input.args.historicSqlMinExecutions ?? input.args.historicSqlMinCalls ?? 5,
minExecutions: input.args.historicSqlMinExecutions ?? 5,
},
};
}
@ -922,6 +926,81 @@ function writePrefixedLines(write: (chunk: string) => void, output: string): voi
}
}
function envWithCurrentNodeFirst(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv {
return {
...env,
PATH: `${dirname(process.execPath)}${delimiter}${env.PATH ?? ''}`,
};
}
function errorTextProperty(error: unknown, property: 'stderr' | 'stdout'): string {
if (typeof error !== 'object' || error === null || !(property in error)) {
return '';
}
const value = (error as Record<typeof property, unknown>)[property];
return typeof value === 'string' ? value : '';
}
function commandFailureOutput(error: unknown): string {
const stderr = errorTextProperty(error, 'stderr');
const stdout = errorTextProperty(error, 'stdout');
const message = error instanceof Error ? error.message : String(error);
return [stderr.trim(), stdout.trim(), message.trim()].filter((line) => line.length > 0).join('\n');
}
type PackageJsonScriptStatus = 'has-script' | 'exists' | 'missing';
async function packageJsonScriptStatus(
packageJsonPath: string,
scriptName: string,
): Promise<PackageJsonScriptStatus> {
try {
const parsed = JSON.parse(await readFile(packageJsonPath, 'utf-8')) as unknown;
if (typeof parsed !== 'object' || parsed === null || !('scripts' in parsed)) {
return 'exists';
}
const scripts = (parsed as { scripts?: unknown }).scripts;
return typeof scripts === 'object' && scripts !== null && scriptName in scripts ? 'has-script' : 'exists';
} catch {
return 'missing';
}
}
async function nativeSqliteRebuildCommand(): Promise<{ cwd: string; args: string[] }> {
let dir = dirname(fileURLToPath(import.meta.url));
let packageRoot: string | undefined;
while (true) {
const status = await packageJsonScriptStatus(join(dir, 'package.json'), 'native:rebuild');
if (status === 'has-script') {
return { cwd: dir, args: ['run', 'native:rebuild'] };
}
if (status === 'exists') {
packageRoot ??= dir;
}
const parent = dirname(dir);
if (parent === dir) {
return { cwd: packageRoot ?? process.cwd(), args: ['rebuild', 'better-sqlite3'] };
}
dir = parent;
}
}
async function defaultRebuildNativeSqlite(io: KtxCliIo): Promise<number> {
const command = await nativeSqliteRebuildCommand();
try {
await execFileAsync('pnpm', command.args, {
cwd: command.cwd,
env: envWithCurrentNodeFirst(),
maxBuffer: 1024 * 1024 * 16,
});
return 0;
} catch (error) {
writePrefixedLines((chunk) => io.stderr.write(chunk), commandFailureOutput(error));
return typeof (error as { code?: unknown })?.code === 'number' ? (error as { code: number }).code : 1;
}
}
function flushPrefixedBufferedCommandOutput(io: KtxCliIo, bufferedIo: BufferedCommandIo): void {
writePrefixedLines((chunk) => io.stdout.write(chunk), bufferedIo.stdoutText());
writePrefixedLines((chunk) => io.stderr.write(chunk), bufferedIo.stderrText());
@ -1435,8 +1514,8 @@ async function validateAndScanConnection(input: {
writeSetupSection(input.io, `Scanning ${input.connectionId}`, [
'Running structural scan…',
]);
const scanIo = createBufferedCommandIo();
const scanCode = await scanConnection(input.projectDir, input.connectionId, scanIo);
let scanIo = createBufferedCommandIo();
let scanCode = await scanConnection(input.projectDir, input.connectionId, scanIo);
if (scanCode !== 0) {
const nativeSqliteDetail = nativeSqliteAbiMismatchDetail(`${scanIo.stderrText()}\n${scanIo.stdoutText()}`);
if (nativeSqliteDetail) {
@ -1446,10 +1525,32 @@ async function validateAndScanConnection(input: {
`Structural scan failed for ${input.connectionId}.`,
'Native SQLite is built for a different Node.js ABI.',
`Detail: ${nativeSqliteDetail}`,
'Fix: pnpm run native:rebuild',
`Retry: ktx scan --project-dir ${input.projectDir} ${input.connectionId}`,
'Rebuilding Native SQLite with pnpm run native:rebuild…',
].join('\n'),
);
const rebuildNativeSqlite = input.deps.rebuildNativeSqlite ?? defaultRebuildNativeSqlite;
const rebuildCode = await rebuildNativeSqlite(input.io);
if (rebuildCode === 0) {
writePrefixedLines(
(chunk) => input.io.stderr.write(chunk),
'Native SQLite rebuild complete. Retrying structural scan…',
);
const retryScanIo = createBufferedCommandIo();
scanCode = await scanConnection(input.projectDir, input.connectionId, retryScanIo);
scanIo = retryScanIo;
}
if (scanCode !== 0) {
writePrefixedLines(
(chunk) => input.io.stderr.write(chunk),
[
rebuildCode === 0
? `Structural scan still failed for ${input.connectionId} after rebuilding Native SQLite.`
: `Native SQLite rebuild failed for ${input.connectionId}.`,
'Fix: pnpm run native:rebuild',
`Retry: ktx scan --project-dir ${input.projectDir} ${input.connectionId}`,
].join('\n'),
);
}
} else {
flushPrefixedBufferedCommandOutput(input.io, scanIo);
writePrefixedLines(
@ -1460,7 +1561,9 @@ async function validateAndScanConnection(input: {
].join('\n'),
);
}
return false;
if (scanCode !== 0) {
return false;
}
}
const scanOutput = scanIo.stdoutText();
const reportPath = readOutputValue(scanOutput, 'Report');

4
packages/cli/src/setup-demo-tour.ts
View file

@ -230,7 +230,7 @@ function renderDemoContextCompletionSummary(): string {
'',
' KTX created:',
` ${cyan('📊')} 46 semantic layer definitions`,
` ${cyan('📝')} 28 knowledge pages`,
` ${cyan('📝')} 28 wiki pages`,
'',
` ${dim('Press Enter to continue, Escape to go back')}`,
'',
@ -355,7 +355,7 @@ export async function runDemoTour(
if (step === 'databases') {
direction = await renderDemoCard('Database connection', ['PostgreSQL — Orbit Analytics (56 tables, 2 schemas)'], io, undefined, waitNav, projectDir);
} else if (step === 'sources') {
direction = await renderDemoCard('Context sources', ['dbt — 34 transformation models', 'Metabase — 80 dashboard cards', 'Notion — 9 knowledge pages'], io, undefined, waitNav, projectDir);
direction = await renderDemoCard('Context sources', ['dbt — 34 transformation models', 'Metabase — 80 dashboard cards', 'Notion — 9 wiki pages'], io, undefined, waitNav, projectDir);
} else if (step === 'context') {
io.stdout.write(renderDemoBanner(projectDir) + '\n\n');
if (deps.skipReplayAnimation) {

26
packages/cli/src/setup-sources.test.ts
View file

@ -624,6 +624,32 @@ describe('setup sources step', () => {
expect(options).toContainEqual({ value: 'notion', label: 'Notion' });
});
it('shows already configured context sources in the interactive checklist', async () => {
await addPrimarySource();
await addConnection('notion-main', {
driver: 'notion',
auth_token_ref: 'env:NOTION_TOKEN',
crawl_mode: 'all_accessible',
});
const io = makeIo();
const testPrompts = prompts({ multiselect: [['back']] });
await expect(
runKtxSetupSourcesStep(
{ projectDir, inputMode: 'auto', runInitialSourceIngest: false, skipSources: false },
io.io,
{ prompts: testPrompts },
),
).resolves.toEqual({ status: 'back', projectDir });
expect(testPrompts.multiselect).toHaveBeenCalledWith(
expect.objectContaining({
initialValues: ['notion'],
options: expect.arrayContaining([{ value: 'notion', label: 'Notion', hint: 'configured: notion-main' }]),
}),
);
});
it('uses a source-specific editable connection name for new interactive connections', async () => {
await addPrimarySource();
const validateDbt = vi.fn(async () => ({ ok: true as const, detail: 'project=analytics schemas=2' }));

35
packages/cli/src/setup-sources.ts
View file

@ -76,6 +76,7 @@ export interface KtxSetupSourcesPromptAdapter {
multiselect(options: {
message: string;
options: KtxSetupPromptOption[];
initialValues?: string[];
required?: boolean;
}): Promise<string[]>;
select(options: { message: string; options: KtxSetupPromptOption[] }): Promise<string>;
@ -504,8 +505,8 @@ function sourcePathFromFileRepoUrl(repoUrl: string, subpath?: string): string {
}
function repoAuthToken(connection: KtxProjectConnectionConfig | Record<string, unknown>): string | null {
const ref = stringField(connection.auth_token_ref) ?? stringField(connection.authTokenRef);
const literal = stringField(connection.authToken) ?? stringField(connection.auth_token);
const ref = stringField(connection.auth_token_ref);
const literal = stringField(connection.auth_token);
return literal ?? resolveKtxConfigReference(ref, process.env) ?? null;
}
@ -523,8 +524,8 @@ async function collectYamlFilesRecursive(sourceRoot: string): Promise<Array<{ co
}
async function defaultValidateDbt(connection: KtxProjectConnectionConfig): Promise<SourceValidationResult> {
let sourceDir = stringField(connection.source_dir) ?? stringField(connection.sourceDir);
const repoUrl = stringField(connection.repo_url) ?? stringField(connection.repoUrl);
let sourceDir = stringField(connection.source_dir);
const repoUrl = stringField(connection.repo_url);
if (!sourceDir && repoUrl?.startsWith('file:')) {
sourceDir = sourcePathFromFileRepoUrl(repoUrl, stringField(connection.path));
}
@ -584,7 +585,7 @@ async function defaultValidateLooker(projectDir: string, connectionId: string):
}
async function defaultValidateLookml(connection: KtxProjectConnectionConfig): Promise<SourceValidationResult> {
const repoUrl = stringField(connection.repoUrl) ?? stringField(connection.repo_url);
const repoUrl = stringField(connection.repoUrl);
if (!repoUrl) {
return { ok: false, message: 'LookML setup requires repoUrl.' };
}
@ -1285,6 +1286,22 @@ function existingConnectionIdsBySource(
.sort((left, right) => left.localeCompare(right));
}
function sourceChecklistForConnections(connections: Record<string, KtxProjectConnectionConfig>): {
options: Array<{ value: KtxSetupSourceType; label: string; hint?: string }>;
initialValues: KtxSetupSourceType[];
} {
const initialValues: KtxSetupSourceType[] = [];
const options = SOURCE_OPTIONS.map((option) => {
const existingIds = existingConnectionIdsBySource(connections, option.value);
if (existingIds.length === 0) {
return option;
}
initialValues.push(option.value);
return { ...option, hint: `configured: ${existingIds.join(', ')}` };
});
return { options, initialValues };
}
function defaultConnectionIdForSource(
connections: Record<string, KtxProjectConnectionConfig>,
source: KtxSetupSourceType,
@ -1443,13 +1460,19 @@ export async function runKtxSetupSourcesStep(
}
while (true) {
const contextSourceChecklist = sourceChecklistForConnections(
(await loadKtxProject({ projectDir: args.projectDir })).config.connections,
);
const selected = args.source
? [args.source]
: args.inputMode === 'disabled'
? []
: await prompts.multiselect({
message: withMultiselectNavigation('Which context sources should KTX ingest?'),
options: [...SOURCE_OPTIONS],
options: contextSourceChecklist.options,
...(contextSourceChecklist.initialValues.length > 0
? { initialValues: contextSourceChecklist.initialValues }
: {}),
required: false,
});
if (selected.includes('back')) {

2
packages/cli/src/setup.ts
View file

@ -94,7 +94,6 @@ export type KtxSetupArgs =
disableHistoricSql?: boolean;
historicSqlWindowDays?: number;
historicSqlMinExecutions?: number;
historicSqlMinCalls?: number;
historicSqlServiceAccountPatterns?: string[];
historicSqlRedactionPatterns?: string[];
skipDatabases: boolean;
@ -634,7 +633,6 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
...(args.historicSqlMinExecutions !== undefined
? { historicSqlMinExecutions: args.historicSqlMinExecutions }
: {}),
...(args.historicSqlMinCalls !== undefined ? { historicSqlMinCalls: args.historicSqlMinCalls } : {}),
...(args.historicSqlServiceAccountPatterns
? { historicSqlServiceAccountPatterns: args.historicSqlServiceAccountPatterns }
: {}),

2
packages/cli/src/sl.test.ts
View file

@ -129,7 +129,7 @@ describe('runKtxSl', () => {
connectionId: 'warehouse',
name: 'orders',
score: expect.any(Number),
matchReasons: expect.arrayContaining(['token']),
matchReasons: expect.any(Array),
}),
],
},

2
packages/context/prompts/memory_agent_backfill.md
View file

@ -10,7 +10,7 @@ Capture only when the signal is unambiguous: a metric definition stated plainly,
<workflow>
1. Read the wiki and SL indexes to avoid creating duplicates.
2. If the content has wiki-style signal, load the `knowledge_capture` skill and follow its workflow.
2. If the content has wiki-style signal, load the `wiki_capture` skill and follow its workflow.
3. If the content has SL-style signal, load the `sl` skill and follow its Part 3 workflow.
4. Prefer updating existing entries over creating new ones — backfills often duplicate existing knowledge.
5. When done, exit the loop.

6
packages/context/prompts/memory_agent_bundle_ingest_reconcile.md
View file

@ -1,5 +1,5 @@
<role>
You are the reconciliation agent for a multi-file ingest bundle. Stage 3 WorkUnits have already run against this job's session worktree; your input is the deterministic Stage Index listing every write each WU made, plus an Eviction Set listing raw files present in the prior sync but absent in this one. Your job is to (a) decide what happens to each evicted artifact (remove vs retain with a deprecation marker), (b) sweep the Stage Index for any cross-WU conflicts the individual WUs missed, and (c) emit conflict + eviction records that the runner will fold into the final IngestReport.
You are the reconciliation agent for a multi-file ingest bundle. Stage 3 WorkUnits have already run against this job's session worktree; your input is the deterministic Stage Index listing every write each WU made, plus an Eviction Set listing raw files present in the prior sync but absent in this one. Your job is to (a) remove artifacts produced by deleted raw files, (b) sweep the Stage Index for any cross-WU conflicts the individual WUs missed, and (c) emit conflict + eviction records that the runner will fold into the final IngestReport.
</role>
<stance>
@ -7,12 +7,12 @@ Parsimonious. Stage 3 WUs already loaded `ingest_triage` and handled conflicts t
</stance>
<workflow>
1. Load `ingest_triage`, then `sl_capture` + `knowledge_capture`.
1. Load `ingest_triage`, then `sl_capture` + `wiki_capture`.
2. Call `stage_list()` for the full index of this job's writes. If it is empty AND you have no evictions, exit — the runner short-circuits this case but the skill still teaches you to bail fast.
3. If the system prompt includes `<canonical_pins>`, apply those pins before flagging a same-name or near-duplicate conflict. A pinned `canonicalArtifactKey` keeps the contested name when it is present in the Stage Index; competing variants keep or receive disambiguated names.
4. Sweep both exact-key conflicts and near-duplicate writes. Compare WUs that wrote overlapping SL source names, overlapping wiki keys, the same `tables:` or `sl_refs:` action details, or obviously equivalent topic titles under different wiki keys. Call `stage_diff` to see the actual difference, and use `wiki_read`/`sl_read_source` when two different keys appear to describe the same table, metric, or source-of-truth mapping. If they're the same content, leave one canonical artifact and record the duplicate as subsumed. If they differ per `ingest_triage` rules, apply the correct resolution (rename + capture; election of canonical; silent replace for expression-only re-ingest change; or pinned canonical), then call `emit_conflict_resolution` with the artifact key and decision.
5. For any `wiki_write`, `wiki_remove`, `sl_write_source`, or `sl_edit_source` call you make during reconciliation, include `rawPaths` with only the raw paths that directly caused that reconciliation action.
6. Call `eviction_list()` for deleted raw paths. For each eviction: if inbound refs are empty, remove the artifact (`sl_delete`, `wiki_remove`) and include that evicted raw path in `rawPaths`; if inbound refs exist, retain with a deprecation marker and include that evicted raw path in `rawPaths`. Then call `emit_eviction_decision` for every removed or retained artifact.
6. Call `eviction_list()` for deleted raw paths. For each listed artifact, remove it (`sl_delete`, `wiki_remove`) and include the evicted raw path in `rawPaths`. Then call `emit_eviction_decision` with `action: "removed"` for every removed artifact.
7. If the Stage 4 sweep discovers a raw file whose only honest outcome is standalone SQL, wiki-only capture, or a human flag, call `emit_unmapped_fallback` with the raw path, reason, and fallback kind.
8. Use `read_raw_span` to zoom into specific raw files when you need to resolve what two contested measures or wiki pages actually describe.
9. Exit when you've processed every item.

2
packages/context/prompts/memory_agent_bundle_ingest_work_unit.md
View file

@ -8,7 +8,7 @@ Assertive. The bundle was explicitly submitted for ingest. Default to capturing
<workflow>
1. Read this WorkUnit's section at the end of the user prompt. It lists your `rawFiles`, any unchanged `dependencyPaths` you may need to resolve references, the `peerFileIndex` (paths only; you CANNOT read them), the source's `skillNames`, and any `priorProvenance` rows telling you what earlier syncs produced from these files.
2. Load the per-source review skill first (e.g. `lookml_ingest`, `metricflow_ingest`, `dbt_ingest`), then `sl_capture` and `knowledge_capture`, and `ingest_triage` last. The triage skill tells you how to react when `discover_data` reveals that a prior WU already wrote something overlapping.
2. Load the per-source review skill first (e.g. `lookml_ingest`, `metricflow_ingest`, `dbt_ingest`), then `sl_capture` and `wiki_capture`, and `ingest_triage` last. The triage skill tells you how to react when `discover_data` reveals that a prior WU already wrote something overlapping.
3. If the system prompt includes `<canonical_pins>`, read those pins before choosing artifact keys. A pin's `canonicalArtifactKey` is the preferred artifact for its `contestedKey`: prefer editing the pinned canonical artifact when it already exists or when this raw file clearly updates it. Do not create a duplicate contested artifact when a pin says another artifact is canonical; use a specific disambiguated key only when the raw file describes a genuinely different domain.
4. For each raw file: call `read_raw_file` (or `read_raw_span` for slicing large files) to load content. Before writing a new SL source or wiki page, call `discover_data` for each candidate source, table, metric, or topic name to find prior-WU writes, existing wiki pages, SL sources, and raw warehouse matches; apply `ingest_triage` when you hit one, and apply any matching canonical pin before deciding whether to edit, rename, or skip.
5. For every `wiki_write`, `wiki_remove`, `sl_write_source`, or `sl_edit_source` call, include `rawPaths` with only the raw file paths that directly support that action. If one artifact synthesizes several files, list each contributing raw file. Do not include unrelated files from the same WorkUnit.

2
packages/context/prompts/memory_agent_external_ingest.md
View file

@ -10,7 +10,7 @@ A single artifact typically produces multiple actions: one SL source per table/v
<workflow>
1. Review the wiki and SL indexes in the prompt. Prefer updating existing entries over creating duplicates.
2. Load the `sl` skill for SL-writes and `knowledge_capture` for wiki-writes. Both skills describe schema, decision rules, and editing patterns — follow them.
2. Load the `sl` skill for SL-writes and `wiki_capture` for wiki-writes. Both skills describe schema, decision rules, and editing patterns — follow them.
3. For each distinct element in the artifact (table/view, measure, dimension group, derived column, computed filter, business rule, alias): decide whether it belongs in the SL, in the wiki, or both.
4. Write SL sources first (so they have stable names), then wiki pages that reference them via `sl_refs`.
5. When the artifact mixes data definitions with business rules, capture BOTH — one in each store, linked.

2
packages/context/prompts/memory_agent_research.md
View file

@ -19,7 +19,7 @@ Skip:
<workflow>
1. Read the wiki index and the SL sources index in the prompt below.
2. Identify durable knowledge OR reusable data patterns in the turn.
3. If the turn has wiki-style signal (preferences, definitions, conventions), load the `knowledge_capture` skill and follow its workflow.
3. If the turn has wiki-style signal (preferences, definitions, conventions), load the `wiki_capture` skill and follow its workflow.
4. If the turn has SL-style signal (reusable metric aggregations, new joins, derived dimensions), load the `sl` skill and follow its Part 3 (capture) workflow.
5. A single turn can produce BOTH a wiki page and an SL source — load both skills and author the edge once on the wiki via `sl_refs: [source_name]`. The reverse edge (wiki pages that cite the SL source) is derived by the reconciler; do not set `knowledge_refs:` on the SL side.
6. When you're done, exit the loop without calling any more tools. Do NOT emit a final text summary.

4
packages/context/skills/ingest_triage/SKILL.md
View file

@ -32,8 +32,8 @@ Apply the rules below before every write that could collide with an existing art
| Definitional contradiction | Same name, substantively different formulas (different aggregation, different filters, different columns) | **Rename + capture**: disambiguate ALL variants with suffix derived from the domain (`churn_risk_engagement_based`, `churn_risk_billing_based`) and write a unified wiki page listing every variant with provenance. The contested name does NOT land in the SL. **Always flag.** |
5. **Eviction (Stage 4 only)**: for each entry in `eviction_list()`:
- `inbound_refs: []` → remove the artifact (`sl_delete` for SL sources, `wiki_remove` for wiki pages).
- `inbound_refs: [...]` → retain the artifact, set `deprecated: true` on SL sources (via `sl_edit_source`), write a wiki note "origin file removed in <syncId>; preserved because referenced by: …". Flag in the IngestReport so the user can plan migration.
- Remove the artifact (`sl_delete` for SL sources, `wiki_remove` for wiki pages).
- Record the removal with `emit_eviction_decision` and `action: "removed"`.
## Why same-ingest vs re-ingest differs

2
packages/context/skills/metabase_ingest/SKILL.md
View file

@ -98,7 +98,7 @@ measures:
expr: "<expression>"
```
Overlay shape: `name:` plus any of `measures:`, `segments:`, `description:`, `joins:`, `disable_joins:`. Never include `sql:`, `table:`, `grain:`, or `columns:` on a manifest-backed name — those would shadow the manifest's schema and drop its joins. Overlay `joins:` are merged additively with the manifest's joins (deduped by `to` + `on`); use `disable_joins: ["<on-clause>"]` to suppress a specific manifest join. After the overlay exists, use `sl_edit_source` for further tweaks. See `sl_capture` skill for the canonical overlay rule.
Overlay shape: `name:` plus any of `measures:`, `segments:`, `descriptions:`, `joins:`, `disable_joins:`. Never include `sql:`, `table:`, `grain:`, or `columns:` on a manifest-backed name — those would shadow the manifest's schema and drop its joins. Overlay `joins:` are merged additively with the manifest's joins (deduped by `to` + `on`); use `disable_joins: ["<on-clause>"]` to suppress a specific manifest join. After the overlay exists, use `sl_edit_source` for further tweaks. See `sl_capture` skill for the canonical overlay rule.
**Join discovery:** When your card's SQL references warehouse tables (e.g. in `FROM` or `JOIN` clauses), call `sl_discover({ query: '<table>' })` before writing. The matching manifest entry's `name` is the value you use in `joins: [- to: <name>]` only when the card output exposes a local key that matches the target source grain (for example `account_id = mart_account_segments.account_id`). Do not declare a KTX join just because the card SQL joins that table internally. If the output only exposes display fields such as `account_name`, keep the SQL source self-contained or project the key before adding the join. Use `many_to_one` for FK-to-dimension joins, `one_to_many` for the reverse.

10
packages/context/skills/metricflow_ingest/SKILL.md
View file

@ -140,7 +140,7 @@ metrics:
```
Do NOT emit SL for this. Instead:
- Write a wiki page at `knowledge/global/<metric_name>-intent.md` quoting the full YAML body and a one-line explanation of the intended semantics (base event → conversion event within window).
- Write a wiki page at `wiki/global/<metric_name>-intent.md` quoting the full YAML body and a one-line explanation of the intended semantics (base event → conversion event within window).
- Call `emit_unmapped_fallback` with `rawPath` set to the MetricFlow file path, `reason: "conversion_metric_unsupported"`, and `fallback: "flagged"`.
When KTX SL gains conversion primitives, re-ingesting will find the prior wiki note (via `priorProvenance`) and replace it with an SL source.
@ -177,7 +177,8 @@ semantic_models:
# KTX overlay at <connId>/orders.yaml:
# <!-- from: raw-sources/.../models/orders.yml#L1-10 -->
name: orders
description: Order fact table.
descriptions:
user: Order fact table.
measures:
- {name: order_count, expr: "count(order_id)"}
- {name: gross_amount, expr: "sum(amount)"}
@ -221,7 +222,8 @@ metrics:
# <!-- from: raw-sources/.../models/orders_ext.yml#L1-8 -->
# <!-- from: raw-sources/.../metrics/orders_final.yml#L1-10 -->
name: orders_ext
description: Extended order fact including refund handling; `revenue` = gross - refund.
descriptions:
user: Extended order fact including refund handling; `revenue` = gross - refund.
measures:
- {name: order_count, expr: "count(order_id)"}
- {name: gross_amount, expr: "sum(amount)"}
@ -288,7 +290,7 @@ measures:
- {name: margin, expr: "sum(revenue_cents) - sum(cost_cents)"}
```
Also write a wiki page at `knowledge/global/margin-metric.md` explaining the cross-source origin.
Also write a wiki page at `wiki/global/margin-metric.md` explaining the cross-source origin.
## Example 4 — filtered metric creates a new measure

3
packages/context/skills/sl/SKILL.md
View file

@ -29,7 +29,8 @@ Enrich a manifest-backed table with measures, computed columns, joins, and segme
```yaml
name: fct_orders # must match an existing manifest table
description: "Overlay adding business measures to the orders fact table."
descriptions:
user: "Overlay adding business measures to the orders fact table."
measures:
- name: total_revenue
expr: sum(amount)

11
packages/context/skills/sl_capture/SKILL.md
View file

@ -100,13 +100,13 @@ measures:
**Extract repeated filter bundles into named segments.** If the same predicate appears on multiple measures of the same source, lift it to a `segments[]` entry and have each measure reference it. One edit updates every measure that depends on it.
**Never write a standalone file on a manifest-backed name.** If `sl_discover({ tableName })` finds an existing schema for that name, you MUST write an overlay (`name:` + `measures:`/`segments:`/`description:` only — no `sql:`, `table:`, `grain:`, `columns:`, `joins:`). A standalone with `sql:` or `table:` on a manifest-backed name clobbers the inherited columns and joins; `sl_write_source` and `sl_validate` both reject this shape with a clear fix hint. Always run `sl_discover` before your first write on any existing name.
**Never write a standalone file on a manifest-backed name.** If `sl_discover({ tableName })` finds an existing schema for that name, you MUST write an overlay (`name:` + `measures:`/`segments:`/`descriptions:` only — no `sql:`, `table:`, `grain:`, `columns:`, `joins:`). A standalone with `sql:` or `table:` on a manifest-backed name clobbers the inherited columns and joins; `sl_write_source` and `sl_validate` both reject this shape with a clear fix hint. Always run `sl_discover` before your first write on any existing name.
**Prefer overlay decomposition over standalone SQL sources.** Before reaching for `source_type: sql`, check whether the metric decomposes into measures on existing overlays (including cross-source derived measures). Use `source_type: sql` only when:
- The metric requires per-user/per-entity derivation that cannot be expressed as a single `expr` (e.g., `EXISTS` over a time-windowed subset), OR
- The metric requires multi-step CTEs whose intermediate grain is not a column in any existing source.
When an `sql` source is unavoidable, note in its `description` which SL gap forced the choice so it can be retired once the primitive ships. It must target a name NOT in the manifest — pick a distinct one (e.g. `mrr_waterfall_rollup`, not `fct_orders`).
When an `sql` source is unavoidable, note in its `descriptions` map which SL gap forced the choice so it can be retired once the primitive ships. It must target a name NOT in the manifest — pick a distinct one (e.g. `mrr_waterfall_rollup`, not `fct_orders`).
## Slim standalone sources via `inherits_columns_from`
@ -116,7 +116,8 @@ Discover the manifest key with `sl_discover` — pass the bare name (`CONSIGNMEN
```yaml
name: aav_consignments
description: AAV consignments — filtered view of MARTS.CONSIGNMENTS for the auto-auction-vaulting channel.
descriptions:
user: AAV consignments — filtered view of MARTS.CONSIGNMENTS for the auto-auction-vaulting channel.
source_type: sql
sql: |
SELECT CONSIGNED_ITEM_ID, CASH_ADV_AMOUNT, ALT_VALUE_COMBINED, my_derived_flag
@ -127,10 +128,10 @@ sql: |
inherits_columns_from: CONSIGNMENTS
grain: [CONSIGNED_ITEM_ID]
columns:
- { name: CONSIGNED_ITEM_ID } # type/description inherited from manifest
- { name: CONSIGNED_ITEM_ID } # type/descriptions inherited from manifest
- { name: CASH_ADV_AMOUNT }
- { name: ALT_VALUE_COMBINED }
- { name: my_derived_flag, type: boolean, expr: "CASH_ADV_AMOUNT > 0", description: "Computed locally — has any cash advance." }
- { name: my_derived_flag, type: boolean, expr: "CASH_ADV_AMOUNT > 0", descriptions: { user: "Computed locally — has any cash advance." } }
measures:
- name: total_cash_advance
expr: sum(CASH_ADV_AMOUNT)

6
packages/context/skills/knowledge_capture/SKILL.md → packages/context/skills/wiki_capture/SKILL.md
View file

@ -1,10 +1,10 @@
---
name: knowledge_capture
description: KTX's knowledge base — wiki pages for durable, reusable business knowledge. Covers capture workflow for user preferences, metric definitions, organizational conventions, and cross-references between knowledge pages and semantic-layer sources. Loaded by the post-turn memory-agent only. The research agent reads wiki via `wiki_read`/`wiki_search` but does not write it.
name: wiki_capture
description: KTX's knowledge base — wiki pages for durable, reusable business knowledge. Covers capture workflow for user preferences, metric definitions, organizational conventions, and cross-references between wiki pages and semantic-layer sources. Loaded by the post-turn memory-agent only. The research agent reads wiki via `wiki_read`/`wiki_search` but does not write it.
callers: [memory_agent]
---
# Knowledge Capture
# Wiki Capture
## Role

4
packages/context/src/ingest/action-identity.test.ts
View file

@ -19,13 +19,13 @@ describe('memory action target identity', () => {
{
target: 'wiki',
type: 'created',
key: 'knowledge/global/orders.md',
key: 'wiki/global/orders.md',
detail: '',
targetConnectionId: 'ignored',
},
'looker-run',
),
).toBe('wiki:looker-run:knowledge/global/orders.md');
).toBe('wiki:looker-run:wiki/global/orders.md');
});
it('resolves action target connection only for SL actions', () => {

2
packages/context/src/ingest/adapters/dbt/dbt.adapter.ts
View file

@ -16,7 +16,7 @@ interface DbtSourceAdapterOptions {
export class DbtSourceAdapter implements SourceAdapter {
readonly source = 'dbt' as const;
/** Runner merges: ingest_triage, sl_capture, knowledge_capture (see ingest-bundle.runner.ts) */
/** Runner merges: ingest_triage, sl_capture, wiki_capture (see ingest-bundle.runner.ts) */
readonly skillNames: string[] = ['dbt_ingest'];
constructor(private readonly options: DbtSourceAdapterOptions = {}) {}

4
packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.test.ts
View file

@ -13,7 +13,7 @@ async function tempDir(): Promise<string> {
const sqlAnalysis: SqlAnalysisPort = {
async analyzeForFingerprint() {
throw new Error('legacy analyzeForFingerprint must not be used');
throw new Error('analyzeForFingerprint must not be used');
},
async analyzeBatch() {
return new Map();
@ -66,7 +66,7 @@ describe('HistoricSqlSourceAdapter', () => {
};
const batchSqlAnalysis: SqlAnalysisPort = {
async analyzeForFingerprint() {
throw new Error('legacy analyzeForFingerprint must not be used');
throw new Error('analyzeForFingerprint must not be used');
},
async analyzeBatch() {
return new Map([

7
packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.ts
View file

@ -1,5 +1,3 @@
import { rm } from 'node:fs/promises';
import { join } from 'node:path';
import type { ChunkResult, DiffSet, FetchContext, ScopeDescriptor, SourceAdapter } from '../../types.js';
import { chunkHistoricSqlUnifiedStagedDir, describeHistoricSqlUnifiedScope } from './chunk-unified.js';
import { detectHistoricSqlStagedDir } from './detect.js';
@ -28,11 +26,6 @@ export class HistoricSqlSourceAdapter implements SourceAdapter {
pullConfig,
now: this.deps.now?.(),
});
if (this.deps.legacyPostgresBaselineRootDir) {
await rm(join(this.deps.legacyPostgresBaselineRootDir, ctx.connectionId, ['pgss', 'baseline.json'].join('-')), {
force: true,
});
}
}
chunk(stagedDir: string, diffSet?: DiffSet): Promise<ChunkResult> {

2
packages/context/src/ingest/adapters/historic-sql/local-ingest-acceptance.test.ts
View file

@ -277,7 +277,7 @@ describe('historic-SQL local ingest retrieval acceptance', () => {
await expect(readFile(join(project.projectDir, 'semantic-layer/warehouse/_schema/public.yaml'), 'utf-8')).resolves
.toContain('Analysts repeatedly inspect paid order lifecycle by customer segment.');
await expect(readFile(join(project.projectDir, 'knowledge/global/historic-sql-paid-order-lifecycle.md'), 'utf-8'))
await expect(readFile(join(project.projectDir, 'wiki/global/historic-sql-paid-order-lifecycle.md'), 'utf-8'))
.resolves.toContain('Paid Order Lifecycle');
const reloaded = await loadKtxProject({ projectDir: project.projectDir });

2
packages/context/src/ingest/adapters/historic-sql/post-processor.ts
View file

@ -10,7 +10,7 @@ async function commitProjectionChanges(workdir: string): Promise<void> {
const status = await git.status();
const paths = status.files
.map((file) => file.path)
.filter((path) => path.startsWith('semantic-layer/') || path.startsWith('knowledge/global/historic-sql'));
.filter((path) => path.startsWith('semantic-layer/') || path.startsWith('wiki/global/historic-sql'));
if (paths.length === 0) {
return;
}

33
packages/context/src/ingest/adapters/historic-sql/projection.test.ts
View file

@ -106,7 +106,7 @@ describe('projectHistoricSqlEvidence', () => {
await writeJson(workdir, 'raw-sources/warehouse/historic-sql/sync-1/tables/public.customers.json', { table: 'public.customers' });
await writeText(
workdir,
'knowledge/global/historic-sql-old-order-lifecycle.md',
'wiki/global/historic-sql-old-order-lifecycle.md',
[
'---',
YAML.stringify({
@ -127,7 +127,7 @@ describe('projectHistoricSqlEvidence', () => {
);
await writeText(
workdir,
'knowledge/global/historic-sql-retired-pattern.md',
'wiki/global/historic-sql-retired-pattern.md',
[
'---',
YAML.stringify({
@ -164,10 +164,10 @@ describe('projectHistoricSqlEvidence', () => {
const result = await projectHistoricSqlEvidence({ workdir, connectionId: 'warehouse', syncId: 'sync-1', runId: 'run-1' });
expect(result.patternPagesWritten).toBe(1);
await expect(readFile(join(workdir, 'knowledge/global/historic-sql-old-order-lifecycle.md'), 'utf-8')).resolves.toContain(
await expect(readFile(join(workdir, 'wiki/global/historic-sql-old-order-lifecycle.md'), 'utf-8')).resolves.toContain(
'Order Lifecycle Analysis',
);
await expect(readFile(join(workdir, 'knowledge/global/historic-sql-retired-pattern.md'), 'utf-8')).resolves.toContain(
await expect(readFile(join(workdir, 'wiki/global/historic-sql-retired-pattern.md'), 'utf-8')).resolves.toContain(
'stale_since: "2026-05-11T00:00:00.000Z"',
);
});
@ -192,7 +192,7 @@ describe('projectHistoricSqlEvidence', () => {
await writeJson(workdir, 'raw-sources/warehouse/historic-sql/sync-1/tables/public.customers.json', { table: 'public.customers' });
await writeText(
workdir,
'knowledge/global/historic-sql-order-lifecycle-analysis.md',
'wiki/global/historic-sql-order-lifecycle-analysis.md',
[
'---',
YAML.stringify({
@ -230,7 +230,7 @@ describe('projectHistoricSqlEvidence', () => {
const result = await projectHistoricSqlEvidence({ workdir, connectionId: 'warehouse', syncId: 'sync-1', runId: 'run-1' });
expect(result.patternPagesWritten).toBe(1);
const page = await readFile(join(workdir, 'knowledge/global/historic-sql-order-lifecycle-analysis.md'), 'utf-8');
const page = await readFile(join(workdir, 'wiki/global/historic-sql-order-lifecycle-analysis.md'), 'utf-8');
expect(page).toContain('Analysts compare order status with customer segment again.');
expect(page).not.toContain('Archived body');
expect(page).not.toContain('archived');
@ -254,7 +254,7 @@ describe('projectHistoricSqlEvidence', () => {
});
await writeText(
workdir,
'knowledge/global/historic-sql-retired-pattern.md',
'wiki/global/historic-sql-retired-pattern.md',
[
'---',
YAML.stringify({
@ -279,12 +279,12 @@ describe('projectHistoricSqlEvidence', () => {
expect(result.archivedPatternPages).toBe(0);
expect(result.stalePatternPagesMarked).toBe(0);
await expect(readFile(join(workdir, 'knowledge/global/historic-sql-retired-pattern.md'), 'utf-8')).resolves.toContain(
await expect(readFile(join(workdir, 'wiki/global/historic-sql-retired-pattern.md'), 'utf-8')).resolves.toContain(
'Archived retired body',
);
});
it('marks missing table usage stale and deletes legacy historic SQL query pages', async () => {
it('marks missing table usage stale without deleting old query pages', async () => {
const workdir = await tempWorkdir();
await writeText(
workdir,
@ -322,22 +322,22 @@ describe('projectHistoricSqlEvidence', () => {
});
await writeText(
workdir,
'knowledge/global/historic-sql-legacy-template.md',
'wiki/global/historic-sql-old-template.md',
[
'---',
YAML.stringify({
summary: 'Legacy template page',
summary: 'Old template page',
tags: ['historic-sql', 'query-pattern'],
refs: [],
sl_refs: ['orders'],
usage_mode: 'auto',
source: 'historic-sql',
tables: ['public.orders'],
fingerprints: ['legacy:1'],
fingerprints: ['old:1'],
}).trimEnd(),
'---',
'',
'Legacy body',
'Old body',
'',
].join('\n'),
);
@ -345,7 +345,6 @@ describe('projectHistoricSqlEvidence', () => {
const result = await projectHistoricSqlEvidence({ workdir, connectionId: 'warehouse', syncId: 'sync-1', runId: 'run-1' });
expect(result.staleTablesMarked).toBe(1);
expect(result.legacyPagesDeleted).toBe(1);
expect(result.touchedSources).toEqual([{ connectionId: 'warehouse', sourceName: 'orders' }]);
const shard = YAML.parse(await readFile(join(workdir, 'semantic-layer/warehouse/_schema/public.yaml'), 'utf-8'));
expect(shard.tables.orders.usage).toEqual({
@ -357,8 +356,8 @@ describe('projectHistoricSqlEvidence', () => {
commonJoins: [],
staleSince: '2026-05-11T00:00:00.000Z',
});
await expect(readFile(join(workdir, 'knowledge/global/historic-sql-legacy-template.md'), 'utf-8')).rejects.toMatchObject({
code: 'ENOENT',
});
await expect(readFile(join(workdir, 'wiki/global/historic-sql-old-template.md'), 'utf-8')).resolves.toContain(
'Old body',
);
});
});

16
packages/context/src/ingest/adapters/historic-sql/projection.ts
View file

@ -1,4 +1,4 @@
import { access, mkdir, readdir, readFile, rename, rm, writeFile } from 'node:fs/promises';
import { access, mkdir, readdir, readFile, rename, writeFile } from 'node:fs/promises';
import { dirname, join, relative } from 'node:path';
import YAML from 'yaml';
import { rawSourcesDirForSync } from '../../raw-sources-paths.js';
@ -20,7 +20,6 @@ export interface HistoricSqlProjectionResult {
patternPagesWritten: number;
stalePatternPagesMarked: number;
archivedPatternPages: number;
legacyPagesDeleted: number;
touchedSources: Array<{ connectionId: string; sourceName: string }>;
warnings: string[];
}
@ -152,11 +151,6 @@ function isHistoricPatternPage(page: HistoricSqlPatternPage): boolean {
);
}
function isLegacyQueryPage(page: HistoricSqlPatternPage): boolean {
const tags = Array.isArray(page.frontmatter.tags) ? page.frontmatter.tags : [];
return page.frontmatter.source === 'historic-sql' && tags.includes('query-pattern') && !tags.includes('pattern');
}
function isArchivedPatternPage(page: HistoricSqlPatternPage): boolean {
const tags = Array.isArray(page.frontmatter.tags) ? page.frontmatter.tags : [];
return tags.includes('archived');
@ -228,7 +222,6 @@ export async function projectHistoricSqlEvidence(input: HistoricSqlProjectionInp
patternPagesWritten: 0,
stalePatternPagesMarked: 0,
archivedPatternPages: 0,
legacyPagesDeleted: 0,
touchedSources: [],
warnings: [],
};
@ -283,7 +276,7 @@ export async function projectHistoricSqlEvidence(input: HistoricSqlProjectionInp
}
}
const wikiRoot = join(input.workdir, 'knowledge/global');
const wikiRoot = join(input.workdir, 'wiki/global');
await mkdir(wikiRoot, { recursive: true });
const allPages = await loadPatternPages(wikiRoot);
const activePages = allPages.filter((page) => !isArchivedPatternPage(page));
@ -333,10 +326,5 @@ export async function projectHistoricSqlEvidence(input: HistoricSqlProjectionInp
result.stalePatternPagesMarked += 1;
}
for (const page of allPages.filter(isLegacyQueryPage)) {
await rm(page.path, { force: true });
result.legacyPagesDeleted += 1;
}
return result;
}

12
packages/context/src/ingest/adapters/historic-sql/types.test.ts
View file

@ -8,7 +8,7 @@ import {
} from './types.js';
describe('historic-sql unified contracts', () => {
it('parses minExecutions and accepts minCalls as a one-release alias', () => {
it('parses minExecutions and service-account filters', () => {
expect(historicSqlUnifiedPullConfigSchema.parse({ dialect: 'postgres', minExecutions: 9 })).toMatchObject({
dialect: 'postgres',
minExecutions: 9,
@ -18,7 +18,15 @@ describe('historic-sql unified contracts', () => {
staleArchiveAfterDays: 90,
});
expect(historicSqlUnifiedPullConfigSchema.parse({ dialect: 'postgres', minCalls: 7 }).minExecutions).toBe(7);
const parsed = historicSqlUnifiedPullConfigSchema.parse({
dialect: 'postgres',
minExecutions: 7,
filters: {
serviceAccounts: { patterns: ['^svc_'], mode: 'exclude' },
},
});
expect(parsed.minExecutions).toBe(7);
expect(parsed.filters.serviceAccounts).toEqual({ patterns: ['^svc_'], mode: 'exclude' });
});
it('validates aggregate templates from warehouse readers', () => {

24
packages/context/src/ingest/adapters/historic-sql/types.ts
View file

@ -8,26 +8,7 @@ export type HistoricSqlDialect = z.infer<typeof historicSqlDialectSchema>;
const filterModeSchema = z.enum(['exclude', 'include', 'mark-only']);
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value);
}
export const historicSqlUnifiedPullConfigSchema = z.preprocess((value) => {
if (!isRecord(value)) {
return value;
}
const next: Record<string, unknown> = { ...value };
if (next.minExecutions === undefined && typeof next.minCalls === 'number') {
next.minExecutions = next.minCalls;
}
if (!next.filters && Array.isArray(next.serviceAccountUserPatterns)) {
next.filters = {
serviceAccounts: { patterns: next.serviceAccountUserPatterns, mode: 'exclude' },
dropTrivialProbes: true,
};
}
return next;
}, z.object({
export const historicSqlUnifiedPullConfigSchema = z.object({
dialect: historicSqlDialectSchema,
windowDays: z.number().int().positive().default(90),
minExecutions: z.number().int().nonnegative().default(5),
@ -48,7 +29,7 @@ export const historicSqlUnifiedPullConfigSchema = z.preprocess((value) => {
}).default({ dropTrivialProbes: true }),
redactionPatterns: z.array(z.string()).default([]),
staleArchiveAfterDays: z.number().int().positive().default(90),
}));
});
export type HistoricSqlUnifiedPullConfig = z.infer<typeof historicSqlUnifiedPullConfigSchema>;
@ -157,6 +138,5 @@ export interface HistoricSqlSourceAdapterDeps {
sqlAnalysis: SqlAnalysisPort;
reader: HistoricSqlReader;
queryClient: unknown;
legacyPostgresBaselineRootDir?: string;
now?: () => Date;
}

8
packages/context/src/ingest/adapters/looker/local-looker.adapter.ts
View file

@ -26,13 +26,11 @@ export function lookerCredentialsFromLocalConnection(
if (!connection || String(connection.driver).toLowerCase() !== 'looker') {
throw new Error(`Connection "${connectionId}" is not a Looker connection`);
}
const baseUrl = stringField(connection.base_url) ?? stringField(connection.baseUrl) ?? stringField(connection.url);
const clientId = stringField(connection.client_id) ?? stringField(connection.clientId);
const baseUrl = stringField(connection.base_url);
const clientId = stringField(connection.client_id);
const clientSecret =
stringField(connection.client_secret) ??
stringField(connection.clientSecret) ??
(stringField(connection.client_secret_ref) ? resolveEnvReference(String(connection.client_secret_ref), env) : null) ??
(stringField(connection.clientSecretRef) ? resolveEnvReference(String(connection.clientSecretRef), env) : null);
(stringField(connection.client_secret_ref) ? resolveEnvReference(String(connection.client_secret_ref), env) : null);
if (!baseUrl) {
throw new Error(`Connection "${connectionId}" is missing Looker base_url`);

11
packages/context/src/ingest/adapters/metabase/client-port.test.ts
View file

@ -87,10 +87,13 @@ it('allows the concrete client result shapes used by the relocated Metabase clie
const datasetQuery: MetabaseDatasetQuery = {
type: 'native',
database: 42,
native: {
query: 'SELECT * FROM orders WHERE created_at > {{ created_at }}',
'template-tags': { created_at: templateTag },
},
stages: [
{
'lib/type': 'mbql.stage/native',
native: 'SELECT * FROM orders WHERE created_at > {{ created_at }}',
'template-tags': { created_at: templateTag },
},
],
};
const card: MetabaseCard = {
id: 1,

2
packages/context/src/ingest/adapters/metabase/client-port.ts
View file

@ -117,7 +117,7 @@ interface MetabaseNativeStage {
}
interface MetabaseLegacyNativeQuery {
query: string;
query?: string;
'template-tags'?: Record<string, MetabaseTemplateTag>;
}

Some files were not shown because too many files have changed in this diff Show more