mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-07 07:55:13 +02:00
Polish documentation copy
This commit is contained in:
Luca Martial
parent
ce23aca4c4
commit
5568b3d37a
65 changed files with 478 additions and 478 deletions
10
README.md
10
README.md
|
|
@ -19,14 +19,14 @@ KTX turns warehouse metadata, semantic definitions, and business knowledge into
|
|||
reviewable project files that agents can use while planning, querying, and
|
||||
updating analytics work.
|
||||
|
||||
A KTX project is a directory of plain files — YAML semantic sources, Markdown
|
||||
wiki pages, and SQLite state — that you commit to git and review in PRs,
|
||||
A KTX project is a directory of plain files - YAML semantic sources, Markdown
|
||||
wiki pages, and SQLite state - that you commit to git and review in PRs,
|
||||
just like dbt models.
|
||||
|
||||
## Who KTX is for
|
||||
|
||||
KTX is built for analytics engineers and data teams who want data agents to
|
||||
work on real analytics systems — not just generate one-off SQL.
|
||||
work on real analytics systems - not just generate one-off SQL.
|
||||
|
||||
Use KTX when you want agents to:
|
||||
|
||||
|
|
@ -120,7 +120,7 @@ my-project/
|
|||
```
|
||||
|
||||
Semantic sources and wiki pages are committed to git. The `.ktx/` directory
|
||||
holds ephemeral state and is git-ignored — delete it and KTX rebuilds on the
|
||||
holds ephemeral state and is git-ignored - delete it and KTX rebuilds on the
|
||||
next run.
|
||||
|
||||
### Build demo warehouse context
|
||||
|
|
@ -163,7 +163,7 @@ source packages for development, not public release artifacts.
|
|||
KTX integrates with coding agents through CLI skills. The setup wizard
|
||||
configures this automatically.
|
||||
|
||||
**CLI skills** — the agent calls `ktx` commands directly through a skill file
|
||||
**CLI skills** - the agent calls `ktx` commands directly through a skill file
|
||||
installed in your agent's config (e.g., `.claude/skills/ktx/SKILL.md`):
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
KTX Light Theme — Warm Cream & Taupe
|
||||
KTX Light Theme - Warm Cream & Taupe
|
||||
═══════════════════════════════════════════ */
|
||||
:root {
|
||||
--color-fd-background: #faf9f6;
|
||||
|
|
@ -42,7 +42,7 @@
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
KTX Dark Theme — Deep Ocean Slate
|
||||
KTX Dark Theme - Deep Ocean Slate
|
||||
═══════════════════════════════════════════ */
|
||||
.dark {
|
||||
--color-fd-background: #0f1719;
|
||||
|
|
@ -79,7 +79,7 @@ body {
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
Typography — Outfit display, Inter body
|
||||
Typography - Outfit display, Inter body
|
||||
═══════════════════════════════════════════ */
|
||||
h1, h2, h3, h4 {
|
||||
font-family: var(--font-display), var(--font-sans), sans-serif;
|
||||
|
|
@ -114,7 +114,7 @@ h2 {
|
|||
border-color: rgba(255, 255, 255, 0.08) !important;
|
||||
}
|
||||
|
||||
/* Code blocks — give them a subtle traffic-light feel */
|
||||
/* Code blocks - give them a subtle traffic-light feel */
|
||||
figure[data-rehype-pretty-code-figure],
|
||||
figure:has(> pre) {
|
||||
position: relative;
|
||||
|
|
@ -166,7 +166,7 @@ pre {
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
Code blocks — context-aware modes
|
||||
Code blocks - context-aware modes
|
||||
═══════════════════════════════════════════ */
|
||||
|
||||
/* Shared wrapper base */
|
||||
|
|
@ -504,7 +504,7 @@ th {
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
Sidebar — Typographic sections + active rail
|
||||
Sidebar - Typographic sections + active rail
|
||||
═══════════════════════════════════════════ */
|
||||
#nd-sidebar {
|
||||
border-right: 1px solid var(--color-fd-border);
|
||||
|
|
@ -516,7 +516,7 @@ th {
|
|||
backdrop-filter: blur(10px);
|
||||
}
|
||||
|
||||
/* Section folder trigger — uppercase tracked label
|
||||
/* Section folder trigger - uppercase tracked label
|
||||
Fumadocs 15 section wrappers are bare <div data-state> (no class, no id);
|
||||
content panels and other Radix collapsibles always carry a class attribute,
|
||||
so :not([class]) tightly scopes these rules to section triggers only. */
|
||||
|
|
@ -601,7 +601,7 @@ th {
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
Cards — refined with multi-layer shadow & lift
|
||||
Cards - refined with multi-layer shadow & lift
|
||||
═══════════════════════════════════════════ */
|
||||
[data-card="true"] {
|
||||
border-radius: 12px !important;
|
||||
|
|
@ -683,7 +683,7 @@ th {
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
Page title area — give docs pages a hero feel
|
||||
Page title area - give docs pages a hero feel
|
||||
═══════════════════════════════════════════ */
|
||||
[data-page-header] h1,
|
||||
article > h1:first-of-type {
|
||||
|
|
@ -724,7 +724,7 @@ article a:not([data-card]):hover {
|
|||
}
|
||||
|
||||
/* ═══════════════════════════════════════════
|
||||
Background atmosphere — gradient blobs (subtle)
|
||||
Background atmosphere - gradient blobs (subtle)
|
||||
═══════════════════════════════════════════ */
|
||||
body::before {
|
||||
content: "";
|
||||
|
|
@ -973,7 +973,7 @@ body > * {
|
|||
100% { left: 200%; }
|
||||
}
|
||||
|
||||
/* Glow text — use sparingly on hero key phrase */
|
||||
/* Glow text - use sparingly on hero key phrase */
|
||||
.glow-text {
|
||||
position: relative;
|
||||
color: var(--color-fd-primary);
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ export function CodeBlock(props: Props) {
|
|||
const isOutput = !isTerminal && WIZARD_GLYPHS.test(codeText);
|
||||
const hasTitle = typeof title === "string" && title.length > 0;
|
||||
|
||||
// Mode A — Terminal (commands the user types)
|
||||
// Mode A - Terminal (commands the user types)
|
||||
if (isTerminal) {
|
||||
return (
|
||||
<div className="not-prose ktx-code ktx-code-terminal group">
|
||||
|
|
@ -77,7 +77,7 @@ export function CodeBlock(props: Props) {
|
|||
);
|
||||
}
|
||||
|
||||
// Mode D — Output preview (wizard prompts, terminal output)
|
||||
// Mode D - Output preview (wizard prompts, terminal output)
|
||||
if (isOutput) {
|
||||
return (
|
||||
<div className="not-prose ktx-code ktx-code-output group relative">
|
||||
|
|
@ -90,7 +90,7 @@ export function CodeBlock(props: Props) {
|
|||
);
|
||||
}
|
||||
|
||||
// Mode B — VS Code tab (filename present)
|
||||
// Mode B - VS Code tab (filename present)
|
||||
if (hasTitle) {
|
||||
return (
|
||||
<div className="not-prose ktx-code ktx-code-tab group">
|
||||
|
|
@ -107,7 +107,7 @@ export function CodeBlock(props: Props) {
|
|||
);
|
||||
}
|
||||
|
||||
// Mode C — Minimal default
|
||||
// Mode C - Minimal default
|
||||
return (
|
||||
<div className="not-prose ktx-code ktx-code-minimal group relative">
|
||||
{language && <span className="ktx-code-minimal-lang">{language}</span>}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ export function CopyButton({ text, className = "" }: Props) {
|
|||
setCopied(true);
|
||||
setTimeout(() => setCopied(false), 1500);
|
||||
} catch {
|
||||
// Older browsers or denied permission — fail silently
|
||||
// Older browsers or denied permission - fail silently
|
||||
}
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -19,14 +19,14 @@ ktx setup [options]
|
|||
|------|-------------|---------|
|
||||
| `--project-dir <path>` | KTX project directory | `KTX_PROJECT_DIR`, nearest `ktx.yaml`, or cwd |
|
||||
| `--yes` | Accept safe defaults in non-interactive setup | `false` |
|
||||
| `--no-input` | Disable interactive terminal input | — |
|
||||
| `--no-input` | Disable interactive terminal input | - |
|
||||
|
||||
### Agent Integration
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--agents` | Install agent integration only | `false` |
|
||||
| `--target <target>` | Agent target (`claude-code`, `codex`, `cursor`, `opencode`, `universal`) | — |
|
||||
| `--target <target>` | Agent target (`claude-code`, `codex`, `cursor`, `opencode`, `universal`) | - |
|
||||
| `--global` | Install agent integration into the global target scope (Claude Code and Codex only) | `false` |
|
||||
|
||||
The setup wizard is the public configuration interface. It prompts for LLM
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: "ktx sl"
|
|||
description: "List, search, validate, or query semantic-layer sources."
|
||||
---
|
||||
|
||||
Interact with your project's semantic layer. Semantic sources are YAML definitions that describe your tables, columns, measures, joins, and grain — the vocabulary agents use to generate correct SQL.
|
||||
Interact with your project's semantic layer. Semantic sources are YAML definitions that describe your tables, columns, measures, joins, and grain - the vocabulary agents use to generate correct SQL.
|
||||
|
||||
## Command signature
|
||||
|
||||
|
|
@ -26,7 +26,7 @@ ktx sl <subcommand> [options]
|
|||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection-id <id>` | Filter by KTX connection id | — |
|
||||
| `--connection-id <id>` | Filter by KTX connection id | - |
|
||||
| `--output <mode>` | Output mode: `pretty` (default in TTY), `plain` (TSV), or `json` | `pretty` |
|
||||
| `--json` | Shortcut for `--output=json` (overrides `--output`) | `false` |
|
||||
|
||||
|
|
@ -34,8 +34,8 @@ ktx sl <subcommand> [options]
|
|||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection-id <id>` | Filter by KTX connection id | — |
|
||||
| `--limit <number>` | Maximum search results | — |
|
||||
| `--connection-id <id>` | Filter by KTX connection id | - |
|
||||
| `--limit <number>` | Maximum search results | - |
|
||||
| `--output <mode>` | Output mode: `pretty` (default in TTY), `plain` (TSV), or `json` | `pretty` |
|
||||
| `--json` | Shortcut for `--output=json` (overrides `--output`) | `false` |
|
||||
|
||||
|
|
@ -43,24 +43,24 @@ ktx sl <subcommand> [options]
|
|||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection-id <id>` | KTX connection id (required) | — |
|
||||
| `--connection-id <id>` | KTX connection id (required) | - |
|
||||
|
||||
### `sl query`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection-id <id>` | KTX connection id | — |
|
||||
| `--query-file <path>` | JSON semantic-layer query file | — |
|
||||
| `--measure <measure>` | Measure to query; repeatable (at least one required) | — |
|
||||
| `--dimension <dimension>` | Dimension to include; repeatable | — |
|
||||
| `--filter <filter>` | Filter expression; repeatable | — |
|
||||
| `--segment <segment>` | Segment to include; repeatable | — |
|
||||
| `--order-by <field[:direction]>` | Order field, optionally suffixed with `:asc` or `:desc`; repeatable | — |
|
||||
| `--limit <n>` | Query limit | — |
|
||||
| `--connection-id <id>` | KTX connection id | - |
|
||||
| `--query-file <path>` | JSON semantic-layer query file | - |
|
||||
| `--measure <measure>` | Measure to query; repeatable (at least one required) | - |
|
||||
| `--dimension <dimension>` | Dimension to include; repeatable | - |
|
||||
| `--filter <filter>` | Filter expression; repeatable | - |
|
||||
| `--segment <segment>` | Segment to include; repeatable | - |
|
||||
| `--order-by <field[:direction]>` | Order field, optionally suffixed with `:asc` or `:desc`; repeatable | - |
|
||||
| `--limit <n>` | Query limit | - |
|
||||
| `--include-empty` | Include empty rows | `false` |
|
||||
| `--format <format>` | Output format: `json` or `sql` | `json` |
|
||||
| `--execute` | Execute the compiled query against the database | `false` |
|
||||
| `--max-rows <n>` | Maximum rows to return when executing | — |
|
||||
| `--max-rows <n>` | Maximum rows to return when executing | - |
|
||||
|
||||
## Examples
|
||||
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ ktx status [options]
|
|||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--no-input` | Disable interactive terminal input | — |
|
||||
| `--no-input` | Disable interactive terminal input | - |
|
||||
|
||||
## Examples
|
||||
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ ktx wiki <subcommand> [options]
|
|||
|------|-------------|---------|
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--user-id <id>` | Local user id | `local` |
|
||||
| `--limit <number>` | Maximum search results | — |
|
||||
| `--limit <number>` | Maximum search results | - |
|
||||
|
||||
## Examples
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: Contributing
|
|||
description: How to contribute to KTX.
|
||||
---
|
||||
|
||||
KTX is an open-source project and welcomes contributions — bug fixes, new connectors, documentation improvements, and feature proposals. This page covers how to set up a development environment, navigate the repository, run tests, and submit changes.
|
||||
KTX is an open-source project and welcomes contributions - bug fixes, new connectors, documentation improvements, and feature proposals. This page covers how to set up a development environment, navigate the repository, run tests, and submit changes.
|
||||
|
||||
## Development setup
|
||||
|
||||
|
|
@ -14,9 +14,9 @@ an analytics project, use the published
|
|||
|
||||
### Prerequisites
|
||||
|
||||
- **Node.js 22+** and **pnpm** — for the TypeScript workspace
|
||||
- **Python 3.11+** and **uv** — for the Python semantic layer and daemon
|
||||
- **Git** — for version control
|
||||
- **Node.js 22+** and **pnpm** - for the TypeScript workspace
|
||||
- **Python 3.11+** and **uv** - for the Python semantic layer and daemon
|
||||
- **Git** - for version control
|
||||
|
||||
### Clone and install
|
||||
|
||||
|
|
@ -72,8 +72,8 @@ packages/
|
|||
connector-posthog/ # PostHog connector
|
||||
|
||||
python/
|
||||
ktx-sl/ # Semantic layer — grain-aware query planning and SQL generation
|
||||
ktx-daemon/ # Daemon — portable API server around the semantic layer
|
||||
ktx-sl/ # Semantic layer - grain-aware query planning and SQL generation
|
||||
ktx-daemon/ # Daemon - portable API server around the semantic layer
|
||||
|
||||
examples/ # Example projects and fixtures
|
||||
scripts/ # Workspace scripts (benchmarks, verification, release)
|
||||
|
|
@ -179,17 +179,17 @@ The `package.json` should follow the pattern of existing connectors:
|
|||
|
||||
Your connector class must implement `KtxScanConnector`, which requires:
|
||||
|
||||
- **`id`** — a string identifier, typically `"<driver>:<connectionId>"`
|
||||
- **`driver`** — the `KtxConnectionDriver` value for your database
|
||||
- **`capabilities`** — a `KtxConnectorCapabilities` object declaring what your connector supports: `tableSampling`, `columnSampling`, `columnStats`, `readOnlySql`, `nestedAnalysis`, `eventStreamDiscovery`, `formalForeignKeys`, `estimatedRowCounts`
|
||||
- **`introspect()`** — discovers tables, columns, types, and constraints, returning a `KtxSchemaSnapshot`
|
||||
- **`id`** - a string identifier, typically `"<driver>:<connectionId>"`
|
||||
- **`driver`** - the `KtxConnectionDriver` value for your database
|
||||
- **`capabilities`** - a `KtxConnectorCapabilities` object declaring what your connector supports: `tableSampling`, `columnSampling`, `columnStats`, `readOnlySql`, `nestedAnalysis`, `eventStreamDiscovery`, `formalForeignKeys`, `estimatedRowCounts`
|
||||
- **`introspect()`** - discovers tables, columns, types, and constraints, returning a `KtxSchemaSnapshot`
|
||||
|
||||
Optional methods for richer scanning:
|
||||
|
||||
- **`sampleColumn()`** — sample values from a specific column
|
||||
- **`sampleTable()`** — sample rows from a table
|
||||
- **`columnStats()`** — compute column statistics
|
||||
- **`executeReadOnly()`** — execute arbitrary read-only SQL
|
||||
- **`sampleColumn()`** - sample values from a specific column
|
||||
- **`sampleTable()`** - sample rows from a table
|
||||
- **`columnStats()`** - compute column statistics
|
||||
- **`executeReadOnly()`** - execute arbitrary read-only SQL
|
||||
|
||||
### Step 3: Add a dialect
|
||||
|
||||
|
|
@ -212,7 +212,7 @@ Use `packages/connector-sqlite/` as a minimal reference and `packages/connector-
|
|||
## Code conventions
|
||||
|
||||
- **TypeScript**: strict types, no `any`, no `as unknown as`. Use `zod` schemas for runtime validation at CLI and config boundaries. Follow the `camelCaseSchema` / `PascalCaseType` naming convention for Zod schemas and inferred types.
|
||||
- **Python**: type hints on all new code, `pathlib` over `os.path`, explicit exception types over broad `except Exception`, `logger.exception()` for caught exceptions. Use `sqlglot` for SQL parsing — never regex.
|
||||
- **Python**: type hints on all new code, `pathlib` over `os.path`, explicit exception types over broad `except Exception`, `logger.exception()` for caught exceptions. Use `sqlglot` for SQL parsing - never regex.
|
||||
- **Dependencies**: `pnpm` for Node packages (never `npm` or `bun`), `uv` for Python (never `pip`).
|
||||
- **Dead code**: remove it. Don't leave commented-out code, unused wrappers, or empty directories.
|
||||
|
||||
|
|
@ -220,11 +220,11 @@ Use `packages/connector-sqlite/` as a minimal reference and `packages/connector-
|
|||
|
||||
Before submitting a pull request:
|
||||
|
||||
1. **Run the relevant checks** — at minimum, `pnpm run type-check` and `pnpm run test` for TypeScript changes, `uv run pytest -q` and `uv run pre-commit run --files [FILES]` for Python changes.
|
||||
2. **Build if you changed exports** — run `pnpm run build` to verify package exports and `dist/` expectations still align.
|
||||
3. **Keep changes focused** — one logical change per PR. Don't bundle unrelated refactors.
|
||||
4. **Follow existing patterns** — match the style and conventions of surrounding code. The codebase favors explicit over clever.
|
||||
5. **Don't commit artifacts** — `node_modules/`, `.venv/`, `dist/`, coverage output, and local databases should not be committed.
|
||||
1. **Run the relevant checks** - at minimum, `pnpm run type-check` and `pnpm run test` for TypeScript changes, `uv run pytest -q` and `uv run pre-commit run --files [FILES]` for Python changes.
|
||||
2. **Build if you changed exports** - run `pnpm run build` to verify package exports and `dist/` expectations still align.
|
||||
3. **Keep changes focused** - one logical change per PR. Don't bundle unrelated refactors.
|
||||
4. **Follow existing patterns** - match the style and conventions of surrounding code. The codebase favors explicit over clever.
|
||||
5. **Don't commit artifacts** - `node_modules/`, `.venv/`, `dist/`, coverage output, and local databases should not be committed.
|
||||
|
||||
For larger features or architectural changes, open an issue first to discuss the approach.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,23 +1,23 @@
|
|||
---
|
||||
title: Context as Code
|
||||
description: Treat analytics context like code — version it, review it, merge it.
|
||||
description: Treat analytics context like code - version it, review it, merge it.
|
||||
---
|
||||
|
||||
## The idea
|
||||
|
||||
dbt proved that analytics transformations belong in version control. Before dbt, SQL lived in BI tools, scheduling systems, and spreadsheets — scattered, unreviewed, impossible to audit. "Analytics as code" changed that: put your models in git, review them in PRs, deploy them by merging.
|
||||
dbt proved that analytics transformations belong in version control. Before dbt, SQL lived in BI tools, scheduling systems, and spreadsheets - scattered, unreviewed, impossible to audit. "Analytics as code" changed that: put your models in git, review them in PRs, deploy them by merging.
|
||||
|
||||
KTX applies the same principle to analytics context. Metric definitions, business rules, join relationships, wiki pages — these are artifacts that determine whether an agent produces correct results. They change over time. They need review. They need history. They need to be treated like code.
|
||||
KTX applies the same principle to analytics context. Metric definitions, business rules, join relationships, wiki pages - these are artifacts that determine whether an agent produces correct results. They change over time. They need review. They need history. They need to be treated like code.
|
||||
|
||||
A KTX project is a git repository. Semantic sources are YAML files. Wiki pages are Markdown files. Changes are commits. Updates are pull requests. Deployment is a merge. The entire lifecycle of your analytics context follows the same workflow your team already uses for dbt models, application code, and infrastructure.
|
||||
|
||||
## Auto-ingestion
|
||||
|
||||
Most analytics context already exists — it's in your dbt manifests, LookML models, Metabase questions, and team Notion pages. KTX pulls from these sources automatically through adapters.
|
||||
Most analytics context already exists - it's in your dbt manifests, LookML models, Metabase questions, and team Notion pages. KTX pulls from these sources automatically through adapters.
|
||||
|
||||
An ingestion run works like this:
|
||||
|
||||
1. **Adapters extract metadata.** Each configured source — dbt, LookML, Metabase, MetricFlow, Notion, or your live database — provides structured metadata about models, metrics, dimensions, questions, and documentation.
|
||||
1. **Adapters extract metadata.** Each configured source - dbt, LookML, Metabase, MetricFlow, Notion, or your live database - provides structured metadata about models, metrics, dimensions, questions, and documentation.
|
||||
|
||||
2. **The LLM agent reconciles.** KTX doesn't blindly overwrite existing context. An LLM agent compares incoming metadata against your current semantic sources and wiki pages. It decides what to create, what to update, and what to leave alone. If your dbt project added a new model, the agent writes a new semantic source. If a Metabase question references a metric you've already defined, the agent skips the duplicate.
|
||||
|
||||
|
|
@ -66,17 +66,17 @@ metadata, and documentation updates are ready for review each morning.
|
|||
|
||||
Once merged, agents querying through the KTX CLI see the updated context immediately. No deployment step, no cache invalidation, no restart. The files are the source of truth, and agents read them on every request.
|
||||
|
||||
This workflow gives you the same review guarantees you have for dbt models. No semantic source reaches production without a human approving it. But unlike maintaining context manually, the heavy lifting — discovering new tables, drafting source definitions, extracting business rules from documentation — is done by the ingestion agent. You review and approve. You don't write from scratch.
|
||||
This workflow gives you the same review guarantees you have for dbt models. No semantic source reaches production without a human approving it. But unlike maintaining context manually, the heavy lifting - discovering new tables, drafting source definitions, extracting business rules from documentation - is done by the ingestion agent. You review and approve. You don't write from scratch.
|
||||
|
||||
## Feedback loops
|
||||
|
||||
Context improves over time through two feedback channels.
|
||||
|
||||
**Analyst corrections.** When an analytics engineer spots something wrong — a measure formula that doesn't match the business definition, a join that should be `many_to_one` instead of `one_to_many`, a wiki page that's out of date — they edit the YAML or Markdown directly and commit. These corrections become part of the project's git history, and the next ingestion run respects them. If you manually fix a measure definition, KTX won't overwrite it on the next ingest.
|
||||
**Analyst corrections.** When an analytics engineer spots something wrong - a measure formula that doesn't match the business definition, a join that should be `many_to_one` instead of `one_to_many`, a wiki page that's out of date - they edit the YAML or Markdown directly and commit. These corrections become part of the project's git history, and the next ingestion run respects them. If you manually fix a measure definition, KTX won't overwrite it on the next ingest.
|
||||
|
||||
**Agent feedback.** When an agent queries the semantic layer and gets unexpected results — a query that returns no rows because of a bad filter, a join path that produces duplicated results — it can flag the issue. These signals feed back into the context: wiki pages can note known data quality issues, and source definitions can be tightened with better filters, join paths, or grain declarations.
|
||||
**Agent feedback.** When an agent queries the semantic layer and gets unexpected results - a query that returns no rows because of a bad filter, a join path that produces duplicated results - it can flag the issue. These signals feed back into the context: wiki pages can note known data quality issues, and source definitions can be tightened with better filters, join paths, or grain declarations.
|
||||
|
||||
Each of these channels makes the next ingestion cycle better. Analyst corrections teach the system what your team considers authoritative. Agent feedback surfaces gaps in coverage. Context is not a static artifact — it's a living system that converges toward accuracy with every iteration.
|
||||
Each of these channels makes the next ingestion cycle better. Analyst corrections teach the system what your team considers authoritative. Agent feedback surfaces gaps in coverage. Context is not a static artifact - it's a living system that converges toward accuracy with every iteration.
|
||||
|
||||
## Deterministic replay
|
||||
|
||||
|
|
@ -84,9 +84,9 @@ Every ingestion session in KTX produces a full transcript: every tool call the L
|
|||
|
||||
This matters for three reasons.
|
||||
|
||||
**Debugging.** When a semantic source looks wrong — the grain is off, a join points to the wrong table, a measure formula doesn't match the business definition — you can trace it back to the ingestion session that created it. The transcript shows exactly which adapter provided the input, how the LLM interpreted it, and why it made the decision it did. You don't have to guess.
|
||||
**Debugging.** When a semantic source looks wrong - the grain is off, a join points to the wrong table, a measure formula doesn't match the business definition - you can trace it back to the ingestion session that created it. The transcript shows exactly which adapter provided the input, how the LLM interpreted it, and why it made the decision it did. You don't have to guess.
|
||||
|
||||
**Trust.** Analytics teams need to trust the context that agents consume. Deterministic replay means you can verify any part of the context layer by re-examining the session that produced it. If a stakeholder asks "where did this revenue definition come from?", you have a complete audit trail — from the dbt manifest entry, through the LLM's reconciliation logic, to the YAML file that was written.
|
||||
**Trust.** Analytics teams need to trust the context that agents consume. Deterministic replay means you can verify any part of the context layer by re-examining the session that produced it. If a stakeholder asks "where did this revenue definition come from?", you have a complete audit trail - from the dbt manifest entry, through the LLM's reconciliation logic, to the YAML file that was written.
|
||||
|
||||
**Reproducibility.** Because ingestion sessions are recorded as structured transcripts (tool calls and responses, not just logs), they can be replayed for testing and validation. If you change your ingestion configuration or upgrade the LLM, you can replay previous sessions to see how the output would differ. This gives you a safety net for changes that affect how context is generated.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ description: What a context layer is, why agents need one, and how KTX compares
|
|||
|
||||
## The problem
|
||||
|
||||
Give an agent access to your database and it will generate SQL. It might even produce a decent chart. But ask it a real analytics question — "what's our net revenue trend by segment?" — and things fall apart.
|
||||
Give an agent access to your database and it will generate SQL. It might even produce a decent chart. But ask it a real analytics question - "what's our net revenue trend by segment?" - and things fall apart.
|
||||
|
||||
The agent doesn't know that `orders.amount` includes refunds and needs a status filter. It doesn't know that `customers` should join to `orders` on `customer_id`, not `id`. It doesn't know that your team stopped using `legacy_segments` six months ago, or that "enterprise" means contracts over $100k, not just big logos. It sees column names and types. It doesn't see your business.
|
||||
|
||||
|
|
@ -17,15 +17,15 @@ Analytics engineers already know this pain. It's the same reason you write dbt t
|
|||
|
||||
The industry has moved through three distinct approaches to getting AI and data to work together.
|
||||
|
||||
**Wave one: database access.** Connect an LLM to a database, let it generate SQL. This works for simple lookups — "how many orders last week?" — but breaks on anything that requires business knowledge. The agent guesses at joins, invents metrics, and hallucinates table relationships. Every query is a coin flip.
|
||||
**Wave one: database access.** Connect an LLM to a database, let it generate SQL. This works for simple lookups - "how many orders last week?" - but breaks on anything that requires business knowledge. The agent guesses at joins, invents metrics, and hallucinates table relationships. Every query is a coin flip.
|
||||
|
||||
**Wave two: semantic layers and text-to-SQL.** Add structure. Define metrics in MetricFlow or Cube, expose schemas, build text-to-SQL pipelines. This is better — the agent knows that `revenue` means `sum(amount) where status != 'refunded'` — but building and maintaining that structure by hand is manual, time-consuming, and still limited. Semantic layers define what to calculate, not why, when, or how to interpret the result. The agent can compute net revenue but doesn't know about the February refund anomaly, the segment reclassification, or the fact that `enterprise` changed definition last quarter.
|
||||
**Wave two: semantic layers and text-to-SQL.** Add structure. Define metrics in MetricFlow or Cube, expose schemas, build text-to-SQL pipelines. This is better - the agent knows that `revenue` means `sum(amount) where status != 'refunded'` - but building and maintaining that structure by hand is manual, time-consuming, and still limited. Semantic layers define what to calculate, not why, when, or how to interpret the result. The agent can compute net revenue but doesn't know about the February refund anomaly, the segment reclassification, or the fact that `enterprise` changed definition last quarter.
|
||||
|
||||
**Wave three: agentic context.** AI is no longer just answering questions — it's generating dashboards, writing semantic definitions, proposing dbt models, creating tests and documentation. For that to work, agents need more than metric definitions. They need the full picture: business rules, known data quality issues, relationship maps, historical context, and the institutional knowledge that lives in your team's heads. They need a context layer.
|
||||
**Wave three: agentic context.** AI is no longer just answering questions - it's generating dashboards, writing semantic definitions, proposing dbt models, creating tests and documentation. For that to work, agents need more than metric definitions. They need the full picture: business rules, known data quality issues, relationship maps, historical context, and the institutional knowledge that lives in your team's heads. They need a context layer.
|
||||
|
||||
## What a context layer is
|
||||
|
||||
A context layer is the infrastructure that gives agents the business knowledge they need to produce correct analytics artifacts. It includes a semantic layer — that's a critical component — but it's not the whole thing.
|
||||
A context layer is the infrastructure that gives agents the business knowledge they need to produce correct analytics artifacts. It includes a semantic layer - that's a critical component - but it's not the whole thing.
|
||||
|
||||
KTX organizes context into four pillars:
|
||||
|
||||
|
|
@ -67,7 +67,7 @@ measures:
|
|||
expr: count(id)
|
||||
```
|
||||
|
||||
**Wiki pages** are Markdown documents that capture business definitions, rules, and operating context — the kind of context that doesn't fit in a schema definition. Pages have structured frontmatter (summary, tags, semantic layer references) and free-form content. Agents search them when they need to understand why a metric works a certain way, not just how to compute it.
|
||||
**Wiki pages** are Markdown documents that capture business definitions, rules, and operating context - the kind of context that doesn't fit in a schema definition. Pages have structured frontmatter (summary, tags, semantic layer references) and free-form content. Agents search them when they need to understand why a metric works a certain way, not just how to compute it.
|
||||
|
||||
```markdown
|
||||
---
|
||||
|
|
@ -91,9 +91,9 @@ canonical revenue reporting.
|
|||
|
||||
**Scan artifacts** are the raw output of KTX's database scanner: table and column metadata, inferred foreign key relationships (even without declared constraints), column statistics, and enrichment reports. They form the foundation that semantic sources are built on.
|
||||
|
||||
**Provenance** is the record of how context was created and changed. Every ingestion session records a full transcript — which adapter ran, what the LLM decided, which sources were created or updated, and why. This is what makes the system auditable: you can trace any semantic source back to the ingestion decision that created it.
|
||||
**Provenance** is the record of how context was created and changed. Every ingestion session records a full transcript - which adapter ran, what the LLM decided, which sources were created or updated, and why. This is what makes the system auditable: you can trace any semantic source back to the ingestion decision that created it.
|
||||
|
||||
Together, these four pillars give agents enough context to produce analytics artifacts that match what your team would produce — not just syntactically valid SQL, but the right query for the question.
|
||||
Together, these four pillars give agents enough context to produce analytics artifacts that match what your team would produce - not just syntactically valid SQL, but the right query for the question.
|
||||
|
||||
## How KTX compares
|
||||
|
||||
|
|
@ -115,7 +115,7 @@ If you do not have a semantic layer, KTX can build an agent-native one from your
|
|||
|
||||
## The plain-files philosophy
|
||||
|
||||
A KTX project is a directory of plain files. No server to run, no database to manage, no proprietary store to back up. Everything is YAML, Markdown, and SQLite — formats you can read, diff, and version-control with tools you already use.
|
||||
A KTX project is a directory of plain files. No server to run, no database to manage, no proprietary store to back up. Everything is YAML, Markdown, and SQLite - formats you can read, diff, and version-control with tools you already use.
|
||||
|
||||
```
|
||||
my-project/
|
||||
|
|
@ -140,7 +140,7 @@ my-project/
|
|||
└── cache/ # Runtime cache (git-ignored)
|
||||
```
|
||||
|
||||
Semantic sources and wiki pages are committed to git. The SQLite database holds ephemeral state — schema ingest results, embedding indexes, session logs — and is git-ignored. If you delete it, KTX rebuilds it on the next run.
|
||||
Semantic sources and wiki pages are committed to git. The SQLite database holds ephemeral state - schema ingest results, embedding indexes, session logs - and is git-ignored. If you delete it, KTX rebuilds it on the next run.
|
||||
|
||||
This means your analytics context travels with your code. You can fork it, branch it, review it in a PR, and merge it with the same tools you use for dbt models. There's no sync problem between a remote server and your local state. There's no migration to run. The files are the source of truth.
|
||||
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ description: How KTX gives analytics agents trusted context for warehouse work.
|
|||
## Who KTX is for
|
||||
|
||||
KTX is built for analytics engineers and data teams who want data agents to
|
||||
work on real analytics systems — not just generate one-off SQL.
|
||||
work on real analytics systems - not just generate one-off SQL.
|
||||
|
||||
Use KTX when you want agents to:
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: Quickstart
|
|||
description: Set up KTX and build your first context in under 10 minutes.
|
||||
---
|
||||
|
||||
This guide walks you through `ktx setup` — an interactive wizard that configures your LLM provider, connects your database, optionally ingests from your existing tools, builds context, and installs agent integration.
|
||||
This guide walks you through `ktx setup` - an interactive wizard that configures your LLM provider, connects your database, optionally ingests from your existing tools, builds context, and installs agent integration.
|
||||
|
||||
If you are a coding assistant trying to decide which KTX docs page to read, start with the [Agent Quickstart](/docs/ai-resources/agent-quickstart). This page is the human setup walkthrough.
|
||||
|
||||
|
|
@ -11,8 +11,8 @@ If you are a coding assistant trying to decide which KTX docs page to read, star
|
|||
|
||||
Use this sequence when you are setting up KTX in an analytics project:
|
||||
|
||||
1. `npm install -g @kaelio/ktx` — install the published KTX CLI from npm.
|
||||
2. `ktx setup` — create or resume a KTX project.
|
||||
1. `npm install -g @kaelio/ktx` - install the published KTX CLI from npm.
|
||||
2. `ktx setup` - create or resume a KTX project.
|
||||
|
||||
The setup wizard is stateful. If it exits before completion, rerun `ktx setup` in the same project directory to resume from the first incomplete step.
|
||||
|
||||
|
|
@ -118,7 +118,7 @@ under `connections.<id>.context.queryHistory` in `ktx.yaml`.
|
|||
|
||||
## Step 4: Add context sources
|
||||
|
||||
Context sources let KTX ingest metadata from your existing analytics tools. This step is optional — you can skip it and add sources later.
|
||||
Context sources let KTX ingest metadata from your existing analytics tools. This step is optional - you can skip it and add sources later.
|
||||
|
||||
```
|
||||
◆ Which context sources should KTX ingest?
|
||||
|
|
@ -248,7 +248,7 @@ Agent integration ready: yes (claude-code:project)
|
|||
|
||||
## Next steps
|
||||
|
||||
- **Build more context** — learn about [database ingest](/docs/guides/building-context), relationship detection, and source ingestion workflows in the Building Context guide.
|
||||
- **Refine your semantic layer** — the [Writing Context](/docs/guides/writing-context) guide covers source YAML, measures, joins, and wiki pages.
|
||||
- **Understand the architecture** — read [The Context Layer](/docs/concepts/the-context-layer) to learn why a context layer is more than a semantic layer.
|
||||
- **Connect more agents** — see the [Agent Clients](/docs/integrations/agent-clients) integration page for per-tool setup details.
|
||||
- **Build more context** - learn about [database ingest](/docs/guides/building-context), relationship detection, and source ingestion workflows in the Building Context guide.
|
||||
- **Refine your semantic layer** - the [Writing Context](/docs/guides/writing-context) guide covers source YAML, measures, joins, and wiki pages.
|
||||
- **Understand the architecture** - read [The Context Layer](/docs/concepts/the-context-layer) to learn why a context layer is more than a semantic layer.
|
||||
- **Connect more agents** - see the [Agent Clients](/docs/integrations/agent-clients) integration page for per-tool setup details.
|
||||
|
|
|
|||
|
|
@ -50,13 +50,13 @@ ktx status
|
|||
|
||||
### Relationship detection
|
||||
|
||||
Many databases lack declared foreign keys. KTX infers relationships by scoring column pairs across seven signals — name similarity, type compatibility, value overlap, embedding similarity, profile uniqueness, null rate, and structural priors. The weighted score determines each candidate's status:
|
||||
Many databases lack declared foreign keys. KTX infers relationships by scoring column pairs across seven signals - name similarity, type compatibility, value overlap, embedding similarity, profile uniqueness, null rate, and structural priors. The weighted score determines each candidate's status:
|
||||
|
||||
| Score range | Status | Meaning |
|
||||
|-------------|--------|---------|
|
||||
| ≥ 0.85 | `accepted` | High confidence — applied automatically |
|
||||
| 0.55 – 0.84 | `review` | Plausible — needs human review |
|
||||
| < 0.55 | `rejected` | Low confidence — not applied |
|
||||
| ≥ 0.85 | `accepted` | High confidence - applied automatically |
|
||||
| 0.55 – 0.84 | `review` | Plausible - needs human review |
|
||||
| < 0.55 | `rejected` | Low confidence - not applied |
|
||||
|
||||
Deep database ingest can include relationship evidence where the connector can
|
||||
provide it. Relationship review and calibration subcommands are not part of the
|
||||
|
|
@ -64,14 +64,14 @@ current public CLI surface.
|
|||
|
||||
## Ingestion
|
||||
|
||||
Ingestion pulls semantic context from your existing analytics tools — dbt projects, Looker models, Metabase questions, and more — and writes it into your KTX project as semantic sources and wiki pages.
|
||||
Ingestion pulls semantic context from your existing analytics tools - dbt projects, Looker models, Metabase questions, and more - and writes it into your KTX project as semantic sources and wiki pages.
|
||||
|
||||
### How it works
|
||||
|
||||
Each ingest run follows this flow:
|
||||
|
||||
1. An **adapter** extracts metadata from your tool (dbt manifest, LookML files, Metabase API, etc.)
|
||||
2. An **LLM agent** reconciles the extracted metadata with your existing context — it merges intelligently rather than overwriting
|
||||
2. An **LLM agent** reconciles the extracted metadata with your existing context - it merges intelligently rather than overwriting
|
||||
3. **Semantic sources** (YAML) and **wiki pages** (Markdown) are written to your project directory
|
||||
|
||||
### Running an ingest
|
||||
|
|
|
|||
|
|
@ -3,22 +3,22 @@ title: Writing Context
|
|||
description: Write and refine semantic sources and wiki pages.
|
||||
---
|
||||
|
||||
After building context through scanning and ingestion, you'll want to refine it — edit semantic sources to match your business logic, add wiki pages that capture tribal knowledge, and query your data through the semantic layer to verify everything works.
|
||||
After building context through scanning and ingestion, you'll want to refine it - edit semantic sources to match your business logic, add wiki pages that capture tribal knowledge, and query your data through the semantic layer to verify everything works.
|
||||
|
||||
## Agent workflow summary
|
||||
|
||||
Agents should refine context in this order:
|
||||
|
||||
1. `ktx sl list --json` — discover available sources and connection ids.
|
||||
2. `ktx sl search <query> --json` — find source candidates for a concept.
|
||||
1. `ktx sl list --json` - discover available sources and connection ids.
|
||||
2. `ktx sl search <query> --json` - find source candidates for a concept.
|
||||
3. Edit the source YAML directly in `semantic-layer/<connection-id>/`.
|
||||
4. `ktx sl validate <source> --connection-id <id>` — verify columns, joins, and table references.
|
||||
5. `ktx sl query ... --format sql` — compile a representative query without executing it.
|
||||
6. `ktx wiki search ...` — check business context captured by ingest or memory.
|
||||
4. `ktx sl validate <source> --connection-id <id>` - verify columns, joins, and table references.
|
||||
5. `ktx sl query ... --format sql` - compile a representative query without executing it.
|
||||
6. `ktx wiki search ...` - check business context captured by ingest or memory.
|
||||
|
||||
## Semantic Sources
|
||||
|
||||
Semantic sources are YAML files that describe your tables, columns, measures, and joins. They're the core of the context layer — the structured definitions that agents use to generate correct SQL.
|
||||
Semantic sources are YAML files that describe your tables, columns, measures, and joins. They're the core of the context layer - the structured definitions that agents use to generate correct SQL.
|
||||
|
||||
### Listing sources
|
||||
|
||||
|
|
@ -44,7 +44,7 @@ YAML file under `semantic-layer/<connection-id>/`.
|
|||
|
||||
### The source schema
|
||||
|
||||
A semantic source defines a single queryable entity — usually a table or a SQL expression. Here's a fully annotated example:
|
||||
A semantic source defines a single queryable entity - usually a table or a SQL expression. Here's a fully annotated example:
|
||||
|
||||
```yaml
|
||||
name: orders
|
||||
|
|
@ -146,7 +146,7 @@ Column visibility controls what agents see:
|
|||
|------------|----------|
|
||||
| `public` | Included in agent queries and listings (default) |
|
||||
| `internal` | Available for joins and measures but not shown to agents |
|
||||
| `hidden` | Excluded entirely — useful for ETL columns |
|
||||
| `hidden` | Excluded entirely - useful for ETL columns |
|
||||
|
||||
### Editing a source
|
||||
|
||||
|
|
@ -161,7 +161,7 @@ Validation checks a source definition against the actual database schema:
|
|||
ktx sl validate orders --connection-id my-postgres
|
||||
```
|
||||
|
||||
This catches mismatches — columns that don't exist in the table, type mismatches, invalid join targets — before an agent tries to use the source.
|
||||
This catches mismatches - columns that don't exist in the table, type mismatches, invalid join targets - before an agent tries to use the source.
|
||||
|
||||
### Querying
|
||||
|
||||
|
|
@ -207,20 +207,20 @@ Query flags:
|
|||
| `--max-rows <n>` | Maximum rows to return when executing |
|
||||
| `--include-empty` | Include empty/null rows in results |
|
||||
|
||||
The query planner is grain-aware — it understands the cardinality of joins and avoids chasm traps (double-counting caused by many-to-many fan-outs). When you query measures that span multiple sources, KTX generates sub-queries at the correct grain before joining.
|
||||
The query planner is grain-aware - it understands the cardinality of joins and avoids chasm traps (double-counting caused by many-to-many fan-outs). When you query measures that span multiple sources, KTX generates sub-queries at the correct grain before joining.
|
||||
|
||||
### Workflow: edit and validate a source
|
||||
|
||||
1. Open `semantic-layer/my-postgres/orders.yaml`.
|
||||
2. Edit the file to add columns, measures, joins, or descriptions.
|
||||
3. `ktx sl validate orders --connection-id my-postgres` — check the definition against the live schema.
|
||||
4. `ktx sl query --connection-id my-postgres --measure total_revenue --dimension order_date --format sql` — compile a representative query.
|
||||
3. `ktx sl validate orders --connection-id my-postgres` - check the definition against the live schema.
|
||||
4. `ktx sl query --connection-id my-postgres --measure total_revenue --dimension order_date --format sql` - compile a representative query.
|
||||
|
||||
If validation fails, fix the YAML before asking an agent to use the source. Common validation failures are missing columns, invalid join targets, and measure expressions that reference fields outside the source.
|
||||
|
||||
## Wiki Pages
|
||||
|
||||
Wiki pages are Markdown files that capture business context — definitions, rules, gotchas, and anything an agent needs to understand beyond what the schema tells it.
|
||||
Wiki pages are Markdown files that capture business context - definitions, rules, gotchas, and anything an agent needs to understand beyond what the schema tells it.
|
||||
|
||||
### What they are
|
||||
|
||||
|
|
@ -242,8 +242,8 @@ wiki/
|
|||
└── known-data-issues.md
|
||||
```
|
||||
|
||||
- **Global pages** apply across all connections — business definitions, metric standards, company terminology.
|
||||
- **User-scoped pages** are private to a user ID — personal notes, local gotchas, or context you do not want shared globally.
|
||||
- **Global pages** apply across all connections - business definitions, metric standards, company terminology.
|
||||
- **User-scoped pages** are private to a user ID - personal notes, local gotchas, or context you do not want shared globally.
|
||||
|
||||
### Editing pages
|
||||
|
||||
|
|
@ -274,7 +274,7 @@ ktx wiki list
|
|||
ktx wiki search "revenue recognition"
|
||||
```
|
||||
|
||||
Search uses both full-text matching and semantic similarity — it finds relevant pages even when the exact terms don't match. Agents call this automatically when they need business context to answer a question.
|
||||
Search uses both full-text matching and semantic similarity - it finds relevant pages even when the exact terms don't match. Agents call this automatically when they need business context to answer a question.
|
||||
|
||||
### Workflow: add searchable business context
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: Context Sources
|
|||
description: Ingest semantic context from dbt, MetricFlow, LookML, Metabase, Looker, and Notion.
|
||||
---
|
||||
|
||||
Context sources feed your existing analytics tooling into KTX. During ingestion, KTX extracts metadata from each source and uses an LLM agent to reconcile it with your existing semantic layer and knowledge base — merging intelligently rather than overwriting.
|
||||
Context sources feed your existing analytics tooling into KTX. During ingestion, KTX extracts metadata from each source and uses an LLM agent to reconcile it with your existing semantic layer and knowledge base - merging intelligently rather than overwriting.
|
||||
|
||||
All context sources are configured in `ktx.yaml` under `connections` with their respective `driver` value.
|
||||
|
||||
|
|
@ -250,7 +250,7 @@ mappings:
|
|||
syncMode: ONLY # ONLY = restrict to mapped DBs
|
||||
```
|
||||
|
||||
Find Metabase database IDs in **Admin > Databases** — the ID is in the URL when editing a database.
|
||||
Find Metabase database IDs in **Admin > Databases** - the ID is in the URL when editing a database.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -353,7 +353,7 @@ Create an integration at [notion.so/my-integrations](https://www.notion.so/my-in
|
|||
|
||||
| Field | Description | Default |
|
||||
|-------|-------------|---------|
|
||||
| `crawl_mode` | `all_accessible` or `selected_roots` | — |
|
||||
| `crawl_mode` | `all_accessible` or `selected_roots` | - |
|
||||
| `root_page_ids` | Page IDs to crawl from (for `selected_roots`) | `[]` |
|
||||
| `root_database_ids` | Database IDs to include | `[]` |
|
||||
| `max_pages_per_run` | Pages processed per sync | `1000` |
|
||||
|
|
@ -369,7 +369,7 @@ Create an integration at [notion.so/my-integrations](https://www.notion.so/my-in
|
|||
|
||||
### Notes
|
||||
|
||||
- Notion is knowledge-only — it does not produce semantic layer sources
|
||||
- Notion is knowledge-only - it does not produce semantic layer sources
|
||||
- Rate limits apply; large workspaces may require multiple ingestion runs
|
||||
- Incremental sync cursors are stored in `.ktx/db.sqlite`; don't add
|
||||
`last_successful_cursor` to `ktx.yaml`
|
||||
|
|
|
|||
|
|
@ -154,9 +154,9 @@ For multiple schemas:
|
|||
| Primary keys | Yes | Via table constraints |
|
||||
| Foreign keys | No | Not available in Snowflake |
|
||||
| Row count estimates | Yes | From `INFORMATION_SCHEMA.TABLES.ROW_COUNT` |
|
||||
| Column statistics | No | — |
|
||||
| Column statistics | No | - |
|
||||
| Query history | Yes | Via `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY` when enabled |
|
||||
| Table sampling | Yes | — |
|
||||
| Table sampling | Yes | - |
|
||||
|
||||
### Query history
|
||||
|
||||
|
|
@ -228,12 +228,12 @@ mapping metadata. The BigQuery connector still authenticates with the
|
|||
| Feature | Supported | Notes |
|
||||
|---------|-----------|-------|
|
||||
| Tables & views | Yes | Including materialized views and external tables |
|
||||
| Primary keys | No | — |
|
||||
| Primary keys | No | - |
|
||||
| Foreign keys | No | Not available in BigQuery |
|
||||
| Row count estimates | Yes | From table metadata |
|
||||
| Column statistics | No | — |
|
||||
| Column statistics | No | - |
|
||||
| Query history | Yes | Via region-scoped `INFORMATION_SCHEMA.JOBS_BY_PROJECT` when enabled |
|
||||
| Table sampling | Yes | — |
|
||||
| Table sampling | Yes | - |
|
||||
|
||||
### Query history
|
||||
|
||||
|
|
@ -307,9 +307,9 @@ connections:
|
|||
| Primary keys | Yes | Via `system.columns` |
|
||||
| Foreign keys | No | Not a ClickHouse concept |
|
||||
| Row count estimates | Yes | Via `system.parts` aggregation |
|
||||
| Column statistics | No | — |
|
||||
| Query history | No | — |
|
||||
| Table sampling | Yes | — |
|
||||
| Column statistics | No | - |
|
||||
| Query history | No | - |
|
||||
| Table sampling | Yes | - |
|
||||
|
||||
### Dialect notes
|
||||
|
||||
|
|
@ -364,8 +364,8 @@ connections:
|
|||
| Primary keys | Yes | Via `KEY_COLUMN_USAGE` |
|
||||
| Foreign keys | Yes | Via `REFERENTIAL_CONSTRAINTS` |
|
||||
| Row count estimates | Yes | From `TABLE_ROWS` (InnoDB estimate) |
|
||||
| Column statistics | No | — |
|
||||
| Query history | No | — |
|
||||
| Column statistics | No | - |
|
||||
| Query history | No | - |
|
||||
| Table sampling | Yes | Uses `RAND()` filter |
|
||||
|
||||
### Dialect notes
|
||||
|
|
@ -430,10 +430,10 @@ For multiple schemas:
|
|||
| Primary keys | Yes | Via `TABLE_CONSTRAINTS` and `KEY_COLUMN_USAGE` |
|
||||
| Foreign keys | Yes | Via `REFERENTIAL_CONSTRAINTS` |
|
||||
| Row count estimates | Yes | Via `sys.dm_db_partition_stats` |
|
||||
| Column statistics | No | — |
|
||||
| Query history | No | — |
|
||||
| Table sampling | Yes | — |
|
||||
| Nested analysis | No | — |
|
||||
| Column statistics | No | - |
|
||||
| Query history | No | - |
|
||||
| Table sampling | Yes | - |
|
||||
| Nested analysis | No | - |
|
||||
|
||||
### Dialect notes
|
||||
|
||||
|
|
@ -478,7 +478,7 @@ url: sqlite:///path/to/db.sqlite
|
|||
|
||||
### Authentication
|
||||
|
||||
No authentication required — SQLite is file-based. The file must be readable by the process running KTX.
|
||||
No authentication required - SQLite is file-based. The file must be readable by the process running KTX.
|
||||
|
||||
### Features
|
||||
|
||||
|
|
@ -488,10 +488,10 @@ No authentication required — SQLite is file-based. The file must be readable b
|
|||
| Primary keys | Yes | Via `PRAGMA table_info()` |
|
||||
| Foreign keys | Yes | Via `PRAGMA foreign_key_list()` (requires `PRAGMA foreign_keys = ON`) |
|
||||
| Row count estimates | Yes | Exact count via `SELECT COUNT(*)` |
|
||||
| Column statistics | No | — |
|
||||
| Query history | No | — |
|
||||
| Table sampling | Yes | — |
|
||||
| Nested analysis | No | — |
|
||||
| Column statistics | No | - |
|
||||
| Query history | No | - |
|
||||
| Table sampling | Yes | - |
|
||||
| Nested analysis | No | - |
|
||||
|
||||
### Dialect notes
|
||||
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ describe('renderDemoBanner', () => {
|
|||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run test -- --testPathPattern setup-demo-tour`
|
||||
Expected: FAIL — module not found
|
||||
Expected: FAIL - module not found
|
||||
|
||||
- [ ] **Step 3: Implement `renderDemoBanner` and `waitForDemoNavigation`**
|
||||
|
||||
|
|
@ -58,7 +58,7 @@ function dim(text: string): string {
|
|||
export function renderDemoBanner(): string {
|
||||
const lines = [
|
||||
'',
|
||||
`┌ ${cyan('Demo mode')} — data has been pre-processed and KTX context is already built.`,
|
||||
`┌ ${cyan('Demo mode')} - data has been pre-processed and KTX context is already built.`,
|
||||
`│ This walkthrough illustrates the setup steps. Selections are pre-filled and read-only.`,
|
||||
'',
|
||||
];
|
||||
|
|
@ -145,7 +145,7 @@ describe('renderDemoCardContent', () => {
|
|||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run test -- --testPathPattern setup-demo-tour`
|
||||
Expected: FAIL — `renderDemoCardContent` not exported
|
||||
Expected: FAIL - `renderDemoCardContent` not exported
|
||||
|
||||
- [ ] **Step 3: Implement `renderDemoCardContent` and `renderDemoCard`**
|
||||
|
||||
|
|
@ -243,7 +243,7 @@ describe('DEMO_REPLAY_TARGETS', () => {
|
|||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run test -- --testPathPattern setup-demo-tour`
|
||||
Expected: FAIL — exports not found
|
||||
Expected: FAIL - exports not found
|
||||
|
||||
- [ ] **Step 3: Implement replay timeline and target definitions**
|
||||
|
||||
|
|
@ -388,7 +388,7 @@ function renderDemoContextCompletionSummary(): string {
|
|||
'',
|
||||
`${cyan('★')} KTX finished ingesting demo data`,
|
||||
'',
|
||||
' Placeholder — final counts will come from pre-packaged demo results.',
|
||||
' Placeholder - final counts will come from pre-packaged demo results.',
|
||||
'',
|
||||
` ${dim('Press Enter to continue, Escape to go back')}`,
|
||||
'',
|
||||
|
|
@ -459,7 +459,7 @@ describe('renderDemoCompletionSummary', () => {
|
|||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run test -- --testPathPattern setup-demo-tour`
|
||||
Expected: FAIL — exports not found
|
||||
Expected: FAIL - exports not found
|
||||
|
||||
- [ ] **Step 3: Implement transition and completion rendering**
|
||||
|
||||
|
|
@ -469,7 +469,7 @@ Add to `setup-demo-tour.ts`:
|
|||
export function renderDemoAgentTransition(): string {
|
||||
const lines = [
|
||||
'',
|
||||
`┌ Demo project is ready — let's connect your agent`,
|
||||
`┌ Demo project is ready - let's connect your agent`,
|
||||
'│',
|
||||
'│ Your KTX context has been built with demo data.',
|
||||
'│ Select an agent to start using it.',
|
||||
|
|
@ -583,7 +583,7 @@ describe('runDemoTour', () => {
|
|||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run test -- --testPathPattern setup-demo-tour`
|
||||
Expected: FAIL — `runDemoTour` not exported or wrong signature
|
||||
Expected: FAIL - `runDemoTour` not exported or wrong signature
|
||||
|
||||
- [ ] **Step 3: Implement `runDemoTour`**
|
||||
|
||||
|
|
@ -677,7 +677,7 @@ Expected: PASS
|
|||
- [ ] **Step 5: Run type-check**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run type-check`
|
||||
Expected: PASS — all types align with existing interfaces
|
||||
Expected: PASS - all types align with existing interfaces
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
|
|
@ -736,7 +736,7 @@ async function runKtxSetupDemoFromEntryMenu(
|
|||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Update imports — remove unused `defaultDemoProjectDir` import if no longer needed elsewhere in setup.ts**
|
||||
- [ ] **Step 3: Update imports - remove unused `defaultDemoProjectDir` import if no longer needed elsewhere in setup.ts**
|
||||
|
||||
Check if `defaultDemoProjectDir` is used elsewhere in `setup.ts`. If it's only used
|
||||
in `runKtxSetupDemoFromEntryMenu`, remove the import. If used elsewhere, keep it.
|
||||
|
|
@ -749,7 +749,7 @@ called from the entry menu path.
|
|||
- [ ] **Step 4: Run type-check and tests**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run type-check && pnpm --filter @ktx/cli run test`
|
||||
Expected: PASS — existing tests continue to work, demo tour is now wired in
|
||||
Expected: PASS - existing tests continue to work, demo tour is now wired in
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
|
|
@ -807,7 +807,7 @@ git commit -m "fix(cli): demo tour adjustments from smoke test"
|
|||
|
||||
When the user provides the real pre-packaged demo results, update these locations:
|
||||
|
||||
1. **`renderDemoContextCompletionSummary()`** in `setup-demo-tour.ts` — replace placeholder text with actual counts (business areas, query definitions, knowledge pages) from the demo data
|
||||
2. **`buildDemoReplayTimeline()`** in `setup-demo-tour.ts` — adjust timing and progress details to match the real ingestion profile
|
||||
3. **`demo-assets.ts`** — update `REQUIRED_SEEDED_ASSET_PATHS` and `demoConfig()` if the demo dataset changes from SQLite/Orbit to Postgres/dbt/Metabase/Notion
|
||||
4. **Pre-packaged asset files** in `packages/cli/assets/demo/` — replace with the new demo dataset
|
||||
1. **`renderDemoContextCompletionSummary()`** in `setup-demo-tour.ts` - replace placeholder text with actual counts (business areas, query definitions, knowledge pages) from the demo data
|
||||
2. **`buildDemoReplayTimeline()`** in `setup-demo-tour.ts` - adjust timing and progress details to match the real ingestion profile
|
||||
3. **`demo-assets.ts`** - update `REQUIRED_SEEDED_ASSET_PATHS` and `demoConfig()` if the demo dataset changes from SQLite/Orbit to Postgres/dbt/Metabase/Notion
|
||||
4. **Pre-packaged asset files** in `packages/cli/assets/demo/` - replace with the new demo dataset
|
||||
|
|
|
|||
|
|
@ -654,11 +654,11 @@ In `docs/content/docs/cli-reference/ktx-setup.mdx`, replace the Historic SQL fla
|
|||
```markdown
|
||||
| `--enable-historic-sql` | Enable Historic SQL when the selected database supports it | `false` |
|
||||
| `--disable-historic-sql` | Disable Historic SQL for the selected database | `false` |
|
||||
| `--historic-sql-window-days <number>` | Historic SQL query-history window in days | — |
|
||||
| `--historic-sql-min-executions <number>` | Minimum executions for a Historic SQL template | — |
|
||||
| `--historic-sql-min-calls <number>` | Alias for `--historic-sql-min-executions` for one release | — |
|
||||
| `--historic-sql-service-account-pattern <pattern>` | Historic SQL service-account regex; repeatable | — |
|
||||
| `--historic-sql-redaction-pattern <pattern>` | Historic SQL SQL-literal redaction regex; repeatable | — |
|
||||
| `--historic-sql-window-days <number>` | Historic SQL query-history window in days | - |
|
||||
| `--historic-sql-min-executions <number>` | Minimum executions for a Historic SQL template | - |
|
||||
| `--historic-sql-min-calls <number>` | Alias for `--historic-sql-min-executions` for one release | - |
|
||||
| `--historic-sql-service-account-pattern <pattern>` | Historic SQL service-account regex; repeatable | - |
|
||||
| `--historic-sql-redaction-pattern <pattern>` | Historic SQL SQL-literal redaction regex; repeatable | - |
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update primary source Historic SQL docs**
|
||||
|
|
|
|||
|
|
@ -874,7 +874,7 @@ Expected: PASS. The output includes `# fail 0`.
|
|||
|
||||
- [ ] **Step 2: Verify stale artifact strings are gone from production/docs files**
|
||||
|
||||
Run (scans only production and docs files, not test files — test files keep guard assertions that reference the removed strings):
|
||||
Run (scans only production and docs files, not test files - test files keep guard assertions that reference the removed strings):
|
||||
|
||||
```bash
|
||||
rg -n "uv', \\['build', '--package', 'ktx-sl'|uv', \\['build', '--package', 'ktx-daemon'|ktx_sl-0\\.1\\.0|ktx_daemon-0\\.1\\.0|pythonArtifactInstallArgs|pythonVerifySource|verifyPythonArtifacts|standalone Python distributions|installs the Python artifacts directly" scripts/package-artifacts.mjs scripts/release-readiness.mjs README.md examples/package-artifacts/README.md release-policy.json
|
||||
|
|
|
|||
|
|
@ -199,7 +199,7 @@ Modify the raw schema markdown in
|
|||
.slice(0, limit)
|
||||
.map(
|
||||
(hit) =>
|
||||
`- ${hit.kind}: ${hit.display} [connectionName=${hit.connectionName}] (matched on ${hit.matchedOn}) — ` +
|
||||
`- ${hit.kind}: ${hit.display} [connectionName=${hit.connectionName}] (matched on ${hit.matchedOn}) - ` +
|
||||
`follow up with \`entity_details({connectionName: "${hit.connectionName}", targets: [{display: "${hit.display}"}]})\``,
|
||||
)
|
||||
.join('\n'),
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a build-time script that prints the full `ktx` CLI command tree (name, aliases, description per node) as an indented text tree, for docs and discovery — without adding a runtime `ktx` subcommand.
|
||||
**Goal:** Add a build-time script that prints the full `ktx` CLI command tree (name, aliases, description per node) as an indented text tree, for docs and discovery - without adding a runtime `ktx` subcommand.
|
||||
|
||||
**Architecture:** Commander.js exposes every registered command as a `Command` instance with `.commands`, `.name()`, `.aliases()`, `.description()` — we walk that tree. The current `runCommanderKtxCli` in `packages/cli/src/cli-program.ts` builds the program inline; we extract that assembly into a pure `buildKtxProgram(...)` helper that any caller can use to materialize the configured root `Command` without parsing argv. A new pure module `command-tree.ts` walks the `Command` into plain data and renders it as indented text. A new TypeScript entrypoint `print-command-tree.ts` compiles alongside `bin.ts` into `dist/print-command-tree.js`, instantiates the program with stub IO/deps, and writes the rendered tree to stdout. A pnpm script under `@ktx/cli` exposes it as `pnpm --filter @ktx/cli run docs:commands`.
|
||||
**Architecture:** Commander.js exposes every registered command as a `Command` instance with `.commands`, `.name()`, `.aliases()`, `.description()` - we walk that tree. The current `runCommanderKtxCli` in `packages/cli/src/cli-program.ts` builds the program inline; we extract that assembly into a pure `buildKtxProgram(...)` helper that any caller can use to materialize the configured root `Command` without parsing argv. A new pure module `command-tree.ts` walks the `Command` into plain data and renders it as indented text. A new TypeScript entrypoint `print-command-tree.ts` compiles alongside `bin.ts` into `dist/print-command-tree.js`, instantiates the program with stub IO/deps, and writes the rendered tree to stdout. A pnpm script under `@ktx/cli` exposes it as `pnpm --filter @ktx/cli run docs:commands`.
|
||||
|
||||
**Tech Stack:** TypeScript (NodeNext ESM), Node 22, Commander 14 via `@commander-js/extra-typings`, vitest 4.
|
||||
|
||||
|
|
@ -12,14 +12,14 @@
|
|||
|
||||
## File Map
|
||||
|
||||
- **Modify:** `packages/cli/src/cli-program.ts` — extract `buildKtxProgram` from `runCommanderKtxCli`.
|
||||
- **Create:** `packages/cli/src/cli-program.test.ts` — vitest tests for the new helper.
|
||||
- **Create:** `packages/cli/src/command-tree.ts` — pure `walkCommandTree` + `formatCommandTree`.
|
||||
- **Create:** `packages/cli/src/command-tree.test.ts` — vitest tests against ad-hoc Command trees.
|
||||
- **Create:** `packages/cli/src/print-command-tree.ts` — script entrypoint; thin glue.
|
||||
- **Create:** `packages/cli/src/print-command-tree.test.ts` — vitest test that calls the script's exported `main()` with a fake stdout and asserts the rendered tree includes known top-level commands.
|
||||
- **Modify:** `packages/cli/package.json` — add `docs:commands` script and include the new entry in tsc build output (no change needed if `tsconfig` already globs `src/**/*.ts`, but verify).
|
||||
- **Modify:** `packages/cli/README.md` (if it exists; otherwise skip) — document `pnpm run docs:commands`.
|
||||
- **Modify:** `packages/cli/src/cli-program.ts` - extract `buildKtxProgram` from `runCommanderKtxCli`.
|
||||
- **Create:** `packages/cli/src/cli-program.test.ts` - vitest tests for the new helper.
|
||||
- **Create:** `packages/cli/src/command-tree.ts` - pure `walkCommandTree` + `formatCommandTree`.
|
||||
- **Create:** `packages/cli/src/command-tree.test.ts` - vitest tests against ad-hoc Command trees.
|
||||
- **Create:** `packages/cli/src/print-command-tree.ts` - script entrypoint; thin glue.
|
||||
- **Create:** `packages/cli/src/print-command-tree.test.ts` - vitest test that calls the script's exported `main()` with a fake stdout and asserts the rendered tree includes known top-level commands.
|
||||
- **Modify:** `packages/cli/package.json` - add `docs:commands` script and include the new entry in tsc build output (no change needed if `tsconfig` already globs `src/**/*.ts`, but verify).
|
||||
- **Modify:** `packages/cli/README.md` (if it exists; otherwise skip) - document `pnpm run docs:commands`.
|
||||
|
||||
Files that change together (cli-program + its test, command-tree + its test, print-command-tree + its test) live next to each other under `packages/cli/src/`, matching the existing convention (e.g. `bin.ts`, `cli-runtime.ts`, `runtime.ts` + `runtime.test.ts`).
|
||||
|
||||
|
|
@ -27,7 +27,7 @@ Files that change together (cli-program + its test, command-tree + its test, pri
|
|||
|
||||
## Task 1: Extract `buildKtxProgram` from `runCommanderKtxCli`
|
||||
|
||||
Refactor only — no behavior change. The current code in `cli-program.ts` interleaves program construction with `parseAsync` dispatch. Splitting them lets the new script reuse construction without invoking the CLI.
|
||||
Refactor only - no behavior change. The current code in `cli-program.ts` interleaves program construction with `parseAsync` dispatch. Splitting them lets the new script reuse construction without invoking the CLI.
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/cli/src/cli-program.ts:197-275` (function `runCommanderKtxCli`)
|
||||
|
|
@ -88,7 +88,7 @@ describe('buildKtxProgram', () => {
|
|||
|
||||
Run: `pnpm --filter @ktx/cli exec vitest run src/cli-program.test.ts`
|
||||
|
||||
Expected: FAIL — `buildKtxProgram is not exported from './cli-program.js'` (or similar TS/ESM error).
|
||||
Expected: FAIL - `buildKtxProgram is not exported from './cli-program.js'` (or similar TS/ESM error).
|
||||
|
||||
- [ ] **Step 3: Extract `buildKtxProgram` from `runCommanderKtxCli`**
|
||||
|
||||
|
|
@ -160,19 +160,19 @@ Then rewrite the body of `runCommanderKtxCli` (lines 197-275) to delegate progra
|
|||
};
|
||||
```
|
||||
|
||||
Keep the `context` re-declaration only if subsequent code (the `if (argv.length === 0)` branch that calls `runBareInteractiveCommand(program, io, context)`) still needs it. It does — `runBareInteractiveCommand` consumes `context`. Keep `context` exactly as it was after the deletion; do not change `runBareInteractiveCommand`'s signature or behavior. Drop the now-removed individual `register*` calls and their `profileMark` lines from `runCommanderKtxCli`.
|
||||
Keep the `context` re-declaration only if subsequent code (the `if (argv.length === 0)` branch that calls `runBareInteractiveCommand(program, io, context)`) still needs it. It does - `runBareInteractiveCommand` consumes `context`. Keep `context` exactly as it was after the deletion; do not change `runBareInteractiveCommand`'s signature or behavior. Drop the now-removed individual `register*` calls and their `profileMark` lines from `runCommanderKtxCli`.
|
||||
|
||||
- [ ] **Step 4: Run the new test to verify it passes**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli exec vitest run src/cli-program.test.ts`
|
||||
|
||||
Expected: PASS — both `it` blocks green.
|
||||
Expected: PASS - both `it` blocks green.
|
||||
|
||||
- [ ] **Step 5: Run the full CLI test suite to confirm no regression**
|
||||
|
||||
Run: `pnpm --filter @ktx/cli run test 2>&1 | tee /tmp/ktx-cli-test-output.log`
|
||||
|
||||
Expected: PASS overall. Inspect the log if any previously-passing test now fails — most likely a missing register call (compare to lines 221-249 of the pre-change file).
|
||||
Expected: PASS overall. Inspect the log if any previously-passing test now fails - most likely a missing register call (compare to lines 221-249 of the pre-change file).
|
||||
|
||||
- [ ] **Step 6: Type-check**
|
||||
|
||||
|
|
@ -191,7 +191,7 @@ git commit -m "refactor(cli): extract buildKtxProgram for reuse outside runComma
|
|||
|
||||
## Task 2: Pure tree walker `walkCommandTree`
|
||||
|
||||
Take a Commander `Command` and produce plain data: `{ name, description, aliases, children }`. No formatting yet. Pure function — depends only on the public `Command` API.
|
||||
Take a Commander `Command` and produce plain data: `{ name, description, aliases, children }`. No formatting yet. Pure function - depends only on the public `Command` API.
|
||||
|
||||
**Files:**
|
||||
- Create: `packages/cli/src/command-tree.ts`
|
||||
|
|
@ -254,7 +254,7 @@ describe('walkCommandTree', () => {
|
|||
|
||||
Run: `pnpm --filter @ktx/cli exec vitest run src/command-tree.test.ts`
|
||||
|
||||
Expected: FAIL — `walkCommandTree` cannot be resolved.
|
||||
Expected: FAIL - `walkCommandTree` cannot be resolved.
|
||||
|
||||
- [ ] **Step 3: Implement `walkCommandTree`**
|
||||
|
||||
|
|
@ -296,7 +296,7 @@ Expected: no errors.
|
|||
|
||||
## Task 3: Indented-text renderer `formatCommandTree`
|
||||
|
||||
Render a `CommandTreeNode` as plain text. Each node on its own line: `<indent><name>[ (alias1, alias2)][ — description]`. Indent is two spaces per depth level. Children sorted alphabetically by name to keep output stable across changes that reorder registrar calls.
|
||||
Render a `CommandTreeNode` as plain text. Each node on its own line: `<indent><name>[ (alias1, alias2)][ - description]`. Indent is two spaces per depth level. Children sorted alphabetically by name to keep output stable across changes that reorder registrar calls.
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/cli/src/command-tree.ts`
|
||||
|
|
@ -312,12 +312,12 @@ import { formatCommandTree } from './command-tree.js';
|
|||
describe('formatCommandTree', () => {
|
||||
it('renders a single node with no children', () => {
|
||||
const node = { name: 'solo', description: 'just me', aliases: [], children: [] };
|
||||
expect(formatCommandTree(node)).toBe('solo — just me\n');
|
||||
expect(formatCommandTree(node)).toBe('solo - just me\n');
|
||||
});
|
||||
|
||||
it('renders aliases in parentheses before the description', () => {
|
||||
const node = { name: 'cmd', description: 'does things', aliases: ['c', 'co'], children: [] };
|
||||
expect(formatCommandTree(node)).toBe('cmd (c, co) — does things\n');
|
||||
expect(formatCommandTree(node)).toBe('cmd (c, co) - does things\n');
|
||||
});
|
||||
|
||||
it('omits the dash when description is empty', () => {
|
||||
|
|
@ -338,10 +338,10 @@ describe('formatCommandTree', () => {
|
|||
],
|
||||
};
|
||||
expect(formatCommandTree(tree)).toBe(
|
||||
'root — top\n' +
|
||||
' alpha (al) — a\n' +
|
||||
' inner — i\n' +
|
||||
' beta — b\n',
|
||||
'root - top\n' +
|
||||
' alpha (al) - a\n' +
|
||||
' inner - i\n' +
|
||||
' beta - b\n',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
@ -351,7 +351,7 @@ describe('formatCommandTree', () => {
|
|||
|
||||
Run: `pnpm --filter @ktx/cli exec vitest run src/command-tree.test.ts`
|
||||
|
||||
Expected: FAIL — `formatCommandTree` is not exported.
|
||||
Expected: FAIL - `formatCommandTree` is not exported.
|
||||
|
||||
- [ ] **Step 3: Implement `formatCommandTree`**
|
||||
|
||||
|
|
@ -367,7 +367,7 @@ export function formatCommandTree(node: CommandTreeNode): string {
|
|||
function appendNode(node: CommandTreeNode, depth: number, lines: string[]): void {
|
||||
const indent = ' '.repeat(depth);
|
||||
const aliasPart = node.aliases.length > 0 ? ` (${node.aliases.join(', ')})` : '';
|
||||
const descPart = node.description.length > 0 ? ` — ${node.description}` : '';
|
||||
const descPart = node.description.length > 0 ? ` - ${node.description}` : '';
|
||||
lines.push(`${indent}${node.name}${aliasPart}${descPart}`);
|
||||
|
||||
const sortedChildren = [...node.children].sort((a, b) => a.name.localeCompare(b.name));
|
||||
|
|
@ -419,7 +419,7 @@ describe('renderKtxCommandTree', () => {
|
|||
const output = renderKtxCommandTree();
|
||||
|
||||
const lines = output.split('\n');
|
||||
expect(lines[0]).toMatch(/^ktx( |$|\s—)/);
|
||||
expect(lines[0]).toMatch(/^ktx( |$|\s-)/);
|
||||
|
||||
// Top-level commands are indented exactly two spaces.
|
||||
const topLevel = lines
|
||||
|
|
@ -443,7 +443,7 @@ describe('renderKtxCommandTree', () => {
|
|||
|
||||
Run: `pnpm --filter @ktx/cli exec vitest run src/print-command-tree.test.ts`
|
||||
|
||||
Expected: FAIL — module not found.
|
||||
Expected: FAIL - module not found.
|
||||
|
||||
- [ ] **Step 3: Implement the script**
|
||||
|
||||
|
|
@ -495,7 +495,7 @@ if (invokedAsScript) {
|
|||
|
||||
Run: `pnpm --filter @ktx/cli exec vitest run src/print-command-tree.test.ts`
|
||||
|
||||
Expected: PASS — both assertions green.
|
||||
Expected: PASS - both assertions green.
|
||||
|
||||
- [ ] **Step 5: Type-check**
|
||||
|
||||
|
|
@ -572,9 +572,9 @@ git commit -m "chore(cli): add docs:commands pnpm script"
|
|||
|
||||
After all tasks, confirm:
|
||||
|
||||
- [ ] `pnpm --filter @ktx/cli run type-check` — clean
|
||||
- [ ] `pnpm --filter @ktx/cli run test` — green, including new tests in `cli-program.test.ts`, `command-tree.test.ts`, `print-command-tree.test.ts`
|
||||
- [ ] `pnpm --filter @ktx/cli run docs:commands` — prints `ktx` followed by indented subcommand tree
|
||||
- [ ] `git status --short` — only the files listed in the File Map are modified or created; no incidental edits
|
||||
- [ ] `pnpm --filter @ktx/cli run type-check` - clean
|
||||
- [ ] `pnpm --filter @ktx/cli run test` - green, including new tests in `cli-program.test.ts`, `command-tree.test.ts`, `print-command-tree.test.ts`
|
||||
- [ ] `pnpm --filter @ktx/cli run docs:commands` - prints `ktx` followed by indented subcommand tree
|
||||
- [ ] `git status --short` - only the files listed in the File Map are modified or created; no incidental edits
|
||||
|
||||
If any check fails, fix in place and re-run before declaring done.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Demo Guided Tour — Design Spec
|
||||
# Demo Guided Tour - Design Spec
|
||||
|
||||
## Problem
|
||||
|
||||
|
|
@ -40,7 +40,7 @@ Copy pre-packaged assets (demo DB, replay, context artifacts)
|
|||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ Demo banner (persistent, shown on every step) │
|
||||
│ │
|
||||
│ Demo mode — data has been pre-processed and KTX context is │
|
||||
│ Demo mode - data has been pre-processed and KTX context is │
|
||||
│ already built. This walkthrough illustrates the setup steps. │
|
||||
│ Selections are pre-filled and read-only. │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
|
|
@ -67,7 +67,7 @@ Context build replay
|
|||
│
|
||||
▼
|
||||
Transition message:
|
||||
"Demo project is ready — let's connect your agent"
|
||||
"Demo project is ready - let's connect your agent"
|
||||
│
|
||||
▼
|
||||
Interactive agents step (real runKtxSetupAgentsStep())
|
||||
|
|
@ -89,7 +89,7 @@ Final summary:
|
|||
Shown at the top of every read-only step. Uses clack box-drawing style:
|
||||
|
||||
```
|
||||
┌ Demo mode — data has been pre-processed and KTX context is already built.
|
||||
┌ Demo mode - data has been pre-processed and KTX context is already built.
|
||||
│ This walkthrough illustrates the setup steps. Selections are pre-filled and read-only.
|
||||
```
|
||||
|
||||
|
|
@ -170,7 +170,7 @@ Completion summary uses the existing format:
|
|||
★ KTX finished ingesting your data
|
||||
|
||||
✓ Analyzed X business areas
|
||||
✓ Reconciled — 0 conflicts
|
||||
✓ Reconciled - 0 conflicts
|
||||
|
||||
KTX created:
|
||||
📊 X query definitions
|
||||
|
|
@ -187,7 +187,7 @@ The exact counts and artifact names come from the pre-packaged demo results
|
|||
A brief message bridges from the read-only tour to the interactive step:
|
||||
|
||||
```
|
||||
┌ Demo project is ready — let's connect your agent
|
||||
┌ Demo project is ready - let's connect your agent
|
||||
│
|
||||
│ Your KTX context has been built with demo data.
|
||||
│ Select an agent to start using it.
|
||||
|
|
@ -240,13 +240,13 @@ the pre-packaged replay file at an accelerated playback rate.
|
|||
| `packages/cli/src/setup.ts` | Add `demoMode` flag to setup loop; skip models/embeddings; dispatch to demo cards for databases/sources; show demo banner; demo completion summary |
|
||||
| `packages/cli/src/setup-demo-cards.ts` | New file: `renderDemoCard()` helper, demo banner renderer, demo step definitions |
|
||||
| `packages/cli/src/setup-context.ts` | Support replay mode for demo: feed pre-packaged events at accelerated pace through existing progress view |
|
||||
| `packages/cli/src/demo.ts` | Remove or simplify `runKtxSetupDemoFromEntryMenu()` — now dispatches to the main setup loop with `demoMode: true` |
|
||||
| `packages/cli/src/demo.ts` | Remove or simplify `runKtxSetupDemoFromEntryMenu()` - now dispatches to the main setup loop with `demoMode: true` |
|
||||
| `packages/cli/src/demo-assets.ts` | Update asset list if new demo data is provided; ensure demo project setup writes valid `ktx.yaml` for agent use |
|
||||
|
||||
## Open Items
|
||||
|
||||
- **Demo data**: User will provide improved pre-packaged results (Postgres,
|
||||
dbt, Metabase, Notion). Current demo assets may need updating.
|
||||
- **Replay speed**: Exact acceleration factor TBD — should feel brisk but
|
||||
- **Replay speed**: Exact acceleration factor TBD - should feel brisk but
|
||||
give users time to read source names and status transitions. Start with
|
||||
~2x real-time and adjust.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Historic SQL Ingestion — Redesign
|
||||
# Historic SQL Ingestion - Redesign
|
||||
|
||||
**Status:** draft
|
||||
**Date:** 2026-05-11
|
||||
|
|
@ -16,12 +16,12 @@ Concrete pain points observed:
|
|||
- The output is **rigid and shallow**: deterministic slot classification (constant / categorical / runtime) and triage-signal buckets do not produce narrative an agent can use. The current downstream skills (`historic_sql_ingest`, `historic_sql_curator`) try to recover narrative from these templates but at high cost.
|
||||
- Lots of moving parts (baseline files, reset detection, atomic per-connection commit, slot heuristics, ranking formula) for what is fundamentally "find interesting queries and tell agents about them."
|
||||
|
||||
The end goal — per the user — is for ingested content to be **searchable by `ktx wiki search` and `ktx sl search` to help consumer research agents do data analysis and agentic BI**.
|
||||
The end goal - per the user - is for ingested content to be **searchable by `ktx wiki search` and `ktx sl search` to help consumer research agents do data analysis and agentic BI**.
|
||||
|
||||
## 2. Design principles
|
||||
|
||||
1. **LLMs are the right tool for narrative and clustering.** Deterministic heuristics (slot classification, ranking formulas, categorical expansion) get replaced by LLM judgement applied to aggregated, bucketed inputs.
|
||||
2. **The adapter stays LLM-free.** The existing convention — adapters are deterministic, skills do LLM work — is preserved.
|
||||
2. **The adapter stays LLM-free.** The existing convention - adapters are deterministic, skills do LLM work - is preserved.
|
||||
3. **One pipeline across dialects.** A single reader interface, a single staging shape, a single set of skills. Dialect-specific behavior lives only in the snapshot query.
|
||||
4. **No work where no signal changed.** Daily reruns should LLM only the things that actually changed.
|
||||
5. **Lean context for caller agents.** Each retrieval tier (search hit → source read → pattern read) carries only what the agent needs to make the next decision. The principle lives in prompt instructions, not in defensive schema constraints.
|
||||
|
|
@ -53,9 +53,9 @@ Reader (unified) ─▶ Aggregated snapshot ─▶ Batch SQL parse ─▶ Bu
|
|||
└──────────────────────────┬───────────────────────────────────────────────────┘
|
||||
▼
|
||||
onPullSucceeded() projection (no LLM):
|
||||
Pass A — merge `usage` into _schema/{shard}.yaml (per-shard atomic, scan-managed keys)
|
||||
Pass B — write/update pattern wiki pages (slug stability + stale handling)
|
||||
Pass C — trigger SL search re-index for changed sources
|
||||
Pass A - merge `usage` into _schema/{shard}.yaml (per-shard atomic, scan-managed keys)
|
||||
Pass B - write/update pattern wiki pages (slug stability + stale handling)
|
||||
Pass C - trigger SL search re-index for changed sources
|
||||
```
|
||||
|
||||
## 4. Hot path (LLM-free)
|
||||
|
|
@ -78,7 +78,7 @@ interface HistoricSqlReader {
|
|||
|
||||
### 4.2 Snapshot queries (one per dialect)
|
||||
|
||||
**Postgres** — `pg_stat_statements` collapsed to `queryid`:
|
||||
**Postgres** - `pg_stat_statements` collapsed to `queryid`:
|
||||
|
||||
```sql
|
||||
SELECT queryid::text AS template_id,
|
||||
|
|
@ -95,7 +95,7 @@ HAVING SUM(calls) >= @min_executions
|
|||
|
||||
`firstSeen` derives from `pg_stat_statements_info.stats_reset`; `lastSeen` is `now()`. `p50RuntimeMs` / `p95RuntimeMs` collapse to `mean_ms`. `errorRate = 0` (PG doesn't track failures in PGSS).
|
||||
|
||||
**BigQuery** — warehouse-side aggregation over `INFORMATION_SCHEMA.JOBS_BY_PROJECT`:
|
||||
**BigQuery** - warehouse-side aggregation over `INFORMATION_SCHEMA.JOBS_BY_PROJECT`:
|
||||
|
||||
```sql
|
||||
SELECT query_hash AS template_id,
|
||||
|
|
@ -115,7 +115,7 @@ GROUP BY query_hash
|
|||
HAVING COUNT(*) >= @min_executions
|
||||
```
|
||||
|
||||
**Snowflake** — analogous, over `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY`:
|
||||
**Snowflake** - analogous, over `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY`:
|
||||
|
||||
```sql
|
||||
SELECT query_hash AS template_id,
|
||||
|
|
@ -154,22 +154,22 @@ Per-row parse failures are non-fatal: the template loses table grounding (exclud
|
|||
|
||||
### 4.4 Filtering (three layers)
|
||||
|
||||
**Layer A — Warehouse-side (in the SQL above):**
|
||||
**Layer A - Warehouse-side (in the SQL above):**
|
||||
|
||||
- Noise prefixes (`SHOW`, `DESCRIBE`, `EXPLAIN`, `USE`, `SET`).
|
||||
- System catalogs (`INFORMATION_SCHEMA`, `SNOWFLAKE.ACCOUNT_USAGE`, `pg_*`, `system.*`).
|
||||
- DDL / non-analytical statement types via `statement_type` / `query_type` columns (PG falls back to prefix regex).
|
||||
- Trivial probes (`SELECT 1`, `SELECT NOW()`, `SELECT VERSION()`) — configurable.
|
||||
- Trivial probes (`SELECT 1`, `SELECT NOW()`, `SELECT VERSION()`) - configurable.
|
||||
- Minimum executions threshold (`@min_executions`, default 5).
|
||||
- Trailing window (`@window_days`, default 90) — BQ/SF only.
|
||||
- Trailing window (`@window_days`, default 90) - BQ/SF only.
|
||||
|
||||
**Layer B — Post-fetch, in-memory:**
|
||||
**Layer B - Post-fetch, in-memory:**
|
||||
|
||||
- Service-account exclusion/inclusion via configurable regex patterns; three modes (`exclude` default, `include`, `mark-only`).
|
||||
- Orchestrator boilerplate (dbt/Looker/Metabase markers) — default `mark-only` (do not drop; dbt-generated queries are often the actual business logic).
|
||||
- Orchestrator boilerplate (dbt/Looker/Metabase markers) - default `mark-only` (do not drop; dbt-generated queries are often the actual business logic).
|
||||
- Failed-query filter (BQ/SF only): templates with `errorRate > 0.9 AND executions < 10`.
|
||||
|
||||
**Layer C — Post-parse:**
|
||||
**Layer C - Post-parse:**
|
||||
|
||||
- Zero-table templates (parsed cleanly but touch no real tables) are dropped from per-table bucketization and from patterns.
|
||||
|
||||
|
|
@ -187,7 +187,7 @@ In-memory pass: a single template touching N tables ends up in N table buckets.
|
|||
patterns-input.json
|
||||
```
|
||||
|
||||
`manifest.json` is small (summary, window, counts, warnings — schema in §9).
|
||||
`manifest.json` is small (summary, window, counts, warnings - schema in §9).
|
||||
|
||||
`tables/{schema}.{name}.json` contains **bucketed** content so that DiffSet content hashes are stable when nothing material changed:
|
||||
|
||||
|
|
@ -223,7 +223,7 @@ Bucket bands are defined deterministically in code (e.g. `executionsBucket`: `<1
|
|||
|
||||
### 4.7 `chunk()` (trivial, convention-following)
|
||||
|
||||
One `WorkUnit` per `tables/*.json` file (handled by `historic_sql_table_digest`) + one `WorkUnit` referencing `patterns-input.json` (handled by `historic_sql_patterns`). No custom diff logic — the framework's `DiffSetComputerPort` already filters to changed files.
|
||||
One `WorkUnit` per `tables/*.json` file (handled by `historic_sql_table_digest`) + one `WorkUnit` referencing `patterns-input.json` (handled by `historic_sql_patterns`). No custom diff logic - the framework's `DiffSetComputerPort` already filters to changed files.
|
||||
|
||||
## 5. Cold path (LLM, via skills)
|
||||
|
||||
|
|
@ -262,7 +262,7 @@ No hard length/cap constraints in the schema. Concision is a behavioral instruct
|
|||
|
||||
### 5.2 `historic_sql_patterns`
|
||||
|
||||
One invocation per run (or a small handful if `patterns-input.json` exceeds a context budget — split deterministically by `tablesTouched` cardinality stratification).
|
||||
One invocation per run (or a small handful if `patterns-input.json` exceeds a context budget - split deterministically by `tablesTouched` cardinality stratification).
|
||||
|
||||
**Prompt:** identifies recurring analytical intents that span ≥2 tables with ≥mid executionsBucket and ≥2-5 distinct users. Output is a list of `PatternOutput`.
|
||||
|
||||
|
|
@ -288,18 +288,18 @@ export const patternOutputSchema = z.object({
|
|||
|
||||
After all skills complete and evidence is committed, run two passes. Both are pure data transformations, no LLM calls.
|
||||
|
||||
**Pass A — `_schema` shard reconciliation:**
|
||||
**Pass A - `_schema` shard reconciliation:**
|
||||
|
||||
1. Collect all `historic_sql_table_usage` evidence written this run.
|
||||
2. Group by `shardKey` (`catalog.schema`).
|
||||
3. For each shard:
|
||||
- Load existing `_schema/{shardKey}.yaml`.
|
||||
- For each table entry: if new evidence exists, merge under `usage` via `mergeUsagePreservingExternal()` (only `historicSql`-managed keys touched; user-added keys preserved — same pattern as `mergeDescriptionsPreservingExternal` at `local-enrichment-artifacts.ts:237-242`).
|
||||
- For each table entry: if new evidence exists, merge under `usage` via `mergeUsagePreservingExternal()` (only `historicSql`-managed keys touched; user-added keys preserved - same pattern as `mergeDescriptionsPreservingExternal` at `local-enrichment-artifacts.ts:237-242`).
|
||||
- For tables previously present with `historicSql`-managed `usage` but absent from this run's snapshot: set `usage.staleSince = lastSnapshotSeenAt`, clear other historicSql-managed fields.
|
||||
- Atomic write to `_schema/{shardKey}.yaml`.
|
||||
4. Trigger SL search re-index for changed sources via the existing flow (`sl-search.service.ts:91-99` detects search-text drift).
|
||||
|
||||
**Pass B — wiki pattern pages:**
|
||||
**Pass B - wiki pattern pages:**
|
||||
|
||||
1. Collect all `historic_sql_pattern` evidence written this run.
|
||||
2. Load existing wiki pages with tags `['historic-sql', 'pattern']` for this connection.
|
||||
|
|
@ -312,21 +312,21 @@ After all skills complete and evidence is committed, run two passes. Both are pu
|
|||
|
||||
## 6. Search-surface plumbing
|
||||
|
||||
### 6.1 `ktx wiki search` — no plumbing required
|
||||
### 6.1 `ktx wiki search` - no plumbing required
|
||||
|
||||
Pattern pages are written to `knowledge/global/historic-sql/{slug}.md` and are discovered by the existing `searchLocalKnowledgePages()` walk. Tags `['historic-sql', 'pattern']` enable faceted search.
|
||||
|
||||
### 6.2 `ktx sl search` — small extension
|
||||
### 6.2 `ktx sl search` - small extension
|
||||
|
||||
**6.2.1 — `SemanticLayerSource.usage` field**
|
||||
**6.2.1 - `SemanticLayerSource.usage` field**
|
||||
|
||||
Add an optional `usage` field to `SemanticLayerSource` in `packages/context/src/sl/schemas.ts`, reusing the same `tableUsageOutputSchema` from `skill-schemas.ts`. Single source of truth end-to-end.
|
||||
|
||||
**6.2.2 — `_schema` → `SemanticLayerSource` projection carries `usage`**
|
||||
**6.2.2 - `_schema` → `SemanticLayerSource` projection carries `usage`**
|
||||
|
||||
The existing projection step in `local-sl.ts` (or wherever the manifest reader builds `SemanticLayerSource` objects) needs one new field copy: `entry.usage → source.usage`.
|
||||
|
||||
**6.2.3 — `buildSemanticLayerSourceSearchText()` extension**
|
||||
**6.2.3 - `buildSemanticLayerSourceSearchText()` extension**
|
||||
|
||||
Extend the function at `sl-search.service.ts:8-74` to include usage content in the FTS5/embedding text:
|
||||
|
||||
|
|
@ -344,11 +344,11 @@ if (source.usage) {
|
|||
}
|
||||
```
|
||||
|
||||
**6.2.4 — Re-index trigger**
|
||||
**6.2.4 - Re-index trigger**
|
||||
|
||||
Already wired. Per-source content-hash detection at `sl-search.service.ts:91-99` ensures only sources whose `usage` changed re-embed.
|
||||
|
||||
**6.2.5 — Query-mode result enrichment**
|
||||
**6.2.5 - Query-mode result enrichment**
|
||||
|
||||
Extend the search result shape returned by `agent sl list --query` to include `score` and an FTS5 `snippet()` per hit. Implementation: small SQL change in `sqlite-sl-sources-index.ts` to select `snippet(local_sl_sources_fts, ...)` alongside the source row.
|
||||
|
||||
|
|
@ -510,7 +510,7 @@ export const stagedManifestSchema = z.object({
|
|||
});
|
||||
```
|
||||
|
||||
In `packages/context/src/ingest/adapters/historic-sql/skill-schemas.ts` — the **single source of truth for LLM I/O shapes**, imported by the prompt builder, the evidence parser, the projection step, the `SemanticLayerSource` type, and the `_schema` manifest entry type:
|
||||
In `packages/context/src/ingest/adapters/historic-sql/skill-schemas.ts` - the **single source of truth for LLM I/O shapes**, imported by the prompt builder, the evidence parser, the projection step, the `SemanticLayerSource` type, and the `_schema` manifest entry type:
|
||||
|
||||
```typescript
|
||||
export const tableUsageOutputSchema = z.object({
|
||||
|
|
@ -541,10 +541,10 @@ export type PatternOutput = z.infer<typeof patternOutputSchema>;
|
|||
|
||||
**Extensions to existing types:**
|
||||
|
||||
- `packages/context/src/sl/schemas.ts` — `SemanticLayerSource.usage: tableUsageOutputSchema.optional()`.
|
||||
- `packages/context/src/ingest/adapters/live-database/manifest.ts` — `LiveDatabaseManifestTableEntry.usage?: TableUsageOutput`.
|
||||
- `packages/context/src/sl/schemas.ts` - `SemanticLayerSource.usage: tableUsageOutputSchema.optional()`.
|
||||
- `packages/context/src/ingest/adapters/live-database/manifest.ts` - `LiveDatabaseManifestTableEntry.usage?: TableUsageOutput`.
|
||||
|
||||
The `_schema/{shard}.yaml` manifest version need not bump — `usage` is an additive, optional field. Validators must allow unknown future keys (audit during step 1 of §10).
|
||||
The `_schema/{shard}.yaml` manifest version need not bump - `usage` is an additive, optional field. Validators must allow unknown future keys (audit during step 1 of §10).
|
||||
|
||||
## 10. Cutover plan
|
||||
|
||||
|
|
@ -554,22 +554,22 @@ Hard cutover. No parallel codepaths. Single coordinated PR (or PR train).
|
|||
|
||||
Within `packages/context/src/ingest/adapters/historic-sql/`:
|
||||
|
||||
- `stage.ts` — rewritten
|
||||
- `stage-pgss.ts` — **deleted** (no baseline tracking)
|
||||
- `stage-pgss.test.ts`, `stage-pgss-golden.test.ts` — **deleted**
|
||||
- `historic-sql.adapter.ts` — rewritten
|
||||
- `historic-sql.adapter.test.ts` — rewritten
|
||||
- `chunk.ts` / `chunk.test.ts` — rewritten (becomes trivial)
|
||||
- `detect.ts` / `detect.test.ts` — trivial update
|
||||
- `postgres-pgss-query-history-reader.ts` — rewritten as `postgres-pgss-reader.ts`; baseline-tracking code removed
|
||||
- `bigquery-query-history-reader.ts` / `snowflake-query-history-reader.ts` — rewritten; cursor logic removed; warehouse-side GROUP BY
|
||||
- `types.ts` — rewritten
|
||||
- `stage.ts` - rewritten
|
||||
- `stage-pgss.ts` - **deleted** (no baseline tracking)
|
||||
- `stage-pgss.test.ts`, `stage-pgss-golden.test.ts` - **deleted**
|
||||
- `historic-sql.adapter.ts` - rewritten
|
||||
- `historic-sql.adapter.test.ts` - rewritten
|
||||
- `chunk.ts` / `chunk.test.ts` - rewritten (becomes trivial)
|
||||
- `detect.ts` / `detect.test.ts` - trivial update
|
||||
- `postgres-pgss-query-history-reader.ts` - rewritten as `postgres-pgss-reader.ts`; baseline-tracking code removed
|
||||
- `bigquery-query-history-reader.ts` / `snowflake-query-history-reader.ts` - rewritten; cursor logic removed; warehouse-side GROUP BY
|
||||
- `types.ts` - rewritten
|
||||
- **new** `skill-schemas.ts`
|
||||
- `errors.ts` — keep (probe errors); prune unused
|
||||
- `errors.ts` - keep (probe errors); prune unused
|
||||
|
||||
Old skills `historic_sql_ingest` and `historic_sql_curator` — audit; if only consumed by historic-sql, delete.
|
||||
Old skills `historic_sql_ingest` and `historic_sql_curator` - audit; if only consumed by historic-sql, delete.
|
||||
|
||||
`expandCategoricalTemplates`, `classifySlot`, `rankTemplate`, slot-related types — gone.
|
||||
`expandCategoricalTemplates`, `classifySlot`, `rankTemplate`, slot-related types - gone.
|
||||
|
||||
### 10.2 Existing artifacts
|
||||
|
||||
|
|
@ -596,7 +596,7 @@ Old skills `historic_sql_ingest` and `historic_sql_curator` — audit; if only c
|
|||
- `historic_sql_table_digest` + `historic_sql_patterns`.
|
||||
- `onPullSucceeded` projection passes.
|
||||
- One-time legacy cleanup.
|
||||
5. **Delete the old codepath** — same PR as step 3, ideally.
|
||||
5. **Delete the old codepath** - same PR as step 3, ideally.
|
||||
6. **Docs + setup wizard** updates.
|
||||
|
||||
### 10.4 Verification before merging
|
||||
|
|
@ -612,11 +612,11 @@ Old skills `historic_sql_ingest` and `historic_sql_curator` — audit; if only c
|
|||
### 10.5 Out of scope
|
||||
|
||||
- Embedding-based pattern clustering (rejected in favor of LLM-driven intent detection).
|
||||
- Wiki shard pages (rejected — patterns are sparse; per-page is correct).
|
||||
- Wiki shard pages (rejected - patterns are sparse; per-page is correct).
|
||||
- Incremental dialect-by-dialect rollout behind a flag.
|
||||
- A `ktx historic-sql migrate` command — cleanup runs automatically once.
|
||||
- A `ktx historic-sql migrate` command - cleanup runs automatically once.
|
||||
- Framework-level `raw-sources/` retention policy (separate concern; not introduced here).
|
||||
- Per-table wiki pages (the very problem `_schema` shards exist to avoid — see §11).
|
||||
- Per-table wiki pages (the very problem `_schema` shards exist to avoid - see §11).
|
||||
|
||||
### 10.6 Risks
|
||||
|
||||
|
|
@ -632,23 +632,23 @@ Old skills `historic_sql_ingest` and `historic_sql_curator` — audit; if only c
|
|||
|
||||
Documented so future readers don't relitigate.
|
||||
|
||||
**Per-table wiki pages** — one `.md` per table under `knowledge/global/historic-sql/`. Rejected: reintroduces the per-table-file proliferation problem (`writeLocalKnowledgePage` writes one file per page) that `_schema` shards exist to avoid. ~800 markdown files for a 1000-table warehouse, ~100 churning daily.
|
||||
**Per-table wiki pages** - one `.md` per table under `knowledge/global/historic-sql/`. Rejected: reintroduces the per-table-file proliferation problem (`writeLocalKnowledgePage` writes one file per page) that `_schema` shards exist to avoid. ~800 markdown files for a 1000-table warehouse, ~100 churning daily.
|
||||
|
||||
**Single-file all-usage page** — one giant page containing every table. Rejected: ~700 KB blob; FTS5 snippets all come from the same source; `wiki read` returns an unusable mass.
|
||||
**Single-file all-usage page** - one giant page containing every table. Rejected: ~700 KB blob; FTS5 snippets all come from the same source; `wiki read` returns an unusable mass.
|
||||
|
||||
**One file per table in a new `_usage/` directory** — same file-count problem as per-table wiki, plus needs new search plumbing.
|
||||
**One file per table in a new `_usage/` directory** - same file-count problem as per-table wiki, plus needs new search plumbing.
|
||||
|
||||
**New parallel `_usage/{shard}.yaml` shards** — same sharding benefit as merging into `_schema` but without riding SL search. Plumbing required without offsetting win.
|
||||
**New parallel `_usage/{shard}.yaml` shards** - same sharding benefit as merging into `_schema` but without riding SL search. Plumbing required without offsetting win.
|
||||
|
||||
**One wiki page per `catalog.schema`** — workable, but pages get large (200 tables per page) and only rides wiki search, not SL search. The chosen design rides both.
|
||||
**One wiki page per `catalog.schema`** - workable, but pages get large (200 tables per page) and only rides wiki search, not SL search. The chosen design rides both.
|
||||
|
||||
**Single staged `snapshot.json`** — to reduce `raw-sources/` accumulation. Rejected: required custom diff logic in `chunk()`, broke framework convention, saved bounded disk for a framework-level concern (sync retention). Per-table staged files with bucketed content is cleaner.
|
||||
**Single staged `snapshot.json`** - to reduce `raw-sources/` accumulation. Rejected: required custom diff logic in `chunk()`, broke framework convention, saved bounded disk for a framework-level concern (sync retention). Per-table staged files with bucketed content is cleaner.
|
||||
|
||||
**Embedding-based pattern clustering** — using sentence-transformer embeddings to cluster templates into themes before naming via LLM. Rejected: reintroduces clustering hyperparameters and determinism the redesign aims to avoid. The LLM does the grouping in one call from the full template list, no embedding step.
|
||||
**Embedding-based pattern clustering** - using sentence-transformer embeddings to cluster templates into themes before naming via LLM. Rejected: reintroduces clustering hyperparameters and determinism the redesign aims to avoid. The LLM does the grouping in one call from the full template list, no embedding step.
|
||||
|
||||
**Skip pattern pages entirely** — ship only `_schema` enrichment for a leaner v1. Rejected: leaves `ktx wiki search` empty of historic-sql content (loses one of two stated consumption surfaces) and forces agents to synthesize cross-cutting intents from fragmented per-table mentions.
|
||||
**Skip pattern pages entirely** - ship only `_schema` enrichment for a leaner v1. Rejected: leaves `ktx wiki search` empty of historic-sql content (loses one of two stated consumption surfaces) and forces agents to synthesize cross-cutting intents from fragmented per-table mentions.
|
||||
|
||||
**TypeScript-native SQL parser** instead of sqlglot via daemon — `node-sql-parser`, `pgsql-parser` (WASM), etc. Rejected: materially worse dialect coverage on Snowflake/BigQuery edge cases; duplicates parser logic when KTX already uses sqlglot elsewhere (`python/ktx-daemon/src/ktx_daemon/lookml.py`); AGENTS.md explicitly mandates sqlglot. Batch endpoint on the existing daemon achieves the perf win.
|
||||
**TypeScript-native SQL parser** instead of sqlglot via daemon - `node-sql-parser`, `pgsql-parser` (WASM), etc. Rejected: materially worse dialect coverage on Snowflake/BigQuery edge cases; duplicates parser logic when KTX already uses sqlglot elsewhere (`python/ktx-daemon/src/ktx_daemon/lookml.py`); AGENTS.md explicitly mandates sqlglot. Batch endpoint on the existing daemon achieves the perf win.
|
||||
|
||||
**Hard length/count caps in zod output schemas** (e.g. `narrative.max(250)`, `commonFilters.max(5)`). Rejected: arbitrary thresholds, brittle retry-on-violation paths, defensive coding for a soft concern. Concision belongs in prompt instructions; the schema validates shape.
|
||||
|
||||
|
|
@ -671,7 +671,7 @@ For a large warehouse (~800 touched tables): first-run ~$20–30, daily ~$0.20
|
|||
|
||||
## 13. Open questions
|
||||
|
||||
- Exact bucket thresholds for `executionsBucket`, `distinctUsersBucket`, etc. — to be chosen during implementation based on what produces stable hashes in practice.
|
||||
- Exact bucket thresholds for `executionsBucket`, `distinctUsersBucket`, etc. - to be chosen during implementation based on what produces stable hashes in practice.
|
||||
- Final naming of the daemon endpoint (`/sql/analyze-batch` vs alternatives).
|
||||
- Whether `historic_sql_ingest` / `historic_sql_curator` skills are consumed elsewhere — audit during step 1.
|
||||
- Whether to delete legacy wiki pages automatically or behind a confirmation flag — design assumes automatic.
|
||||
- Whether `historic_sql_ingest` / `historic_sql_curator` skills are consumed elsewhere - audit during step 1.
|
||||
- Whether to delete legacy wiki pages automatically or behind a confirmation flag - design assumes automatic.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
**Date:** 2026-05-12
|
||||
**Author:** Andrey Avtomonov
|
||||
**Status:** Design — pending implementation plan
|
||||
**Status:** Design - pending implementation plan
|
||||
|
||||
## Background and motivation
|
||||
|
||||
|
|
@ -16,7 +16,7 @@ A real-world inspection (project `/tmp/ktx-proj-1`) surfaced two failure modes t
|
|||
Root cause analysis (`packages/context/skills/notion_synthesize/SKILL.md`, `packages/context/src/ingest/tools/emit-unmapped-fallback.tool.ts`, `packages/context/src/wiki/tools/wiki-write.tool.ts`) showed three contributing factors:
|
||||
|
||||
- The synthesis LLM has no verification primitive that distinguishes a real warehouse identifier from a fabricated one. `sl_discover` only finds objects already promoted into the semantic layer; raw warehouse scans (which already exist on disk under `raw-sources/<conn>/live-database/<sync>/`) are not surfaced to the LLM at all.
|
||||
- `wiki_write` performs no body-text validation — anything the LLM emits is written.
|
||||
- `wiki_write` performs no body-text validation - anything the LLM emits is written.
|
||||
- The skill prompt itself uses `orbit_analytics.customer` as a canonical example string (`SKILL.md:70`), reinforcing the same fictional name the LLM ends up emitting.
|
||||
|
||||
Kaelio's server-side ingest WU agent (`/Users/andrey/conductor/workspaces/kaelio-main2/douala/server/src/tools/toolset-factory.service.ts`) had four verification tools that KTX dropped during the open-source extraction: `discover_data`, `entity_details`, `dictionary_search`, and `sql_execution`. The underlying connector infrastructure (`KtxScanConnector`, dialect classes, `assertReadOnlySql`, `SemanticLayerService.executeQuery`) is present in KTX, so the gap is at the tool layer, not the platform layer.
|
||||
|
|
@ -115,7 +115,7 @@ export type SupportedDriver = 'postgres'|'postgresql'|'mysql'|'sqlserver'|'snowf
|
|||
export function getDialectForDriver(driver: SupportedDriver): KtxDialect;
|
||||
```
|
||||
|
||||
Sync dispatch. The connectors' existing dialect classes already expose the same shape — `formatTableName(KtxTableRef)`, `quoteIdentifier(string)`, `mapToDimensionType(nativeType)`. The implementation plan introduces a minimal `KtxDialect` interface that these classes already satisfy structurally; no connector-internal changes required. Used by tools only for display-string parsing and error-message formatting; tools never construct executable SQL.
|
||||
Sync dispatch. The connectors' existing dialect classes already expose the same shape - `formatTableName(KtxTableRef)`, `quoteIdentifier(string)`, `mapToDimensionType(nativeType)`. The implementation plan introduces a minimal `KtxDialect` interface that these classes already satisfy structurally; no connector-internal changes required. Used by tools only for display-string parsing and error-message formatting; tools never construct executable SQL.
|
||||
|
||||
## Tool contracts
|
||||
|
||||
|
|
@ -139,7 +139,7 @@ Type: table | Native columns: 11 | PK: account_id | FKs: parent_account_id → o
|
|||
Description: One row per customer account…
|
||||
|
||||
Columns:
|
||||
- account_id (text, nullable=false, PK) — sample: ["acct_001","acct_002",…]
|
||||
- account_id (text, nullable=false, PK) - sample: ["acct_001","acct_002",…]
|
||||
- parent_account_id (text, nullable=true, FK → orbit_raw.accounts.account_id)
|
||||
- account_name (text, nullable=false)
|
||||
- …
|
||||
|
|
@ -147,7 +147,7 @@ Columns:
|
|||
Profile: rowCount=4321 distinctCount(account_id)=4321 nullRate(parent_account_id)=0.62
|
||||
```
|
||||
|
||||
When `column` is provided in a target, output is scoped to that one column. When a target doesn't resolve, output is `Not found in scan. Closest matches: …` with up to 5 candidates from `searchByName`. When the connection has no `live-database` scan, output is `No live-database scan available for connection "<name>"; run \`ktx scan\` first.` — distinct from the "not found" state.
|
||||
When `column` is provided in a target, output is scoped to that one column. When a target doesn't resolve, output is `Not found in scan. Closest matches: …` with up to 5 candidates from `searchByName`. When the connection has no `live-database` scan, output is `No live-database scan available for connection "<name>"; run \`ktx scan\` first.` - distinct from the "not found" state.
|
||||
|
||||
Structured output: `{ resolved: TableDetail[], missing: Array<{target, candidates}>, scanAvailable: boolean }`.
|
||||
|
||||
|
|
@ -165,14 +165,14 @@ input = {
|
|||
|
||||
Pipeline:
|
||||
|
||||
1. `assertReadOnlySql(sql)` — regex rejects anything starting with `insert|update|delete|merge|alter|drop|create|truncate|grant|revoke|copy|call|do|vacuum|analyze|refresh`.
|
||||
2. `limitSqlForExecution(sql, rowLimit)` — wraps as `select * from (<llm_sql>) as ktx_query_result limit N`.
|
||||
1. `assertReadOnlySql(sql)` - regex rejects anything starting with `insert|update|delete|merge|alter|drop|create|truncate|grant|revoke|copy|call|do|vacuum|analyze|refresh`.
|
||||
2. `limitSqlForExecution(sql, rowLimit)` - wraps as `select * from (<llm_sql>) as ktx_query_result limit N`.
|
||||
3. `SemanticLayerService.executeQuery(connectionName, wrappedSql)`.
|
||||
4. Format as markdown table; first ~20 rows inline; if truncated, append `… +N more rows`.
|
||||
|
||||
Structured output: `{ headers, rows, rowCount, truncated, sql, wrappedSql }`.
|
||||
|
||||
Connector errors surface verbatim (e.g., Postgres `relation "orbit_analytics.customer" does not exist`). That error message is the most valuable verification signal — it tells the LLM the identifier is fictional.
|
||||
Connector errors surface verbatim (e.g., Postgres `relation "orbit_analytics.customer" does not exist`). That error message is the most valuable verification signal - it tells the LLM the identifier is fictional.
|
||||
|
||||
Refuses `connectionName` not in `allowedConnectionNames`. Each connector's driver-level read-only enforcement (Postgres read-only transaction, BigQuery query-only jobs) is a second defence under the regex gate.
|
||||
|
||||
|
|
@ -189,9 +189,9 @@ input = {
|
|||
|
||||
Composes three searches and groups output into three sections, omitting empty sections:
|
||||
|
||||
1. **Wiki Pages** — `wiki_search({query, limit})`. Routing hint: *use `wiki_read(blockKey)` for full content*.
|
||||
2. **Semantic Layer Sources** — `sl_discover({query, connectionName})`. Routing hint: *use `sl_read_source(sourceName)` for the YAML, or `entity_details` for warehouse-shape details*.
|
||||
3. **Raw Warehouse Schema** — `WarehouseCatalogService.searchByName(connectionName, query, limit)`. Routing hint: *use `entity_details({connectionName, targets: [{display}]})` for full DDL + sample values*.
|
||||
1. **Wiki Pages** - `wiki_search({query, limit})`. Routing hint: *use `wiki_read(blockKey)` for full content*.
|
||||
2. **Semantic Layer Sources** - `sl_discover({query, connectionName})`. Routing hint: *use `sl_read_source(sourceName)` for the YAML, or `entity_details` for warehouse-shape details*.
|
||||
3. **Raw Warehouse Schema** - `WarehouseCatalogService.searchByName(connectionName, query, limit)`. Routing hint: *use `entity_details({connectionName, targets: [{display}]})` for full DDL + sample values*.
|
||||
|
||||
When `sourceName` is set, delegates entirely to `sl_discover` inspect mode and skips other sections. When all three sections are empty, output is `No matches for "<query>" across wiki, semantic layer, or raw warehouse schema. Try broader terms; this concept may not exist yet.`
|
||||
|
||||
|
|
@ -215,7 +215,7 @@ const warehouseTools = createWarehouseVerificationTools({
|
|||
// alongside emit_unmapped_fallback.
|
||||
```
|
||||
|
||||
`createWarehouseVerificationTools` returns `Record<string, Tool>` with three keys. The set is wired into every adapter's synthesis stage — no per-adapter opt-in.
|
||||
`createWarehouseVerificationTools` returns `Record<string, Tool>` with three keys. The set is wired into every adapter's synthesis stage - no per-adapter opt-in.
|
||||
|
||||
## Skill-prompt updates
|
||||
|
||||
|
|
@ -227,12 +227,12 @@ const warehouseTools = createWarehouseVerificationTools({
|
|||
## Identifier Verification Protocol
|
||||
|
||||
Before writing a wiki page or SL source on any topic:
|
||||
1. `discover_data({query: "<topic>"})` — see what wikis, SL sources, and raw tables
|
||||
1. `discover_data({query: "<topic>"})` - see what wikis, SL sources, and raw tables
|
||||
already exist. Prefer updating existing pages over creating new ones.
|
||||
|
||||
Before emitting any `schema.table` or `schema.table.column` into a wiki body,
|
||||
SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
|
||||
2. `entity_details({connectionName, targets: [{display: "<identifier>"}]})` —
|
||||
2. `entity_details({connectionName, targets: [{display: "<identifier>"}]})` -
|
||||
confirm the identifier resolves; inspect native types, FK/PK, and sampleValues.
|
||||
3. For literal values from the source (status codes, plan tiers): check whether
|
||||
they appear in `entity_details`' `sampleValues` for the relevant column.
|
||||
|
|
@ -241,7 +241,7 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
|
|||
4. If the candidate identifier still doesn't resolve, do one of:
|
||||
(a) Use `sql_execution` with `SELECT 1 FROM <ref> LIMIT 0`. If it errors,
|
||||
the identifier is fictional.
|
||||
(b) Wrap the identifier in `[unverified — from <rawPath>]` in the wiki body,
|
||||
(b) Wrap the identifier in `[unverified - from <rawPath>]` in the wiki body,
|
||||
citing the exact raw path that mentioned it.
|
||||
(c) When recording `emit_unmapped_fallback` with `no_physical_table`,
|
||||
include the failing probe error in `clarification`.
|
||||
|
|
@ -271,10 +271,10 @@ Two skills are deliberately excluded from updates: `ingest_triage` (read-only tr
|
|||
|
||||
### Cleanups beyond the four-tool addition
|
||||
|
||||
- `notion_synthesize/SKILL.md:70` — remove `orbit_analytics.customer` (placeholder).
|
||||
- `packages/context/src/ingest/tools/emit-unmapped-fallback.tool.ts:67` — same example string in the Zod `.describe()` — replace with `<schema>.<table>`.
|
||||
- `dbt_ingest/SKILL.md:24` — fix `wiki_sl_search` and `sl_describe_table` (neither tool exists in KTX).
|
||||
- `packages/context/src/sl/tools/sl-warehouse-validation.ts:93` — inline error message references the non-existent `sl_describe_table`. Replace with `sl_read_source`.
|
||||
- `notion_synthesize/SKILL.md:70` - remove `orbit_analytics.customer` (placeholder).
|
||||
- `packages/context/src/ingest/tools/emit-unmapped-fallback.tool.ts:67` - same example string in the Zod `.describe()` - replace with `<schema>.<table>`.
|
||||
- `dbt_ingest/SKILL.md:24` - fix `wiki_sl_search` and `sl_describe_table` (neither tool exists in KTX).
|
||||
- `packages/context/src/sl/tools/sl-warehouse-validation.ts:93` - inline error message references the non-existent `sl_describe_table`. Replace with `sl_read_source`.
|
||||
|
||||
## Testing strategy
|
||||
|
||||
|
|
@ -294,7 +294,7 @@ Two skills are deliberately excluded from updates: `ingest_triage` (read-only tr
|
|||
|
||||
- Extend `packages/context/src/ingest/ingest-bundle.runner.test.ts` to verify the three new tools are present in both WU-stage and reconcile-stage tool maps and refuse out-of-scope `connectionName` values.
|
||||
- New fixture-based test: stage a small `raw-sources/<conn>/live-database/<sync>/` directory with 2 tables + 1 enrichment profile, then call each tool through the runner's tool map and assert the markdown contains the expected fields. Uses the same fake-LLM harness as `notion.adapter.test.ts`.
|
||||
- One end-to-end regression test reproducing the `orbit_analytics.customer` hallucination: a fake Notion page mentioning the fictional table is fed to the synthesis stage; the run produces a wiki page where the fictional name is wrapped in `[unverified — …]` or omitted, not promoted to `tables:` frontmatter.
|
||||
- One end-to-end regression test reproducing the `orbit_analytics.customer` hallucination: a fake Notion page mentioning the fictional table is fed to the synthesis stage; the run produces a wiki page where the fictional name is wrapped in `[unverified - …]` or omitted, not promoted to `tables:` frontmatter.
|
||||
|
||||
### Prompt-bundling tests
|
||||
|
||||
|
|
@ -306,7 +306,7 @@ Extend `packages/context/src/memory/memory-runtime-assets.test.ts`:
|
|||
|
||||
### Performance guards
|
||||
|
||||
`WarehouseCatalogService` caches the per-connection table list per stage (one WorkUnit's lifetime). Tests assert second call is a cache hit. No DB index for `searchByName` in this iteration — linear scan over scan artefacts is acceptable up to ~50K columns. If volume warrants it later, a follow-up PR adds a SQLite FTS index.
|
||||
`WarehouseCatalogService` caches the per-connection table list per stage (one WorkUnit's lifetime). Tests assert second call is a cache hit. No DB index for `searchByName` in this iteration - linear scan over scan artefacts is acceptable up to ~50K columns. If volume warrants it later, a follow-up PR adds a SQLite FTS index.
|
||||
|
||||
## Rollout
|
||||
|
||||
|
|
@ -323,7 +323,7 @@ Skill prompts land last so they can reference the tools that already exist.
|
|||
|
||||
## Out of scope
|
||||
|
||||
- **Hard write-time validation in `wiki_write` / `emit_unmapped_fallback`.** A complementary spec covers regex-based identifier validation at the write boundary. Defence-in-depth — separate concern.
|
||||
- **Hard write-time validation in `wiki_write` / `emit_unmapped_fallback`.** A complementary spec covers regex-based identifier validation at the write boundary. Defence-in-depth - separate concern.
|
||||
- **SQLite FTS index for `searchByName`.** Deferred until the linear scan benchmark fails.
|
||||
- **`raw_schema_search` as a standalone tool.** `discover_data`'s raw section covers the concept-search case.
|
||||
- **`semantic_query` in the synthesis toolset.** `semantic_query` will exist in KTX for the research/chat-time agent; it is deliberately excluded from synthesis because synthesis creates SL sources rather than queries them.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
**Date:** 2026-05-13
|
||||
**Author:** Andrey Avtomonov
|
||||
**Status:** Design — pending implementation plan
|
||||
**Status:** Design - pending implementation plan
|
||||
|
||||
## Background
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ refs:
|
|||
|
||||
## Customer Update Communication Standard
|
||||
|
||||
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
|
||||
**Source:** Notion - People & Operating Norms, last edited 2026-05-07
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -12,23 +12,23 @@ refs:
|
|||
|
||||
## New Hire Week-One Onboarding Policy
|
||||
|
||||
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
|
||||
**Source:** Notion - People & Operating Norms, last edited 2026-05-07
|
||||
**Owner:** Manager (not People Ops)
|
||||
|
||||
---
|
||||
|
||||
## Policy
|
||||
|
||||
Every new hire must understand **four things by end of week one**. The manager — not People Ops — is responsible for supplying this context.
|
||||
Every new hire must understand **four things by end of week one**. The manager - not People Ops - is responsible for supplying this context.
|
||||
|
||||
## Required Week-One Knowledge
|
||||
|
||||
| # | What the new hire must understand |
|
||||
|---|---|
|
||||
| 1 | **What Orbit sells** — the core procurement workflow product and value proposition |
|
||||
| 2 | **Why procurement workflow gets messy inside a customer** — the pain points that make Orbit necessary |
|
||||
| 3 | **Which team handles which part of the customer lifecycle** — team lanes and ownership boundaries |
|
||||
| 4 | **What their first useful project is** — a concrete, scoped piece of work they can contribute to immediately |
|
||||
| 1 | **What Orbit sells** - the core procurement workflow product and value proposition |
|
||||
| 2 | **Why procurement workflow gets messy inside a customer** - the pain points that make Orbit necessary |
|
||||
| 3 | **Which team handles which part of the customer lifecycle** - team lanes and ownership boundaries |
|
||||
| 4 | **What their first useful project is** - a concrete, scoped piece of work they can contribute to immediately |
|
||||
|
||||
## Ownership
|
||||
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ tables:
|
|||
# Activation KPI Glossary
|
||||
|
||||
**Owner team:** Growth
|
||||
**Source:** Notion — Orbit Demo Home / Data Team - Onboarding / Activation KPI Glossary, last edited 2026-05-07
|
||||
**Source:** Notion - Orbit Demo Home / Data Team - Onboarding / Activation KPI Glossary, last edited 2026-05-07
|
||||
|
||||
Use this when a question is about signup-to-habit behavior. Orbit uses activation language across Growth, Product, and CS conversations.
|
||||
|
||||
|
|
@ -41,7 +41,7 @@ A customer is **activated** when **all three** of the following happen **within
|
|||
| 2. Email Verified | `customer.email_verified_at` is not null | `orbit_analytics.customer` |
|
||||
| 3. First Project | At least one row in `orbit_analytics.project` for the customer | `orbit_analytics.project` |
|
||||
| 4. Team Invite | At least one row in `orbit_analytics.invite` for the customer | `orbit_analytics.invite` |
|
||||
| 5. Activated | All of (2), (3), and (4) within 14 days of (1) | — |
|
||||
| 5. Activated | All of (2), (3), and (4) within 14 days of (1) | - |
|
||||
|
||||
## Conversion-Rate KPIs
|
||||
|
||||
|
|
@ -51,7 +51,7 @@ A customer is **activated** when **all three** of the following happen **within
|
|||
| **D14 Activation Rate** | `activated_customers_within_14_days / signups_in_cohort` |
|
||||
| **Time-to-Activate** | `median(activated_at - created_at)` in hours |
|
||||
|
||||
Growth conversations typically use D7 and D14 Activation Rate. Product and CS may ask about individual funnel steps — confirm whether they mean the full activation definition or only one stage.
|
||||
Growth conversations typically use D7 and D14 Activation Rate. Product and CS may ask about individual funnel steps - confirm whether they mean the full activation definition or only one stage.
|
||||
|
||||
## Source Notes
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ sl_refs:
|
|||
- mart_account_activity
|
||||
---
|
||||
|
||||
# Activation Policy Change — January 2026
|
||||
# Activation Policy Change - January 2026
|
||||
|
||||
**Governed metric key:** `activated_accounts`
|
||||
**Owner team:** growth
|
||||
|
|
@ -23,8 +23,8 @@ sl_refs:
|
|||
|
||||
The activation workflow changed on **2026-01-15**. All activation events are tagged with `policy_version`:
|
||||
|
||||
- `pre_2026_01_15` — events before the workflow update
|
||||
- `post_2026_01_15` — events after the workflow update
|
||||
- `pre_2026_01_15` - events before the workflow update
|
||||
- `post_2026_01_15` - events after the workflow update
|
||||
|
||||
## Activation Event Types
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ sl_refs:
|
|||
- mart_account_segments
|
||||
---
|
||||
|
||||
# ARR — Contract-First Definition
|
||||
# ARR - Contract-First Definition
|
||||
|
||||
**Governed metric key:** `arr`
|
||||
**Owner team:** finance
|
||||
|
|
@ -30,10 +30,10 @@ The dbt test on `mart_arr_daily.arr_cents` asserts the value equals **1,874,200,
|
|||
|
||||
## Intermediate model
|
||||
|
||||
`int_active_contract_arr` — active contract ARR as of 2026-03-31 (grain: `contract_id`).
|
||||
`int_active_contract_arr` - active contract ARR as of 2026-03-31 (grain: `contract_id`).
|
||||
|
||||
## Related
|
||||
|
||||
- `stg_contracts` — contract records (status: draft, active, cancelled, expired)
|
||||
- `stg_subscriptions` — fallback ARR source (status: active, cancelled, past_due, trialing)
|
||||
- `mart_arr_daily` — board-prep daily ARR mart
|
||||
- `stg_contracts` - contract records (status: draft, active, cancelled, expired)
|
||||
- `stg_subscriptions` - fallback ARR source (status: active, cancelled, past_due, trialing)
|
||||
- `mart_arr_daily` - board-prep daily ARR mart
|
||||
|
|
|
|||
|
|
@ -15,14 +15,14 @@ refs:
|
|||
|
||||
# Orbit Company Overview
|
||||
|
||||
**Source:** Notion — Orbit Demo Home / Company Overview + Orbit Demo Home (root), last edited 2026-05-07
|
||||
**Source:** Notion - Orbit Demo Home / Company Overview + Orbit Demo Home (root), last edited 2026-05-07
|
||||
|
||||
## What Orbit Sells
|
||||
|
||||
Orbit sells procurement workflow and spend-control software. The core value proposition: route purchase requests, collect approvals, onboard suppliers, and issue purchase orders without turning every exception into a status hunt.
|
||||
|
||||
**Primary buyers:** Finance, Procurement, Business Operations.
|
||||
**Daily users:** department admins, office managers, IT leads, legal ops partners — anyone who has to get a vendor through the building.
|
||||
**Daily users:** department admins, office managers, IT leads, legal ops partners - anyone who has to get a vendor through the building.
|
||||
|
||||
## Product Workflow
|
||||
|
||||
|
|
|
|||
|
|
@ -21,18 +21,18 @@ sl_refs:
|
|||
|
||||
## Risk Levels
|
||||
|
||||
`low`, `medium`, `high` — derived from two signal types:
|
||||
`low`, `medium`, `high` - derived from two signal types:
|
||||
|
||||
1. **Support ticket signals** (`stg_support_tickets`): open or pending tickets with severity `high` or `critical` increase risk.
|
||||
2. **Procurement activity signals** (`stg_purchase_requests`, `stg_purchase_orders`): recent qualifying procurement actions reduce risk.
|
||||
|
||||
## Intermediate Model
|
||||
|
||||
`int_customer_health_signals` — combines open critical ticket count and recent procurement action count per account.
|
||||
`int_customer_health_signals` - combines open critical ticket count and recent procurement action count per account.
|
||||
|
||||
## Mart
|
||||
|
||||
`mart_customer_health` — account-grain risk mart as of **2026-03-31**.
|
||||
`mart_customer_health` - account-grain risk mart as of **2026-03-31**.
|
||||
|
||||
- `account_id`: dbt not_null, unique
|
||||
- `risk_level`: dbt accepted_values [low, medium, high]
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ refs:
|
|||
|
||||
## Customer Stakeholder Needs by Role
|
||||
|
||||
**Source:** Notion — Product & Customers, last edited 2026-05-07
|
||||
**Source:** Notion - Product & Customers, last edited 2026-05-07
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -26,7 +26,7 @@ These are recurring, role-specific customer needs observed across accounts. Use
|
|||
| Role | Primary Need | Implication |
|
||||
|---|---|---|
|
||||
| **Finance** | Committed spend visibility earlier in the procurement cycle | Surface budget commitments at request approval, not at PO creation |
|
||||
| **Department leaders** | Request speed — faster time from request to approval | Reduce approval routing friction; minimize back-and-forth |
|
||||
| **Department leaders** | Request speed - faster time from request to approval | Reduce approval routing friction; minimize back-and-forth |
|
||||
| **Procurement** | Supplier file complete before the first invoice | Supplier onboarding must be finished before PO is issued, not after |
|
||||
| **Legal** | Fewer emergency reviews | Route contracts with legal implications earlier; avoid last-minute escalations |
|
||||
| **Customer Success (internal)** | Renewal risk visible before the account is already annoyed | CS needs leading indicators of dissatisfaction, not lagging ones |
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ tables:
|
|||
|
||||
**Table:** `orbit_analytics.customer`
|
||||
**Grain:** one row per signed-up customer
|
||||
**Source:** Notion — Orbit Demo Home / Data Team - Onboarding / Orbit Customers Source, last edited 2026-05-07
|
||||
**Source:** Notion - Orbit Demo Home / Data Team - Onboarding / Orbit Customers Source, last edited 2026-05-07
|
||||
|
||||
Use this when a question needs customer identity, plan tier, signup timing, recent activity, or the standard customer joins.
|
||||
|
||||
|
|
@ -29,7 +29,7 @@ Use this when a question needs customer identity, plan tier, signup timing, rece
|
|||
| Column | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | number | Primary key, surrogate key |
|
||||
| `email` | string | Login email, unique — **do not use as join key** |
|
||||
| `email` | string | Login email, unique - **do not use as join key** |
|
||||
| `name` | string | Display name |
|
||||
| `country` | string | ISO 3166-1 alpha-2 code |
|
||||
| `plan_tier` | string | One of `free`, `pro`, `enterprise` |
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ refs:
|
|||
|
||||
## How We Work
|
||||
|
||||
**Source:** Notion — Orbit Demo Home / How We Work, last edited 2026-05-07
|
||||
**Source:** Notion - Orbit Demo Home / How We Work, last edited 2026-05-07
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -30,7 +30,7 @@ refs:
|
|||
|---|---|
|
||||
| **Monday** | Commitments and dependency checks |
|
||||
| **Tuesday – Thursday** | Customer calls, product work, implementation, and building |
|
||||
| **Friday** | Closing loops — review what shipped, what slipped, and write down any decisions |
|
||||
| **Friday** | Closing loops - review what shipped, what slipped, and write down any decisions |
|
||||
|
||||
Use this rhythm when scheduling work, meetings, or reviews. Do not schedule decision-making meetings on Fridays; use Friday to record decisions already made.
|
||||
|
||||
|
|
@ -64,11 +64,11 @@ These are explicitly codified rules Orbit has identified as recurring failure mo
|
|||
- **Escalations are coordination tools, not indicators of individual failure.** Escalating is the correct behavior when a problem exceeds the current team's ability to resolve it alone.
|
||||
- When escalating, the person escalating must:
|
||||
1. Bring in the right people (those with authority or context to unblock).
|
||||
2. Summarize current state clearly — what has been tried, what is blocked, and why.
|
||||
2. Summarize current state clearly - what has been tried, what is blocked, and why.
|
||||
3. Name the customer impact explicitly.
|
||||
4. Keep updates moving until the risk is resolved or a workaround is established.
|
||||
- Escalations that stall because no one owns the next update are a process failure, not a customer failure.
|
||||
- An escalation is closed when the risk is resolved or a documented workaround is in place — not when the immediate noise stops.
|
||||
- An escalation is closed when the risk is resolved or a documented workaround is in place - not when the immediate noise stops.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ refs:
|
|||
|
||||
## Known Product Gaps and Friction Points
|
||||
|
||||
**Source:** Notion — Product & Customers (Notes from Recent Customer Calls), last edited 2026-05-07
|
||||
**Source:** Notion - Product & Customers (Notes from Recent Customer Calls), last edited 2026-05-07
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -33,8 +33,8 @@ tables:
|
|||
|
||||
## Key measures (SL: `mart_account_activity`)
|
||||
|
||||
- `avg_pre_policy_activation_rate` — `avg(pre_policy_30_day_activation_rate)`
|
||||
- `avg_post_policy_activation_rate` — `avg(post_policy_30_day_activation_rate)`
|
||||
- `avg_pre_policy_activation_rate` - `avg(pre_policy_30_day_activation_rate)`
|
||||
- `avg_post_policy_activation_rate` - `avg(post_policy_30_day_activation_rate)`
|
||||
|
||||
## Common query patterns
|
||||
|
||||
|
|
|
|||
|
|
@ -37,10 +37,10 @@ tables:
|
|||
|
||||
## Key measures (SL: `mart_account_segments`)
|
||||
|
||||
- `account_count` — `count(*)`
|
||||
- `total_contract_arr_cents` — `sum(contract_arr_cents)`
|
||||
- `active_contract_arr_cents` — `sum(contract_arr_cents)` where `contract_status = 'active'`
|
||||
- `active_contract_arr_millions` — active ARR in $M
|
||||
- `account_count` - `count(*)`
|
||||
- `total_contract_arr_cents` - `sum(contract_arr_cents)`
|
||||
- `active_contract_arr_cents` - `sum(contract_arr_cents)` where `contract_status = 'active'`
|
||||
- `active_contract_arr_millions` - active ARR in $M
|
||||
|
||||
## Common query patterns
|
||||
|
||||
|
|
|
|||
|
|
@ -31,8 +31,8 @@ tables:
|
|||
|
||||
## Key measures (SL: `mart_arr_daily`)
|
||||
|
||||
- `total_arr_cents` — `sum(arr_cents)`
|
||||
- `arr_millions` — `round(sum(arr_cents) / 100000000.0, 3)` — ARR in $M
|
||||
- `total_arr_cents` - `sum(arr_cents)`
|
||||
- `arr_millions` - `round(sum(arr_cents) / 100000000.0, 3)` - ARR in $M
|
||||
|
||||
## Common query patterns
|
||||
|
||||
|
|
|
|||
|
|
@ -38,8 +38,8 @@ tables:
|
|||
|
||||
## Key measures (SL: `mart_nrr_quarterly`)
|
||||
|
||||
- `avg_nrr` — `avg(net_revenue_retention)` across all rows
|
||||
- `avg_nrr_enterprise` — `avg(net_revenue_retention)` filtered to `segment = 'enterprise'`
|
||||
- `avg_nrr` - `avg(net_revenue_retention)` across all rows
|
||||
- `avg_nrr_enterprise` - `avg(net_revenue_retention)` filtered to `segment = 'enterprise'`
|
||||
- `total_expansion_arr_cents`, `total_contraction_arr_cents`, `total_churned_arr_cents`
|
||||
|
||||
## Common query patterns
|
||||
|
|
|
|||
|
|
@ -32,8 +32,8 @@ tables:
|
|||
|
||||
## Key measures (SL: `mart_procurement_activity`)
|
||||
|
||||
- `total_active_requesters` — `sum(active_requesters)`
|
||||
- `active_requesters_200k_threshold` — `sum(active_requesters)` where `contract_arr_threshold_cents = 20000000`
|
||||
- `total_active_requesters` - `sum(active_requesters)`
|
||||
- `active_requesters_200k_threshold` - `sum(active_requesters)` where `contract_arr_threshold_cents = 20000000`
|
||||
|
||||
## Common query patterns
|
||||
|
||||
|
|
@ -44,4 +44,4 @@ tables:
|
|||
|
||||
- `active_requesters` counts non-internal, non-test requesters on large active contracts. See [orbit-procurement-qualifying-actions](orbit-procurement-qualifying-actions).
|
||||
- The standard threshold is `contract_arr_threshold_cents = 20000000` ($200k ARR).
|
||||
- Always filter by `contract_arr_threshold_cents` — the table contains rows for multiple threshold values.
|
||||
- Always filter by `contract_arr_threshold_cents` - the table contains rows for multiple threshold values.
|
||||
|
|
|
|||
|
|
@ -36,12 +36,12 @@ tables:
|
|||
|
||||
## Key measures (SL: `mart_revenue_daily`)
|
||||
|
||||
- `total_gross_revenue_cents` — `sum(gross_revenue_cents)`
|
||||
- `total_credits_cents` — `sum(credits_cents)`
|
||||
- `total_refunds_cents` — `sum(refunds_cents)`
|
||||
- `total_net_revenue_cents` — `sum(net_revenue_cents)`
|
||||
- `net_revenue_millions` — `round(sum(net_revenue_cents) / 100000000.0, 3)`
|
||||
- `gross_revenue_millions` — `round(sum(gross_revenue_cents) / 100000000.0, 3)`
|
||||
- `total_gross_revenue_cents` - `sum(gross_revenue_cents)`
|
||||
- `total_credits_cents` - `sum(credits_cents)`
|
||||
- `total_refunds_cents` - `sum(refunds_cents)`
|
||||
- `total_net_revenue_cents` - `sum(net_revenue_cents)`
|
||||
- `net_revenue_millions` - `round(sum(net_revenue_cents) / 100000000.0, 3)`
|
||||
- `gross_revenue_millions` - `round(sum(gross_revenue_cents) / 100000000.0, 3)`
|
||||
|
||||
## Common query patterns
|
||||
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ sl_refs:
|
|||
- mart_nrr_quarterly
|
||||
---
|
||||
|
||||
# Orbit Metabase SQL Library — Patterns & Conventions
|
||||
# Orbit Metabase SQL Library - Patterns & Conventions
|
||||
|
||||
Collection **7 "SQL Library"** (parent: Orbit Showcase, collection 5) contains reference queries that demonstrate how to write Metabase native SQL against the Orbit analytics marts. Cards here are intentionally illustrative; several have `dashboardCount: 0` and are not embedded in live dashboards.
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ sl_refs:
|
|||
- mart_nrr_quarterly
|
||||
---
|
||||
|
||||
# NRR — Discount Expiration Treatment
|
||||
# NRR - Discount Expiration Treatment
|
||||
|
||||
**Governed metric key:** `net_revenue_retention`
|
||||
**Owner team:** analytics
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ sl_refs:
|
|||
- mart_procurement_activity
|
||||
---
|
||||
|
||||
# Procurement — Qualifying Actions & Weekly Active Requesters
|
||||
# Procurement - Qualifying Actions & Weekly Active Requesters
|
||||
|
||||
**Governed metric key:** `weekly_active_requesters`
|
||||
**Owner team:** product
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ refs:
|
|||
|
||||
## Orbit Product Design Principles
|
||||
|
||||
**Source:** Notion — Product & Customers, last edited 2026-05-07
|
||||
**Source:** Notion - Product & Customers, last edited 2026-05-07
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ refs:
|
|||
|
||||
## Product Review Checklist
|
||||
|
||||
**Source:** Notion — Product & Customers, last edited 2026-05-07
|
||||
**Source:** Notion - Product & Customers, last edited 2026-05-07
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ sl_refs:
|
|||
- mart_revenue_daily
|
||||
---
|
||||
|
||||
# Revenue — Gross-to-Net Reconciliation
|
||||
# Revenue - Gross-to-Net Reconciliation
|
||||
|
||||
**Governed metric key:** `net_revenue`
|
||||
**Owner team:** finance
|
||||
|
|
@ -25,7 +25,7 @@ sl_refs:
|
|||
net_revenue = gross_revenue - credits - refunds
|
||||
```
|
||||
|
||||
All amounts are in **cents** (USD only — `stg_invoices.currency` is asserted to be `USD`).
|
||||
All amounts are in **cents** (USD only - `stg_invoices.currency` is asserted to be `USD`).
|
||||
|
||||
## Components
|
||||
|
||||
|
|
@ -38,12 +38,12 @@ All amounts are in **cents** (USD only — `stg_invoices.currency` is asserted t
|
|||
|
||||
## Intermediate model
|
||||
|
||||
`int_revenue_components` — daily gross, credit, refund, and net revenue components.
|
||||
`int_revenue_components` - daily gross, credit, refund, and net revenue components.
|
||||
|
||||
## Quality Gates
|
||||
|
||||
- `reconciliation_check` must be `true` on every row of `mart_revenue_daily`.
|
||||
- `assert_february_2026_net_revenue` — a dbt singular test covering February 2026 net revenue total.
|
||||
- `assert_february_2026_net_revenue` - a dbt singular test covering February 2026 net revenue total.
|
||||
|
||||
## Line Item Types (`stg_invoice_line_items`)
|
||||
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ refs:
|
|||
|
||||
## Sales Ops → Customer Success Implementation Handoff
|
||||
|
||||
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
|
||||
**Source:** Notion - People & Operating Norms, last edited 2026-05-07
|
||||
**Owner:** Sales Ops (sender), Customer Success (receiver)
|
||||
|
||||
---
|
||||
|
|
@ -27,7 +27,7 @@ Sales Ops must complete the handoff **before the first implementation call**. Cu
|
|||
|
||||
| Field | Notes |
|
||||
|---|---|
|
||||
| Current plan | Starter / Growth / Enterprise — use canonical plan name |
|
||||
| Current plan | Starter / Growth / Enterprise - use canonical plan name |
|
||||
| Account segment | self_serve / commercial / enterprise (see `orbit-plan-segment-normalization`) |
|
||||
| Contract shape | Term, ARR, any discounts or custom terms |
|
||||
| Renewal contact | Named person on the customer side responsible for renewal |
|
||||
|
|
@ -38,7 +38,7 @@ Sales Ops must complete the handoff **before the first implementation call**. Cu
|
|||
|
||||
- **Sales Ops** is responsible for populating and delivering the handoff before the first implementation call.
|
||||
- **Customer Success** is responsible for flagging missing fields to Sales Ops before the call, not during or after.
|
||||
- If a field is unknown at handoff time, Sales Ops must note it explicitly as "unknown — to be resolved by [date]" rather than leaving it blank.
|
||||
- If a field is unknown at handoff time, Sales Ops must note it explicitly as "unknown - to be resolved by [date]" rather than leaving it blank.
|
||||
|
||||
## Common Failure Mode
|
||||
|
||||
|
|
@ -51,7 +51,7 @@ Handoffs that omit contract shape or renewal contact force CS to re-engage Sales
|
|||
- Enterprise accounts with parent/child account structures require extra care during handoff.
|
||||
- Small assumptions made during handoff in these accounts tend to produce large downstream problems (billing mismatches, approval routing failures, supplier onboarding gaps).
|
||||
- When the account has parent/child complexity, Sales Ops must explicitly flag it in the handoff and document the account hierarchy before the first implementation call.
|
||||
- CS should treat any undocumented parent/child relationship as a blocker — do not proceed with implementation setup until the structure is confirmed.
|
||||
- CS should treat any undocumented parent/child relationship as a blocker - do not proceed with implementation setup until the structure is confirmed.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ You are backfilling knowledge from a historical chat transcript or archived SQL
|
|||
</role>
|
||||
|
||||
<stance>
|
||||
Moderately conservative. Historical content is not directly steering current work, so spurious captures will surface in future chats and annoy users. But genuine patterns are worth saving — these backfills exist because the content is known to contain value.
|
||||
Moderately conservative. Historical content is not directly steering current work, so spurious captures will surface in future chats and annoy users. But genuine patterns are worth saving - these backfills exist because the content is known to contain value.
|
||||
|
||||
Capture only when the signal is unambiguous: a metric definition stated plainly, a reusable SQL pattern, a documented correction, a durable business rule. Skip casual chatter and ambiguous interpretations.
|
||||
</stance>
|
||||
|
|
@ -12,10 +12,10 @@ Capture only when the signal is unambiguous: a metric definition stated plainly,
|
|||
1. Read the wiki and SL indexes to avoid creating duplicates.
|
||||
2. If the content has wiki-style signal, load the `wiki_capture` skill and follow its workflow.
|
||||
3. If the content has SL-style signal, load the `sl` skill and follow its Part 3 workflow.
|
||||
4. Prefer updating existing entries over creating new ones — backfills often duplicate existing knowledge.
|
||||
4. Prefer updating existing entries over creating new ones - backfills often duplicate existing knowledge.
|
||||
5. When done, exit the loop.
|
||||
</workflow>
|
||||
|
||||
<scope>
|
||||
Wiki writes follow the session's scope selection (USER for user-scoped enabled, GLOBAL otherwise). The `wiki_write` tool picks automatically — focus on capture judgment.
|
||||
Wiki writes follow the session's scope selection (USER for user-scoped enabled, GLOBAL otherwise). The `wiki_write` tool picks automatically - focus on capture judgment.
|
||||
</scope>
|
||||
|
|
|
|||
|
|
@ -3,12 +3,12 @@ You are the reconciliation agent for a multi-file ingest bundle. Stage 3 WorkUni
|
|||
</role>
|
||||
|
||||
<stance>
|
||||
Parsimonious. Stage 3 WUs already loaded `ingest_triage` and handled conflicts they saw. Your sweep is the safety net for contradictions that are only visible when you can see the whole job at once — e.g. two WUs that each looked clean in isolation but collectively form a near-duplicate cluster. Do not redo work Stage 3 already did.
|
||||
Parsimonious. Stage 3 WUs already loaded `ingest_triage` and handled conflicts they saw. Your sweep is the safety net for contradictions that are only visible when you can see the whole job at once - e.g. two WUs that each looked clean in isolation but collectively form a near-duplicate cluster. Do not redo work Stage 3 already did.
|
||||
</stance>
|
||||
|
||||
<workflow>
|
||||
1. Load `ingest_triage`, then `sl_capture` + `wiki_capture`.
|
||||
2. Call `stage_list()` for the full index of this job's writes. If it is empty AND you have no evictions, exit — the runner short-circuits this case but the skill still teaches you to bail fast.
|
||||
2. Call `stage_list()` for the full index of this job's writes. If it is empty AND you have no evictions, exit - the runner short-circuits this case but the skill still teaches you to bail fast.
|
||||
3. If the system prompt includes `<canonical_pins>`, apply those pins before flagging a same-name or near-duplicate conflict. A pinned `canonicalArtifactKey` keeps the contested name when it is present in the Stage Index; competing variants keep or receive disambiguated names.
|
||||
4. Sweep both exact-key conflicts and near-duplicate writes. Compare WUs that wrote overlapping SL source names, overlapping wiki keys, the same `tables:` or `sl_refs:` action details, or obviously equivalent topic titles under different wiki keys. Call `stage_diff` to see the actual difference, and use `wiki_read`/`sl_read_source` when two different keys appear to describe the same table, metric, or source-of-truth mapping. If they're the same content, leave one canonical artifact and record the duplicate as subsumed. If they differ per `ingest_triage` rules, apply the correct resolution (rename + capture; election of canonical; silent replace for expression-only re-ingest change; or pinned canonical), then call `emit_conflict_resolution` with the artifact key and decision.
|
||||
5. For any `wiki_write`, `wiki_remove`, `sl_write_source`, or `sl_edit_source` call you make during reconciliation, include `rawPaths` with only the raw paths that directly caused that reconciliation action.
|
||||
|
|
@ -24,6 +24,6 @@ Wiki keys must be flat slugs, not directory paths. If a Stage 3 page used a path
|
|||
</scope>
|
||||
|
||||
<do_not>
|
||||
- Do not overwrite a Stage 3 WU's resolution that already matches `ingest_triage` output — that's churn.
|
||||
- Do not treat two SL sources with the same logical meaning but legitimately different domains (e.g. `finance.revenue` and `marketing.revenue`) as a conflict — that's by design.
|
||||
- Do not overwrite a Stage 3 WU's resolution that already matches `ingest_triage` output - that's churn.
|
||||
- Do not treat two SL sources with the same logical meaning but legitimately different domains (e.g. `finance.revenue` and `marketing.revenue`) as a conflict - that's by design.
|
||||
</do_not>
|
||||
|
|
|
|||
|
|
@ -10,19 +10,19 @@ A single artifact typically produces multiple actions: one SL source per table/v
|
|||
|
||||
<workflow>
|
||||
1. Review the wiki and SL indexes in the prompt. Prefer updating existing entries over creating duplicates.
|
||||
2. Load the `sl` skill for SL-writes and `wiki_capture` for wiki-writes. Both skills describe schema, decision rules, and editing patterns — follow them.
|
||||
2. Load the `sl` skill for SL-writes and `wiki_capture` for wiki-writes. Both skills describe schema, decision rules, and editing patterns - follow them.
|
||||
3. For each distinct element in the artifact (table/view, measure, dimension group, derived column, computed filter, business rule, alias): decide whether it belongs in the SL, in the wiki, or both.
|
||||
4. Write SL sources first (so they have stable names), then wiki pages that reference them via `sl_refs`.
|
||||
5. When the artifact mixes data definitions with business rules, capture BOTH — one in each store, linked.
|
||||
5. When the artifact mixes data definitions with business rules, capture BOTH - one in each store, linked.
|
||||
6. When you're done, exit the loop without calling any more tools.
|
||||
</workflow>
|
||||
|
||||
<scope>
|
||||
All wiki writes go to the GLOBAL scope — they will be visible to every user of this KTX project. Phrase wiki pages as objective business knowledge, not personal preference. The `wiki_write` tool handles scope selection automatically for external ingest.
|
||||
All wiki writes go to the GLOBAL scope - they will be visible to every user of this KTX project. Phrase wiki pages as objective business knowledge, not personal preference. The `wiki_write` tool handles scope selection automatically for external ingest.
|
||||
</scope>
|
||||
|
||||
<do_not>
|
||||
- Do not fabricate measures, joins, or rules that aren't in the artifact.
|
||||
- Do not invent column names. If a type is unclear, omit it rather than guess.
|
||||
- Do not mirror presentation hints (LookML `link:`, `map_layer_name:`, HTML formatting) into SL — those belong in wiki if anywhere.
|
||||
- Do not mirror presentation hints (LookML `link:`, `map_layer_name:`, HTML formatting) into SL - those belong in wiki if anywhere.
|
||||
</do_not>
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
<role>
|
||||
You capture durable knowledge from an analytics assistant's chat turn. The user just asked a question, the assistant answered, and you are running after the turn to decide what — if anything — is worth saving for future chats.
|
||||
You capture durable knowledge from an analytics assistant's chat turn. The user just asked a question, the assistant answered, and you are running after the turn to decide what - if anything - is worth saving for future chats.
|
||||
</role>
|
||||
|
||||
<criteria>
|
||||
|
|
@ -21,7 +21,7 @@ Skip:
|
|||
2. Identify durable knowledge OR reusable data patterns in the turn.
|
||||
3. If the turn has wiki-style signal (preferences, definitions, conventions), load the `wiki_capture` skill and follow its workflow.
|
||||
4. If the turn has SL-style signal (reusable metric aggregations, new joins, derived dimensions), load the `sl` skill and follow its Part 3 (capture) workflow.
|
||||
5. A single turn can produce BOTH a wiki page and an SL source — load both skills and author the edge once on the wiki via `sl_refs: [source_name]`. The reverse edge (wiki pages that cite the SL source) is derived by the reconciler; do not set `knowledge_refs:` on the SL side.
|
||||
5. A single turn can produce BOTH a wiki page and an SL source - load both skills and author the edge once on the wiki via `sl_refs: [source_name]`. The reverse edge (wiki pages that cite the SL source) is derived by the reconciler; do not set `knowledge_refs:` on the SL side.
|
||||
6. When you're done, exit the loop without calling any more tools. Do NOT emit a final text summary.
|
||||
</workflow>
|
||||
|
||||
|
|
|
|||
|
|
@ -12,11 +12,11 @@ Reusable templates and scripts are durable knowledge regardless of subject matte
|
|||
|
||||
Analytics evidence (BI tools like Looker, Metabase, Tableau) is durable knowledge of *how the organization defines its metrics and segments*. The `signals.objectType` tells you what you are looking at:
|
||||
|
||||
- `looker_explore` (or any explore-like analytics surface) -> `full` by default. Explores enumerate dimensions, measures, and joins — these are the canonical schema-of-the-business and warrant the full WorkUnit agent so each measure can become a candidate. Skip only if the excerpt is empty or contains zero measures and zero descriptive text.
|
||||
- `looker_explore` (or any explore-like analytics surface) -> `full` by default. Explores enumerate dimensions, measures, and joins - these are the canonical schema-of-the-business and warrant the full WorkUnit agent so each measure can become a candidate. Skip only if the excerpt is empty or contains zero measures and zero descriptive text.
|
||||
- `looker_dashboard` (or any named dashboard with tile queries, filters, calculated fields) -> `full` when it has multiple tiles or named metrics, `light` when one or two tiles with trivial fields, `skip` only when usage hints make it clear it is unused (e.g. `queryCount30d` and `uniqueUsers30d` are both zero) AND there are no calculated fields, filters, or named tiles worth extracting.
|
||||
- `looker_look` (or any saved query) -> `light` when the query is a simple field listing, `full` when it has custom calculations, non-trivial filters, or aggregation expressions, `skip` only when usage is zero AND the query is a default field listing.
|
||||
|
||||
Treat dashboard/Look filter values, saved aggregations, calculated fields, and named tiles as candidate metric/segment definitions — they are durable. Do **not** mark BI evidence as `skip` solely because it is "configuration" or "tied to a data model"; that is exactly the durable knowledge we want to capture.
|
||||
Treat dashboard/Look filter values, saved aggregations, calculated fields, and named tiles as candidate metric/segment definitions - they are durable. Do **not** mark BI evidence as `skip` solely because it is "configuration" or "tied to a data model"; that is exactly the durable knowledge we want to capture.
|
||||
|
||||
Examples:
|
||||
|
||||
|
|
|
|||
|
|
@ -6,13 +6,13 @@ callers: [memory_agent]
|
|||
|
||||
# dbt → KTX (bundle ingest)
|
||||
|
||||
Use this skill for **uploaded** dbt projects (`dbt_project.yml` at stage root, `models/**`, `sources/**`, `schema.yml`). There is **no** `fetch()` in v1 — scheduled `dbt parse` / `manifest.json` pulls are out of scope; host-provided dbt sync may still backfill structured test metadata into `_schema` on the next sync.
|
||||
Use this skill for **uploaded** dbt projects (`dbt_project.yml` at stage root, `models/**`, `sources/**`, `schema.yml`). There is **no** `fetch()` in v1 - scheduled `dbt parse` / `manifest.json` pulls are out of scope; host-provided dbt sync may still backfill structured test metadata into `_schema` on the next sync.
|
||||
|
||||
## Mapping (models / sources → SL)
|
||||
|
||||
| dbt | KTX | Notes |
|
||||
|-----|--------|--------|
|
||||
| `models:` entry with `columns:` | **Overlay** on the manifest table with the same name (after `discover_data` / `entity_details`) | One SL source per physical table; model name may differ from DB name — resolve with `read_raw_file` + warehouse context. |
|
||||
| `models:` entry with `columns:` | **Overlay** on the manifest table with the same name (after `discover_data` / `entity_details`) | One SL source per physical table; model name may differ from DB name - resolve with `read_raw_file` + warehouse context. |
|
||||
| `sources:` → `tables:` | Same as models; use `identifier` when present instead of logical `name`. | Schema + name must match how the connection sees tables. |
|
||||
| Column `description` | `descriptions.user` or merged `descriptions` map on the column | Do not overwrite `dbt` description keys from sync. |
|
||||
| `data_tests: not_null` / `unique` | Short hint in column `descriptions` or notes: “dbt: not null”, “dbt: unique” | Full structured metadata lands in manifest via **sync**; the skill keeps bundle-time SL text useful for the agent. |
|
||||
|
|
@ -73,4 +73,4 @@ If the same bundle also has MetricFlow `semantic_models:` / `metrics:`, the **`m
|
|||
- Do not invent column names, grain keys, or measure expressions from dbt model names, descriptions, tests, or common naming patterns.
|
||||
- Do not write `columns:`, `grain:`, or `measures:` for a dbt model unless those exact column names are confirmed by dbt YAML columns or warehouse schema discovery.
|
||||
- Do not invent joins from `relationships` tests if the target model/table is not found in SL or the warehouse.
|
||||
- Do not read `peerFileIndex` paths — use `read_raw_file` only on `rawFiles` and `dependencyPaths` from the WorkUnit.
|
||||
- Do not read `peerFileIndex` paths - use `read_raw_file` only on `rawFiles` and `dependencyPaths` from the WorkUnit.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ description: Classify and resolve conflicts detected during bundle ingest (struc
|
|||
callers: [memory_agent]
|
||||
---
|
||||
|
||||
# Ingest Triage — conflict classification and resolution
|
||||
# Ingest Triage - conflict classification and resolution
|
||||
|
||||
This skill is loaded in two contexts:
|
||||
- By a Stage 3 WorkUnit agent when `sl_discover` reveals that a prior WU (or a prior sync) already wrote something that overlaps with what the current WU is about to write.
|
||||
|
|
@ -15,12 +15,12 @@ Apply the rules below before every write that could collide with an existing art
|
|||
## Decision tree
|
||||
|
||||
1. **Is this the same artifact I'm producing now, or a different one with the same name?**
|
||||
Read both. If names match and content matches (modulo whitespace): no conflict — skip the write, the prior one stands.
|
||||
Read both. If names match and content matches (modulo whitespace): no conflict - skip the write, the prior one stands.
|
||||
|
||||
2. **If content differs, is it an expression-only change (e.g. a different `sql:` body for the same measure name, same grain, same columns)?**
|
||||
Re-ingest change (expression-only): silently replace via `sl_edit_source`. No flag.
|
||||
|
||||
3. **If the difference is structural — grain, columns, filter, join shape — is the current bundle the re-ingest of a previously-ingested bundle (i.e. `priorProvenance` has a row for this raw file and artifact)?**
|
||||
3. **If the difference is structural - grain, columns, filter, join shape - is the current bundle the re-ingest of a previously-ingested bundle (i.e. `priorProvenance` has a row for this raw file and artifact)?**
|
||||
Re-ingest change (semantic break): replace + flag. Record in the IngestReport's `conflicts_resolved` list with `flagged_for_human: true`.
|
||||
|
||||
4. **If there's no prior-sync row (both are from THIS job), check for same-ingest contradictions:**
|
||||
|
|
@ -37,11 +37,11 @@ Apply the rules below before every write that could collide with an existing art
|
|||
|
||||
## Why same-ingest vs re-ingest differs
|
||||
|
||||
Within ONE bundle there's no user signal telling us which duplicate wins — we capture all variants and flag. Across bundles, re-uploading IS the signal that the new state is intended — we replace silently for expression changes and flag for semantic breaks.
|
||||
Within ONE bundle there's no user signal telling us which duplicate wins - we capture all variants and flag. Across bundles, re-uploading IS the signal that the new state is intended - we replace silently for expression changes and flag for semantic breaks.
|
||||
|
||||
## Naming disambiguation hints
|
||||
|
||||
When you rename to disambiguate, prefer domain suffixes that match the containing view/table/collection name: `customers.churn_risk_score` → `customers.churn_risk_engagement_based` (if the `customer_churn` view computes it from engagement); `billing.churn_risk_score` → `billing.churn_risk_billing_based`. Avoid numeric suffixes (`churn_risk_1`, `churn_risk_2`) — they disclose nothing.
|
||||
When you rename to disambiguate, prefer domain suffixes that match the containing view/table/collection name: `customers.churn_risk_score` → `customers.churn_risk_engagement_based` (if the `customer_churn` view computes it from engagement); `billing.churn_risk_score` → `billing.churn_risk_billing_based`. Avoid numeric suffixes (`churn_risk_1`, `churn_risk_2`) - they disclose nothing.
|
||||
|
||||
## Applying canonical pins
|
||||
|
||||
|
|
@ -62,7 +62,7 @@ When you perform rename + capture, also write one page named `<canonical-concept
|
|||
- One paragraph per variant: what it computes, where it came from (raw file + line range), when to use it.
|
||||
- A closing "Choosing between these" paragraph if the variants are legitimately domain-specific.
|
||||
|
||||
Do not attempt to rank variants or pick a "best" — that's user-override territory.
|
||||
Do not attempt to rank variants or pick a "best" - that's user-override territory.
|
||||
|
||||
## Silence rules
|
||||
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ LookML views map to SL sources, `measure:` to measures, `explore: { join: }` to
|
|||
| `view: X { sql_always_where: <p> }` | **Standalone** with `sql: SELECT * FROM <base> WHERE <p>` | Enforcement, not opt-in |
|
||||
| `explore: { join: Y { sql_on: …; relationship: … } }` | `joins:` entry `{ to: Y, on: "<local> = Y.<col>", relationship: … }` | On the overlay or standalone |
|
||||
| `conditionally_filter` / `always_filter` | `segments: [{ name, expr }]` | Callers reference by name |
|
||||
| Manifest entry | `_schema/*.yaml` | **Never edit** — auto-imported |
|
||||
| Manifest entry | `_schema/*.yaml` | **Never edit** - auto-imported |
|
||||
|
||||
Type map: `date`/`datetime`/`timestamp` → `time`; `yesno` → `boolean`; `number` → `number`; `string` → `string`. Ignore `drill_fields:` (UI only).
|
||||
|
||||
|
|
@ -92,14 +92,14 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
|
|||
`sql_execution({connectionName: "warehouse", sql: "SELECT 1 FROM analytics.orders LIMIT 0"})`.
|
||||
Replace `warehouse`, `analytics`, and `orders` with the verified connection,
|
||||
schema or dataset, and table from the WorkUnit evidence.
|
||||
3. Use only those names in `sql:`, `columns:`, and `grain:`. Map each `dimension_group` to ONE `{ name: <physical_col>, type: time, role: time }` entry — never one per timeframe.
|
||||
3. Use only those names in `sql:`, `columns:`, and `grain:`. Map each `dimension_group` to ONE `{ name: <physical_col>, type: time, role: time }` entry - never one per timeframe.
|
||||
|
||||
| LookML input | KTX `columns:` entry |
|
||||
|---|---|
|
||||
| `dimension_group: month { type: time; timeframes: [month]; sql: ${TABLE}.month_date ;; }` | `{ name: month_date, type: time, role: time }` |
|
||||
| `dimension_group: date { type: time; timeframes: [raw, date, week, month]; sql: ${TABLE}.date ;; }` | `{ name: date, type: time, role: time }` — single entry, NOT `date_raw`/`date_date`/`date_week` |
|
||||
| `dimension_group: date { type: time; timeframes: [raw, date, week, month]; sql: ${TABLE}.date ;; }` | `{ name: date, type: time, role: time }` - single entry, NOT `date_raw`/`date_date`/`date_week` |
|
||||
|
||||
**After every `sl_write_source`**: call `sl_validate`. It runs `SELECT * FROM (<your sql:>) LIMIT 0` against the connection. If a column name was invented, the warehouse's `Unrecognized name: …` error comes back verbatim. Treat that as a hard failure — re-read the real columns with `sl_discover` and rewrite.
|
||||
**After every `sl_write_source`**: call `sl_validate`. It runs `SELECT * FROM (<your sql:>) LIMIT 0` against the connection. If a column name was invented, the warehouse's `Unrecognized name: …` error comes back verbatim. Treat that as a hard failure - re-read the real columns with `sl_discover` and rewrite.
|
||||
|
||||
## Provenance markers
|
||||
|
||||
|
|
@ -110,13 +110,13 @@ When a wiki mixes LookML source prose with `sl_discover` output, tag sections:
|
|||
Customers fan out many-to-one into `accounts` via `account_id`.
|
||||
<!-- /from -->
|
||||
<!-- from: bq_schema -->
|
||||
`customers.admin_user_id` is nullable — orphan rows exist.
|
||||
`customers.admin_user_id` is nullable - orphan rows exist.
|
||||
<!-- /from -->
|
||||
```
|
||||
|
||||
Invisible in most renderers; lets a future pass audit provenance.
|
||||
|
||||
## Example 1 — overlay (thin wrapper)
|
||||
## Example 1 - overlay (thin wrapper)
|
||||
|
||||
LookML (excerpt):
|
||||
|
||||
|
|
@ -154,7 +154,7 @@ joins:
|
|||
relationship: many_to_one
|
||||
```
|
||||
|
||||
## Example 2 — standalone from `derived_table`
|
||||
## Example 2 - standalone from `derived_table`
|
||||
|
||||
```lookml
|
||||
view: lab_results {
|
||||
|
|
@ -188,7 +188,7 @@ measures:
|
|||
- { name: avg_delta, expr: "avg(delta)" }
|
||||
```
|
||||
|
||||
## Example 3 — standalone with `sql_always_where`
|
||||
## Example 3 - standalone with `sql_always_where`
|
||||
|
||||
```lookml
|
||||
view: rpt_daily_braze_email {
|
||||
|
|
|
|||
|
|
@ -79,12 +79,12 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
|
|||
|
||||
For each card:
|
||||
1. Analyze `resolvedSql` + `resultMetadata`: identify base tables, aggregations, joins, filters, column types.
|
||||
2. **REQUIRED before any write**: call `sl_discover` for every candidate target source name. The response tells you whether the name is manifest-backed (`Type: table` or `Type: sql`). For manifest-backed names you MUST use the overlay shape (`name:` + `measures:`/`segments:`/`description:` only — no `sql:`, `table:`, `grain:`, or `columns:`); the tool will reject a standalone write and you'll have wasted the call. If `sl_discover` returns nothing for the name, you can write a standalone source. Also call `sl_read_source` on existing sources you intend to extend so you don't duplicate measures.
|
||||
2. **REQUIRED before any write**: call `sl_discover` for every candidate target source name. The response tells you whether the name is manifest-backed (`Type: table` or `Type: sql`). For manifest-backed names you MUST use the overlay shape (`name:` + `measures:`/`segments:`/`description:` only - no `sql:`, `table:`, `grain:`, or `columns:`); the tool will reject a standalone write and you'll have wasted the call. If `sl_discover` returns nothing for the name, you can write a standalone source. Also call `sl_read_source` on existing sources you intend to extend so you don't duplicate measures.
|
||||
3. Include `rawPaths: ["cards/<id>.json"]` on every `sl_write_source`, `sl_edit_source`, and `wiki_write` call. If one artifact generalizes multiple near-duplicate cards, include each contributing card path and no unrelated cards.
|
||||
4. Decide:
|
||||
- Simple aggregation on a table that already has a source → `sl_edit_source` to add a measure.
|
||||
- Join between tables that should be linked in the SL graph → `sl_edit_source` to add a join.
|
||||
- Complex derived SQL (CTEs, multi-layer aggregation, scoring models) → `sl_write_source` with `source_type: sql`. When the SQL projects/filters from a single manifest-backed base table, set `inherits_columns_from: <manifest_key>` so columns inherit type and description from the manifest — see `sl_capture` skill for the slim form. Use `sl_discover` to discover the manifest key from the table reference in the SQL (it accepts `MARTS.CONSIGNMENTS`, `ANALYTICS.MARTS.CONSIGNMENTS`, or `CONSIGNMENTS`).
|
||||
- Complex derived SQL (CTEs, multi-layer aggregation, scoring models) → `sl_write_source` with `source_type: sql`. When the SQL projects/filters from a single manifest-backed base table, set `inherits_columns_from: <manifest_key>` so columns inherit type and description from the manifest - see `sl_capture` skill for the slim form. Use `sl_discover` to discover the manifest key from the table reference in the SQL (it accepts `MARTS.CONSIGNMENTS`, `ANALYTICS.MARTS.CONSIGNMENTS`, or `CONSIGNMENTS`).
|
||||
- New base table not yet in the semantic layer → `sl_write_source` with `source_type: table`.
|
||||
- Trivial query (`SELECT *`, simple `COUNT(*)` with no business logic) → do nothing; the runner will record this card as `action_type='skipped'`.
|
||||
- Duplicate of an existing measure → same as trivial; do nothing for this card.
|
||||
|
|
@ -98,11 +98,11 @@ measures:
|
|||
expr: "<expression>"
|
||||
```
|
||||
|
||||
Overlay shape: `name:` plus any of `measures:`, `segments:`, `descriptions:`, `joins:`, `disable_joins:`. Never include `sql:`, `table:`, `grain:`, or `columns:` on a manifest-backed name — those would shadow the manifest's schema and drop its joins. Overlay `joins:` are merged additively with the manifest's joins (deduped by `to` + `on`); use `disable_joins: ["<on-clause>"]` to suppress a specific manifest join. After the overlay exists, use `sl_edit_source` for further tweaks. See `sl_capture` skill for the canonical overlay rule.
|
||||
Overlay shape: `name:` plus any of `measures:`, `segments:`, `descriptions:`, `joins:`, `disable_joins:`. Never include `sql:`, `table:`, `grain:`, or `columns:` on a manifest-backed name - those would shadow the manifest's schema and drop its joins. Overlay `joins:` are merged additively with the manifest's joins (deduped by `to` + `on`); use `disable_joins: ["<on-clause>"]` to suppress a specific manifest join. After the overlay exists, use `sl_edit_source` for further tweaks. See `sl_capture` skill for the canonical overlay rule.
|
||||
|
||||
**Join discovery:** When your card's SQL references warehouse tables (e.g. in `FROM` or `JOIN` clauses), call `sl_discover({ query: '<table>' })` before writing. The matching manifest entry's `name` is the value you use in `joins: [- to: <name>]` only when the card output exposes a local key that matches the target source grain (for example `account_id = mart_account_segments.account_id`). Do not declare a KTX join just because the card SQL joins that table internally. If the output only exposes display fields such as `account_name`, keep the SQL source self-contained or project the key before adding the join. Use `many_to_one` for FK-to-dimension joins, `one_to_many` for the reverse.
|
||||
|
||||
**Hard rule on join columns (prevents broken joins):** For every join you declare, the local column on the left of `on:` MUST be (a) present in your source's projected output and (b) a key/ID column, never a display value. If the natural FK isn't in your SELECT, add it to SELECT before declaring the join. Joining `account_name = mart_account_segments.account_id` is always wrong — names are not identifiers and the equality produces zero matches. The validator rejects this with a "display value to identifier" error; the tool will refuse to save it. Add `account_id` to your SELECT and join on `account_id = mart_account_segments.account_id`, or omit the join entirely.
|
||||
**Hard rule on join columns (prevents broken joins):** For every join you declare, the local column on the left of `on:` MUST be (a) present in your source's projected output and (b) a key/ID column, never a display value. If the natural FK isn't in your SELECT, add it to SELECT before declaring the join. Joining `account_name = mart_account_segments.account_id` is always wrong - names are not identifiers and the equality produces zero matches. The validator rejects this with a "display value to identifier" error; the tool will refuse to save it. Add `account_id` to your SELECT and join on `account_id = mart_account_segments.account_id`, or omit the join entirely.
|
||||
|
||||
## priorProvenance
|
||||
|
||||
|
|
@ -114,7 +114,7 @@ If the WU prompt includes a `priorProvenance` section for a card, it tells you w
|
|||
|
||||
## Deduplication
|
||||
|
||||
Before writing, scan all cards in this WU for near-duplicate groups — cards whose `resolvedSql` shares the same CTEs, base tables, joins, and aggregation structure but differs only in:
|
||||
Before writing, scan all cards in this WU for near-duplicate groups - cards whose `resolvedSql` shares the same CTEs, base tables, joins, and aggregation structure but differs only in:
|
||||
- Trailing filters (e.g. `date_trunc(week, date)` vs `date_trunc(month, date)`).
|
||||
- Minor `WHERE` clause variations.
|
||||
- Column aliases or output column subsets.
|
||||
|
|
@ -124,7 +124,7 @@ When you find a group of near-duplicates:
|
|||
1. Create ONE generalized source from the most comprehensive card in the group.
|
||||
2. Strip card-specific trailing filters from the SQL so the source covers all variants (e.g. keep daily grain instead of filtering to week/month).
|
||||
3. If each card had a distinct measure or filter, add them as separate measures on the single source.
|
||||
4. For all cards except the canonical one, do nothing — they'll be recorded as `action_type='skipped'` automatically by the runner.
|
||||
4. For all cards except the canonical one, do nothing - they'll be recorded as `action_type='skipped'` automatically by the runner.
|
||||
|
||||
Do NOT merge cards with fundamentally different business logic, even if they share CTEs.
|
||||
|
||||
|
|
@ -132,7 +132,7 @@ Do NOT merge cards with fundamentally different business logic, even if they sha
|
|||
|
||||
When a card's `resolvedSql` contains `GROUP BY` with aggregation functions (`SUM`, `COUNT`, `AVG`, …):
|
||||
|
||||
1. **Detect**: simple aggregation on base tables/joins — `SELECT` with `GROUP BY`, no complex CTEs or window functions.
|
||||
1. **Detect**: simple aggregation on base tables/joins - `SELECT` with `GROUP BY`, no complex CTEs or window functions.
|
||||
2. **Decompose**: strip the `GROUP BY` and aggregation functions. Keep `FROM`, `JOIN`, and `WHERE` intact.
|
||||
3. **Expose row-level columns**: include the grouped-by columns AND the raw columns being aggregated (e.g. `money_out` instead of `SUM(money_out) AS total_money_out`).
|
||||
4. **Define aggregations as measures**: convert each aggregation into a KSL measure (e.g. `sum(money_out)`).
|
||||
|
|
@ -144,17 +144,17 @@ Exception: keep the pre-aggregated SQL when the query involves multi-CTE pipelin
|
|||
|
||||
Every card carries a `resolvedSql` field. Check the staged card's `resolutionStatus` first:
|
||||
|
||||
- `resolutionStatus: "resolved"` — `{{#N}}` references are inlined and `[[ ... ]]` optional clauses have been dropped locally. If the resolved SQL contains no other parameters the SQL is executable as-is. If the card had **required** (non-bracketed) `{{ var }}` placeholders, the SQL is prefixed with a placeholder-warning comment block listing every dummy substitution Metabase made — see "Step A" below.
|
||||
- `resolutionStatus: "fallback"` — Metabase failed to resolve. The SQL still contains `{{#N}}`, `{{#N-name}} alias`, `{{ var }}`, and `[[ ... ]]` syntax. Do the translation steps below before writing a source.
|
||||
- `resolutionStatus: "resolved"` - `{{#N}}` references are inlined and `[[ ... ]]` optional clauses have been dropped locally. If the resolved SQL contains no other parameters the SQL is executable as-is. If the card had **required** (non-bracketed) `{{ var }}` placeholders, the SQL is prefixed with a placeholder-warning comment block listing every dummy substitution Metabase made - see "Step A" below.
|
||||
- `resolutionStatus: "fallback"` - Metabase failed to resolve. The SQL still contains `{{#N}}`, `{{#N-name}} alias`, `{{ var }}`, and `[[ ... ]]` syntax. Do the translation steps below before writing a source.
|
||||
|
||||
### Step A — Handle dummy-substituted placeholders (resolved cards only)
|
||||
### Step A - Handle dummy-substituted placeholders (resolved cards only)
|
||||
|
||||
When a card has a required `{{ var }}` outside any `[[ ]]` block, the resolver substitutes a **dummy value** purely so Metabase's parser will accept the query. The resulting SQL is prefixed with a comment like:
|
||||
|
||||
```sql
|
||||
-- PLACEHOLDER_WARNING: this SQL was extracted from a Metabase card with
|
||||
-- unbound template parameters. The placeholders below were substituted with DUMMY
|
||||
-- values to satisfy Metabase's parser — they DO NOT represent intended filters.
|
||||
-- values to satisfy Metabase's parser - they DO NOT represent intended filters.
|
||||
-- Drop the corresponding clauses (or expose them as runtime SL filters) before
|
||||
-- persisting this SQL as a semantic-layer source.
|
||||
-- {{ auction_end }} (type=dimension, widget=date/all-options) → '2020-01-01~2020-12-31'
|
||||
|
|
@ -165,7 +165,7 @@ WHERE start_date >= '2020-01-01' AND start_date < '2021-01-01' AND status = 'pla
|
|||
|
||||
For each listed placeholder: locate the WHERE clause(s) in the SQL that reference the dummy literal and **drop them**, then strip the warning comment. SL chat-time filters compose narrowing predicates dynamically, so the source should represent the unfiltered dataset.
|
||||
|
||||
For `fallback` cards, dropping is simpler — the SQL still has the `[[ ... ]]` brackets and `{{ var }}` placeholders intact:
|
||||
For `fallback` cards, dropping is simpler - the SQL still has the `[[ ... ]]` brackets and `{{ var }}` placeholders intact:
|
||||
|
||||
```sql
|
||||
-- before:
|
||||
|
|
@ -177,18 +177,18 @@ WHERE 1=1
|
|||
WHERE 1=1
|
||||
```
|
||||
|
||||
### Step B — Inline `{{#N}}` references (fallback cards only)
|
||||
### Step B - Inline `{{#N}}` references (fallback cards only)
|
||||
|
||||
Resolved cards already have `{{#N}}` inlined for you. For `fallback` cards, each `{{#N}}` (or `{{#N-some-slug}}`) in the SQL refers to another card's `resolvedSql`. The referenced card is in the WU's `rawFiles` or `dependencyPaths`. Read it with `read_raw_file`, then inline its SQL.
|
||||
|
||||
If the reference has an alias (`from {{#5996-listing-interactions}} tb`), the **outer** SQL probably uses that alias (`select tb.* ...`, `tb.column_name`, etc.). When you inline, you must EITHER:
|
||||
|
||||
1. **Pick a single base table inside the inlined SQL and rename its alias to the outer alias.** Useful when the inlined card is `SELECT * FROM listings JOIN ...` — set the LISTINGS alias to `tb` and `tb.*` keeps working in the outer query.
|
||||
1. **Pick a single base table inside the inlined SQL and rename its alias to the outer alias.** Useful when the inlined card is `SELECT * FROM listings JOIN ...` - set the LISTINGS alias to `tb` and `tb.*` keeps working in the outer query.
|
||||
2. **Replace the outer alias references with explicit columns from the inlined SQL.** Useful when the inlined card has multiple JOINs and `tb.*` is ambiguous.
|
||||
|
||||
Never leave the outer alias dangling: after inlining, **grep your SQL for the outer alias name and rewrite or remove every reference**. A leftover `tb.*` with no `tb` table is the most common failure mode here.
|
||||
|
||||
### Step C — Inlining cleanup checklist
|
||||
### Step C - Inlining cleanup checklist
|
||||
|
||||
After Steps A and B, your SQL must:
|
||||
|
||||
|
|
@ -209,11 +209,11 @@ For `source_type: sql`:
|
|||
- If `sl_discover` resolves the table, it is not outside the manifest. Do not write an `unmapped-table-*` fallback for resolved `orbit_raw`, `mart`, or other manifest-backed sources just because they appear inside card SQL.
|
||||
- If `sl_discover` cannot resolve a referenced table at all, write a single-line `wiki_write` with key `unmapped-table-<table_name>` and `rawPaths: ["cards/<id>.json"]` so the gap is documented, then call `emit_unmapped_fallback` with the staged card path as `rawPath`, `reason: "missing_target_table"`, `tableRef: "<table_name>"`, and `fallback: "wiki_only"`. Do not use this fallback if `sl_discover` resolved the table/source.
|
||||
|
||||
Joins on manifest-backed names compose: the manifest's joins are inherited automatically, and any overlay `joins:` are merged on top (deduped by `to` + `on`). Use `disable_joins: ["<on-clause>"]` in the overlay to suppress a specific manifest join. If `sl_discover` shows a manifest-backed source with `Joins: 0` and the warehouse FK metadata is genuinely absent, declaring application-level joins via the overlay is fair game — bootstrap with `sl_write_source` (overlay shape above), then refine via `sl_edit_source`.
|
||||
Joins on manifest-backed names compose: the manifest's joins are inherited automatically, and any overlay `joins:` are merged on top (deduped by `to` + `on`). Use `disable_joins: ["<on-clause>"]` in the overlay to suppress a specific manifest join. If `sl_discover` shows a manifest-backed source with `Joins: 0` and the warehouse FK metadata is genuinely absent, declaring application-level joins via the overlay is fair game - bootstrap with `sl_write_source` (overlay shape above), then refine via `sl_edit_source`.
|
||||
|
||||
## Cross-card references (`{{#N}}`)
|
||||
|
||||
Resolved cards (`resolutionStatus: "resolved"`) have these inlined for you. Unresolved cards (`resolutionStatus: "fallback"`) need manual handling — see "SQL translation from raw native to KSL" above.
|
||||
Resolved cards (`resolutionStatus: "resolved"`) have these inlined for you. Unresolved cards (`resolutionStatus: "fallback"`) need manual handling - see "SQL translation from raw native to KSL" above.
|
||||
|
||||
## Provenance markers
|
||||
|
||||
|
|
@ -237,7 +237,7 @@ Source definitions must follow ktx-sl YAML conventions:
|
|||
- `columns`: all columns with correct types (`string`, `number`, `time`, `boolean`).
|
||||
- Time columns: mark with `role: time`.
|
||||
- `joins`: use correct `relationship` types (`many_to_one` for FK→PK, `one_to_many` for reverse).
|
||||
- `joins.on`: `local_column = TARGET_SOURCE.target_column` — the right side MUST include the target source name.
|
||||
- `joins.on`: `local_column = TARGET_SOURCE.target_column` - the right side MUST include the target source name.
|
||||
- `measures.expr`: aggregation expression (e.g. `"sum(amount)"`); optional `filter` for business rules; required `description`.
|
||||
|
||||
Measure naming: descriptive `snake_case` (e.g. `total_revenue`, `avg_order_value`).
|
||||
|
|
@ -250,4 +250,4 @@ Measure naming: descriptive `snake_case` (e.g. `total_revenue`, `avg_order_value
|
|||
- If two measures differ only by a filter (e.g. `revenue` vs `paid_revenue`), they are distinct.
|
||||
- Use the card's `name` + `description` to write meaningful measure descriptions.
|
||||
- When multiple cards in a WU are near-duplicates, create ONE generalized source; the runner will skip the rest automatically.
|
||||
- Process every card in the WU — don't stop early.
|
||||
- Process every card in the WU - don't stop early.
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ A MetricFlow `semantic_model` maps to an SL source; MetricFlow `measures` map to
|
|||
| `semantic_model: X { model: ref('t') }` with measures + dimensions | **Overlay** at `<connId>/X.yaml` with `measures`, `columns` (computed), `joins` | The `model:` ref resolves to a manifest table. |
|
||||
| `semantic_model: X { model: source('s','t') }` | **Overlay** at `<connId>/X.yaml` over table `t`. | Same shape; `source()` still resolves to a physical table. |
|
||||
| `semantic_model: X { model: <literal> }` with no manifest entry | **Standalone** with explicit `sql:`, `grain:`, `columns:` | Happens when the dbt manifest isn't available. |
|
||||
| `semantic_model: Y { extends: X }` | **Merge** Y's measures/dimensions/entities into X's overlay, or write a single overlay named for the most-derived child (Y) containing both X's and Y's primitives | Do not emit a second overlay for X — flatten. |
|
||||
| `semantic_model: Y { extends: X }` | **Merge** Y's measures/dimensions/entities into X's overlay, or write a single overlay named for the most-derived child (Y) containing both X's and Y's primitives | Do not emit a second overlay for X - flatten. |
|
||||
| `measures: [{ name, agg, expr }]` | `measures: [{ name, expr: "<agg>(<expr>)" }]` | Aggregation inlined. `agg: count_distinct` → `count(distinct ...)`. |
|
||||
| `entities: [{ name, type: primary }]` | `grain: [<entity_name-or-expr>]` on the overlay/standalone | Primary/unique entities drive grain. |
|
||||
| `entities: [{ name, type: foreign }]` | `joins:` entry joining to the primary-entity's semantic_model | Only when a matching primary is discoverable. |
|
||||
|
|
@ -24,10 +24,10 @@ A MetricFlow `semantic_model` maps to an SL source; MetricFlow `measures` map to
|
|||
| `metrics: [{ type: derived, type_params: { expr, metrics } }]` | **Derived measure** on whichever source owns the referenced measures, with `expr:` referencing measure names | If the metric spans models, still write it once on the source owning the "primary" measure (the one the agent judges most central). Mention the cross-model chain in the description. |
|
||||
| `metrics: [{ type: ratio, type_params: { numerator, denominator } }]` | Same as derived; `expr: "numerator / NULLIF(denominator, 0)"` if no explicit expr | Safe-division by default. |
|
||||
| `metrics: [{ type: cumulative, type_params: { window, grain_to_date } }]` | **Standalone** source with a window-function SQL; reference the resulting column as a normal measure | KTX SL has no first-class cumulative primitive (spec Non-goals). |
|
||||
| `metrics: [{ type: conversion }]` | **Flag for human** — do NOT write. Emit a wiki note describing the intended semantics. | No KTX equivalent in v1. |
|
||||
| `metrics: [{ type: conversion }]` | **Flag for human** - do NOT write. Emit a wiki note describing the intended semantics. | No KTX equivalent in v1. |
|
||||
| Metric not mappable | Wiki page `<metric_name>-definition.md` with the full YAML body quoted | Capture the intent even if we can't emit SL. |
|
||||
|
||||
Type map: MetricFlow `time` to KTX `time`; `categorical` to `string`; `number` to `number`; `boolean` to `boolean`. Follow `expr` over `name` when both differ — `expr` is the physical column.
|
||||
Type map: MetricFlow `time` to KTX `time`; `categorical` to `string`; `number` to `number`; `boolean` to `boolean`. Follow `expr` over `name` when both differ - `expr` is the physical column.
|
||||
|
||||
Verify each MetricFlow model source table with entity_details before producing
|
||||
the corresponding sl_write_source.
|
||||
|
|
@ -67,7 +67,7 @@ Within one WorkUnit, multiple semantic_models linked by `extends:` are guarantee
|
|||
1. Start with the most-derived child (the one that no other semantic_model extends).
|
||||
2. Walk the `extends:` chain upward, accumulating measures, dimensions, entities.
|
||||
3. Write ONE overlay/standalone, named for the most-derived child's SL-appropriate name (not the base).
|
||||
4. Parents that lack their own distinctive content should NOT get a separate overlay. If a parent has unique measures a child doesn't inherit, consider whether the base is used elsewhere — if yes, write both; if no, still one overlay.
|
||||
4. Parents that lack their own distinctive content should NOT get a separate overlay. If a parent has unique measures a child doesn't inherit, consider whether the base is used elsewhere - if yes, write both; if no, still one overlay.
|
||||
5. Measure/dimension name collisions: child wins, but note the overridden parent in the overlay's description or in a sibling wiki page.
|
||||
|
||||
The spec's worked example has `orders`, `orders_ext` (extends orders), and `metrics/orders_final.yml` (defines `revenue` referencing both). The right output is ONE overlay named `orders_ext` (or `orders` if the team's naming favors the base) containing `order_count`, `gross_amount`, `refund_amount`, and a derived `revenue` measure. Provenance tags point to all three source files.
|
||||
|
|
@ -88,9 +88,9 @@ call `sql_execution` with the same warehouse connection name, for example:
|
|||
`sql:` must be sourced from raw files, `entity_details`, or a successful SQL
|
||||
probe.
|
||||
|
||||
After every `sl_write_source`, call `sl_validate`. The warehouse will reject invented columns with `Unrecognized name: <name>` — treat as a hard failure and re-read the schema.
|
||||
After every `sl_write_source`, call `sl_validate`. The warehouse will reject invented columns with `Unrecognized name: <name>` - treat as a hard failure and re-read the schema.
|
||||
|
||||
## Cumulative metrics — sql-standalone fallback
|
||||
## Cumulative metrics - sql-standalone fallback
|
||||
|
||||
KTX SL has no first-class `window:` or `grain_to_date:` primitive in v1 (spec Non-goals). Translate a MetricFlow cumulative metric to a standalone SL source with a window-function SQL:
|
||||
|
||||
|
|
@ -125,7 +125,7 @@ measures:
|
|||
|
||||
Pick the time column based on the semantic_model's `defaults.agg_time_dimension` (e.g. `ordered_at`). If the MetricFlow config omits it, probe the base table for time-typed columns and choose the most obvious. After writing the standalone SQL source, call `emit_unmapped_fallback` with `rawPath` set to the MetricFlow file path, `reason: "cumulative_metric_unsupported"`, and `fallback: "sql_standalone"`.
|
||||
|
||||
## Conversion metrics — flag for human
|
||||
## Conversion metrics - flag for human
|
||||
|
||||
```yaml
|
||||
metrics:
|
||||
|
|
@ -159,7 +159,7 @@ name: orders_ext
|
|||
|
||||
Line ranges (`#L<start>-<end>`) point to the exact YAML span within the file (the `semantic_models:` entry for its own `name`). Use `read_raw_span` to identify those ranges before writing.
|
||||
|
||||
## Example 1 — single semantic_model to overlay
|
||||
## Example 1 - single semantic_model to overlay
|
||||
|
||||
```yaml
|
||||
# MetricFlow:
|
||||
|
|
@ -185,7 +185,7 @@ measures:
|
|||
grain: [order_id]
|
||||
```
|
||||
|
||||
## Example 2 — extends chain → one flattened overlay
|
||||
## Example 2 - extends chain → one flattened overlay
|
||||
|
||||
```yaml
|
||||
# MetricFlow:
|
||||
|
|
@ -232,7 +232,7 @@ measures:
|
|||
grain: [order_id]
|
||||
```
|
||||
|
||||
## Example 3 — derived metric spanning two semantic_models
|
||||
## Example 3 - derived metric spanning two semantic_models
|
||||
|
||||
```yaml
|
||||
# models/sales.yml
|
||||
|
|
@ -256,7 +256,7 @@ metrics:
|
|||
metrics: [{name: revenue}, {name: cost}]
|
||||
```
|
||||
|
||||
Because the WorkUnit bundles all three files (cross-component union via the metric), write the derived measure on ONE of the two sources — pick the source whose domain "owns" the metric (here, `sales` — margin is inherently a sales metric). Cross-source references aren't native in KTX SL; treat the metric's operands as already-resolvable in the target source's query context OR emit a standalone SQL that joins the two tables:
|
||||
Because the WorkUnit bundles all three files (cross-component union via the metric), write the derived measure on ONE of the two sources - pick the source whose domain "owns" the metric (here, `sales` - margin is inherently a sales metric). Cross-source references aren't native in KTX SL; treat the metric's operands as already-resolvable in the target source's query context OR emit a standalone SQL that joins the two tables:
|
||||
|
||||
```yaml
|
||||
# <connId>/sales.yaml
|
||||
|
|
@ -269,7 +269,7 @@ measures:
|
|||
```
|
||||
|
||||
```yaml
|
||||
# <connId>/margin.yaml — standalone because it spans two tables
|
||||
# <connId>/margin.yaml - standalone because it spans two tables
|
||||
# <!-- from: .../models/sales.yml#L1-8 -->
|
||||
# <!-- from: .../models/costs.yml#L1-8 -->
|
||||
# <!-- from: .../metrics/margin.yml#L1-8 -->
|
||||
|
|
@ -292,7 +292,7 @@ measures:
|
|||
|
||||
Also write a wiki page at `wiki/global/margin-metric.md` explaining the cross-source origin.
|
||||
|
||||
## Example 4 — filtered metric creates a new measure
|
||||
## Example 4 - filtered metric creates a new measure
|
||||
|
||||
```yaml
|
||||
metrics:
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@ Search existing wiki pages for the same `tables:` or `sl_refs:` frontmatter and
|
|||
- Do not create SL sources under the Notion connection just because a page mentions a warehouse, dbt, Looker, or Metabase object. Use the mapped warehouse/source connection after discovery, or emit an unmapped fallback and write wiki-only.
|
||||
- Distinguish fallback reasons precisely: if a non-Notion warehouse/dbt connection exists but `sl_discover` cannot find the named table/source, use `no_physical_table`; reserve `no_connection_mapping` for cases where there is no plausible non-Notion target connection at all.
|
||||
- If `sl_discover` resolves the table/source, do not call `emit_unmapped_fallback` for that table. Use the resolved source for `sl_refs`, overlay edits, or wiki-only documentation.
|
||||
- When calling `emit_unmapped_fallback`, pass the table or source identifier as `tableRef` (e.g. `tableRef: "<schema>.<table>"`) — the tool generates the canonical detail string from the reason code and `tableRef`. Use the optional `clarification` field only to add context that does not contradict the reason. Do not restate the reason in `clarification`.
|
||||
- When calling `emit_unmapped_fallback`, pass the table or source identifier as `tableRef` (e.g. `tableRef: "<schema>.<table>"`) - the tool generates the canonical detail string from the reason code and `tableRef`. Use the optional `clarification` field only to add context that does not contradict the reason. Do not restate the reason in `clarification`.
|
||||
|
||||
## Identifier Verification Protocol
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: sl
|
||||
description: KTX's semantic layer — a structured catalog of sources (tables/views), measures, joins, and segments expressed as YAML. Covers the schema and how to query it via `sl_query`. Use when the task involves querying pre-defined metrics (ARR, churn, retention, LTV, MAU) or reading SL source YAML to understand the catalog. Capture is handled by the `sl_capture` skill (memory-agent only).
|
||||
description: KTX's semantic layer - a structured catalog of sources (tables/views), measures, joins, and segments expressed as YAML. Covers the schema and how to query it via `sl_query`. Use when the task involves querying pre-defined metrics (ARR, churn, retention, LTV, MAU) or reading SL source YAML to understand the catalog. Capture is handled by the `sl_capture` skill (memory-agent only).
|
||||
---
|
||||
|
||||
# Semantic Layer
|
||||
|
|
@ -8,10 +8,10 @@ description: KTX's semantic layer — a structured catalog of sources (tables/vi
|
|||
KTX's semantic layer (SL) is a structured catalog. Each **source** represents a table, a SQL view, or an overlay that enriches a manifest-backed table with measures, computed columns, joins, and named segments. The catalog is the single source of truth for reusable business metrics.
|
||||
|
||||
This skill covers two parts:
|
||||
- **Part 1** — Schema reference (what an SL source looks like).
|
||||
- **Part 2** — Querying via `sl_query`.
|
||||
- **Part 1** - Schema reference (what an SL source looks like).
|
||||
- **Part 2** - Querying via `sl_query`.
|
||||
|
||||
Capture (when and how to add new patterns to the SL) is a separate concern handled by the memory-agent — see the `sl_capture` skill if you are running in capture mode. The research agent **reads** and **queries** the SL via the tools described here; it does not write to it.
|
||||
Capture (when and how to add new patterns to the SL) is a separate concern handled by the memory-agent - see the `sl_capture` skill if you are running in capture mode. The research agent **reads** and **queries** the SL via the tools described here; it does not write to it.
|
||||
|
||||
For capture-time identifier verification, load `sl_capture`. Synthesis writer
|
||||
skills must verify warehouse identifiers with `discover_data`,
|
||||
|
|
@ -19,7 +19,7 @@ skills must verify warehouse identifiers with `discover_data`,
|
|||
|
||||
---
|
||||
|
||||
## Part 1 — Schema reference
|
||||
## Part 1 - Schema reference
|
||||
|
||||
An SL source is a YAML file at `semantic-layer/<connectionId>/<source_name>.yaml`. There are three flavors:
|
||||
|
||||
|
|
@ -34,7 +34,7 @@ descriptions:
|
|||
measures:
|
||||
- name: total_revenue
|
||||
expr: sum(amount)
|
||||
description: Total order revenue — filter by status or region at query time
|
||||
description: Total order revenue - filter by status or region at query time
|
||||
columns: # computed dimensions only
|
||||
- name: is_large_order
|
||||
type: boolean
|
||||
|
|
@ -49,7 +49,7 @@ joins:
|
|||
```
|
||||
|
||||
Rules:
|
||||
- Do **not** repeat base-table columns, grain, `table`, or `source_type` in an overlay — those are inherited.
|
||||
- Do **not** repeat base-table columns, grain, `table`, or `source_type` in an overlay - those are inherited.
|
||||
- Overlay columns MUST be computed (`expr` + `type`).
|
||||
- `exclude_columns` hides specific manifest columns; `disable_joins` suppresses specific auto-detected joins.
|
||||
|
||||
|
|
@ -106,7 +106,7 @@ measures:
|
|||
expr: count(*)
|
||||
```
|
||||
|
||||
An SQL source is a one-shot answer: the aggregation is frozen, callers cannot re-group or re-filter by columns the SQL has collapsed, and the source is disconnected from the join graph. Prefer overlays + measures over SQL sources when possible — the `sl_capture` skill covers when SQL is justified.
|
||||
An SQL source is a one-shot answer: the aggregation is frozen, callers cannot re-group or re-filter by columns the SQL has collapsed, and the source is disconnected from the join graph. Prefer overlays + measures over SQL sources when possible - the `sl_capture` skill covers when SQL is justified.
|
||||
|
||||
### Columns
|
||||
|
||||
|
|
@ -119,7 +119,7 @@ Every standalone column requires `name` and `type`. Overlays have computed colum
|
|||
|
||||
### Grain
|
||||
|
||||
`grain: [col_a, col_b]` — the set of columns that uniquely identify one row. The query engine uses grain to prevent fan-out in joins. Overlays inherit grain from the manifest unless they override.
|
||||
`grain: [col_a, col_b]` - the set of columns that uniquely identify one row. The query engine uses grain to prevent fan-out in joins. Overlays inherit grain from the manifest unless they override.
|
||||
|
||||
### Joins
|
||||
|
||||
|
|
@ -128,7 +128,7 @@ joins:
|
|||
- to: customers # target source name
|
||||
on: "customer_id = customers.id" # local_col = TARGET.target_col
|
||||
relationship: many_to_one # or one_to_many, one_to_one
|
||||
alias: primary_customer # optional — lets you join the same target twice
|
||||
alias: primary_customer # optional - lets you join the same target twice
|
||||
```
|
||||
|
||||
- `on` format: `local_col = TARGET.target_col`. Always qualify the right side with the target source name.
|
||||
|
|
@ -140,13 +140,13 @@ joins:
|
|||
measures:
|
||||
- name: total_arr
|
||||
expr: sum(arr_amount)
|
||||
description: Sum of ARR — filter by plan_name at query time
|
||||
description: Sum of ARR - filter by plan_name at query time
|
||||
filter: "is_active = true"
|
||||
segments: [paid_non_refunded]
|
||||
```
|
||||
|
||||
- `name` (required, snake_case).
|
||||
- `expr` (required): any valid SQL aggregate — `sum(x)`, `count(*)`, `count(distinct user_id)`, `avg(score)`.
|
||||
- `expr` (required): any valid SQL aggregate - `sum(x)`, `count(*)`, `count(distinct user_id)`, `avg(score)`.
|
||||
- `description` (required on capture): what the measure computes and how to use it.
|
||||
- `filter` (optional): SQL predicate applied as a WHERE clause specific to this measure.
|
||||
- `segments` (optional): names of segments defined on the same source. The engine AND-composes each segment's `expr` into this measure's effective filter.
|
||||
|
|
@ -162,23 +162,23 @@ segments:
|
|||
description: Orders that were paid and not refunded
|
||||
```
|
||||
|
||||
Named, reusable boolean predicates scoped to one source. Reference by bare name in a measure's `segments: []`, or by dotted form `source.segment_name` in an `sl_query`. Segments are predicates only — they are NOT selectable as dimensions. If you need to group by the predicate, add a `columns[]` entry instead.
|
||||
Named, reusable boolean predicates scoped to one source. Reference by bare name in a measure's `segments: []`, or by dotted form `source.segment_name` in an `sl_query`. Segments are predicates only - they are NOT selectable as dimensions. If you need to group by the predicate, add a `columns[]` entry instead.
|
||||
|
||||
### Cross-references with the wiki
|
||||
|
||||
The reverse edge (wiki pages that cite this source) is derived automatically from each wiki's `sl_refs:` — you don't emit anything on the SL side. Author the edge once on the wiki via `sl_refs:`; the post-write reconciler populates the knowledge↔SL index.
|
||||
The reverse edge (wiki pages that cite this source) is derived automatically from each wiki's `sl_refs:` - you don't emit anything on the SL side. Author the edge once on the wiki via `sl_refs:`; the post-write reconciler populates the knowledge↔SL index.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Querying via `sl_query`
|
||||
## Part 2 - Querying via `sl_query`
|
||||
|
||||
The `sl_query` tool generates correct SQL from a structured query. It handles joins, fan-out prevention, aggregation correctness, and filter classification automatically. Prefer it over writing raw SQL whenever the SL has the relevant sources.
|
||||
|
||||
### When to prefer sl_query over raw SQL
|
||||
|
||||
- A pre-defined measure already exists (`source.measure_name` appears in the catalog).
|
||||
- The question combines fields from multiple sources — the engine resolves the join path automatically.
|
||||
- The question asks for a standard metric (revenue, ARR, churn, retention, LTV, conversion, MAU, etc.) — even if no pre-defined measure exists, a runtime aggregation over a catalog column is usually correct.
|
||||
- The question combines fields from multiple sources - the engine resolves the join path automatically.
|
||||
- The question asks for a standard metric (revenue, ARR, churn, retention, LTV, conversion, MAU, etc.) - even if no pre-defined measure exists, a runtime aggregation over a catalog column is usually correct.
|
||||
|
||||
Use raw SQL (`sql_execution`) only when:
|
||||
- The computation requires multi-step CTEs whose intermediate grain is not a column in any source.
|
||||
|
|
@ -201,17 +201,17 @@ Use raw SQL (`sql_execution`) only when:
|
|||
- **`measures`**: mix pre-defined refs (`source.measure`) and runtime aggregations (`sum(source.column)`).
|
||||
- **`dimensions`**: column refs or `{ field, granularity }` objects for time grains (`day`, `week`, `month`, `quarter`, `year`).
|
||||
- **`filters`**: free-form SQL predicates. The engine auto-classifies each as WHERE or HAVING based on whether it references an aggregated measure.
|
||||
- **`segments`**: dotted `source.segment_name`. Each segment is AND-ed into the effective filter of every measure whose base source matches. Segments never become a global WHERE — use `filters` for cross-source predicates.
|
||||
- **`segments`**: dotted `source.segment_name`. Each segment is AND-ed into the effective filter of every measure whose base source matches. Segments never become a global WHERE - use `filters` for cross-source predicates.
|
||||
- **`order_by`**: string or `{ field, direction }`. Direction defaults to `asc`.
|
||||
- **`limit`**: integer row cap.
|
||||
|
||||
### Join resolution
|
||||
|
||||
You don't specify a base table. The engine infers the set of sources needed from the fields you reference and resolves the shortest join path through the catalog's declared joins. If no path exists between two sources, the query fails with a path-not-found error — check `discover_data` or `sl_discover` to see which sources are connected.
|
||||
You don't specify a base table. The engine infers the set of sources needed from the fields you reference and resolves the shortest join path through the catalog's declared joins. If no path exists between two sources, the query fails with a path-not-found error - check `discover_data` or `sl_discover` to see which sources are connected.
|
||||
|
||||
### Worked examples
|
||||
|
||||
Cross-source query — engine resolves `account_health_scores → accounts ← opportunities` automatically:
|
||||
Cross-source query - engine resolves `account_health_scores → accounts ← opportunities` automatically:
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
---
|
||||
name: sl_capture
|
||||
description: How to capture new reusable patterns into KTX's semantic layer — when a measure, segment, or join belongs in the catalog and how to write it generically so it stays small and useful over time. Loaded by the post-turn memory-agent only. The research agent does not write to the SL.
|
||||
description: How to capture new reusable patterns into KTX's semantic layer - when a measure, segment, or join belongs in the catalog and how to write it generically so it stays small and useful over time. Loaded by the post-turn memory-agent only. The research agent does not write to the SL.
|
||||
callers: [memory_agent]
|
||||
---
|
||||
|
||||
# Semantic Layer — Capture
|
||||
# Semantic Layer - Capture
|
||||
|
||||
This skill covers **when** and **how** to capture new patterns into the semantic layer. For schema reference and query grammar, load the `sl` skill first.
|
||||
|
||||
|
|
@ -13,8 +13,8 @@ When the current turn produces a reusable pattern (business metric, derived view
|
|||
## SQL dialect
|
||||
|
||||
The user-facing prompt includes a `Warehouse:` line under the SL Sources index
|
||||
(e.g. `Warehouse: BIGQUERY`). All `expr` strings — measure expressions, segment
|
||||
predicates, computed-column SQL — execute on that warehouse and must use its
|
||||
(e.g. `Warehouse: BIGQUERY`). All `expr` strings - measure expressions, segment
|
||||
predicates, computed-column SQL - execute on that warehouse and must use its
|
||||
syntax. Date arithmetic in particular varies by dialect:
|
||||
|
||||
- **BigQuery**: `transaction_date >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)` (when the column is `TIMESTAMP`); `event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY)` (when `DATE`).
|
||||
|
|
@ -22,7 +22,7 @@ syntax. Date arithmetic in particular varies by dialect:
|
|||
- **Snowflake**: `transaction_date >= dateadd(day, -90, current_timestamp())`.
|
||||
|
||||
Match the column's manifest type (`type: time` → TIMESTAMP/DATETIME on the
|
||||
warehouse) — comparing TIMESTAMP to a DATE-arithmetic result fails on
|
||||
warehouse) - comparing TIMESTAMP to a DATE-arithmetic result fails on
|
||||
BigQuery. After every `sl_edit_source`/`sl_write_source`, the inline validator runs a
|
||||
`LIMIT 1` warehouse probe per measure and surfaces dialect mismatches; if
|
||||
you see an error trailer, fix the expression and retry rather than leaving
|
||||
|
|
@ -68,12 +68,12 @@ Callers filter `region = 'US'` at query time.
|
|||
**Bake constants in only when the filter has named business meaning that won't change** (`enterprise_arr` for a contractually defined tier), cannot be expressed via the source's dimensions, or comes from a regulated/fixed list.
|
||||
|
||||
**Time anchors and value lists belong in callers' filters, not in measure expressions or source SQL.**
|
||||
- Anti-pattern (date anchor inlined): `expr: count(distinct case when transaction_date >= '2026-04-12' then customer_id end)` — the date will need editing every time the question shifts, and every reader has to discover it.
|
||||
- Anti-pattern (value list inlined in source SQL): `WHERE product_category_1 IN ('Testosterone', 'Weight Loss', …)` — locks the source to today's catalog and blocks callers from broadening or narrowing.
|
||||
- Anti-pattern (date anchor inlined): `expr: count(distinct case when transaction_date >= '2026-04-12' then customer_id end)` - the date will need editing every time the question shifts, and every reader has to discover it.
|
||||
- Anti-pattern (value list inlined in source SQL): `WHERE product_category_1 IN ('Testosterone', 'Weight Loss', …)` - locks the source to today's catalog and blocks callers from broadening or narrowing.
|
||||
- Preferred: a generic measure (`count(distinct customer_id)`) plus either a named segment that captures the *meaning* of the anchor (`gh_new_products_since_launch`) or a query-time filter. Callers compose; the source stays small.
|
||||
- A date is durable to bake in only when it represents a regulatory cutover, a contractually fixed boundary, or a one-time event that reshapes how the source itself is read.
|
||||
|
||||
**If you create a segment whose expr matches a measure's filter, the measure MUST reference the segment via `segments: [segment_name]` rather than re-inlining the predicate.** This is the canonical pattern even with a single measure — duplicating the predicate inline defeats the purpose of naming it.
|
||||
**If you create a segment whose expr matches a measure's filter, the measure MUST reference the segment via `segments: [segment_name]` rather than re-inlining the predicate.** This is the canonical pattern even with a single measure - duplicating the predicate inline defeats the purpose of naming it.
|
||||
|
||||
Anti-pattern:
|
||||
```yaml
|
||||
|
|
@ -100,24 +100,24 @@ measures:
|
|||
|
||||
**Extract repeated filter bundles into named segments.** If the same predicate appears on multiple measures of the same source, lift it to a `segments[]` entry and have each measure reference it. One edit updates every measure that depends on it.
|
||||
|
||||
**Never write a standalone file on a manifest-backed name.** If `sl_discover({ query: "<table-or-source-name>" })` finds an existing schema for that name, you MUST write an overlay (`name:` + `measures:`/`segments:`/`descriptions:` only — no `sql:`, `table:`, `grain:`, `columns:`, `joins:`). A standalone with `sql:` or `table:` on a manifest-backed name clobbers the inherited columns and joins; `sl_write_source` and `sl_validate` both reject this shape with a clear fix hint. Always run `sl_discover` before your first write on any existing name.
|
||||
**Never write a standalone file on a manifest-backed name.** If `sl_discover({ query: "<table-or-source-name>" })` finds an existing schema for that name, you MUST write an overlay (`name:` + `measures:`/`segments:`/`descriptions:` only - no `sql:`, `table:`, `grain:`, `columns:`, `joins:`). A standalone with `sql:` or `table:` on a manifest-backed name clobbers the inherited columns and joins; `sl_write_source` and `sl_validate` both reject this shape with a clear fix hint. Always run `sl_discover` before your first write on any existing name.
|
||||
|
||||
**Prefer overlay decomposition over standalone SQL sources.** Before reaching for `source_type: sql`, check whether the metric decomposes into measures on existing overlays (including cross-source derived measures). Use `source_type: sql` only when:
|
||||
- The metric requires per-user/per-entity derivation that cannot be expressed as a single `expr` (e.g., `EXISTS` over a time-windowed subset), OR
|
||||
- The metric requires multi-step CTEs whose intermediate grain is not a column in any existing source.
|
||||
|
||||
When an `sql` source is unavoidable, note in its `descriptions` map which SL gap forced the choice so it can be retired once the primitive ships. It must target a name NOT in the manifest — pick a distinct one (e.g. `mrr_waterfall_rollup`, not `fct_orders`).
|
||||
When an `sql` source is unavoidable, note in its `descriptions` map which SL gap forced the choice so it can be retired once the primitive ships. It must target a name NOT in the manifest - pick a distinct one (e.g. `mrr_waterfall_rollup`, not `fct_orders`).
|
||||
|
||||
## Slim standalone sources via `inherits_columns_from`
|
||||
|
||||
When a standalone SQL source filters or projects from a single manifest-backed base table (the common pattern for derived views like `aav_consignments` over `MARTS.CONSIGNMENTS`), set `inherits_columns_from:` to the base table's manifest key and list only column **names** in `columns:`. Compose-time enrichment fills `type`, `descriptions`, and `role` from the matching manifest column.
|
||||
|
||||
Discover the manifest key with `sl_discover` — pass the bare name (`CONSIGNMENTS`), the fully-qualified path (`ANALYTICS.MARTS.CONSIGNMENTS`), or any suffix; the tool resolves all forms and prints the canonical key in its output.
|
||||
Discover the manifest key with `sl_discover` - pass the bare name (`CONSIGNMENTS`), the fully-qualified path (`ANALYTICS.MARTS.CONSIGNMENTS`), or any suffix; the tool resolves all forms and prints the canonical key in its output.
|
||||
|
||||
```yaml
|
||||
name: aav_consignments
|
||||
descriptions:
|
||||
user: AAV consignments — filtered view of MARTS.CONSIGNMENTS for the auto-auction-vaulting channel.
|
||||
user: AAV consignments - filtered view of MARTS.CONSIGNMENTS for the auto-auction-vaulting channel.
|
||||
source_type: sql
|
||||
sql: |
|
||||
SELECT CONSIGNED_ITEM_ID, CASH_ADV_AMOUNT, ALT_VALUE_COMBINED, my_derived_flag
|
||||
|
|
@ -131,7 +131,7 @@ columns:
|
|||
- { name: CONSIGNED_ITEM_ID } # type/descriptions inherited from manifest
|
||||
- { name: CASH_ADV_AMOUNT }
|
||||
- { name: ALT_VALUE_COMBINED }
|
||||
- { name: my_derived_flag, type: boolean, expr: "CASH_ADV_AMOUNT > 0", descriptions: { user: "Computed locally — has any cash advance." } }
|
||||
- { name: my_derived_flag, type: boolean, expr: "CASH_ADV_AMOUNT > 0", descriptions: { user: "Computed locally - has any cash advance." } }
|
||||
measures:
|
||||
- name: total_cash_advance
|
||||
expr: sum(CASH_ADV_AMOUNT)
|
||||
|
|
@ -139,12 +139,12 @@ measures:
|
|||
|
||||
Rules:
|
||||
|
||||
- Inheritance fills only **blank** fields. If you set a `description` locally, it wins — useful when the base description is misleading in the filtered view.
|
||||
- Inheritance fills only **blank** fields. If you set a `description` locally, it wins - useful when the base description is misleading in the filtered view.
|
||||
- A column not in the manifest (a derived/aliased column, or one from a different table in a `JOIN`) needs its own `type` and `description` declared.
|
||||
- If `inherits_columns_from` doesn't resolve, the source still loads, but every column without a type triggers a validator error on the warehouse probe — `sl_discover` first to confirm the key.
|
||||
- Don't use `inherits_columns_from` for sources backed by `table:` (those should be overlays — see the rule against shadowing the manifest above).
|
||||
- If `inherits_columns_from` doesn't resolve, the source still loads, but every column without a type triggers a validator error on the warehouse probe - `sl_discover` first to confirm the key.
|
||||
- Don't use `inherits_columns_from` for sources backed by `table:` (those should be overlays - see the rule against shadowing the manifest above).
|
||||
|
||||
## Refinement — replace, don't append
|
||||
## Refinement - replace, don't append
|
||||
|
||||
When the user corrects a prior answer, the existing measure is wrong by the user's own standard. Replace it, don't add a parallel measure.
|
||||
|
||||
|
|
@ -208,14 +208,14 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
|
|||
|
||||
## Tool sequence
|
||||
|
||||
1. `sl_discover` — see what source files exist.
|
||||
2. `sl_discover({ query: "<table-or-source-name>" })` — **REQUIRED before the first write on any name**. Shows columns/joins/grain from the manifest. If the call returns a schema, you MUST write an overlay, not a standalone. Skipping this is the #1 cause of accidentally shadowing the manifest.
|
||||
3. `sl_read_source({ connectionId, sourceName })` — read the raw YAML before editing.
|
||||
1. `sl_discover` - see what source files exist.
|
||||
2. `sl_discover({ query: "<table-or-source-name>" })` - **REQUIRED before the first write on any name**. Shows columns/joins/grain from the manifest. If the call returns a schema, you MUST write an overlay, not a standalone. Skipping this is the #1 cause of accidentally shadowing the manifest.
|
||||
3. `sl_read_source({ connectionId, sourceName })` - read the raw YAML before editing.
|
||||
4. For modifications: `sl_edit_source({ connectionId, sourceName, yaml_edits: [{ oldText, newText, reason }] })` with exact-string replacements. `oldText` must match exactly and be unique in the file.
|
||||
5. For new sources or full rewrites: `sl_write_source({ connectionId, sourceName, source })` with the full structured source definition.
|
||||
6. For join discovery: use `sql_execution({connectionName: "warehouse", sql: "SELECT count(*) FROM public.orders o JOIN public.customers c ON c.id = o.customer_id LIMIT 20"})` with the target warehouse connection name and dialect-correct table names to verify the join key exists in both tables and assess cardinality before declaring the join.
|
||||
7. Cross-reference knowledge: author the edge once on the **wiki** side via `sl_refs: [source_name]` in the page's front-matter. The reverse edge (wiki pages that cite an SL source) is derived automatically by the reconciler — do not add a `knowledge_refs:` field to SL YAMLs.
|
||||
8. `sl_validate` — run after writing or editing to surface schema issues, duplicate measure names, and cross-source validation errors. Read-only; the writes are already committed (the squash-at-end flow will collapse them into one commit).
|
||||
7. Cross-reference knowledge: author the edge once on the **wiki** side via `sl_refs: [source_name]` in the page's front-matter. The reverse edge (wiki pages that cite an SL source) is derived automatically by the reconciler - do not add a `knowledge_refs:` field to SL YAMLs.
|
||||
8. `sl_validate` - run after writing or editing to surface schema issues, duplicate measure names, and cross-source validation errors. Read-only; the writes are already committed (the squash-at-end flow will collapse them into one commit).
|
||||
|
||||
## Editing patterns
|
||||
|
||||
|
|
@ -224,13 +224,13 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
|
|||
- Do NOT modify existing measures or their descriptions unless the current turn explicitly corrects them.
|
||||
- During bundle/external ingest, include `rawPaths` on every `sl_write_source`/`sl_edit_source` call with only the raw files that directly support the SL action.
|
||||
|
||||
## Worked example — additive overlay
|
||||
## Worked example - additive overlay
|
||||
|
||||
Conversation:
|
||||
- User: "What was the average order value last quarter?"
|
||||
- Assistant fell back to SQL: `SELECT AVG(amount) FROM orders WHERE order_date >= ...`
|
||||
|
||||
Existing index: `orders [measures=0, joins=0] — candidate for enrichment`.
|
||||
Existing index: `orders [measures=0, joins=0] - candidate for enrichment`.
|
||||
|
||||
```
|
||||
sl_discover()
|
||||
|
|
@ -253,9 +253,9 @@ sl_validate({ connectionId: "warehouse" })
|
|||
→ clean
|
||||
```
|
||||
|
||||
The overlay only contains `name` and `measures` — no columns, grain, or table. Those are inherited from the manifest.
|
||||
The overlay only contains `name` and `measures` - no columns, grain, or table. Those are inherited from the manifest.
|
||||
|
||||
## Worked example — refinement (replace)
|
||||
## Worked example - refinement (replace)
|
||||
|
||||
Prior turn:
|
||||
- [user] "How many active users do we have per region?"
|
||||
|
|
@ -281,7 +281,7 @@ sl_validate({ connectionId: "warehouse" })
|
|||
|
||||
If you only added a new measure, the old incorrect `active_count` would stay and future queries would keep answering the wrong question.
|
||||
|
||||
## Worked example — new join
|
||||
## Worked example - new join
|
||||
|
||||
Prior turn: user asked to correlate LTV with protocol count; assistant joined `fct_orders` with `fct_mau_multiprotocol` on `admin_user_id` in raw SQL.
|
||||
|
||||
|
|
@ -315,6 +315,6 @@ Always verify joins with `sql_execution` before adding them.
|
|||
- A measure whose filter matches a segment MUST reference the segment via `segments: [name]`.
|
||||
- Extract repeated predicates into named segments.
|
||||
- Use computed dimensions for derived categories.
|
||||
- When the user corrects a prior answer, replace — don't append.
|
||||
- When the user corrects a prior answer, replace - don't append.
|
||||
- Always run `sl_validate` after writing to surface issues.
|
||||
- If nothing is worth capturing, respond without calling any SL write tool.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: wiki_capture
|
||||
description: KTX's knowledge base — wiki pages for durable, reusable business knowledge. Covers capture workflow for user preferences, metric definitions, organizational conventions, and cross-references between wiki pages and semantic-layer sources. Loaded by the post-turn memory-agent only. The research agent reads wiki via `wiki_read`/`wiki_search` but does not write it.
|
||||
description: KTX's knowledge base - wiki pages for durable, reusable business knowledge. Covers capture workflow for user preferences, metric definitions, organizational conventions, and cross-references between wiki pages and semantic-layer sources. Loaded by the post-turn memory-agent only. The research agent reads wiki via `wiki_read`/`wiki_search` but does not write it.
|
||||
callers: [memory_agent]
|
||||
---
|
||||
|
||||
|
|
@ -8,14 +8,14 @@ callers: [memory_agent]
|
|||
|
||||
## Role
|
||||
|
||||
The knowledge base stores durable, reusable business knowledge for an analytics assistant. Each page is a self-contained rule, definition, or convention that answers "how should this concept be handled in this organization?" — written once and reused across chats.
|
||||
The knowledge base stores durable, reusable business knowledge for an analytics assistant. Each page is a self-contained rule, definition, or convention that answers "how should this concept be handled in this organization?" - written once and reused across chats.
|
||||
|
||||
Scope selection is handled by the runtime:
|
||||
- When user-scoped knowledge is enabled AND the caller is a chat turn, writes go to the user's **personal** scope.
|
||||
- When the caller is an admin-driven ingest (`sourceType: 'external_ingest'`), writes go to the **global** scope.
|
||||
- When user-scoped knowledge is disabled, all writes go to the global scope.
|
||||
|
||||
The `wiki_write` tool picks the right scope based on the session. Capture logic does not need to choose — focus on whether the content is worth capturing at all.
|
||||
The `wiki_write` tool picks the right scope based on the session. Capture logic does not need to choose - focus on whether the content is worth capturing at all.
|
||||
|
||||
## What to capture
|
||||
|
||||
|
|
@ -30,8 +30,8 @@ Do NOT capture:
|
|||
- One-off requests ("answer under 100 words").
|
||||
- Temporary instructions scoped to the current chat.
|
||||
- Ad-hoc formatting preferences.
|
||||
- Information already present in the semantic layer (column names, join paths, measure formulas — those belong in SL).
|
||||
- **Query results, snapshots, or time-bounded benchmark tables.** Numbers go stale; pasting "Oct 2025: 25%, Nov 2025: 19.9%, …" creates misinformation as soon as new data lands. Reference the SL source by name (`sl_refs`) and let future query tools pull live data — the wiki captures the *rule* (definition, exclusion, segmentation), the SL source captures the *measure*, and query execution captures the *current values*.
|
||||
- Information already present in the semantic layer (column names, join paths, measure formulas - those belong in SL).
|
||||
- **Query results, snapshots, or time-bounded benchmark tables.** Numbers go stale; pasting "Oct 2025: 25%, Nov 2025: 19.9%, …" creates misinformation as soon as new data lands. Reference the SL source by name (`sl_refs`) and let future query tools pull live data - the wiki captures the *rule* (definition, exclusion, segmentation), the SL source captures the *measure*, and query execution captures the *current values*.
|
||||
- **Interpretive narrative tied to a specific snapshot** ("M1 retention degraded sharply from Dec 2025"). The observation is anchored to data that will move; the actionable convention (e.g., "always exclude in-progress cohorts") may be worth capturing on its own, but the snapshot-specific commentary is not.
|
||||
|
||||
If nothing is worth capturing, respond without calling any tool.
|
||||
|
|
@ -40,13 +40,13 @@ If nothing is worth capturing, respond without calling any tool.
|
|||
|
||||
1. Read the wiki index (provided in the prompt) and decide whether the turn introduces durable knowledge.
|
||||
2. **Before writing**, search for related content so cross-references are accurate:
|
||||
- `discover_data` first when a page relates to data or SL concepts — find
|
||||
- `discover_data` first when a page relates to data or SL concepts - find
|
||||
existing wiki pages, SL sources, and raw warehouse schema together.
|
||||
- `wiki_search` with the topic — find related wiki pages to populate `refs`.
|
||||
- `sl_discover` with the concept — if the page defines a metric (revenue, churn, retention, LTV, ARR, MRR, CAC, attribution, etc.), find matching SL sources or measures to populate `sl_refs`. If no matches, pass `sl_refs: []` so future readers know you checked.
|
||||
- `wiki_search` with the topic - find related wiki pages to populate `refs`.
|
||||
- `sl_discover` with the concept - if the page defines a metric (revenue, churn, retention, LTV, ARR, MRR, CAC, attribution, etc.), find matching SL sources or measures to populate `sl_refs`. If no matches, pass `sl_refs: []` so future readers know you checked.
|
||||
3. If updating an existing page, `wiki_read` it first. Use the returned `structured.content` or markdown body as the exact stored text for targeted replacements; current tags, refs, and sl_refs are returned in structured metadata.
|
||||
4. `wiki_write` to create or update. Prefer merging into an existing page over creating a new one.
|
||||
5. `wiki_remove` only when a page is truly obsolete — not to replace stale content (update it instead).
|
||||
5. `wiki_remove` only when a page is truly obsolete - not to replace stale content (update it instead).
|
||||
|
||||
For bundle/external ingest, include `rawPaths` on every `wiki_write`/`wiki_remove` call with only the raw files that directly support that wiki action. This keeps ingest provenance tied to the actual source file, not every file in the WorkUnit.
|
||||
|
||||
|
|
@ -82,7 +82,7 @@ SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
|
|||
|
||||
- **Keys** are short kebab-case topic identifiers: `leads-source-filter`, `revenue-definition`, `churn-calculation`. No namespacing, no prefixes.
|
||||
- **Summary** is a one-line hook (≤200 chars) shown in the index.
|
||||
- **Content** is concise markdown — actionable rules, not prose.
|
||||
- **Content** is concise markdown - actionable rules, not prose.
|
||||
|
||||
```
|
||||
## [Topic Title]
|
||||
|
|
@ -116,8 +116,8 @@ All three fields use REPLACE semantics on update:
|
|||
|
||||
Two modes:
|
||||
|
||||
- **Full content** — pass `content` to rewrite the whole page. Use when the page structure needs to change.
|
||||
- **Targeted edits** — pass `replacements: [{ oldText, newText }]` to apply exact-string replacements. Use for small updates; preserves the rest of the page.
|
||||
- **Full content** - pass `content` to rewrite the whole page. Use when the page structure needs to change.
|
||||
- **Targeted edits** - pass `replacements: [{ oldText, newText }]` to apply exact-string replacements. Use for small updates; preserves the rest of the page.
|
||||
|
||||
When editing, read the page first so the edit matches exact whitespace and indentation.
|
||||
|
||||
|
|
@ -125,7 +125,7 @@ When editing, read the page first so the edit matches exact whitespace and inden
|
|||
|
||||
Organization (GLOBAL) pages are read-only from a user's personal-scope session. To override a global rule for a single user, write a personal page with the **same key**. At read time the USER page wins.
|
||||
|
||||
## Worked example — capturing a metric with cross-references
|
||||
## Worked example - capturing a metric with cross-references
|
||||
|
||||
User says: "Going forward, the official refund rate is total refunded amount divided by total gross transaction amount."
|
||||
|
||||
|
|
@ -133,7 +133,7 @@ User says: "Going forward, the official refund rate is total refunded amount div
|
|||
wiki_list_tags()
|
||||
→ existing tags include "finance"
|
||||
wiki_search({ query: "refund revenue paid orders" })
|
||||
→ returns `revenue-definition` (related — defines paid-orders filter)
|
||||
→ returns `revenue-definition` (related - defines paid-orders filter)
|
||||
sl_discover({ query: "refund rate" })
|
||||
→ returns fct_orders (score 0.08), fct_gaap_revenue (0.06)
|
||||
sl_read_source({ connectionId: "warehouse", sourceName: "fct_orders" })
|
||||
|
|
@ -155,6 +155,6 @@ Search-then-write order matters. Cross-references are part of the page's identit
|
|||
- Read existing pages before updating them.
|
||||
- Prefer merging into an existing page over creating a new one.
|
||||
- Prefer fewer, richer pages over many thin ones.
|
||||
- Write content as clear, actionable rules — not narrative prose.
|
||||
- Write content as clear, actionable rules - not narrative prose.
|
||||
- Discover cross-references via search before writing, not after.
|
||||
- If nothing is worth capturing, respond without calling any tool.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Semantic Layer Engine
|
||||
|
||||
Python semantic layer that generates SQL from structured JSON queries. No `from` clause — sources are inferred from fully-qualified field names (`source.column`).
|
||||
Python semantic layer that generates SQL from structured JSON queries. No `from` clause - sources are inferred from fully-qualified field names (`source.column`).
|
||||
|
||||
## Quick Start
|
||||
|
||||
|
|
@ -16,7 +16,7 @@ Use `--model` to pass a self-contained YAML model (list of source definitions) i
|
|||
### 1. Create an inline model file
|
||||
|
||||
```yaml
|
||||
# /tmp/model.yaml — a YAML list of source definitions
|
||||
# /tmp/model.yaml - a YAML list of source definitions
|
||||
- name: orders
|
||||
table: public.orders
|
||||
grain: [id]
|
||||
|
|
@ -119,9 +119,9 @@ uv run python -m semantic_layer.cli --model /tmp/model.yaml \
|
|||
|
||||
## Coding Guidelines
|
||||
|
||||
### Expression handling — always use sqlglot AST, never regex on SQL
|
||||
### Expression handling - always use sqlglot AST, never regex on SQL
|
||||
|
||||
- **Parse expressions** with `sqlglot.parse_one(f"SELECT {expr}")` and walk/transform the AST. Never use `str.replace()`, `re.sub()`, or string splitting on SQL fragments — these corrupt string literals, aliases, and nested expressions.
|
||||
- **Parse expressions** with `sqlglot.parse_one(f"SELECT {expr}")` and walk/transform the AST. Never use `str.replace()`, `re.sub()`, or string splitting on SQL fragments - these corrupt string literals, aliases, and nested expressions.
|
||||
- **Quote reserved words first**: always call `quote_reserved_identifiers(expr)` before passing to `sqlglot.parse_one()`. Column/source names like `group`, `key`, `order` will fail to parse otherwise.
|
||||
- **Use the parse cache** in `parser.py` (`ExpressionParser._parse_as_select()`) for read-only AST walks. Direct `sqlglot.parse_one()` calls are fine when you need to `.transform()` the tree.
|
||||
- **Regex is fine for non-SQL tasks**: sanitizing alias names, masking string literals before parse, etc. The rule is: don't use regex to interpret SQL structure.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue