mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-10 08:05:14 +02:00
Merge pull request #27 from Kaelio/luca-martial/pull-latest-main
Refresh KTX docs and introduction page
This commit is contained in:
Luca Martial
GitHub
commit
d65b618279
14 changed files with 341 additions and 583 deletions
362
README.md
362
README.md
|
|
@ -3,7 +3,7 @@
|
|||
</h1>
|
||||
|
||||
<p align="center">
|
||||
<strong>Workspace-first context layer for database agents</strong>
|
||||
<strong>The context layer for analytics agents</strong>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
|
|
@ -14,312 +14,154 @@
|
|||
|
||||
---
|
||||
|
||||
KTX stores warehouse memory in a project directory, generates and validates
|
||||
semantic-layer YAML, indexes knowledge, scans database schemas, and exposes the
|
||||
result through a CLI and MCP server.
|
||||
KTX turns warehouse metadata, semantic definitions, and business knowledge into
|
||||
reviewable project files that agents can use while planning, querying, and
|
||||
updating analytics work.
|
||||
|
||||
KTX projects are plain files: YAML, Markdown, SQLite state, and generated
|
||||
artifacts. You can inspect them, commit them, and serve them to any MCP client.
|
||||
A KTX project is a directory of plain files — YAML semantic sources, Markdown
|
||||
knowledge pages, and SQLite state — that you commit to git and review in PRs,
|
||||
just like dbt models.
|
||||
|
||||
## What KTX provides
|
||||
## Who KTX is for
|
||||
|
||||
- Durable warehouse memory with semantic-layer sources and knowledge pages.
|
||||
- Native scan connectors for SQLite, Postgres, MySQL, ClickHouse, SQL Server,
|
||||
BigQuery, and Snowflake.
|
||||
- Agentic ingest with provenance links, tool transcripts, and replay metadata.
|
||||
- Local semantic-layer query planning and optional query execution.
|
||||
- A stdio MCP server with tools for connections, knowledge, semantic-layer
|
||||
sources, ingest reports, and replay.
|
||||
KTX is built for analytics engineers and data teams who want data agents to
|
||||
work on real analytics systems — not just generate one-off SQL.
|
||||
|
||||
Use KTX when you want agents to:
|
||||
|
||||
- **Generate SQL** from approved measures and joins
|
||||
- **Repair semantic definitions** through reviewable diffs
|
||||
- **Explain metric provenance** with warehouse evidence
|
||||
- **Work alongside** dbt, LookML, MetricFlow, Looker, Metabase, and modern BI
|
||||
platforms
|
||||
|
||||
Works with PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, and
|
||||
SQLite.
|
||||
|
||||
## Quick start
|
||||
|
||||
Run the pre-seeded demo through the public npm package:
|
||||
Install the CLI and run the setup wizard:
|
||||
|
||||
```bash
|
||||
npx @kaelio/ktx setup demo --no-input
|
||||
npx @kaelio/ktx setup demo inspect
|
||||
```
|
||||
|
||||
The default demo uses packaged sample data and prebuilt context. It does not
|
||||
require API keys, network access, or an LLM provider.
|
||||
|
||||
To replay the packaged ingest run, use:
|
||||
|
||||
```bash
|
||||
npx @kaelio/ktx setup demo --mode replay --no-input
|
||||
```
|
||||
|
||||
To run the full agentic demo with an LLM provider, set a provider key for the
|
||||
current process:
|
||||
|
||||
```bash
|
||||
ANTHROPIC_API_KEY=$YOUR_ANTHROPIC_API_KEY \
|
||||
npx @kaelio/ktx setup demo --mode full --no-input
|
||||
```
|
||||
|
||||
Interactive full-demo setup can prompt for a provider key without writing the
|
||||
key to `ktx.yaml`.
|
||||
|
||||
You can also install the CLI in a project or globally:
|
||||
|
||||
```bash
|
||||
npm install @kaelio/ktx
|
||||
npx ktx --help
|
||||
npm install -g @kaelio/ktx
|
||||
ktx --help
|
||||
ktx setup
|
||||
```
|
||||
|
||||
## Build a local project
|
||||
The wizard walks through six steps: configuring your LLM provider, setting up
|
||||
embeddings, connecting your database, adding context sources (dbt, LookML,
|
||||
Metabase, Looker, Notion), building context, and installing agent integration.
|
||||
|
||||
Create a project from a local workspace:
|
||||
If it exits before completion, rerun `ktx setup` to resume where you left off.
|
||||
|
||||
Check your project status:
|
||||
|
||||
```bash
|
||||
npm install @kaelio/ktx
|
||||
PROJECT_DIR="$(mktemp -d)/ktx-demo"
|
||||
npx ktx init "$PROJECT_DIR" --name ktx-demo
|
||||
ktx status
|
||||
```
|
||||
|
||||
Create a SQLite warehouse:
|
||||
```
|
||||
KTX project: /home/user/analytics
|
||||
Project ready: yes
|
||||
LLM ready: yes (claude-sonnet-4-6)
|
||||
Embeddings ready: yes (text-embedding-3-small)
|
||||
Primary sources configured: yes (postgres-warehouse)
|
||||
Context sources configured: yes (dbt-main)
|
||||
KTX context built: yes
|
||||
Agent integration ready: yes (claude-code:project)
|
||||
```
|
||||
|
||||
## What's in a project
|
||||
|
||||
```
|
||||
my-project/
|
||||
├── ktx.yaml # Project configuration
|
||||
├── semantic-layer/
|
||||
│ └── warehouse/
|
||||
│ ├── orders.yaml # Semantic source definitions
|
||||
│ ├── customers.yaml
|
||||
│ └── order_items.yaml
|
||||
├── knowledge/
|
||||
│ ├── global/
|
||||
│ │ ├── revenue.md # Business definitions and rules
|
||||
│ │ └── segment-classification.md
|
||||
│ └── user/
|
||||
│ └── local/
|
||||
├── raw-sources/
|
||||
│ └── warehouse/
|
||||
│ └── live-database/ # Scan artifacts and reports
|
||||
└── .ktx/
|
||||
└── db.sqlite # Local state (git-ignored)
|
||||
```
|
||||
|
||||
Semantic sources and knowledge pages are committed to git. The `.ktx/` directory
|
||||
holds ephemeral state and is git-ignored — delete it and KTX rebuilds on the
|
||||
next run.
|
||||
|
||||
## Serve agents
|
||||
|
||||
KTX integrates with coding agents through CLI skills, an MCP server, or both.
|
||||
The setup wizard configures this automatically — here's what each mode looks
|
||||
like.
|
||||
|
||||
**CLI skills** — the agent calls `ktx` commands directly through a skill file
|
||||
installed in your agent's config (e.g., `.claude/skills/ktx/SKILL.md`):
|
||||
|
||||
```bash
|
||||
python - "$PROJECT_DIR/demo.db" <<'PY'
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
conn = sqlite3.connect(sys.argv[1])
|
||||
conn.executescript("""
|
||||
DROP TABLE IF EXISTS accounts;
|
||||
CREATE TABLE accounts (
|
||||
account_id INTEGER PRIMARY KEY,
|
||||
account_name TEXT NOT NULL,
|
||||
segment TEXT NOT NULL,
|
||||
region TEXT NOT NULL
|
||||
);
|
||||
INSERT INTO accounts VALUES
|
||||
(1, 'Acme Analytics', 'Mid-Market', 'NA'),
|
||||
(2, 'Beacon Bank', 'Enterprise', 'EMEA'),
|
||||
(3, 'Cobalt Coffee', 'SMB', 'NA'),
|
||||
(4, 'Delta Devices', 'Mid-Market', 'APAC'),
|
||||
(5, 'Evergreen Energy', 'Enterprise', 'NA');
|
||||
""")
|
||||
conn.close()
|
||||
PY
|
||||
ktx sl query --measure orders.revenue --dimension orders.status --format sql
|
||||
ktx wiki search "revenue definition"
|
||||
ktx sl validate orders
|
||||
```
|
||||
|
||||
Replace the generated `ktx.yaml`:
|
||||
**MCP server** — the agent calls KTX tools over the Model Context Protocol:
|
||||
|
||||
```bash
|
||||
cat > "$PROJECT_DIR/ktx.yaml" <<YAML
|
||||
project: ktx-demo
|
||||
connections:
|
||||
warehouse:
|
||||
driver: sqlite
|
||||
path: $PROJECT_DIR/demo.db
|
||||
readonly: true
|
||||
storage:
|
||||
state: sqlite
|
||||
search: sqlite-fts5
|
||||
git:
|
||||
auto_commit: true
|
||||
author: "ktx <ktx@example.com>"
|
||||
memory:
|
||||
auto_commit: true
|
||||
YAML
|
||||
```
|
||||
|
||||
Write and validate a semantic-layer source:
|
||||
|
||||
```bash
|
||||
npx ktx sl write accounts --project-dir "$PROJECT_DIR" \
|
||||
--connection-id warehouse --yaml 'name: accounts
|
||||
table: accounts
|
||||
description: CRM accounts with segmentation attributes.
|
||||
grain:
|
||||
- account_id
|
||||
columns:
|
||||
- name: account_id
|
||||
type: number
|
||||
- name: account_name
|
||||
type: string
|
||||
- name: segment
|
||||
type: string
|
||||
- name: region
|
||||
type: string
|
||||
measures:
|
||||
- name: account_count
|
||||
expr: count(account_id)
|
||||
joins: []
|
||||
'
|
||||
|
||||
npx ktx sl validate accounts --project-dir "$PROJECT_DIR" \
|
||||
--connection-id warehouse
|
||||
```
|
||||
|
||||
Generate SQL and execute the query:
|
||||
|
||||
```bash
|
||||
npx ktx sl query --project-dir "$PROJECT_DIR" \
|
||||
--connection-id warehouse \
|
||||
--measure accounts.account_count \
|
||||
--dimension accounts.segment \
|
||||
--order-by accounts.account_count:desc \
|
||||
--limit 5 \
|
||||
--format sql
|
||||
|
||||
npx ktx sl query --project-dir "$PROJECT_DIR" \
|
||||
--connection-id warehouse \
|
||||
--measure accounts.account_count \
|
||||
--dimension accounts.segment \
|
||||
--order-by accounts.account_count:desc \
|
||||
--limit 5 \
|
||||
--execute \
|
||||
--max-rows 5
|
||||
```
|
||||
|
||||
List and test the warehouse connection:
|
||||
|
||||
```bash
|
||||
npx ktx connection list --project-dir "$PROJECT_DIR"
|
||||
npx ktx connection test warehouse --project-dir "$PROJECT_DIR"
|
||||
```
|
||||
|
||||
The connection test prints the configured driver and discovered table count:
|
||||
|
||||
```text
|
||||
Driver: sqlite
|
||||
Tables: 1
|
||||
```
|
||||
|
||||
### Scan the demo warehouse
|
||||
|
||||
Scan artifacts are written under
|
||||
`raw-sources/warehouse/live-database/<syncId>/` in the project directory.
|
||||
|
||||
```bash
|
||||
|
||||
SCAN_OUTPUT="$(npx ktx scan warehouse --project-dir "$PROJECT_DIR")"
|
||||
printf '%s\n' "$SCAN_OUTPUT"
|
||||
SCAN_RUN_ID="$(printf '%s\n' "$SCAN_OUTPUT" | awk '/^Run: / { print $2 }')"
|
||||
npx ktx scan status --project-dir "$PROJECT_DIR" "$SCAN_RUN_ID"
|
||||
npx ktx scan report --project-dir "$PROJECT_DIR" "$SCAN_RUN_ID"
|
||||
```
|
||||
|
||||
For non-SQLite drivers, prefer credential references such as `--url env:NAME`
|
||||
or `--url file:PATH` over literal credential URLs.
|
||||
|
||||
## Managed Python runtime
|
||||
|
||||
KTX installs its Python runtime only when a Python-backed command needs it.
|
||||
The runtime lives outside the npm cache, is versioned by the installed CLI
|
||||
version, and is managed by `ktx runtime` commands.
|
||||
|
||||
KTX requires `uv` on `PATH` to create the managed runtime. Install `uv` with
|
||||
your system package manager or the official installer before running Python-
|
||||
backed KTX commands. KTX doesn't download `uv` automatically; run
|
||||
`ktx runtime doctor` if runtime installation fails:
|
||||
|
||||
```bash
|
||||
npx ktx runtime install --yes
|
||||
npx ktx runtime status
|
||||
npx ktx runtime doctor
|
||||
npx ktx runtime start
|
||||
npx ktx runtime stop
|
||||
npx ktx runtime prune --dry-run
|
||||
npx ktx runtime prune --yes
|
||||
```
|
||||
|
||||
Use `runtime prune --dry-run` to preview stale runtime directories from older
|
||||
CLI versions. Add `--yes` to remove those stale directories after daemon
|
||||
processes are stopped.
|
||||
|
||||
Commands such as `npx @kaelio/ktx sl query ... --yes` can install the core
|
||||
runtime lazily from the bundled wheel. Local embeddings remain lazy; prepare
|
||||
them only when you select local `sentence-transformers` embeddings:
|
||||
|
||||
```bash
|
||||
npx ktx runtime install --feature local-embeddings --yes
|
||||
npx ktx runtime start --feature local-embeddings
|
||||
```
|
||||
|
||||
## Serve MCP
|
||||
|
||||
Start the stdio MCP server from the project directory:
|
||||
|
||||
```bash
|
||||
npx ktx serve --mcp stdio --project-dir "$PROJECT_DIR" \
|
||||
ktx serve --mcp stdio \
|
||||
--user-id local \
|
||||
--semantic-compute \
|
||||
--execute-queries \
|
||||
--yes
|
||||
```
|
||||
|
||||
The `--semantic-compute` flag uses the managed Python runtime when no explicit
|
||||
semantic compute URL is provided. KTX starts or reuses the managed runtime as
|
||||
needed.
|
||||
This exposes tools for connections, knowledge search, semantic-layer sources,
|
||||
validation, queries, ingestion, and replay. The `--semantic-compute` flag starts
|
||||
the managed Python runtime for query planning automatically.
|
||||
|
||||
The MCP server exposes `connection_list`, `knowledge_search`,
|
||||
`knowledge_read`, `knowledge_write`, `sl_list_sources`, `sl_read_source`,
|
||||
`sl_write_source`, `sl_validate`, `sl_query`, `ingest_trigger`,
|
||||
`ingest_status`, `ingest_report`, and `ingest_replay`.
|
||||
Supported agents: Claude Code, Codex, Cursor, OpenCode, and any agent that
|
||||
reads `.agents/` skills or MCP configuration.
|
||||
|
||||
## Workspace packages
|
||||
|
||||
- `packages/context`: core TypeScript context library.
|
||||
- `packages/cli`: CLI wrapper over the context package.
|
||||
- `packages/llm`: LLM and embedding provider helpers.
|
||||
- `packages/connector-bigquery`: BigQuery scan connector.
|
||||
- `packages/connector-clickhouse`: ClickHouse scan connector.
|
||||
- `packages/connector-mysql`: MySQL scan connector.
|
||||
- `packages/connector-postgres`: Postgres scan connector.
|
||||
- `packages/connector-snowflake`: Snowflake scan connector.
|
||||
- `packages/connector-sqlite`: SQLite scan connector.
|
||||
- `packages/connector-sqlserver`: SQL Server scan connector.
|
||||
- `python/ktx-sl`: semantic-layer engine.
|
||||
- `python/ktx-daemon`: portable compute service for semantic-layer operations.
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `packages/cli` | CLI entry point |
|
||||
| `packages/context` | Core context engine |
|
||||
| `packages/llm` | LLM and embedding providers |
|
||||
| `packages/connector-*` | Database connectors (Postgres, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, SQLite) |
|
||||
| `python/ktx-sl` | Semantic-layer query planning |
|
||||
| `python/ktx-daemon` | Portable compute service |
|
||||
|
||||
## Development
|
||||
|
||||
Install dependencies and run checks:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/kaelio/ktx.git
|
||||
cd ktx
|
||||
pnpm install
|
||||
uv sync --all-groups
|
||||
pnpm run build
|
||||
pnpm run check
|
||||
uv sync --all-packages
|
||||
source .venv/bin/activate
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
Use the optional development binary when you want a local `ktx-dev` command:
|
||||
Use the development CLI for local testing:
|
||||
|
||||
```bash
|
||||
pnpm run setup:dev
|
||||
pnpm run link:dev
|
||||
ktx-dev --help
|
||||
```
|
||||
|
||||
The repository uses `pnpm` for TypeScript packages and `uv` for Python
|
||||
packages.
|
||||
|
||||
## Release status
|
||||
|
||||
This repository builds one public npm artifact named `@kaelio/ktx`. The release
|
||||
artifact manifest contains the public npm tarball and the bundled `kaelio-ktx`
|
||||
runtime wheel. The first public npm handoff is policy-gated through
|
||||
`release-policy.json`, which keeps Python package publishing disabled because
|
||||
KTX-owned Python code ships inside the npm package as a bundled wheel. The
|
||||
`python/ktx-sl` and `python/ktx-daemon` directories remain source packages for
|
||||
development, not public release artifacts.
|
||||
|
||||
Build local package artifacts and verify the guarded dry-run publish path with:
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
pnpm run artifacts:check
|
||||
pnpm run release:readiness
|
||||
pnpm run release:npm-publish
|
||||
```
|
||||
|
||||
Run the live npm publish only from the manual `KTX Release` workflow with the
|
||||
`publish_live` input enabled after the `NPM_TOKEN` secret is configured.
|
||||
packages. See [Contributing](docs-site/content/docs/community/contributing.mdx)
|
||||
for full development setup, testing, and PR guidelines.
|
||||
|
||||
## License
|
||||
|
||||
|
|
|
|||
|
|
@ -17,6 +17,10 @@ function isDocsIndex(slug: string[] | undefined) {
|
|||
return slug === undefined || slug.length === 0 || slug.join("/") === "";
|
||||
}
|
||||
|
||||
function isHeroPage(slug: string[] | undefined) {
|
||||
return slug?.join("/") === "getting-started/introduction";
|
||||
}
|
||||
|
||||
export default async function Page(props: {
|
||||
params: Promise<{ slug?: string[] }>;
|
||||
}) {
|
||||
|
|
@ -30,14 +34,22 @@ export default async function Page(props: {
|
|||
|
||||
const MDX = page.data.body;
|
||||
|
||||
const hero = isHeroPage(params.slug);
|
||||
|
||||
return (
|
||||
<DocsPage toc={page.data.toc}>
|
||||
<DocsTitle>{page.data.title}</DocsTitle>
|
||||
<DocsDescription>{page.data.description}</DocsDescription>
|
||||
<DocsPageActions
|
||||
markdownUrl={`${page.url}.md`}
|
||||
mdxSource={page.data.content}
|
||||
/>
|
||||
{!hero && (
|
||||
<>
|
||||
<div className="flex items-start justify-between gap-4">
|
||||
<DocsTitle>{page.data.title}</DocsTitle>
|
||||
<DocsPageActions
|
||||
markdownUrl={`${page.url}.md`}
|
||||
mdxSource={page.data.content}
|
||||
/>
|
||||
</div>
|
||||
<DocsDescription>{page.data.description}</DocsDescription>
|
||||
</>
|
||||
)}
|
||||
<DocsBody>
|
||||
<MDX components={{ ...defaultMdxComponents, pre: CodeBlock }} />
|
||||
</DocsBody>
|
||||
|
|
|
|||
|
|
@ -262,6 +262,74 @@ figure[data-rehype-pretty-code-figure]:has(.ktx-code) {
|
|||
color: #c8c3bc !important;
|
||||
}
|
||||
|
||||
/* ── Mode D: Output preview (wizard prompts, status output) ── */
|
||||
.ktx-code-output {
|
||||
background: var(--color-fd-muted);
|
||||
border: 1px solid var(--color-fd-border);
|
||||
border-left: 3px solid color-mix(in oklch, var(--color-fd-primary) 50%, var(--color-fd-border));
|
||||
position: relative;
|
||||
box-shadow: 0 1px 2px rgba(27, 27, 24, 0.02);
|
||||
}
|
||||
|
||||
.dark .ktx-code-output {
|
||||
background: #111a1e;
|
||||
border-color: rgba(255, 255, 255, 0.05);
|
||||
border-left-color: rgba(34, 211, 238, 0.25);
|
||||
}
|
||||
|
||||
.ktx-code-output:hover {
|
||||
border-color: color-mix(in oklch, var(--color-fd-primary) 25%, var(--color-fd-border));
|
||||
border-left-color: var(--color-fd-primary);
|
||||
}
|
||||
|
||||
.dark .ktx-code-output:hover {
|
||||
border-color: rgba(255, 255, 255, 0.08);
|
||||
border-left-color: rgba(34, 211, 238, 0.45);
|
||||
}
|
||||
|
||||
.ktx-code-output-label {
|
||||
position: absolute;
|
||||
top: 8px;
|
||||
right: 14px;
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
color: var(--color-fd-muted-foreground);
|
||||
font-family: var(--font-display), var(--font-sans), sans-serif;
|
||||
opacity: 0.4;
|
||||
pointer-events: none;
|
||||
z-index: 1;
|
||||
}
|
||||
|
||||
.ktx-code-output-copy {
|
||||
position: absolute !important;
|
||||
top: 6px !important;
|
||||
right: 6px !important;
|
||||
opacity: 0;
|
||||
transform: translateY(-4px);
|
||||
transition: opacity 0.2s var(--ktx-ease), transform 0.2s var(--ktx-ease);
|
||||
z-index: 2;
|
||||
}
|
||||
|
||||
.ktx-code-output:hover .ktx-code-output-copy {
|
||||
opacity: 0.5;
|
||||
transform: translateY(0);
|
||||
}
|
||||
|
||||
.ktx-code-output:hover .ktx-code-output-label {
|
||||
opacity: 0;
|
||||
}
|
||||
|
||||
.ktx-code-body-output {
|
||||
background: transparent !important;
|
||||
color: var(--ktx-ink-soft) !important;
|
||||
}
|
||||
|
||||
.dark .ktx-code-body-output {
|
||||
color: #8a9da6 !important;
|
||||
}
|
||||
|
||||
/* ── Mode B: VS Code tab (filename) ───────── */
|
||||
.ktx-code-tab {
|
||||
background: var(--color-fd-card);
|
||||
|
|
@ -495,14 +563,20 @@ th {
|
|||
opacity: 0.7;
|
||||
}
|
||||
|
||||
/* Hide the vertical indicator lines in sidebar sections */
|
||||
#nd-sidebar div[data-state]::before,
|
||||
#nd-sidebar a[data-active]::before {
|
||||
content: none !important;
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
/* Page link items */
|
||||
#nd-sidebar a[data-active] {
|
||||
font-size: 14px;
|
||||
padding: 6px 12px;
|
||||
border-radius: 6px;
|
||||
margin-left: 0;
|
||||
border-left: 2px solid transparent;
|
||||
transition: background 0.15s ease, color 0.15s ease, border-color 0.15s ease;
|
||||
transition: background 0.15s ease, color 0.15s ease;
|
||||
}
|
||||
|
||||
#nd-sidebar a[data-active="false"]:hover {
|
||||
|
|
@ -512,7 +586,6 @@ th {
|
|||
|
||||
#nd-sidebar a[data-active="true"] {
|
||||
background: color-mix(in oklch, var(--color-fd-primary) 8%, transparent) !important;
|
||||
border-left-color: var(--color-fd-primary) !important;
|
||||
color: var(--color-fd-primary) !important;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -52,12 +52,11 @@ export function CodeBlock(props: Props) {
|
|||
const language = detectLanguage(props, children);
|
||||
const codeText = extractText(children);
|
||||
|
||||
const isTerminal =
|
||||
(language !== null && TERMINAL_LANGS.has(language)) ||
|
||||
WIZARD_GLYPHS.test(codeText);
|
||||
const isTerminal = language !== null && TERMINAL_LANGS.has(language);
|
||||
const isOutput = !isTerminal && WIZARD_GLYPHS.test(codeText);
|
||||
const hasTitle = typeof title === "string" && title.length > 0;
|
||||
|
||||
// Mode A — Terminal
|
||||
// Mode A — Terminal (commands the user types)
|
||||
if (isTerminal) {
|
||||
return (
|
||||
<div className="not-prose ktx-code ktx-code-terminal group">
|
||||
|
|
@ -80,6 +79,19 @@ export function CodeBlock(props: Props) {
|
|||
);
|
||||
}
|
||||
|
||||
// Mode D — Output preview (wizard prompts, terminal output)
|
||||
if (isOutput) {
|
||||
return (
|
||||
<div className="not-prose ktx-code ktx-code-output group relative">
|
||||
<span className="ktx-code-output-label">output</span>
|
||||
<CopyButton text={codeText} className="ktx-code-output-copy" />
|
||||
<pre {...rest} className="ktx-code-body ktx-code-body-output">
|
||||
{children}
|
||||
</pre>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
// Mode B — VS Code tab (filename present)
|
||||
if (hasTitle) {
|
||||
return (
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ type Props = {
|
|||
|
||||
export function DocsPageActions({ markdownUrl, mdxSource }: Props) {
|
||||
return (
|
||||
<div className="not-prose mt-4 mb-8 flex flex-wrap items-center gap-2 border-b border-fd-border pb-6 text-xs">
|
||||
<div className="not-prose flex flex-wrap items-center gap-2 text-xs">
|
||||
<CopyMarkdownButton markdownUrl={markdownUrl} />
|
||||
<a
|
||||
href={markdownUrl}
|
||||
|
|
|
|||
|
|
@ -1,163 +0,0 @@
|
|||
---
|
||||
title: Link Detection
|
||||
description: How KTX's relationship detection performs on real-world schemas.
|
||||
---
|
||||
|
||||
KTX infers foreign key relationships between tables even when the database declares no primary keys or foreign key constraints. This is critical for analytics warehouses, where constraints are rarely enforced. This page documents the methodology, scoring pipeline, and a reproducible benchmark you can run yourself.
|
||||
|
||||
## Agent usage notes
|
||||
|
||||
Use this page when an agent needs to explain, tune, or verify relationship detection.
|
||||
|
||||
| Agent task | Relevant section | Command |
|
||||
|------------|------------------|---------|
|
||||
| Explain why KTX inferred a join | Detection pipeline | `ktx dev scan relationships <run-id> --status all` |
|
||||
| Decide whether to accept or reject a candidate | Scoring and threshold configuration | `ktx dev scan relationships <run-id> --accept <candidate-id>` |
|
||||
| Tune thresholds from reviewed decisions | Broader benchmark suite and calibration | `ktx dev scan relationship-thresholds --connection <connection-id>` |
|
||||
| Reproduce the bundled benchmark | Reproducing the benchmark | `pnpm run relationships:verify-orbit` |
|
||||
|
||||
## What this measures
|
||||
|
||||
Most analytics warehouses — Snowflake, BigQuery, Redshift — don't enforce referential integrity constraints. Tables like `fct_product_events` reference `dim_accounts` by convention (`account_id` → `id`), but nothing in the schema says so.
|
||||
|
||||
KTX's relationship detection discovers these links automatically. The benchmark measures how accurately it recovers known foreign key relationships from a schema with **all declared constraints removed** — the hardest operating mode.
|
||||
|
||||
Metrics tracked:
|
||||
|
||||
- **Accepted** — relationships scored above the accept threshold (default 0.85) and written to the project manifest
|
||||
- **Review** — relationships scored between the review threshold (0.55) and accept threshold, flagged for human review
|
||||
- **Rejected** — relationships scored below the review threshold
|
||||
- **Skipped** — relationships not evaluated (e.g., filtered by candidate limits)
|
||||
|
||||
## Methodology
|
||||
|
||||
### Detection pipeline
|
||||
|
||||
Relationship detection runs as a multi-stage pipeline during `ktx dev scan`:
|
||||
|
||||
1. **Candidate generation** — scans the schema for potential FK relationships using multiple heuristics: exact column name matches, normalized table name matching, name inflection (singular/plural), column suffix patterns (`_id`, `_key`, `_code`, `_uuid`), self-references (`parent_id`, `manager_id`), and optionally embedding similarity and LLM proposals.
|
||||
|
||||
2. **Column profiling** — samples up to 10,000 rows per column (configurable via `profile_sample_rows`) to collect statistics: row counts, null rates, distinct value counts, uniqueness ratios, sample values, and text length ranges.
|
||||
|
||||
3. **Validation** — tests each candidate relationship against actual data by measuring target uniqueness, source coverage, violation ratio, and value overlap between child and parent columns.
|
||||
|
||||
4. **Scoring** — combines 7 weighted signals into a confidence score:
|
||||
|
||||
| Signal | Weight | What it captures |
|
||||
|--------|--------|-----------------|
|
||||
| Name similarity | 0.24 | How closely column/table names match FK conventions |
|
||||
| Value overlap | 0.22 | What percentage of FK values exist in the PK column |
|
||||
| Profile uniqueness | 0.22 | How unique the target column values are |
|
||||
| Type compatibility | 0.10 | Whether data types are compatible (hard gate — score is 0 if incompatible) |
|
||||
| Embedding similarity | 0.10 | Semantic similarity between column names |
|
||||
| Profile null rate | 0.08 | Presence of non-null values |
|
||||
| Structural prior | 0.04 | Baseline structural hints from schema conventions |
|
||||
|
||||
Each signal is normalized to \[0, 1\], multiplied by its weight, and summed. The final confidence is `0.56 + (weighted_sum × 0.65)`, clamped to \[0, 1\].
|
||||
|
||||
5. **Graph resolution** — resolves conflicts when multiple candidates target the same column, detects primary keys (by name pattern and validation), and classifies each relationship into `accepted`, `review`, or `rejected` based on thresholds.
|
||||
|
||||
### Threshold configuration
|
||||
|
||||
```yaml
|
||||
scan:
|
||||
relationships:
|
||||
accept_threshold: 0.85
|
||||
review_threshold: 0.55
|
||||
```
|
||||
|
||||
Relationships scoring above `accept_threshold` are automatically accepted into the project manifest. Those between `review_threshold` and `accept_threshold` are flagged for analyst review. Below `review_threshold`, they're rejected.
|
||||
|
||||
### Test fixture
|
||||
|
||||
The benchmark uses the **Orbit-style product warehouse** — a synthetic schema modeled after a real SaaS analytics warehouse with all declared constraints removed. The fixture is a SQLite database with 6 tables:
|
||||
|
||||
| Table | Role | Estimated rows |
|
||||
|-------|------|---------------|
|
||||
| `dim_accounts` | Dimension | 3 |
|
||||
| `dim_users` | Dimension | 4 |
|
||||
| `dim_workspaces` | Dimension | 4 |
|
||||
| `fct_product_events` | Fact | 5 |
|
||||
| `fct_invoices` | Fact | 3 |
|
||||
| `support_tickets` | Fact | 4 |
|
||||
|
||||
**Ground truth:** 6 primary keys (one `id` column per table) and 9 foreign key relationships, all `many_to_one`:
|
||||
|
||||
| Source column | Target |
|
||||
|--------------|--------|
|
||||
| `dim_users.account_id` | `dim_accounts.id` |
|
||||
| `dim_workspaces.account_id` | `dim_accounts.id` |
|
||||
| `dim_workspaces.user_id` | `dim_users.id` |
|
||||
| `fct_product_events.account_id` | `dim_accounts.id` |
|
||||
| `fct_product_events.user_id` | `dim_users.id` |
|
||||
| `fct_product_events.workspace_id` | `dim_workspaces.id` |
|
||||
| `fct_invoices.account_id` | `dim_accounts.id` |
|
||||
| `support_tickets.account_id` | `dim_accounts.id` |
|
||||
| `support_tickets.user_id` | `dim_users.id` |
|
||||
|
||||
The fixture runs in multiple modes to isolate the contribution of each pipeline stage: with LLM disabled, profiling disabled, validation disabled, and embeddings disabled.
|
||||
|
||||
## Results
|
||||
|
||||
Results for the default configuration will be added after the benchmark run is finalized.
|
||||
|
||||
## Reproducing the benchmark
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js 22+
|
||||
- pnpm
|
||||
- The KTX repository cloned and dependencies installed (`pnpm install`)
|
||||
|
||||
### Running
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
pnpm run relationships:verify-orbit
|
||||
```
|
||||
|
||||
This runs `ktx dev scan` against the bundled SQLite fixture with enrichment disabled, then generates a verification report at:
|
||||
|
||||
```text
|
||||
examples/orbit-relationship-verification/reports/orbit-verification.md
|
||||
```
|
||||
|
||||
The report includes the full relationship summary, enrichment details, artifact paths, and any warnings.
|
||||
|
||||
### Custom project
|
||||
|
||||
To run verification against your own database (e.g., a local Orbit project):
|
||||
|
||||
```bash
|
||||
KTX_ORBIT_PROJECT_DIR=/path/to/your-project pnpm run relationships:verify-orbit
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
The benchmark project configuration lives at `examples/orbit-relationship-verification/ktx.yaml`:
|
||||
|
||||
```yaml
|
||||
scan:
|
||||
enrichment:
|
||||
backend: none
|
||||
relationships:
|
||||
enabled: true
|
||||
llm_proposals: false
|
||||
accept_threshold: 0.85
|
||||
review_threshold: 0.55
|
||||
profile_sample_rows: 10000
|
||||
validation_concurrency: 4
|
||||
```
|
||||
|
||||
Adjust `accept_threshold` and `review_threshold` to see how threshold changes affect the accepted/review/rejected distribution. Lower thresholds accept more relationships (higher recall, lower precision); higher thresholds are more conservative.
|
||||
|
||||
## Broader benchmark suite
|
||||
|
||||
Beyond the Orbit fixture, KTX includes a full benchmark corpus at `packages/context/test/fixtures/relationship-benchmarks/` with fixtures across multiple tiers:
|
||||
|
||||
- **Unit** — minimal schemas testing individual heuristics
|
||||
- **Row-bearing** — small schemas with data for validation testing
|
||||
- **Product** — full warehouse schemas like the Orbit fixture
|
||||
|
||||
Fixtures from public datasets (Chinook, Sakila, AdventureWorks, Northwind) supplement the synthetic fixtures. The benchmark runner measures precision, recall, and F1 for both primary key and foreign key detection across all fixtures and modes.
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
{
|
||||
"title": "Benchmarks",
|
||||
"defaultOpen": true,
|
||||
"pages": ["link-detection"]
|
||||
}
|
||||
|
|
@ -68,7 +68,7 @@ The MCP server is typically configured through `ktx setup --agents` rather than
|
|||
|
||||
| Error | Cause | Recovery |
|
||||
|-------|-------|----------|
|
||||
| Agent cannot start server | The agent config cannot find the `ktx` binary | Run `pnpm run link:dev` or use an absolute command path in the agent config |
|
||||
| Agent cannot start server | The agent config cannot find the `ktx` binary | Install `@kaelio/ktx` globally with `npm install -g @kaelio/ktx` or use an absolute command path in the agent config |
|
||||
| Semantic tools are unavailable | Server was started without `--semantic-compute` | Add `--semantic-compute` or `--semantic-compute-url` to the server args |
|
||||
| Query execution is denied | Server was started without `--execute-queries` | Add `--execute-queries` only for trusted projects where read-only execution is intended |
|
||||
| Context resolves to wrong project | `KTX_PROJECT_DIR` is missing or points elsewhere | Set `KTX_PROJECT_DIR` to the project containing the intended `ktx.yaml` |
|
||||
|
|
|
|||
|
|
@ -7,6 +7,11 @@ KTX is an open-source project and welcomes contributions — bug fixes, new conn
|
|||
|
||||
## Development setup
|
||||
|
||||
This page is for contributors working on the KTX repository. To install KTX for
|
||||
an analytics project, use the published
|
||||
[`@kaelio/ktx`](https://www.npmjs.com/package/@kaelio/ktx) package in the
|
||||
[Quickstart](/docs/getting-started/quickstart).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Node.js 22+** and **pnpm** — for the TypeScript workspace
|
||||
|
|
@ -44,7 +49,9 @@ pnpm run setup:dev
|
|||
pnpm run link:dev
|
||||
```
|
||||
|
||||
This makes the `ktx` command available globally, pointing at your local build.
|
||||
This makes the `ktx-dev` command available globally, pointing at your local
|
||||
build. Use this development binary when you need to test unpublished repository
|
||||
changes.
|
||||
|
||||
## Repository structure
|
||||
|
||||
|
|
|
|||
|
|
@ -29,43 +29,51 @@ This reconciliation step is what separates auto-ingestion from a simple sync. A
|
|||
|
||||
Auto-ingestion is designed to plug into a PR-based workflow. Run ingestion on a branch, review the changed YAML and Markdown files, and merge them the same way you merge dbt models or application code.
|
||||
|
||||
```
|
||||
dbt / Looker / Metabase KTX project repo
|
||||
┌──────────────┐ ┌──────────────────────┐
|
||||
│ Metadata │───ingestion──▶│ Branch: ingest/... │
|
||||
│ changes │ │ │
|
||||
└──────────────┘ │ + 3 new sources │
|
||||
│ ~ 2 updated joins │
|
||||
│ + 1 knowledge page │
|
||||
│ │
|
||||
│ ──── Open PR ──── │
|
||||
│ │
|
||||
│ Review semantic diff │
|
||||
│ Approve & merge │
|
||||
└──────────────────────┘
|
||||
│
|
||||
▼
|
||||
Agents see updated
|
||||
context immediately
|
||||
```text
|
||||
dbt / Looker / Metabase / Notion
|
||||
|
|
||||
v
|
||||
metadata changes
|
||||
|
|
||||
v
|
||||
nightly cron or CI ingest
|
||||
|
|
||||
v
|
||||
branch: ingest/nightly
|
||||
|
|
||||
| + 3 new sources
|
||||
| ~ 2 updated joins
|
||||
| + 1 knowledge page
|
||||
v
|
||||
open PR
|
||||
|
|
||||
v
|
||||
review semantic diff
|
||||
|
|
||||
v
|
||||
approve & merge
|
||||
|
|
||||
v
|
||||
agents see updated context
|
||||
```
|
||||
|
||||
A typical branch shows a semantic diff: "this ingest added 3 new sources from dbt, updated 2 join definitions based on schema changes, and created 1 knowledge page from a Notion doc." Analytics engineers review the diff, verify that the new sources look correct, and merge.
|
||||
|
||||
Teams usually run this on demand while setting up a source, then schedule it once the source is stable. A cron job or CI schedule can run `ktx ingest --all --no-input` overnight on an ingest branch so the latest dbt manifests, BI metadata, and documentation updates are ready for review each morning.
|
||||
|
||||
Once merged, agents querying through KTX's MCP server or CLI see the updated context immediately. No deployment step, no cache invalidation, no restart. The files are the source of truth, and agents read them on every request.
|
||||
|
||||
This workflow gives you the same review guarantees you have for dbt models. No semantic source reaches production without a human approving it. But unlike maintaining context manually, the heavy lifting — discovering new tables, drafting source definitions, extracting business rules from documentation — is done by the ingestion agent. You review and approve. You don't write from scratch.
|
||||
|
||||
## Feedback loops
|
||||
|
||||
Context improves over time through three feedback channels.
|
||||
Context improves over time through two feedback channels.
|
||||
|
||||
**Analyst corrections.** When an analytics engineer spots something wrong — a measure formula that doesn't match the business definition, a join that should be `many_to_one` instead of `one_to_many`, a knowledge page that's out of date — they edit the YAML or Markdown directly and commit. These corrections become part of the project's git history, and the next ingestion run respects them. If you manually fix a measure definition, KTX won't overwrite it on the next ingest.
|
||||
|
||||
**Agent feedback.** When an agent queries the semantic layer and gets unexpected results — a query that returns no rows because of a bad filter, a join path that produces duplicated results — it can flag the issue. These signals feed back into the context: knowledge pages can note known data quality issues, source definitions can be tightened with better filters or grain declarations, and relationship thresholds can be adjusted.
|
||||
**Agent feedback.** When an agent queries the semantic layer and gets unexpected results — a query that returns no rows because of a bad filter, a join path that produces duplicated results — it can flag the issue. These signals feed back into the context: knowledge pages can note known data quality issues, and source definitions can be tightened with better filters, join paths, or grain declarations.
|
||||
|
||||
**Relationship calibration.** KTX infers foreign key relationships between tables automatically, even when the database has no declared constraints. It does this by analyzing column names, types, value distributions, and asking the LLM for proposals. Each inferred relationship gets a confidence score. You control two thresholds: `acceptThreshold` (relationships above this score are accepted automatically, default 0.85) and `reviewThreshold` (relationships between review and accept are flagged for human review, default 0.55). As you accept or reject proposals, the system learns which patterns match your schema conventions.
|
||||
|
||||
Each of these channels makes the next ingestion cycle better. Analyst corrections teach the system what your team considers authoritative. Agent feedback surfaces gaps in coverage. Relationship calibration tunes the discovery process to your warehouse's conventions. Context is not a static artifact — it's a living system that converges toward accuracy with every iteration.
|
||||
Each of these channels makes the next ingestion cycle better. Analyst corrections teach the system what your team considers authoritative. Agent feedback surfaces gaps in coverage. Context is not a static artifact — it's a living system that converges toward accuracy with every iteration.
|
||||
|
||||
## Deterministic replay
|
||||
|
||||
|
|
@ -89,5 +97,5 @@ Use this page when an agent needs to explain review workflows, ingestion diffs,
|
|||
|------------|------------------|-----------|
|
||||
| Explain how generated context should be reviewed | The git workflow | [Building Context](/docs/guides/building-context) |
|
||||
| Diagnose why ingestion changed a semantic source | Auto-ingestion and Deterministic replay | [ktx ingest](/docs/cli-reference/ktx-ingest) |
|
||||
| Explain how context improves over time | Feedback loops | [Link Detection](/docs/benchmarks/link-detection) |
|
||||
| Explain how context improves over time | Feedback loops | [Building Context](/docs/guides/building-context) |
|
||||
| Tell a user what to commit | The git workflow | [Writing Context](/docs/guides/writing-context) |
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ Give an agent access to your database and it will generate SQL. It might even pr
|
|||
|
||||
The agent doesn't know that `orders.amount` includes refunds and needs a status filter. It doesn't know that `customers` should join to `orders` on `customer_id`, not `id`. It doesn't know that your team stopped using `legacy_segments` six months ago, or that "enterprise" means contracts over $100k, not just big logos. It sees column names and types. It doesn't see your business.
|
||||
|
||||
This isn't a model capability problem. GPT-4, Claude, and Gemini can all write correct SQL — when they know what correct means. The gap is context: which tables matter, which joins are valid, which metrics are canonical, what the business terms actually refer to. Without that, agents produce plausible-looking artifacts that are subtly, dangerously wrong. Wrong enough to pass a glance, wrong enough to drive a decision.
|
||||
This isn't a model capability problem. Claude Code, Codex, and your BI agents can write correct SQL when they know what correct means. The gap is context: which tables matter, which joins are valid, which metrics are canonical, what the business terms actually refer to. Without that, agents produce plausible-looking artifacts that are subtly, dangerously wrong. Wrong enough to pass a glance, wrong enough to drive a decision.
|
||||
|
||||
Analytics engineers already know this pain. It's the same reason you write dbt tests, maintain a data dictionary, and spend half of standup explaining why someone's dashboard number doesn't match the board deck. The difference is that agents make decisions at machine speed, so the wrong context propagates faster than a human can catch it.
|
||||
|
||||
|
|
@ -19,9 +19,9 @@ The industry has moved through three distinct approaches to getting AI and data
|
|||
|
||||
**Wave one: database access.** Connect an LLM to a database, let it generate SQL. This works for simple lookups — "how many orders last week?" — but breaks on anything that requires business knowledge. The agent guesses at joins, invents metrics, and hallucinates table relationships. Every query is a coin flip.
|
||||
|
||||
**Wave two: semantic layers and text-to-SQL.** Add structure. Define metrics in MetricFlow or Cube, expose schemas, build text-to-SQL pipelines. This is better — the agent knows that `revenue` means `sum(amount) where status != 'refunded'` — but it's still limited. Semantic layers define what to calculate, not why, when, or how to interpret the result. The agent can compute net revenue but doesn't know about the February refund anomaly, the segment reclassification, or the fact that `enterprise` changed definition last quarter.
|
||||
**Wave two: semantic layers and text-to-SQL.** Add structure. Define metrics in MetricFlow or Cube, expose schemas, build text-to-SQL pipelines. This is better — the agent knows that `revenue` means `sum(amount) where status != 'refunded'` — but building and maintaining that structure by hand is manual, time-consuming, and still limited. Semantic layers define what to calculate, not why, when, or how to interpret the result. The agent can compute net revenue but doesn't know about the February refund anomaly, the segment reclassification, or the fact that `enterprise` changed definition last quarter.
|
||||
|
||||
**Wave three: agentic context.** AI is no longer just answering questions — it's generating dashboards, writing semantic definitions, proposing dbt models, creating tests and documentation. For that to work, agents need more than metric definitions. They need the full picture: business rules, data quality gotchas, relationship maps, historical context, and the institutional knowledge that lives in your team's heads. They need a context layer.
|
||||
**Wave three: agentic context.** AI is no longer just answering questions — it's generating dashboards, writing semantic definitions, proposing dbt models, creating tests and documentation. For that to work, agents need more than metric definitions. They need the full picture: business rules, known data quality issues, relationship maps, historical context, and the institutional knowledge that lives in your team's heads. They need a context layer.
|
||||
|
||||
## What a context layer is
|
||||
|
||||
|
|
@ -29,6 +29,13 @@ A context layer is the infrastructure that gives agents the business knowledge t
|
|||
|
||||
KTX organizes context into four pillars:
|
||||
|
||||
- Semantic sources
|
||||
- Knowledge pages
|
||||
- Scan artifacts
|
||||
- Provenance
|
||||
|
||||
Each pillar covers a different kind of context agents need before they can safely write SQL, update semantic definitions, or explain an analytics result.
|
||||
|
||||
**Semantic sources** are YAML definitions that describe your data in terms agents can reason about. Each source maps to a table or SQL query, declares its grain, defines typed columns, specifies valid joins, and exposes named measures with optional filters. This is where "revenue means `sum(amount)` excluding refunds" lives.
|
||||
|
||||
```yaml
|
||||
|
|
@ -60,7 +67,7 @@ measures:
|
|||
expr: count(id)
|
||||
```
|
||||
|
||||
**Knowledge pages** are Markdown documents that capture business definitions, rules, and gotchas — the kind of context that doesn't fit in a schema definition. Pages have structured frontmatter (summary, tags, semantic layer references) and free-form content. Agents search them when they need to understand why a metric works a certain way, not just how to compute it.
|
||||
**Knowledge pages** are Markdown documents that capture business definitions, rules, and operating context — the kind of context that doesn't fit in a schema definition. Pages have structured frontmatter (summary, tags, semantic layer references) and free-form content. Agents search them when they need to understand why a metric works a certain way, not just how to compute it.
|
||||
|
||||
```markdown
|
||||
---
|
||||
|
|
@ -90,13 +97,12 @@ Together, these four pillars give agents enough context to produce analytics art
|
|||
|
||||
## How KTX compares
|
||||
|
||||
KTX is a context layer, and its structured core is an agent-native semantic layer. That matters. MetricFlow, Cube, and Malloy all give teams ways to model metrics, dimensions, joins, and generated SQL. KTX covers that same semantic-layer job, then adds the surrounding context agents need to use it well: knowledge pages, schema scans, provenance, ingestion, validation, and MCP tools.
|
||||
KTX is a context layer with an agent-native semantic layer at its core. MetricFlow, Cube, and Malloy model metrics, dimensions, joins, and generated SQL. KTX covers that semantic-layer work, then adds the context agents need to use and maintain it: knowledge pages, schema scans, provenance, ingestion, validation, and MCP tools.
|
||||
|
||||
The primary user is different. MetricFlow is centered on dbt-style metric definitions. Cube is centered on a governed semantic runtime for BI, applications, and agents. Malloy is centered on an expressive modeling and query language. KTX is centered on agents that need to read a semantic model, change it, validate it, inspect the generated SQL, and leave a reviewable git diff.
|
||||
The workflow is the difference. Traditional semantic layers are powerful, but they are usually built and maintained through manual modeling work, product-specific runtimes, or language-specific workflows. They are not agent-native by default, which makes them harder for agents to inspect, edit, validate, and review in a tight loop. KTX is designed for agents that need to read context, change semantic files, inspect generated SQL, and leave a reviewable git diff.
|
||||
|
||||
| | KTX semantic layer | MetricFlow | Cube | Malloy |
|
||||
|---|---|---|---|---|
|
||||
| **Design center** | Agent-native semantic modeling inside a broader context layer | Metric definitions and dbt semantic models | Governed serving layer for BI, embedded analytics, APIs, and agents | Semantic modeling and analytical query language |
|
||||
| **Model surface** | Plain YAML sources plus Markdown knowledge pages | YAML semantic models and metrics in a dbt project | YAML or JavaScript cubes, views, access policies, and pre-aggregations | `.malloy` models, query pipelines, notebooks, and annotations |
|
||||
| **What it models** | Sources, columns, measures, segments, joins, grain, filters, default time dimensions, and context references | Semantic models, entities, dimensions, measures, metrics, time grains, and metric types | Cubes, views, measures, dimensions, segments, joins, hierarchies, policies, and rollups | Sources, joins, dimensions, measures, calculations, nested results, and query pipelines |
|
||||
| **Agent edit loop** | First-class. Agents can patch small files, save imperfect drafts, run validation, query through MCP, inspect SQL, and refine in the same workflow | Possible, but the interface is a dbt/metric workflow rather than an agent context workflow | Possible through code-first models and platform APIs, but changes are tied to runtime deployment and governance concerns | Possible, but agents must operate in Malloy's language and compiler model |
|
||||
|
|
@ -105,15 +111,7 @@ The primary user is different. MetricFlow is centered on dbt-style metric defini
|
|||
| **Context around semantics** | Built in: wiki pages, scan artifacts, relationship inference, ingest transcripts, replay, and agent-facing MCP tools | Primarily metric and dbt project context | Descriptions and `meta.ai_context` inside the semantic model, plus platform agent features | Annotations/tags can carry metadata; surrounding context depends on the application |
|
||||
| **Best fit** | Agents maintaining analytics code, metrics, joins, SQL, docs, and semantic definitions | Teams standardizing metrics inside dbt workflows | Production semantic APIs, BI integrations, access control, caching, and concurrency | Expressive modeling and exploratory analysis above SQL |
|
||||
|
||||
**Agent-native by design.** KTX's advantage is not just that the files are YAML. The whole loop is shaped for agents: sources are small, overlays can add measures or computed columns without copying entire generated schemas, writes are permissive so an agent can save a draft, and validation/query tools give immediate feedback. An agent can move from "this metric is wrong" to "here is the semantic diff, generated SQL, and supporting context" without leaving the project.
|
||||
|
||||
**A semantic layer plus the context to use it.** Traditional semantic layers define what to calculate. KTX also stores why the definition exists, where it came from, what schema evidence supports it, and what an agent did when it changed. A measure can live next to a knowledge page about exclusions, a scan artifact that proves the join path, and an ingest transcript that explains the source of the definition. That is the difference between giving an agent a metric catalog and giving it operational memory.
|
||||
|
||||
**Fan-out handling is explicit and reviewable.** KTX asks model authors and agents to declare grain and relationship direction. The planner uses that metadata to avoid silent row multiplication: it detects `one_to_many` fan-out paths, separates independent fact measures into aggregate-locality CTEs, and refuses filters that would be unsafe to apply after pre-aggregation. Cube, MetricFlow, and Malloy all have strong approaches to this class of problem, but KTX's approach is deliberately inspectable in the files and in the generated plan.
|
||||
|
||||
**Where other systems are stronger.** KTX draws a clear product boundary around agent-native context and semantic modeling. Cube is stronger when you need a production semantic API with access policies, pre-aggregations, refresh workers, and high-concurrency serving. MetricFlow is stronger when your primary workflow is dbt-native metric standardization. Malloy is stronger when you want a full analytical language with nested query shapes. KTX is strongest when the semantic layer is the substrate agents will read, edit, validate, and extend as part of day-to-day analytics engineering.
|
||||
|
||||
**When KTX replaces your semantic layer vs. works beside it.** If you do not have a semantic layer, KTX can build an agent-native one from your database schema and enrich it with generated descriptions and knowledge pages. If you already use MetricFlow, LookML, Looker, Metabase, dbt, or Notion, KTX ingests from those tools and merges their context into KTX's files. You can keep your existing BI or metric-serving system while using KTX as the semantic and contextual surface agents work against.
|
||||
If you do not have a semantic layer, KTX can build an agent-native one from your database schema and enrich it with generated descriptions and knowledge pages. If you already use MetricFlow or LookML, KTX ingests from those tools and merges their context into KTX's files. You can keep your existing BI or metric-serving system while using KTX as the semantic and contextual surface agents work against.
|
||||
|
||||
## The plain-files philosophy
|
||||
|
||||
|
|
|
|||
|
|
@ -1,71 +1,86 @@
|
|||
---
|
||||
title: Introduction
|
||||
description: What KTX is and who it's for.
|
||||
description: How KTX gives analytics agents trusted context for warehouse work.
|
||||
---
|
||||
|
||||
Data agents can write SQL. The hard part is making sure they write the SQL your analytics team would have written.
|
||||
|
||||
KTX is the agent-native context layer for analytics engineering. At its core is a semantic layer: YAML sources that define tables, columns, measures, joins, grain, filters, segments, and computed fields. Around that core, KTX adds the context analytics agents need to work safely: warehouse scans, knowledge pages, ingestion from existing tools, provenance, validation, and MCP access.
|
||||
|
||||
KTX projects are plain files — YAML, Markdown, and SQLite — that you commit to git and review in PRs, just like dbt models. Agents can read them, edit them, validate them, query through them, and leave behind a diff your team can review.
|
||||
<div className="not-prose mb-14">
|
||||
<div className="mb-8">
|
||||
<h1
|
||||
className="text-4xl font-extrabold tracking-tight lg:text-5xl"
|
||||
style={{
|
||||
fontFamily: 'var(--font-display)',
|
||||
background: 'linear-gradient(180deg, var(--color-fd-foreground) 0%, color-mix(in oklch, var(--color-fd-foreground) 75%, var(--color-fd-primary)) 100%)',
|
||||
WebkitBackgroundClip: 'text',
|
||||
backgroundClip: 'text',
|
||||
color: 'transparent',
|
||||
WebkitTextFillColor: 'transparent',
|
||||
lineHeight: '1.1',
|
||||
letterSpacing: '0',
|
||||
}}
|
||||
>
|
||||
Make analytics context{'\n'}usable by agents
|
||||
</h1>
|
||||
<p className="mt-4 text-lg text-fd-muted-foreground max-w-2xl" style={{ lineHeight: '1.7' }}>
|
||||
KTX turns warehouse metadata, semantic definitions, and business knowledge
|
||||
into reviewable project files that agents can use while planning, querying,
|
||||
and updating analytics work.
|
||||
</p>
|
||||
</div>
|
||||
<div className="flex flex-wrap gap-3">
|
||||
<a
|
||||
href="/docs/getting-started/quickstart"
|
||||
className="inline-flex h-10 items-center rounded-lg bg-fd-primary px-5 text-sm font-medium text-fd-primary-foreground transition-colors hover:opacity-90"
|
||||
>
|
||||
Get Started
|
||||
</a>
|
||||
<a
|
||||
href="/docs/concepts/the-context-layer"
|
||||
className="inline-flex h-10 items-center rounded-lg border border-fd-border bg-fd-background px-5 text-sm font-medium text-fd-foreground transition-colors hover:bg-fd-muted"
|
||||
>
|
||||
The Context Layer
|
||||
</a>
|
||||
<a
|
||||
href="/docs/guides/building-context"
|
||||
className="inline-flex h-10 items-center rounded-lg border border-fd-border bg-fd-background px-5 text-sm font-medium text-fd-foreground transition-colors hover:bg-fd-muted"
|
||||
>
|
||||
Building Context
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
## Who KTX is for
|
||||
|
||||
KTX is built for analytics engineers and data teams who want data agents to work on real analytics systems, not just generate one-off SQL.
|
||||
KTX is built for analytics engineers and data teams who want data agents to
|
||||
work on real analytics systems — not just generate one-off SQL.
|
||||
|
||||
Use KTX when you want agents to:
|
||||
|
||||
- Generate SQL from approved measures, dimensions, and joins
|
||||
- Repair or extend semantic definitions through reviewable git diffs
|
||||
- Explain where a metric definition came from and what business rules shape it
|
||||
- Use warehouse scans and relationship evidence instead of guessing join paths
|
||||
- Work alongside **dbt**, **LookML**, **MetricFlow**, **Looker**, **Metabase**, **Notion**, and BI platforms
|
||||
- Work with warehouses like **PostgreSQL**, **Snowflake**, **BigQuery**, **ClickHouse**, **MySQL**, or **SQL Server**
|
||||
- **Generate SQL** from approved measures and joins
|
||||
- **Repair semantic definitions** through reviewable diffs
|
||||
- **Explain metric provenance** with warehouse evidence
|
||||
- **Work alongside** dbt, LookML, MetricFlow, Looker, Metabase, and modern BI platforms
|
||||
|
||||
If you've ever watched an agent confidently generate a query that joins on the wrong key or invents a metric that doesn't exist, KTX is the fix.
|
||||
Works with PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, and SQL Server.
|
||||
|
||||
## What KTX gives agents
|
||||
|
||||
- **A semantic layer they can edit** — plain YAML sources with measures, dimensions, joins, grain, segments, filters, and computed columns
|
||||
- **Safe query planning** — grain-aware SQL generation, fan-out detection, chasm-trap handling, and dialect transpilation
|
||||
- **Business context** — Markdown knowledge pages for definitions, rules, exceptions, and data quality notes
|
||||
- **Schema evidence** — warehouse scans with table metadata, column stats, constraints, and inferred relationships
|
||||
- **Provenance** — ingest transcripts and replay metadata that explain where context came from and why it changed
|
||||
- **An agent-facing API** — MCP and CLI tools for reading, writing, validating, searching, and querying context
|
||||
|
||||
## How these docs are organized
|
||||
## Explore the docs
|
||||
|
||||
<Cards>
|
||||
<Card title="Quickstart" href="/docs/getting-started/quickstart">
|
||||
Set up KTX and build your first context in under 10 minutes.
|
||||
</Card>
|
||||
<Card title="AI Resources" href="/docs/ai-resources">
|
||||
Machine-readable docs and prompt recipes for coding assistants.
|
||||
</Card>
|
||||
<Card title="Concepts" href="/docs/concepts/the-context-layer">
|
||||
Understand what a context layer is, why agents need one, and how KTX compares to other semantic layers.
|
||||
Understand what a context layer is and why agents need one.
|
||||
</Card>
|
||||
<Card title="Guides" href="/docs/guides/building-context">
|
||||
Hands-on workflows for scanning, ingesting, writing semantic sources, and serving agents.
|
||||
</Card>
|
||||
<Card title="Integrations" href="/docs/integrations/primary-sources">
|
||||
Setup details for every supported database, context source, and agent client.
|
||||
Hands-on workflows for scanning, ingesting, writing, and serving.
|
||||
</Card>
|
||||
<Card title="CLI Reference" href="/docs/cli-reference/ktx-setup">
|
||||
Exhaustive flag and subcommand reference for every KTX command.
|
||||
Complete flag and subcommand reference for every KTX command.
|
||||
</Card>
|
||||
</Cards>
|
||||
|
||||
## Next steps
|
||||
|
||||
- **Get hands-on** — follow the [Quickstart](/docs/getting-started/quickstart) to set up KTX with your own database in under 10 minutes.
|
||||
- **Help a coding agent use the docs** — start with [AI Resources](/docs/ai-resources) or fetch [`/llms.txt`](/llms.txt).
|
||||
- **Understand the theory** — read [The Context Layer](/docs/concepts/the-context-layer) to learn why schema access alone breaks on real analytics and how KTX addresses it.
|
||||
|
||||
## Agent usage notes
|
||||
|
||||
Use this page as the high-level routing document for KTX docs.
|
||||
|
||||
| Agent task | Read next |
|
||||
|------------|-----------|
|
||||
| Discover machine-readable docs | [AI Resources](/docs/ai-resources) |
|
||||
|
|
|
|||
|
|
@ -9,44 +9,30 @@ If you are a coding assistant trying to decide which KTX docs page to read, star
|
|||
|
||||
## Workflow summary
|
||||
|
||||
Use this sequence when an agent needs to set up KTX from a fresh checkout:
|
||||
Use this sequence when you are setting up KTX in an analytics project:
|
||||
|
||||
1. `pnpm install` — install workspace dependencies.
|
||||
2. `pnpm run setup:dev` — build local packages and prepare the development CLI.
|
||||
3. `pnpm run link:dev` — link the `ktx` command for local use.
|
||||
4. `ktx setup` — create or resume a KTX project.
|
||||
5. `ktx status` — verify project readiness.
|
||||
6. `ktx sl list` — confirm semantic-layer sources are available.
|
||||
7. `ktx sl query ... --format sql` — compile a semantic query without executing it.
|
||||
1. `npm install -g @kaelio/ktx` — install the published KTX CLI from npm.
|
||||
2. `ktx setup` — create or resume a KTX project.
|
||||
|
||||
The setup wizard is stateful. If it exits before completion, rerun `ktx setup` in the same project directory to resume from the first incomplete step.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Node.js 22+** and **pnpm**
|
||||
- An **Anthropic API key** for LLM-powered enrichment and ingestion
|
||||
- A **database connection** — PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, or SQLite
|
||||
- Optionally, a **dbt project**, **LookML repo**, **Metabase instance**, or other context source
|
||||
|
||||
## Install and run setup
|
||||
|
||||
KTX is currently used from a local checkout or linked workspace CLI. Build and link the CLI first:
|
||||
Install the published [`@kaelio/ktx`](https://www.npmjs.com/package/@kaelio/ktx) CLI:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/kaelio/ktx.git
|
||||
cd ktx
|
||||
pnpm install
|
||||
pnpm run setup:dev
|
||||
pnpm run link:dev
|
||||
npm install -g @kaelio/ktx
|
||||
```
|
||||
|
||||
Then run the setup wizard in the directory where you want your KTX project:
|
||||
Then run the setup wizard:
|
||||
|
||||
```bash
|
||||
ktx setup
|
||||
```
|
||||
|
||||
The wizard walks through six steps. You can go back at any point, and if you exit early, running `ktx setup` again resumes where you left off.
|
||||
The local checkout flow is only for contributors working on KTX itself. See [Contributing](/docs/community/contributing) for that setup.
|
||||
|
||||
The wizard walks through six steps. You can go back at any point, and if you exit early, rerunning `ktx setup` resumes where you left off.
|
||||
|
||||
## Step 1: Configure LLM
|
||||
|
||||
|
|
@ -86,10 +72,11 @@ KTX uses embeddings for semantic search over sources, wiki content, schema metad
|
|||
|
||||
**OpenAI embeddings** use `text-embedding-3-small` (1536 dimensions) and require an `OPENAI_API_KEY`.
|
||||
|
||||
**Local embeddings** use `all-MiniLM-L6-v2` (384 dimensions) via the KTX Python daemon. No API key is needed. If you run the daemon as a long-lived HTTP service, start it with:
|
||||
**Local embeddings** use `all-MiniLM-L6-v2` (384 dimensions) via the KTX managed Python runtime. No API key is needed. KTX can install and start the runtime during setup; to prepare it ahead of time, run:
|
||||
|
||||
```bash
|
||||
ktx-daemon serve-http --host 127.0.0.1 --port 8765
|
||||
ktx runtime install --feature local-embeddings --yes
|
||||
ktx runtime start --feature local-embeddings
|
||||
```
|
||||
|
||||
## Step 3: Connect a database
|
||||
|
|
@ -208,12 +195,15 @@ Then select which agents to install for:
|
|||
│ ◻ Codex
|
||||
│ ◻ Cursor
|
||||
│ ◻ OpenCode
|
||||
│ ◻ Custom agent (.agents)
|
||||
```
|
||||
|
||||
**CLI mode** writes a skill file (e.g., `.claude/skills/ktx/SKILL.md`) that teaches the agent to call KTX commands directly.
|
||||
|
||||
**MCP mode** writes an MCP server configuration (e.g., `.mcp.json`) that lets the agent call KTX tools like `sl_query`, `knowledge_search`, and `sl_write_source` over the Model Context Protocol.
|
||||
|
||||
**Custom agent** uses the universal `.agents` target for agents that can read project-local skills or MCP configuration.
|
||||
|
||||
## Generated files
|
||||
|
||||
KTX writes project state as plain files so agents can inspect and edit changes in git.
|
||||
|
|
@ -247,44 +237,14 @@ KTX context built: yes
|
|||
Agent integration ready: yes (claude-code:project)
|
||||
```
|
||||
|
||||
List your semantic sources:
|
||||
|
||||
```bash
|
||||
ktx sl list
|
||||
```
|
||||
|
||||
Query through the semantic layer:
|
||||
|
||||
```bash
|
||||
ktx sl query \
|
||||
--connection-id postgres-warehouse \
|
||||
--measure orders.total_revenue \
|
||||
--dimension orders.status \
|
||||
--order-by orders.total_revenue:desc \
|
||||
--limit 5 \
|
||||
--format sql
|
||||
```
|
||||
|
||||
This outputs the generated SQL. Add `--execute` to run it against your warehouse:
|
||||
|
||||
```bash
|
||||
ktx sl query \
|
||||
--connection-id postgres-warehouse \
|
||||
--measure orders.total_revenue \
|
||||
--dimension orders.status \
|
||||
--order-by orders.total_revenue:desc \
|
||||
--limit 5 \
|
||||
--execute --max-rows 10
|
||||
```
|
||||
|
||||
## Common errors
|
||||
|
||||
| Error or symptom | Likely cause | Recovery |
|
||||
|------------------|--------------|----------|
|
||||
| `ktx: command not found` | The local CLI has not been linked | Run `pnpm run setup:dev` and `pnpm run link:dev` from the KTX checkout, then open a new shell |
|
||||
| `ktx: command not found` | The KTX package is not installed globally, or the shell cannot find the global binary | Run `npm install -g @kaelio/ktx` and open a new shell |
|
||||
| LLM health check fails | Missing, invalid, or unauthorized Anthropic API key | Export `ANTHROPIC_API_KEY` or rerun `ktx setup` and choose the file-backed secret option |
|
||||
| OpenAI embedding check fails | `OPENAI_API_KEY` is missing when OpenAI embeddings are selected | Export `OPENAI_API_KEY`, or rerun setup and choose local sentence-transformers embeddings |
|
||||
| Local embeddings hang or fail | The Python daemon cannot start or the local model runtime is unavailable | Run `uv sync --all-groups`, then start `ktx-daemon serve-http --host 127.0.0.1 --port 8765` and rerun setup |
|
||||
| Local embeddings hang or fail | The managed Python runtime cannot start or the local model runtime is unavailable | Install `uv`, run `ktx runtime doctor`, then run `ktx runtime install --feature local-embeddings --yes` and rerun setup |
|
||||
| Database connection test fails | Credentials, network access, warehouse, database, or schema value is wrong | Test the same URL with the database's native client, then rerun `ktx connection add ... --force` or rerun setup |
|
||||
| `KTX context built: no` in `ktx status` | Setup saved configuration but did not build context | Run `ktx setup context build` or rerun `ktx setup` and choose to build context now |
|
||||
| Agent integration is incomplete | Setup skipped the agents step or the target was not installed | Run `ktx setup --agents --target codex --agent-install-mode both --project` using the target you need |
|
||||
|
|
|
|||
|
|
@ -6,7 +6,6 @@
|
|||
"concepts",
|
||||
"guides",
|
||||
"integrations",
|
||||
"benchmarks",
|
||||
"cli-reference",
|
||||
"ai-resources",
|
||||
"community"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue