diff --git a/README.md b/README.md index 92a2a8dd..45c20bd9 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ KTX -

- The context layer for analytics agents -

+

+ The context layer for analytics agents +

npm version @@ -16,183 +16,82 @@ --- KTX turns warehouse metadata, semantic definitions, and business knowledge into -reviewable project files that agents can use while planning, querying, and -updating analytics work. - -A KTX project is a directory of plain files - YAML semantic sources, Markdown -wiki pages, and SQLite state - that you commit to git and review in PRs, -just like dbt models. - -## Who KTX is for - -KTX is built for analytics engineers and data teams who want data agents to -work on real analytics systems - not just generate one-off SQL. +reviewable project files that agents can use to plan, query, and update +analytics work. Use KTX when you want agents to: -- **Generate SQL** from approved measures and joins -- **Repair semantic definitions** through reviewable diffs -- **Explain metric provenance** with warehouse evidence -- **Work alongside** dbt, LookML, MetricFlow, Looker, Metabase, and modern BI - platforms +- Generate SQL from approved measures and joins +- Repair semantic definitions through reviewable diffs +- Explain metric provenance with warehouse evidence +- Work alongside dbt, MetricFlow, LookML, Looker, Metabase, and Notion -Works with PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, and +Supports PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, and SQLite. -## Quick start - -Install the CLI and run the setup wizard: +## Quick Start ```bash -npm install @kaelio/ktx -npm install -g @kaelio/ktx +pnpm add --global @kaelio/ktx ktx setup -``` - -The wizard walks through six steps: configuring your LLM provider, setting up -embeddings, connecting your database, adding context sources (dbt, LookML, -Metabase, Looker, Notion), building context, and installing agent integration. - -If it exits before completion, rerun `ktx setup` to resume where you left off. - -Check your project status: - -```bash ktx status ``` -``` -KTX project: /home/user/analytics -Project ready: yes -LLM ready: yes (claude-sonnet-4-6) -Embeddings ready: yes (text-embedding-3-small) -Databases configured: yes (postgres-warehouse) -Context sources configured: yes (dbt-main) -KTX context built: yes -Agent integration ready: yes (claude-code:project) -``` +`ktx setup` creates or resumes a local KTX project, configures providers and +connections, builds context, and installs agent integration. -Generate SQL from a semantic-layer source: +## Common Commands -```bash -npx @kaelio/ktx sl query --project-dir "$PROJECT_DIR" \ - --connection-id warehouse \ - --measure accounts.account_count \ - --dimension accounts.segment \ - --format sql -``` +| Command | Purpose | +|---------|---------| +| `ktx setup` | Create, resume, or update a KTX project | +| `ktx status` | Check project readiness | +| `ktx connection list` | List configured connections | +| `ktx connection test ` | Test one connection | +| `ktx ingest ` | Build context for one connection | +| `ktx ingest --all` | Build context for every configured connection | +| `ktx ingest text ` | Capture free-form notes into memory | +| `ktx sl list` | List semantic-layer sources | +| `ktx sl search "revenue"` | Search semantic-layer sources | +| `ktx sl validate --connection-id ` | Validate a semantic source | +| `ktx sl query --measure --format sql` | Compile semantic-layer SQL | +| `ktx wiki search "revenue definition"` | Search local wiki context | +| `ktx mcp start` | Start the local MCP server for agent clients | -List and test a configured warehouse connection: +Project resolution defaults to `KTX_PROJECT_DIR`, then the nearest `ktx.yaml`, +then the current directory. Pass `--project-dir ` when scripting. -```bash -ktx connection list --project-dir "$PROJECT_DIR" -ktx connection test warehouse --project-dir "$PROJECT_DIR" -``` - -The connection test prints the configured driver and connector-specific status: +## Project Layout ```text -Connection test passed: warehouse -Driver: sqlite -Status: ok -``` - -## What's in a project - -``` my-project/ -├── ktx.yaml # Project configuration -├── semantic-layer/ -│ └── warehouse/ -│ ├── orders.yaml # Semantic source definitions -│ ├── customers.yaml -│ └── order_items.yaml -├── wiki/ -│ ├── global/ -│ │ ├── revenue.md # Business definitions and rules -│ │ └── segment-classification.md -│ └── user/ -│ └── local/ -├── raw-sources/ -│ └── warehouse/ -│ └── / # Database ingest artifacts and reports -└── .ktx/ - └── db.sqlite # Local state (git-ignored) +├── ktx.yaml # Project configuration +├── semantic-layer// # YAML semantic sources +├── wiki/global/ # Shared business context +├── wiki/user// # User-scoped notes +├── raw-sources// # Ingest artifacts and reports +└── .ktx/ # Local state and secrets, git-ignored ``` -Semantic sources and wiki pages are committed to git. The `.ktx/` directory -holds ephemeral state and is git-ignored - delete it and KTX rebuilds on the -next run. +Commit `ktx.yaml`, `semantic-layer/`, and `wiki/`. Keep `.ktx/` local. -### Build demo warehouse context +## Agent Usage -Database ingest artifacts are written under `raw-sources/warehouse//` -in the project directory. +Setup can install KTX instructions for Claude Code, Codex, Cursor, OpenCode, +and universal `.agents` clients: ```bash -ktx ingest warehouse --project-dir "$PROJECT_DIR" --fast -ktx status --project-dir "$PROJECT_DIR" +ktx setup --agents --target codex ``` -For non-SQLite drivers, prefer credential references such as `--url env:NAME` -or `--url file:PATH` over literal credential URLs. - -## Managed Python runtime - -KTX installs its Python runtime only when a Python-backed command needs it. -The runtime lives outside the npm cache, is versioned by the installed CLI -version, and is managed by `ktx dev runtime` commands. - -KTX requires `uv` on `PATH` to create the managed runtime. Install `uv` with -your system package manager or the official installer before running Python- -backed KTX commands. KTX doesn't download `uv` automatically; run -`ktx dev runtime status` if runtime installation fails: +Agent-facing workflows typically start with: ```bash -ktx dev runtime install --yes -ktx dev runtime status -ktx dev runtime start -ktx dev runtime stop +ktx sl search "revenue" --json +ktx wiki search "refund policy" --json +ktx sl query --connection-id warehouse --measure orders.revenue --format sql ``` -The release artifact manifest contains the public npm tarball and the bundled `kaelio-ktx` -runtime wheel. The `python/ktx-sl` and `python/ktx-daemon` directories remain -source packages for development, not public release artifacts. - -## Use KTX with agents - -KTX integrates with coding agents through CLI skills. The setup wizard -configures this automatically. - -**CLI skills** - the agent calls `ktx` commands directly through a skill file -installed in your agent's config (e.g., `.claude/skills/ktx/SKILL.md`): - -```bash -ktx sl query --measure orders.revenue --dimension orders.status --format sql -ktx wiki search "revenue definition" -ktx sl validate orders -``` - -Supported agents: Claude Code, Codex, Cursor, OpenCode, and any agent that -reads `.agents/` skills. - -## Workspace packages - -| Package | Purpose | -|---------|---------| -| `packages/cli` | CLI entry point | -| `packages/context` | Core context engine | -| `packages/llm` | LLM and embedding providers | -| `packages/connector-bigquery` | BigQuery scan connector | -| `packages/connector-clickhouse` | ClickHouse scan connector | -| `packages/connector-mysql` | MySQL scan connector | -| `packages/connector-postgres` | Postgres scan connector | -| `packages/connector-snowflake` | Snowflake scan connector | -| `packages/connector-sqlite` | SQLite scan connector | -| `packages/connector-sqlserver` | SQL Server scan connector | -| `python/ktx-sl` | Semantic-layer query planning | -| `python/ktx-daemon` | Portable compute service | - ## Development ```bash @@ -204,7 +103,7 @@ pnpm run build pnpm run check ``` -Use the development CLI for local testing: +Use the development CLI locally: ```bash pnpm run setup:dev @@ -212,30 +111,28 @@ pnpm run link:dev ktx-dev --help ``` -### Debug LLM traces +KTX is a pnpm + uv workspace: -KTX can capture local AI SDK DevTools traces for LLM calls that run through the -KTX provider. Enable it with an environment flag when running an LLM-backed -command: +- TypeScript packages live in `packages/*` +- CLI source lives in `packages/cli` +- Python runtime source lives in `python/ktx-sl` and `python/ktx-daemon` +- Public docs live in `docs-site/content/docs` + +Useful checks: ```bash -KTX_AI_DEVTOOLS_ENABLED=true ktx ingest warehouse --project-dir "$PROJECT_DIR" --deep +pnpm run type-check +pnpm run test +pnpm run dead-code +uv run pytest -q ``` -Traces are written to `.devtools/generations.json` under the current working -directory. To inspect them, run: +## Docs -```bash -pnpm dlx @ai-sdk/devtools -``` - -Then open `http://localhost:4983`. These traces are local-development-only and -store prompts, model outputs, tool arguments/results, and raw provider payloads -in plain text. Do not enable this in production or for sensitive runs. - -The repository uses `pnpm` for TypeScript packages and `uv` for Python -packages. See [Contributing](docs-site/content/docs/community/contributing.mdx) -for full development setup, testing, and PR guidelines. +- [Quickstart](docs-site/content/docs/getting-started/quickstart.mdx) +- [CLI Reference](docs-site/content/docs/cli-reference/index.mdx) +- [Building Context](docs-site/content/docs/guides/building-context.mdx) +- [Contributing](docs-site/content/docs/community/contributing.mdx) ## License diff --git a/docs-site/content/docs/cli-reference/index.mdx b/docs-site/content/docs/cli-reference/index.mdx index c4ef07db..065339ca 100644 --- a/docs-site/content/docs/cli-reference/index.mdx +++ b/docs-site/content/docs/cli-reference/index.mdx @@ -4,8 +4,8 @@ description: "Command map and shared options for the KTX CLI." --- The `ktx` CLI sets up local projects, builds agent-ready context, checks -connections, queries semantic-layer sources, searches wiki pages, and manages -the bundled Python runtime. +connections, queries semantic-layer sources, searches wiki pages, runs the MCP +server, and manages the bundled Python runtime. ## Command Map @@ -26,6 +26,11 @@ ktx validate query status + mcp + start + stop + status + logs dev init [directory] schema @@ -73,4 +78,7 @@ ktx ingest --all # Search semantic-layer sources and wiki pages ktx sl search "revenue" ktx wiki search "revenue recognition" + +# Start the local MCP server for agent clients +ktx mcp start ``` diff --git a/docs-site/content/docs/cli-reference/ktx-mcp.mdx b/docs-site/content/docs/cli-reference/ktx-mcp.mdx new file mode 100644 index 00000000..9f0dd189 --- /dev/null +++ b/docs-site/content/docs/cli-reference/ktx-mcp.mdx @@ -0,0 +1,70 @@ +--- +title: "ktx mcp" +description: "Run the KTX MCP HTTP server for agent clients." +--- + +`ktx mcp` starts, stops, inspects, and tails the local KTX MCP server for a KTX +project. Use it when an agent client connects through MCP instead of generated +CLI instructions. + +## Command signature + +```bash +ktx mcp [options] +``` + +## Subcommands + +| Subcommand | Description | +|-----------|-------------| +| `start` | Start the KTX MCP HTTP server | +| `stop` | Stop the KTX MCP daemon | +| `status` | Show daemon status, URL, PID, token mode, and project path | +| `logs` | Print the daemon log | + +## `mcp start` Options + +| Flag | Description | Default | +|------|-------------|---------| +| `--host ` | Host to bind | `127.0.0.1` | +| `--port ` | Port to bind | `7878` | +| `--token ` | Bearer token for non-loopback binding | `KTX_MCP_TOKEN` | +| `--foreground` | Run the server in the foreground | `false` | +| `--allowed-host ` | Additional allowed Host header; repeatable | - | +| `--allowed-origin ` | Allowed browser Origin header; repeatable | - | + +## `mcp logs` Options + +| Flag | Description | Default | +|------|-------------|---------| +| `--follow` | Follow log output | `false` | + +## Examples + +```bash +# Start the daemon on localhost +ktx mcp start + +# Check status +ktx mcp status + +# Tail logs +ktx mcp logs --follow + +# Run in the foreground on a custom port +ktx mcp start --port 8787 --foreground +``` + +## Security notes + +The default host is loopback-only. If you bind to a non-loopback host, configure +a bearer token with `--token ` or `KTX_MCP_TOKEN` and restrict allowed +hosts and origins for browser clients. + +## Common errors + +| Error | Cause | Recovery | +|-------|-------|----------| +| No KTX project found | Current directory has no `ktx.yaml` and `KTX_PROJECT_DIR` is unset | Run from a KTX project or pass `--project-dir ` | +| Non-loopback host rejected | The server needs token auth before binding beyond localhost | Pass `--token ` or set `KTX_MCP_TOKEN` | +| Client cannot connect | Host, port, token, allowed host, or allowed origin does not match the client | Check `ktx mcp status`, then restart with explicit `--host`, `--port`, `--allowed-host`, and `--allowed-origin` values | diff --git a/docs-site/content/docs/cli-reference/meta.json b/docs-site/content/docs/cli-reference/meta.json index 46aafb4c..4103c025 100644 --- a/docs-site/content/docs/cli-reference/meta.json +++ b/docs-site/content/docs/cli-reference/meta.json @@ -9,6 +9,7 @@ "ktx-sl", "ktx-wiki", "ktx-status", + "ktx-mcp", "ktx-dev" ] } diff --git a/docs-site/tests/docs-index-route.test.mjs b/docs-site/tests/docs-index-route.test.mjs index ddcd3181..7d1c62c0 100644 --- a/docs-site/tests/docs-index-route.test.mjs +++ b/docs-site/tests/docs-index-route.test.mjs @@ -9,6 +9,7 @@ import { setTimeout as delay } from "node:timers/promises"; import { fileURLToPath } from "node:url"; const configuredDocsSiteUrl = process.env.DOCS_SITE_URL; +const docsBasePath = "/ktx"; let docsSiteUrl = configuredDocsSiteUrl; let docsServer; let docsServerOutput = ""; @@ -46,7 +47,7 @@ async function waitForDocsServer() { } try { - await fetch(`${docsSiteUrl}/docs`, { redirect: "manual" }); + await fetch(`${docsSiteUrl}${docsBasePath}/docs`, { redirect: "manual" }); return; } catch { await delay(200); @@ -99,12 +100,14 @@ after(async () => { } }); -test("/docs redirects to the docs introduction", async () => { - const response = await fetch(`${docsSiteUrl}/docs`, { redirect: "manual" }); +test("/ktx/docs redirects to the docs introduction", async () => { + const response = await fetch(`${docsSiteUrl}${docsBasePath}/docs`, { + redirect: "manual", + }); assert.equal(response.status, 307); assert.equal( response.headers.get("location"), - "/docs/getting-started/introduction", + `${docsBasePath}/docs/getting-started/introduction`, ); });