diff --git a/README.md b/README.md index 92a2a8dd..1a66523c 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ KTX -

- The context layer for analytics agents -

+

+ The context layer for analytics agents +

npm version @@ -16,183 +16,82 @@ --- KTX turns warehouse metadata, semantic definitions, and business knowledge into -reviewable project files that agents can use while planning, querying, and -updating analytics work. - -A KTX project is a directory of plain files - YAML semantic sources, Markdown -wiki pages, and SQLite state - that you commit to git and review in PRs, -just like dbt models. - -## Who KTX is for - -KTX is built for analytics engineers and data teams who want data agents to -work on real analytics systems - not just generate one-off SQL. +reviewable project files that agents can use to plan, query, and update +analytics work. Use KTX when you want agents to: -- **Generate SQL** from approved measures and joins -- **Repair semantic definitions** through reviewable diffs -- **Explain metric provenance** with warehouse evidence -- **Work alongside** dbt, LookML, MetricFlow, Looker, Metabase, and modern BI - platforms +- Generate SQL from approved measures and joins +- Repair semantic definitions through reviewable diffs +- Explain metric provenance with warehouse evidence +- Work alongside dbt, MetricFlow, LookML, Looker, Metabase, and Notion -Works with PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, and +Supports PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, and SQLite. -## Quick start - -Install the CLI and run the setup wizard: +## Quick Start ```bash -npm install @kaelio/ktx -npm install -g @kaelio/ktx +pnpm add --global @kaelio/ktx ktx setup -``` - -The wizard walks through six steps: configuring your LLM provider, setting up -embeddings, connecting your database, adding context sources (dbt, LookML, -Metabase, Looker, Notion), building context, and installing agent integration. - -If it exits before completion, rerun `ktx setup` to resume where you left off. - -Check your project status: - -```bash ktx status ``` -``` -KTX project: /home/user/analytics -Project ready: yes -LLM ready: yes (claude-sonnet-4-6) -Embeddings ready: yes (text-embedding-3-small) -Databases configured: yes (postgres-warehouse) -Context sources configured: yes (dbt-main) -KTX context built: yes -Agent integration ready: yes (claude-code:project) -``` +`ktx setup` creates or resumes a local KTX project, configures providers and +connections, builds context, and installs agent integration. -Generate SQL from a semantic-layer source: +## Common Commands -```bash -npx @kaelio/ktx sl query --project-dir "$PROJECT_DIR" \ - --connection-id warehouse \ - --measure accounts.account_count \ - --dimension accounts.segment \ - --format sql -``` +| Command | Purpose | +|---------|---------| +| `ktx setup` | Create, resume, or update a KTX project | +| `ktx status` | Check project readiness | +| `ktx connection list` | List configured connections | +| `ktx connection test ` | Test one connection | +| `ktx ingest ` | Build context for one connection | +| `ktx ingest --all` | Build context for every configured connection | +| `ktx ingest text ` | Capture free-form notes into memory | +| `ktx sl list` | List semantic-layer sources | +| `ktx sl search "revenue"` | Search semantic-layer sources | +| `ktx sl validate --connection-id ` | Validate a semantic source | +| `ktx sl query --measure --format sql` | Compile semantic-layer SQL | +| `ktx wiki search "revenue definition"` | Search local wiki context | +| `ktx mcp start` | Start the local MCP server for agent clients | -List and test a configured warehouse connection: +Project resolution defaults to `KTX_PROJECT_DIR`, then the nearest `ktx.yaml`, +then the current directory. Pass `--project-dir ` when scripting. -```bash -ktx connection list --project-dir "$PROJECT_DIR" -ktx connection test warehouse --project-dir "$PROJECT_DIR" -``` - -The connection test prints the configured driver and connector-specific status: +## Project Layout ```text -Connection test passed: warehouse -Driver: sqlite -Status: ok -``` - -## What's in a project - -``` my-project/ -├── ktx.yaml # Project configuration -├── semantic-layer/ -│ └── warehouse/ -│ ├── orders.yaml # Semantic source definitions -│ ├── customers.yaml -│ └── order_items.yaml -├── wiki/ -│ ├── global/ -│ │ ├── revenue.md # Business definitions and rules -│ │ └── segment-classification.md -│ └── user/ -│ └── local/ -├── raw-sources/ -│ └── warehouse/ -│ └── / # Database ingest artifacts and reports -└── .ktx/ - └── db.sqlite # Local state (git-ignored) +├── ktx.yaml # Project configuration +├── semantic-layer// # YAML semantic sources +├── wiki/global/ # Shared business context +├── wiki/user// # User-scoped notes +├── raw-sources// # Ingest artifacts and reports +└── .ktx/ # Local state and secrets, git-ignored ``` -Semantic sources and wiki pages are committed to git. The `.ktx/` directory -holds ephemeral state and is git-ignored - delete it and KTX rebuilds on the -next run. +Commit `ktx.yaml`, `semantic-layer/`, and `wiki/`. Keep `.ktx/` local. -### Build demo warehouse context +## Agent Usage -Database ingest artifacts are written under `raw-sources/warehouse//` -in the project directory. +Setup can install KTX instructions for Claude Code, Codex, Cursor, OpenCode, +and universal `.agents` clients: ```bash -ktx ingest warehouse --project-dir "$PROJECT_DIR" --fast -ktx status --project-dir "$PROJECT_DIR" +ktx setup --agents --target codex ``` -For non-SQLite drivers, prefer credential references such as `--url env:NAME` -or `--url file:PATH` over literal credential URLs. - -## Managed Python runtime - -KTX installs its Python runtime only when a Python-backed command needs it. -The runtime lives outside the npm cache, is versioned by the installed CLI -version, and is managed by `ktx dev runtime` commands. - -KTX requires `uv` on `PATH` to create the managed runtime. Install `uv` with -your system package manager or the official installer before running Python- -backed KTX commands. KTX doesn't download `uv` automatically; run -`ktx dev runtime status` if runtime installation fails: +Agent-facing workflows typically start with: ```bash -ktx dev runtime install --yes -ktx dev runtime status -ktx dev runtime start -ktx dev runtime stop +ktx sl search "revenue" --json +ktx wiki search "refund policy" --json +ktx sl query --connection-id warehouse --measure orders.revenue --format sql ``` -The release artifact manifest contains the public npm tarball and the bundled `kaelio-ktx` -runtime wheel. The `python/ktx-sl` and `python/ktx-daemon` directories remain -source packages for development, not public release artifacts. - -## Use KTX with agents - -KTX integrates with coding agents through CLI skills. The setup wizard -configures this automatically. - -**CLI skills** - the agent calls `ktx` commands directly through a skill file -installed in your agent's config (e.g., `.claude/skills/ktx/SKILL.md`): - -```bash -ktx sl query --measure orders.revenue --dimension orders.status --format sql -ktx wiki search "revenue definition" -ktx sl validate orders -``` - -Supported agents: Claude Code, Codex, Cursor, OpenCode, and any agent that -reads `.agents/` skills. - -## Workspace packages - -| Package | Purpose | -|---------|---------| -| `packages/cli` | CLI entry point | -| `packages/context` | Core context engine | -| `packages/llm` | LLM and embedding providers | -| `packages/connector-bigquery` | BigQuery scan connector | -| `packages/connector-clickhouse` | ClickHouse scan connector | -| `packages/connector-mysql` | MySQL scan connector | -| `packages/connector-postgres` | Postgres scan connector | -| `packages/connector-snowflake` | Snowflake scan connector | -| `packages/connector-sqlite` | SQLite scan connector | -| `packages/connector-sqlserver` | SQL Server scan connector | -| `python/ktx-sl` | Semantic-layer query planning | -| `python/ktx-daemon` | Portable compute service | - ## Development ```bash @@ -204,7 +103,7 @@ pnpm run build pnpm run check ``` -Use the development CLI for local testing: +Use the development CLI locally: ```bash pnpm run setup:dev @@ -212,30 +111,28 @@ pnpm run link:dev ktx-dev --help ``` -### Debug LLM traces +KTX is a pnpm + uv workspace: -KTX can capture local AI SDK DevTools traces for LLM calls that run through the -KTX provider. Enable it with an environment flag when running an LLM-backed -command: +- TypeScript packages live in `packages/*` +- CLI source lives in `packages/cli` +- Python runtime source lives in `python/ktx-sl` and `python/ktx-daemon` +- Public docs live in `docs-site/content/docs` + +Useful checks: ```bash -KTX_AI_DEVTOOLS_ENABLED=true ktx ingest warehouse --project-dir "$PROJECT_DIR" --deep +pnpm run type-check +pnpm run test +pnpm run dead-code +uv run pytest -q ``` -Traces are written to `.devtools/generations.json` under the current working -directory. To inspect them, run: +## Docs -```bash -pnpm dlx @ai-sdk/devtools -``` - -Then open `http://localhost:4983`. These traces are local-development-only and -store prompts, model outputs, tool arguments/results, and raw provider payloads -in plain text. Do not enable this in production or for sensitive runs. - -The repository uses `pnpm` for TypeScript packages and `uv` for Python -packages. See [Contributing](docs-site/content/docs/community/contributing.mdx) -for full development setup, testing, and PR guidelines. +- [Quickstart](docs-site/content/docs/getting-started/quickstart.mdx) +- [CLI Reference](docs-site/content/docs/cli-reference/index.mdx) +- [Building Context](docs-site/content/docs/guides/building-context.mdx) +- [Contributing](docs-site/content/docs/community/contributing.mdx) ## License