ktx/AGENTS.md

ktx Development Notes

ktx is a standalone open-source context layer for data agents. These instructions apply to all agents working in this repository (Codex, Claude, Gemini, and similar tools). Do not assume an external app server, frontend, database migrations, ORPC contracts, or python-service/ layout exist here.

Critical Rules

Absolute Requirements

• MUST: Use the active agent's task tracker for tasks with 3+ steps or complex operations (TodoWrite in Claude, update_plan in Codex).
• MUST: Read files before editing them.
• MUST: Complete all tracked tasks before finishing.
• MUST: Activate .venv before running Python code when a local virtualenv exists. If no .venv exists, use uv run ... from the relevant project root.
• MUST: After modifying Python files, run the relevant Python tests and run uv run pre-commit run --files [FILES] when a pre-commit config exists. If pre-commit cannot run because config or tool versions are missing, state that explicitly and run the closest available checks.
• MUST: Remove dead code; do not leave commented-out code, unused wrappers, or empty directories.
• MUST: Keep package/public API changes intentional. Do not add compatibility wrappers for old ktx names unless the user explicitly asks for a migration bridge.
• MUST: Avoid compatibility shims for old ktx features, command shapes, configuration formats, or internal APIs. This rule does not prohibit compatibility support for third-party systems and libraries, such as Metabase version differences. Keep the ktx codebase clean instead of preserving stale ktx behavior.
• MUST: Treat ktx as having no public users unless the user says otherwise. Legacy support is not necessary by default; prefer clean breaking changes over compatibility shims, migration bridges, or preserved stale behavior.

Absolute Prohibitions

• MUST NOT: Use raw pip; use uv.
• MUST NOT: Use npm or bun; use pnpm.
• MUST NOT: Run destructive git cleanup commands (git clean, git reset --hard, git checkout .) unless the user explicitly requested that exact operation.
• MUST NOT: Run git stash, git stash pop, git stash apply, or git stash drop without explicit user instruction. Prefer a branch plus commit when the user asks to save work in progress.
• MUST NOT: Reintroduce external app conventions such as ORPC contracts, NestJS controllers, frontend routes, routeTree.gen.ts, or app database migration commands unless those systems are intentionally added to ktx later.

Language Convention

• MUST: Absolute requirement, never deviate.
• MUST NOT: Absolute prohibition.
• SHOULD: Strong recommendation, deviate only with good reason.
• MAY: Optional, at agent's discretion.

Priority Hierarchy

When rules conflict, follow this order:

1. Safety and user intent
2. Correctness: code works and verification passes
3. Single source of truth and DRY design
4. Code quality: types, readable boundaries, focused modules
5. Performance where it matters

Repository Shape

ktx is a pnpm + uv workspace.

• TypeScript package: packages/cli (the sole npm-published package source)
• Core context modules: packages/cli/src/context/
• LLM provider modules: packages/cli/src/llm/
• Database connector modules: packages/cli/src/connectors/<driver>/
• Python semantic layer: python/ktx-sl
• ktx daemon: python/ktx-daemon
• Examples and fixtures: examples/
• Workspace scripts: scripts/
• Local agent skills and internal planning docs are private overlays. Do not commit .agents/, .claude/, or docs/superpowers/ to this public repository.

Some source identifiers still contain historical package-oriented names. Do not mass-rename symbols, paths, or docs unless the task asks for that rename.

Quick Commands

TypeScript Workspace

pnpm install
pnpm run build
pnpm run type-check
pnpm run test
pnpm run check
pnpm run dead-code
pnpm --filter @kaelio/ktx run smoke
pnpm --filter './packages/*' run build
pnpm --filter './packages/*' run test
pnpm --filter './packages/*' run type-check

Python Workspace

uv sync --all-groups
uv run pytest -q
uv run pytest python/ktx-sl/tests -q
uv run pytest python/ktx-daemon/tests -q
uv run pre-commit run --files [FILES]

If pyproject.toml pins a newer uv than the local binary, do not edit the pin just to make checks pass. Report the version mismatch and run checks that do not require changing project configuration.

CLI and Release Checks

pnpm run setup:dev
pnpm run link:dev
pnpm run artifacts:verify
pnpm run release:readiness
pnpm run release:published-smoke

Verification After Changes

Choose the smallest checks that cover the changed surface, then broaden when shared contracts or package exports are affected.

• TypeScript package code: pnpm --filter <package> run type-check and pnpm --filter <package> run test
• Cross-package TypeScript changes: pnpm run type-check and pnpm run test
• Build/export changes: pnpm run build
• Workspace scripts: node --test scripts/*.test.mjs or the specific script test file
• TypeScript dead-code tooling/config changes: pnpm run dead-code
• Python semantic layer: uv run pytest python/ktx-sl/tests -q
• ktx daemon: uv run pytest python/ktx-daemon/tests -q
• Python files: also run uv run pre-commit run --files [FILES] when pre-commit is configured

For test suites that take a while, capture full output once and inspect that file instead of rerunning to apply different filters:

pnpm run test 2>&1 | tee /tmp/ktx-test-output.log

Avoiding Overengineering

For the code-design principles agents must apply when writing or changing behavior — one way to say one thing, behavior follows from inputs (not from which path the caller took), failures must reach a decision-maker, don't build seams without a second piece on the other side, specification and behavior are one artifact, verify the path you claim to have fixed, and naming asymmetries are bugs in waiting — see docs/code-design.md. Treat the MUST / MUST NOT rules there with the same weight as the ones in this file.

Design Reasoning Defaults

When proposing a design, an approach, or any non-trivial change, apply these defaults and run the self-check before presenting it. They encode the corrections users most often have to make; reaching these conclusions autonomously — without being asked the leading question — is the bar.

• MUST: Optimize for the best outcome, not for an unstated constraint. Do not silently adopt "smallest change", "least effort", "cheapest", or "least user intervention" as the goal unless the user said so. Default to the most correct, durable solution, and present cost / effort / scope as information for the user to weigh — not as a ceiling you impose on their behalf.
• MUST: Separate one-time cost from recurring cost before discarding an option. A fixed cost paid once (a setup-time computation, an extra LLM call during setup, a contract change) to make every later run cheaper or more correct is usually worth it. Do not reject it with recurring-cost reasoning; quantify both sides. (Example smell: "don't add an LLM call to a cost-cutting feature" — wrong when the call is one-time and the savings recur.)
• MUST: Treat a user's example as a representative of a class, not as the spec. Design for the general population the example stands for, then stress-test against deliberately different instances — another warehouse, dialect, stack layout, or input shape — before committing. If a design only works because of an incidental property of the example (e.g. "the noise happened to be in a separate schema on this demo"), it is curve-fitting; generalize it or state the assumption explicitly.
• MUST: Prefer deriving from the system's own state over enumerating cases. Favor an allowlist computed from declared/observed state (config, scanned catalog, query log, the user's own inputs) over a denylist of known-bad specifics (particular tables, schemas, tools, or vendors). A hardcoded or hand-maintained list of external specifics is a smell: it rots and fails on the next stack. The only acceptable static patterns are genuinely universal invariants (e.g. DB-engine system catalogs) and ktx's own self-emitted signatures.
• SHOULD: Before inventing an abstraction or hand-rolling structural logic, search for what already exists and reuse it — the codebase's canonical representation (a structured ref/key type) instead of a parallel string scheme, and a mandated/available tool (e.g. sqlglot for SQL structure; see SQL and Structured Parsing) instead of hand-parsing. Normalize ambiguous input to the canonical form at the boundary; do not carry the ambiguity downstream. This is the single-source-of-truth / DRY item from the Priority Hierarchy applied at design time.

Before presenting a design, answer these explicitly:

1. Am I optimizing for a goal the user actually stated, or one I assumed?
2. Does this generalize beyond the example in front of me? Name a real case where it would break.
3. Am I enumerating known-bad cases when I could derive scope from the system's own declared/observed state?
4. Is there an existing canonical representation or mandated tool I should reuse instead of building or parsing my own?
5. Am I discarding the better option on a weak or misapplied constraint (one-time vs recurring cost, "more surface area", "more work now")?

A user question that nudges toward an alternative ("would X help?", "should I always do Y?", "will you hardcode Z?") is a signal that a better option exists. Investigate the implied direction and reason it through before defending the original proposal — and prefer to have asked yourself the question first.

TypeScript Standards

• Use Node 22+ and pnpm workspace commands.
• Keep packages ESM ("type": "module") and preserve NodeNext TypeScript semantics.
• Prefer strict types over any; do not use as unknown as.
• Keep package exports, types, and built dist expectations aligned when changing public APIs.
• Use zod schemas for runtime validation at CLI/config/API boundaries.
• Keep connector modules thin: connector-specific scanning/auth behavior belongs in packages/cli/src/connectors/<driver>/; shared types and orchestration belong in packages/cli/src/context/.
• Avoid circular module dependencies. Shared code should move to the lowest sensible module, not be duplicated across connectors.
• Do not manually edit generated or built output under dist/; edit source and rebuild.

Dead TypeScript Code Checks

ktx uses Biome for local unused-code linting and Knip for workspace graph analysis. These checks are intentionally part of CI and pre-commit because the normal development workflow is agent-based.

• pnpm run dead-code runs three checks: Biome (dead-code:biome), Knip default-mode (dead-code:knip), and Knip production-mode (dead-code:knip:production). All three must pass.
• Default-mode Knip catches dead code reachable from no entry at all (broken graph). Production-mode Knip catches code reachable only via tests — i.e. code that's tested but doesn't ship.
• Pre-commit runs knip --fix (auto-removes the export keyword from symbols that are exported but unused) plus knip --production (alerts on test-only paths). CI runs the same checks without --fix and fails on any finding.
• Treat Knip findings as investigation prompts, not automatic deletion orders.
• Remove private dead code when you confirm there are no imports, dynamic references, generated references, or tests that still need it.
• Preserve public package exports unless the task explicitly includes API pruning.
• Add narrow knip.json ignores only for intentional dynamic or public cases. Do not add broad package-level ignores to silence unrelated findings.
• Update knip.json when adding dynamic entrypoints, generated files, package exports, CLI bins, or framework files that Knip cannot infer.

Internal exports for testability

When a function, type, or constant must be exported solely so a unit test can import it (i.e. it has no production cross-file consumer), annotate the declaration with /** @internal */ JSDoc. Knip's production-mode check ignores @internal exports, so the convention keeps the gate clean without silencing the rest of the file.

/** @internal */
export function reindexHasErrors(result: ReindexResult): boolean { ... }

Do NOT use Vitest in-source testing (if (import.meta.vitest) blocks). Keep tests in separate *.test.ts(x) files.

If the only consumer of an export is its own test and the underlying behavior isn't used in production, delete both the export AND the test — testing dead code is still dead code.

CLI Standards

• Use Commander for CLI command trees, arguments, options, help text, custom parsers, and async action dispatch. Prefer @commander-js/extra-typings for typed command definitions, use InvalidArgumentError for parse failures, and call parseAsync when actions await asynchronous work.
• Use @clack/prompts for interactive flows. Always handle cancellation with isCancel plus cancel, stop active spinners before exiting, and keep prompts grouped or factored so multi-step setup flows share cancellation behavior.
• When CLI behavior is shared by the ktx setup wizard and other ktx commands, reuse or extract components in packages/cli/src instead of duplicating setup-only logic. Prefer neutral helpers such as clack.ts, prompt-navigation.ts, and command-independent prompt adapters over imports from setup command internals.
• Keep command behavior scriptable: prefer flags and config over prompts when values are supplied, and reserve prompts for interactive missing input or explicit setup flows.

Zod Naming Convention

const userSchema = z.object({
  id: z.uuid(),
  email: z.string().email(),
  name: z.string(),
});

type User = z.infer<typeof userSchema>;

Runtime schemas use camelCase plus the Schema suffix. Static inferred types use PascalCase without the suffix.

Python Standards

• Use pyproject.toml; do not add requirements.txt.
• Use type hints for new and changed Python code.
• Use pathlib instead of os.path.
• Use logger.exception() when catching and logging exceptions.
• Prefer explicit exception types over broad except Exception.
• Keep python/ktx-sl focused on semantic-layer planning and SQL generation.
• Keep python/ktx-daemon focused on portable daemon/API behavior around the semantic layer.

SQL and Structured Parsing

• Prefer AST-based parsing over regex for structured input.
• For SQL, use sqlglot; it is already a dependency.
• In python/ktx-sl, follow the local python/ktx-sl/AGENTS.md guidance: parse expressions with sqlglot, quote reserved identifiers before parsing, and generate postgres-shaped SQL before final dialect transpilation.
• Regex may be used for non-structural sanitization, but not to interpret SQL structure.

Telemetry

ktx ships PostHog usage telemetry. Catalog telemetry events use strict schemas. When adding commands or events:

• MUST NOT: Add fields that carry user data — file paths, hostnames, environment values, SQL text, schema/table/column names, error messages, argv, or secrets. Schemas use Zod .strict(), so unknown fields throw at runtime; the privacy rule is enforced by the schema, not by goodwill.
• MUST: Add new event types in packages/cli/src/telemetry/events.ts. pnpm run build mirrors the catalog into the Python daemon schema; a pytest checks Node ↔ Python parity.
• SHOULD: Let Commander's preAction hook auto-emit the command event for any new CLI command — do not call trackTelemetryEvent manually for command-level success/failure.
• MUST: Update the public overview at docs-site/content/docs/community/telemetry.mdx only when the category of collected data changes. Adding another event with no new field types needs no docs change.

Error reports

ktx also sends PostHog Error Tracking $exception events when telemetry is enabled. This channel is separate from the strict catalog event schema and is used only for exception diagnostics.

$exception events may include stack frames, error class names, raw error messages, cause chains, source, handled, fatal, runtime version fields, OS/runtime fields, and the hashed projectId when known. Stack frames may include local file paths and the local username when those appear in paths.

$exception events must never intentionally include secrets, credentials, database URLs, auth headers, raw argv, raw environment values, SQL text, schema/table/column names as explicit properties, customer row data, user prompt text, or raw MCP arguments. Reporters must redact call-site-provided secret snapshots and common static credential patterns before the SDK serializes the exception.

Documentation and Specs

• Keep public documentation in README.md, package READMEs, example READMEs, and the docs-site/ Fumadocs tree.
• Prefer concrete commands, file paths, and acceptance criteria over broad prose.
• When documenting examples, ensure referenced files and commands exist in the standalone ktx tree.
• Remove or rewrite stale external app references unless the doc is explicitly historical.

Product Naming

• MUST: Write the product name as lowercase ktx.
• MUST: In Markdown prose, write **ktx** so the product name stays visually distinct from surrounding text.
• MUST: Use code font for the CLI command, binary, package/path fragments, configuration files, environment variables, source identifiers, and copied terminal output, for example ktx, ktx setup, ktx.yaml, KTX_PROJECT_DIR, and .ktx/.
• MUST: Use plain lowercase ktx in frontmatter, metadata, alt text, headings, nav labels, badges, UI strings, and generated index strings where Markdown emphasis is not rendered or would be visually noisy.
• MUST NOT: Write the bare all-caps spelling for the product name in docs prose. Keep uppercase only when it is part of an exact environment variable, source-code identifier, package/API name, or other literal value that must match the implementation.

Product Category Naming

• MUST: Use context layer as the primary public category for ktx. Preferred phrase: context layer for data agents.
• MUST: Use context engine only as the secondary mechanism term for the active system that builds, reconciles, validates, searches, and serves the context layer.
• MUST: Keep semantic layer as the narrower term for executable metric definitions, semantic sources, joins, measures, and SQL compilation.
• MUST NOT: Replace every semantic layer occurrence with context layer; the semantic layer is one pillar inside the broader context layer.

Preferred pattern:

**ktx** is an open-source context layer for data agents. Its context engine
ingests warehouse metadata, BI definitions, query history, docs, and approved
metrics, then turns them into reviewable files agents can search and execute.

Terminology

For canonical vocabulary used across docs, code, comments, CLI strings, and error messages — including the disambiguation rule for the overloaded word source (semantic / primary / context / source of truth) — see docs/terminology.md. Follow that file when choosing between near-synonyms (e.g. connector vs adapter, data agent vs database agent, context-source ingest vs source ingest). Product-name rules in this section take precedence over anything in that file when they conflict.

Updating docs-site/ After Code Changes

Before finishing a task, decide whether docs-site/content/docs/ needs an update. Update it when your change affects user-visible behavior, including:

• New, renamed, or removed CLI commands, flags, or subcommands (docs-site/content/docs/cli-reference/)
• Changes to ktx.yaml, environment variables, or other configuration users edit
• New or changed connectors, integrations, or supported drivers (docs-site/content/docs/integrations/)
• Changes to setup, install, or getting-started flows (docs-site/content/docs/getting-started/)
• New concepts, agent capabilities, or workflows users should know about (docs-site/content/docs/concepts/, docs-site/content/docs/guides/)

Skip docs updates for purely internal refactors, test-only changes, or fixes that do not change user-facing behavior. When you do update docs, follow the fumadocs-mdx-structure skill and keep examples copy-pasteable. If a change warrants docs but you are out of scope, call it out in your final summary rather than silently skipping it.

Monospace ligatures in docs-site/

• MUST: Disable monospace ligatures on every surface that uses the var(--font-mono) family (Geist Mono). Geist Mono fuses -- into an em-dash glyph that visually eats the adjacent space, so prompts like npx skills add Kaelio/ktx --skill ktx render as Kaelio/ktx--skill ktx.
• MUST: When adding a new container that renders user-visible monospace text outside <code> / <pre> (e.g. a styled <div className="font-mono"> for a copyable prompt), verify the global ligature-off rule in docs-site/app/global.css covers its selector. Either use Tailwind's font-mono utility (already covered) or extend the rule to match the new class — do not silently rely on Geist Mono's defaults.
• SHOULD: Prefer <code> / <pre> (or a font-mono wrapper) for any string that contains CLI flags, paths, or other tokens with --, ->, >=, !=, ==, // so ligatures never alter intent.

LLM and Prompt Development

When creating or modifying agent prompts, system prompts, tool descriptions, or skills:

• Use XML tags for major structure when it helps model reliability: <role>, <workflow>, <examples>, <success_criteria>.
• Use positive framing: tell the model what to do.
• Keep prompts compact and avoid duplicating the same rule in multiple places.
• Include 1-3 concrete examples when examples materially reduce ambiguity.
• Use AI SDK v6 patterns for TypeScript LLM work.
• Use the local ai-sdk skill when working with AI SDK code.

Context7 and External Docs

• Use Context7 when official, current library documentation would materially reduce risk.
• Context7 "Monthly quota exceeded" errors are often transient. Retry before assuming the quota is exhausted.
• If Context7 remains unavailable, state the blocked lookup and use the best available local/source documentation.

When to Ask vs Act

Act without asking when:

• Following explicit user instructions
• Running verification
• Fixing clear bugs or tool failures within the requested scope

Ask first when:

• Requirements are ambiguous
• The next step is destructive or would discard user work
• A breaking public API decision is not already implied by the task
• Missing credentials, live services, or external accounts are required

Git and Worktree Safety

• The worktree may contain unrelated user changes. Do not revert files you did not change unless explicitly asked.
• Before committing, inspect git status --short and commit only intended files.
• Do not commit ignored dependency/build artifacts such as node_modules/, .venv/, dist/, coverage output, or local databases unless the task explicitly concerns packaged artifacts.