Commit graph

102 commits

Author SHA1 Message Date
Andrey Avtomonov
6149f67ce6 docs(docs-site): collapse agent setup explainer into a hover overlay 2026-05-28 16:04:08 +02:00
Andrey Avtomonov
35cecdf65d docs(docs-site): tidy agent setup prompt copy and sizing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:30:51 +02:00
Andrey Avtomonov
b687167bc1 Route ktx stars dashboard 2026-05-28 13:00:49 +02:00
Andrey Avtomonov
2a85346613
fix(docs-site): disable Geist Mono ligatures on every font-mono surface (#228)
Geist Mono fuses `--` into an em-dash glyph that visually swallows the
adjacent space, so prompts like `npx skills add Kaelio/ktx --skill ktx`
rendered as `Kaelio/ktx--skill ktx` on the quickstart page. The existing
ligature-off rule only covered <code>/<pre> and the .ktx-code wrapper —
quickstart.mdx puts the prompt in a plain <div className="font-mono">,
so the rule didn't apply. Extend the selector to also match the
.font-mono Tailwind utility and any inline-style opt-in via the mono
font CSS variable.

Document the convention in AGENTS.md so future docs additions keep
ligatures off on any new monospace container.
2026-05-28 12:51:17 +02:00
Andrey Avtomonov
39f94f39ff
docs: add ktx skills.sh setup skill (#227) 2026-05-28 12:28:10 +02:00
Andrey Avtomonov
a94f35800a
feat(docs-site): redirect ktx.sh/slack to Slack community invite (#224)
Add a host-scoped redirect for /slack on ktx.sh before the existing
catch-all so the path resolves to the community invite link instead of
docs.kaelio.com/ktx/slack.
2026-05-27 18:20:51 +02:00
Andrey Avtomonov
56985b7e09
test: split cli tests from source tree (#216)
* feat(cli): define full warehouse dialect contract

* test(cli): keep dialect edge tests focused

* fix(cli): stabilize dialect contract foundation

* refactor(connectors): own read-only query preparation

* refactor(connectors): resolve dialects through registry

* refactor(connectors): keep concrete dialect classes internal

* chore(workspace): enforce dialect import boundary

* refactor(cli): resolve relationship dialect at scan boundary

* refactor(cli): use dialect display parsing for entity details

* refactor(cli): use dialect display parsing for warehouse catalog

* refactor(cli): use dialect SQL in relationship workflows

* test(cli): verify solid dialect scan workflow closure

* test: split cli tests from source tree

* refactor(cli): standardize BigQuery scope listing

* feat(sqlite): implement connector scope listing

* test(connectors): cover required table listing

* feat(cli): add warehouse driver registry

* refactor(setup): route scope discovery through driver registry

* refactor(cli): route local query execution through driver registry

* refactor(historic-sql): route dialect support through driver registry

* refactor(cli): test warehouse connections through driver registry

* fix(cli): close driver registry type export gaps

* Improve setup daemon diagnostics

* refactor(setup): centralize rail-prefixed diagnostics + query-history fallback

Extract errorMessage, writePrefixedLines, and flushPrefixedBufferedCommandOutput
into clack.ts so the setup wizard, managed daemons, and embedding/agent steps
share one rail-formatted writer. setup-databases.ts also adds a
"disable query history and retry" option when the schema-context build fails
and query history is the likely culprit, surfaced via a new
failed-query-history-unavailable status.

* fix(cli): carry catalog through the picker so BigQuery/Snowflake/SQL Server scope filters match

The setup picker's KtxTableListEntry was a 2-level { schema, name }, so
qualifiedTableId always wrote db.name into enabled_tables. When BigQuery,
Snowflake, or SQL Server later ran fast ingest, their introspect step filtered
the scope set with scopedTableNames(scope, { catalog: projectId|database, db })
— catalog was non-null on the introspect side but null in the scope refs, so
every entry was rejected, the live-database adapter staged zero table files,
and detect() failed with 'Adapter "live-database" did not recognize fetched
source output'.

Align the picker boundary with the canonical 3-level KtxTableRef:

- Add catalog: string | null to KtxTableListEntry.
- BigQuery/Snowflake/SQL Server listTables populate catalog from the
  resolved projectId / database; Postgres/MySQL/ClickHouse/SQLite set null.
- qualifiedTableId emits catalog.schema.name when catalog is non-null
  (resolveEnabledTables already accepts the 3-part shape) and
  schemasFromEnabledTables now goes through parseDottedTableEntry so it
  recovers the schema correctly from both 2-part and 3-part entries.
- Export parseDottedTableEntry from enabled-tables.ts (@internal) for picker
  reuse.

Update listTables expectations in all seven connector tests and the setup /
picker test fixtures. Add a picker regression test that covers the
catalog-bearing round-trip (save + refine).

* fix(cli): allow debug telemetry under opt-out env
2026-05-26 08:49:05 +02:00
Luca Martial
924868841d
docs: standardize fanout terminology (#218) 2026-05-25 11:09:33 -04:00
Andrey Avtomonov
78b8a0c025
feat(connectors): generalize readiness and constraint handling (#212)
* feat(connectors): add postgres maxConnections

* feat(connectors): add mysql maxConnections

* feat(connectors): add sqlserver maxConnections

* feat(connectors): rename snowflake pool config

* docs: document connector maxConnections

* feat(scan): add constraint discovery warning helper

* feat(scan): carry structural warnings through reports

* feat(postgres): soft-fail denied constraint discovery

* feat(mysql): soft-fail denied constraint discovery

* feat(sqlserver): soft-fail denied constraint discovery

* feat(bigquery): soft-fail denied primary key discovery

* feat(snowflake): report denied primary key discovery

* test(scan): verify constraint discovery warnings

* feat(historic-sql): use shared readiness probes

* docs: document query history readiness probes

* test(historic-sql): verify readiness probe registry

* test(ingest): account for live database warnings artifact

* Add skip option for agent setup
2026-05-24 19:30:06 +02:00
Andrey Avtomonov
cfd1749ab9
feat(cli): skip-context-sources menu + clack-style tree picker UX (#213)
* feat(cli): add 'skip context sources' option to database setup menu

After databases are configured, the post-setup menu now offers a 'Skip
context sources' choice equivalent to passing --skip-sources, which
plumbs through KtxSetupDatabasesResult.skipSources to bypass the
context-source step in the same run.

* feat(cli): standardize tree picker UX after clack autocomplete-multiselect

Search is always on (no '/' to enter): typed printable chars feed the
query, Tab toggles selection on the focused node without leaving the
search bar, and Space toggles only after arrow-key navigation
(isNavigating); otherwise it is appended to the query. Esc clears a
non-empty query before quitting, Ctrl+A and Ctrl+N replace bare-letter
bulk bindings, and the cursor refocuses on the first match when the
query change would hide it.
2026-05-24 19:29:37 +02:00
Andrey Avtomonov
96952fb43c
refactor: remove legacy ktx compatibility shims (#211)
* refactor: remove legacy ktx compatibility shims

* fix: restore overlay collision guidance
2026-05-24 16:57:23 +02:00
Andrey Avtomonov
394a985d2a
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204)
* feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure

Snowflake setup previously asked for a single schema as free text, then
ran a multiselect against the discovered schemas — two schema questions
back-to-back, with the first being only a session bootstrap. The SDK's
`schema` is optional, so the bootstrap step is unnecessary.

- Remove the free-text Snowflake schema prompt; only pass `schema` to
  snowflake-sdk when one is configured.
- When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the
  user for a comma-separated list, persist it as `schema_names`, and use
  it as both the table-list filter and the multiselect default. Applies
  to every driver with a scope-discovery spec, not just Snowflake.
- Update docs to lead with `schema_names`; keep `schema_name` as a
  documented single-schema shorthand.

* fix(snowflake): keep introspecting when primary-key discovery is denied

The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and
INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the
connection role may not have. Previously a 'SQL compilation error:
Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist
or not authorized' aborted the entire introspect — schemas, columns,
and row counts were all discarded over a missing nice-to-have.

Wrap the constraint query in try/catch, log a one-line warning per
schema, and return an empty PK map. Columns end up with
primaryKey=false; relationship inference still has FK and profiling
to fall back on.

* fix(scan): unblock relationship discovery on Snowflake

Two adjacent bugs prevented the scan's relationship pipeline from producing
any joins on a Snowflake warehouse:

- relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch
  for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table
  profile query failed with "Unknown function GROUP_CONCAT". Add an explicit
  Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter
  (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected).
- description-generation.ts destructured `connector.sampleTable` and
  `connector.sampleColumn` into bare locals, losing the `this` binding when
  the class-method connectors (Snowflake, Postgres, MySQL) were invoked.
  Every sample call threw "Cannot read properties of undefined (reading
  'assertConnection')" and degraded LLM descriptions to metadata-only
  prompts. Call the methods through the connector instead.

Without these, even after the primary-key probe is allowed to fail softly,
the scan ends up with 0 validated relationships and an empty `joins:` block
in every shard YAML.

* test(scan): cover table-ref helpers

* feat(scan): plumb tableScope through live-database introspection port

* feat(scan): apply tableScope during metadata fetch

* feat(scan): enforce table scope at fetch boundary

* feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206)

* feat(cli): add RSA key-pair auth option to Snowflake setup wizard

Extends the interactive Snowflake setup flow with an authentication-method
prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key
path (env/file/absolute) and an optional passphrase; the resulting connection
config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead
of `password`.

* feat(scan): pool Snowflake sessions

* fix(scan): reuse structural snapshots and cleanup connectors

* feat(scan): parallelize relationship profiling

* feat(scan): batch table description generation

* docs: document Snowflake ingest concurrency knobs

* fix(scan): close Snowflake ingest perf verification gaps

* fix(scan): keep batched description failure bounded

* feat(scan): dispatch query-history probes by connection driver

Extract historic-sql dialect resolution into a shared helper so the
status-project readiness check and the local ingest factory agree on
which connections enable query history and which probe to run. The
status command now picks the postgres/snowflake/bigquery probe based on
the connection's driver instead of always reporting against postgres,
which previously caused snowflake connections with queryHistory.enabled
to surface a misleading "driver is snowflake" failure.

Also drops a noisy console.warn from Snowflake primary-key discovery —
INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only
roles and the FK + profiling paths handle the empty PK map already.

* fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject

The Claude Code agent SDK announces an internal pseudo-tool named
StructuredOutput in the system/init message whenever outputFormat is set
to { type: 'json_schema' }. The runtime's isolation check built its
allowedToolIds set only from MCP tool ids and treated StructuredOutput
as an unexpected host-injected tool, so every generateObject call threw
"Claude Code runtime isolation failed: tools=StructuredOutput ..." and
the table-descriptions and relationship-LLM-proposal enrichment stages
recorded null output across the board.

Whitelist StructuredOutput specifically in generateObject's
allowedToolIds — the check also enforces missing_tools symmetry, so
generateText and runAgentLoop, which do not see StructuredOutput, must
not require it.

generateObject also ran with maxTurns: 1, which the model intermittently
breached when it emitted thinking text before the structured response.
Raised to 5 to give the schema-bound call enough headroom without
allowing unbounded loops. The existing tests now exercise the path with
an init message that announces StructuredOutput so the regression cannot
slip back in.

* chore(scripts): add ktx-reset.sh project-cleanup helper

Convenience script for repeatable ingest testing: takes a project
directory and prunes everything except ktx.yaml and .ktx/secrets/, so
the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
Andrey Avtomonov
b0dd13ce7c
feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205)
* feat: add telemetry phase 1

* feat: add node telemetry event catalog

* feat: add telemetry event helpers

* feat: emit setup and connection telemetry

* feat: emit connection and stack telemetry

* feat: emit ingest and scan telemetry

* feat: emit query telemetry

* feat: emit sampled mcp telemetry

* docs: expand telemetry event catalog

* feat: add telemetry schema sync artifact

* feat: pass telemetry project id to semantic daemon

* feat: add daemon telemetry foundation

* feat: emit semantic daemon telemetry

* feat: emit daemon lifecycle telemetry

* docs: document full telemetry event catalog

* feat(telemetry): dim first-run notice

* feat(telemetry): show first-run notice before command output

* feat(telemetry): wire ktx PostHog project for live ingestion

* docs(telemetry): drop posthog project name and host from storage section

* docs(telemetry): trim to general overview and disclaimer

* docs(agents): add short telemetry guidelines

* feat(telemetry): enable posthog geoip enrichment

* docs(telemetry): drop ip-geoip note from public overview

* refactor(telemetry): drop no-op groupIdentify, rely on capture groups field

* fix(telemetry): respect CI kill switch in python daemon identity

* fix(sql): route table-count analysis to existing analyze-batch endpoint

* fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander

* fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check

* fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests

* fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
Andrey Avtomonov
c87d14a554
feat(cli): redesign database scope picker for searchable schema-first setup (#203)
* feat: add searchable setup prompt pickers

* fix: make snowflake scope discovery single query

* fix: make bigquery table discovery schema scoped

* fix: honor mysql and clickhouse database scope

* feat: wire schema scope discovery for all relational setup drivers

* feat: add schema-first database scope picker

* test: update setup prompt stubs for type-check

* docs: document database scope picker fields

* Fix database setup edit preservation

---------

Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-22 14:22:11 +02:00
Andrey Avtomonov
fd2ba62d92
docs(quickstart): redesign demo-warehouse callout with sticker icons (#202)
* docs(quickstart): redesign demo-warehouse callout with sticker icons

Replaces the plain warning-style callout with a two-column layout: text
and a pill-shaped CTA on the left, a 2x2 cluster of rotated Postgres,
Metabase, dbt, and Notion sticker tiles on the right. Adds the four
connector SVGs under docs-site/public/icons/ to support it.

* chore(docs-site): refresh auto-generated next-env.d.ts
2026-05-21 16:04:58 +02:00
Andrey Avtomonov
4d4296f397
docs: rewrite context-as-code as reviewing-context guide (#201)
* docs: rewrite context-as-code as reviewing-context guide

Move the page from Concepts to Guides and rebuild around an interactive
review-loop diagram. Extract pan/zoom + fit-view controls into a shared
FlowCanvas wrapper and adopt it across all three docs diagrams.

* test: point examples-docs assertion at reviewing-context

Update the doc smoke test that read context-as-code.mdx to read the new
guides/reviewing-context.mdx path. The `ktx ingest --all --no-input`
assertion still holds; the rename was the only break.
2026-05-21 15:42:50 +02:00
Andrey Avtomonov
5211a0317e
docs: add ktx.yaml configuration reference (#200)
Adds a new Configuration section to the docs with a reference page that
covers every top-level block of ktx.yaml: connections, setup, storage,
llm, ingest, scan, agent, and memory. Each block lists fields, defaults,
accepted values, and a short YAML example, with a leading schematic that
groups blocks into inputs, compute, and persistence.
2026-05-21 15:29:20 +02:00
Andrey Avtomonov
2366b00301
chore(workspace): gate dead-code with knip production mode (#196)
* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm

* refactor(workspace): rewrite @ktx/llm imports to relative paths

* refactor(workspace): fold internal packages into cli

* chore(workspace): gate dead-code with knip production mode

Turn on production-mode knip plus an autofix run in pre-commit and the
`pnpm dead-code` script, document the `/** @internal */` convention for
test-only exports in AGENTS.md, annotate test-only exports across the
CLI with that JSDoc, and drop dead exports/wrappers the new gate
surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`,
`createLocalScanEnrichmentProvidersFromConfig`,
`PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports).
Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit
production entries so cross-package barrel leaks are caught.

* refactor(cli): delete internal barrel index.ts files

The 34 `index.ts` re-export barrels inside `packages/cli/src/` were
holdovers from the pre-fold multi-workspace structure. Post-fold-in they
served no production purpose: external consumers go through the single
package main entry, and in-repo callers mostly imported through them
only because the path was short. Internally, knip flagged most barrel
re-exports as production-dead (only reached via tests).

This change:
- Deletes every internal barrel except `packages/cli/src/index.ts`
  (the published package entry).
- Rewrites ~270 source/test files to import each name directly from
  the file that defines it.
- Moves `tools/warehouse-verification/index.ts` to
  `create-warehouse-verification-tools.ts` (the function it defined
  locally) and updates its single consumer.
- Renames `search/backend-conformance.ts` → `.test-utils.ts` to match
  the existing test-helper file convention.
- Deletes 13 dead test-only chains (dbt-descriptions/*,
  live-database/extracted-schema, live-database/structural-sync,
  relationship-* feedback/review chain) plus their tests and a
  cascading orphan integration test.
- Updates test mocks that pointed at deleted barrel paths
  (notion-client, connector barrels in scan/local-scan-connectors
  tests) to mock the source files instead.
- Points the maintainer benchmark script
  (`scripts/relationship-benchmark-report.mjs`) at source files
  instead of `dist/context/scan/index.js`.
- Drops the barrel `!` entries from `knip.json`; adds explicit
  production entries only for the benchmark code reached via dist by
  the maintainer script.

Net: 413 files changed, ~1.2k insertions, ~9.4k deletions.

`pnpm run dead-code` (Biome + knip default + knip production) and
`pnpm run type-check` are clean; 2277 tests pass.

* refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly

Promote the CLI workspace package to the public name `@kaelio/ktx` and
drop the separate `scripts/build-public-npm-package.mjs` wrapper. The
CLI package is now publishable in place (`publishConfig.access: public`,
`provenance: true`), so artifact packing uses `pnpm pack` against
`packages/cli/` instead of assembling a parallel package tree.

Updates all workspace filter invocations, docs, tests, and release
readiness checks to reference the new package name, and folds the
tarball-name helper into `scripts/public-npm-release-metadata.mjs`.

* docs: align "agent clients" and "data agents" terminology

Replace "client agents" with "agent clients" and "database agents" with
"data agents" across AGENTS.md, README.md, the docs-site copy, and the
matching setup-agents test description, matching the canonical
vocabulary in docs/terminology.md.

Also moves packages/cli/tsconfig.json's tsBuildInfoFile from
node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive
node_modules reinstalls.

* refactor(release): single source of truth for package version

Make packages/cli/package.json the single source of truth for the
@kaelio/ktx version. publicNpmPackageVersion() now reads it directly,
so artifact filenames, release-readiness checks, and the Python wheel
version all derive from one field. The duplicate
release-policy.json.publicNpmPackageVersion is removed.

Previously the two fields could drift: tarballs were named
kaelio-ktx-0.4.1.tgz while internally containing
@kaelio/ktx@0.0.0-private.

- update-public-release-version.mjs rewrites both Python pyproject.toml
  files (ktx-daemon, ktx-sl) alongside the npm package.jsons,
  normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2).
- semantic-release-config.cjs adds the two pyproject.toml files to
  @semantic-release/git assets so the release commit back to main
  carries every version source in lockstep.
- The six "?? '0.0.0-private'" fallback literals across the CLI are
  replaced with "?? getKtxCliPackageInfo().version", and
  createDefaultKtxMcpServer makes its version arg required.
- docs/release.md describes the actual commit-back model: the dev tree
  always reflects the most recent release; no sentinel pin to
  maintain.

Verified: pnpm run artifacts:build now produces
kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with
@kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and
2287 vitests + 173 script tests pass.

* refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime

Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and
scan command entrypoints so tests can stub them, and teach
resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime
feature when ktx.yaml selects sentence-transformers.

* chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal

Both symbols are consumed only by status-project.test.ts. Annotating with
/** @internal */ keeps knip's production-mode check clean without changing
runtime behavior.

* fix(cli): use real package metadata in print-command-tree

The stubbed package name embedded a forbidden product identifier that
tripped the boundary check in CI. Read the metadata from package.json
instead — keeps the rendered tree unchanged and removes a duplicate
source of truth.

* feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts

Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer
source counts, computed with `SUM(embedding_json IS NOT NULL)` over
`knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to
"Wiki" (canonical per `docs/terminology.md`) and rename the matching
`localStats.knowledgePages` field to `localStats.wikiPages`.

Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those
duplicated the per-surface rows above. Disk now reports only actual byte
usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` /
`semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry`
helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
Andrey Avtomonov
1c7131c6c2
feat(cli): add --fast flag and Local data section to ktx status (#198)
Add --fast to skip checks requiring external communication (Claude Code
auth probe and Postgres pg_stat_statements probe); skipped checks render
as `-` and carry `"status": "skipped"` in JSON output. Always show a new
Local data section sourced from .ktx/db.sqlite (ingest run counts and
last-completed per connection, knowledge page counts by scope, semantic
layer source/dictionary value counts) plus on-disk sizes for .ktx/db.sqlite,
.ktx/cache/, raw-sources/, wiki/global/, and semantic-layer/. Wrap the
remaining slow probes in a @clack/prompts spinner when stdout is a TTY.
2026-05-21 14:13:03 +02:00
Andrey Avtomonov
56a967278a
chore(docs-site): add dev shortcut and fix hero heading clipping (#190)
* chore(docs-site): add dev shortcut and fix hero heading clipping

- Add `pnpm docs` script that frees port 3000 then runs the docs-site
  dev server, so the docs preview is one command away.
- Bump hero heading line-height to 1.2 and add 0.15em bottom padding
  so the gradient text-clip no longer cuts off descenders.
- Sync auto-generated next-env.d.ts to the current Next types path.

* fix(ci): unblock CI on docs-font branch

- Add lsof to knip ignoreBinaries so the new `pnpm docs` script
  (which uses `lsof -ti:3000` to free port 3000) does not trip
  the Unlisted binaries check.
- Make CLI version assertions read @ktx/cli/package.json at runtime
  instead of hardcoding 0.0.0-private. The 0.4.0 release commit on
  main bumped the package version, breaking 18 hardcoded test cases
  in index.test.ts and admin-reindex.test.ts; reading the version
  dynamically keeps the suite green across future version bumps.

* fix ci release version fixtures
2026-05-21 01:30:45 +02:00
Andrey Avtomonov
ed2d2f9be0
docs(concepts): add Wiki retrieval pillar page (#191)
* docs(concepts): add Wiki retrieval pillar page

Adds a dedicated concept page covering the wiki side of the context
layer: the page contract, the hybrid retrieval pipeline (lexical,
semantic, token lanes fused by RRF), the refs/sl_refs/[[wikilink]]
graph, validation that keeps edges live, and where ingest sources
pages. Wired into concepts nav and cross-linked from the-context-layer
to mirror the existing Semantic querying link.

* test: derive release versions in tests instead of hardcoding 0.1.0-rc.1

After @semantic-release/git started committing version bumps back to the
branch, the 0.4.0 release rewrote package.json, packages/cli/package.json,
and release-policy.json — but the script and CLI tests still pinned the
pre-bump strings (0.0.0-private, 0.1.0-rc.1, 0.1.0rc1), so every new
branch off main failed TypeScript checks and Coverage.

Drive the version off the existing source of truth instead: read
@ktx/cli/package.json via createRequire in the CLI tests, and reuse the
already-imported PUBLIC_NPM_PACKAGE_VERSION / RUNTIME_WHEEL_PACKAGE_VERSION
constants in the script tests. The two assertions that pinned those
constants to specific values become semver shape checks.
2026-05-21 01:26:58 +02:00
Andrey Avtomonov
17647a436a
docs: standardize ktx naming (#187)
* docs: align KTX terminology

* docs: standardize ktx naming
2026-05-20 17:33:38 +02:00
Andrey Avtomonov
ad9c9eda0d
docs(context-layer): replace "committed" badge with git icon (#183)
Swap the small "committed" chip on the two-pillars figure for an inline
Git logo + "git" label so the source-of-truth signal reads at a glance.
Adds GitIcon component matching the existing GitHubIcon/SlackIcon
inline-SVG pattern. Also picks up a Next.js-regenerated next-env.d.ts
routes path.
2026-05-20 14:37:45 +02:00
Andrey Avtomonov
fb82993ce1
docs: rewrite the-context-layer concept and highlight markdown frontmatter (#181)
Restructure the-context-layer.mdx around two committed pillars (semantic
sources + wiki pages) with an inline anatomy card, replace the
semantic-layer-only comparison with a three-way matrix against company
brains and traditional semantic layers, and add a navigable-graph
explanation grounded in sl_refs/refs maintenance. Extend the docs-site
CodeBlock with a markdown highlighter that detects YAML frontmatter,
heading and list markers, and inline code so wiki examples render with
the same token colors as YAML/SQL blocks.
2026-05-20 14:16:33 +02:00
Andrey Avtomonov
a11b9e9757
refactor(release): drop release-policy.json runtime dep and next branch (#180)
* chore: standardize daemon naming on "KTX daemon"

Replace inconsistent names ("KTX Python daemon", "KTX local embeddings
daemon", "KTX managed daemon", "Python daemon") with the single name
"KTX daemon" in CLI output, errors, command descriptions, test
assertions, smoke scripts, docs, AGENTS.md, issue templates, and
codecov flags. The daemon is a portable compute server with endpoints
for SQL analysis, semantic layer, LookML, database introspection, and
embeddings; the previous labels misrepresented it as embeddings-only or
exposed implementation details ("Python", "managed").

The "KTX Python runtime" concept (installed interpreter + packages) is
deliberately left as-is — it is a separate concept from the daemon
process.

* refactor(release): drop release-policy.json runtime dep and next branch

Strips the release-policy.json fallback from release-version.ts so the CLI
reads its version straight from packages/cli/package.json. dev → 0.0.0-private,
installed @kaelio/ktx → the real semver baked into the published package.json.
KtxCliPackageInfo collapses to { name, version, contextPackageName }; /health
no longer depends on version files surviving past a CI run.

Replaces the dual-branch (main + next) semantic-release model with a single-
branch model on main. rcs and stables interleave on the same branch via
{ name: 'main', prerelease: 'rc', channel: 'next' } / ['main']. Drops
@semantic-release/git and @semantic-release/changelog (nothing is committed
back to the repo on any channel) and the workflow's "Prepare next prerelease
branch" step plus the KTX_PRERELEASE_BRANCH plumbing. The git tag plus the
published npm artifact carry the version forward.

Updates docs/release.md, removes the two now-unused devDeps, regenerates
pnpm-lock.yaml. 611/611 @ktx/cli tests, 173/173 script tests, type-check,
biome, knip all clean.

* fix(release): don't throw on non-main branches at config-load time

knip loads .releaserc.cjs on every PR run, where GITHUB_REF_NAME is the
merge ref (e.g. 180/merge). The previous version of releaseBranches threw
immediately when the branch wasn't main, which made knip fail to evaluate
the config and then mis-flag @semantic-release/exec as an unused dep.

semantic-release already refuses to publish when the current branch doesn't
match a configured release branch, so the explicit throw was redundant.
Drop it (and the unused currentBranch helper) and replace the
"rejects releases from non-main" assertion with one that exercises a CI-
shaped GITHUB_REF_NAME and confirms the config loads.
2026-05-20 13:53:14 +02:00
Andrey Avtomonov
da6d05ed55
docs(diagram): enlarge ingestion-flow text and tighten badge widths (#178)
Bumps font sizes across the in-app React Flow diagram and the README's
SVG so block titles, stage names, body copy, and chip labels are easier
to read. Widens stage cards and updates the SVG layout so every stage
body wraps to two lines, and resizes every badge rect to fit its text
with even 12px padding on both sides (notably the PostgreSQL chip).

Also includes a pre-existing README addition noting that KTX runs with
the user's own LLM keys or a Claude Pro/Max subscription.
2026-05-20 10:47:48 +02:00
Andrey Avtomonov
2c9a58bb56
feat(cli): smart defaults and flatter command surface for ktx (#177)
Bare invocations now do the obvious thing instead of erroring out, and mode-as-subcommand patterns collapse into flags on the parent. No new top-level commands.

- `ktx ingest` (bare) ingests every configured connection. The `text` subcommand is gone; capture inline notes with `ktx ingest --text "..."` and files with `ktx ingest --file path` (use `-` for stdin). `--text`/`--file` reject a positional connection id; pass `--connection-id` to tag captured notes.
- `ktx connection` (bare) lists; `ktx connection test` (bare) tests every configured connection.
- `ktx wiki` and `ktx sl` flatten `list`/`search`: bare lists, with a `[query...]` positional searches (multi-word joined with spaces). `sl validate` and `sl query` stay as distinct verbs and now read `--connection-id` from the parent.
- `ktx mcp` (bare) prints daemon status.

Adds a shared `resolveConnectionSelection` helper consumed by ingest and connection test. Updates README, docs-site cli-reference and guides, next-steps strings, agent SKILL templates, and all affected tests. Per-package type-check, unit tests (605), smoke tests, and dead-code checks all pass.
2026-05-20 01:52:37 +02:00
Andrey Avtomonov
6dbb0c8b3a
feat(cli): add ktx admin reindex (#160)
* feat(cli): add admin reindex

* fix: keep lexical-only reindex incremental
2026-05-20 01:36:54 +02:00
Andrey Avtomonov
3db3e724cb
fix(docs): stop semantic-layer flow from trapping page scroll (#175)
Wheel events over the embedded ReactFlow diagram were captured by
zoomOnScroll + preventScrolling, blocking page scroll once the pointer
crossed into the diagram — even at min zoom. Disable wheel-zoom and let
the page handle scroll, keep Cmd/Ctrl + scroll as the zoom escape hatch
(default zoomActivationKeyCode), and remove the inaccessible Controls
that sat at the bottom of the 2340px-tall canvas. Hint badge updated.
2026-05-20 01:29:03 +02:00
Andrey Avtomonov
bf163029d3 Normalize docs punctuation 2026-05-20 00:43:07 +02:00
Luca Martial
8d5186e4ea
docs: add Slack community invite to README and docs (#157)
* docs: add Slack community invite to README and docs

Adds a Slack badge and Community section to the README, a new
Community & Support page under docs-site/content/docs/community/,
and a Community section on the docs introduction page. Routes
chat/questions to Slack and bugs/features to GitHub Issues.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: add Slack icon link to docs navbar

Adds the Slack brand mark as an icon button in the Fumadocs navbar
alongside the existing GitHub link, pointing to the KTX Slack
community invite. Persistent across every docs page so users can
reach the community from anywhere.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: order navbar icons as GitHub then Slack

Moves the GitHub link out of githubUrl and into the explicit links
array so the navbar renders GitHub first, then Slack. Fumadocs
appends githubUrl after links, which previously put Slack first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 18:07:29 -04:00
Andrey Avtomonov
620d6adbe6
docs: rewrite Semantic Querying concept with imperative-vs-declarative diagram (#156)
* docs: rewrite Semantic Querying concept with imperative-vs-declarative diagram

Reframe semantic-layer-internals.mdx around the contract the semantic
layer offers an agent: declare what you want (a Semantic Query), KTX
figures out how to compute it. Replaces the old "Context-Aware SQL"
framing with a clear imperative-vs-declarative narrative.

Adds a React Flow component (semantic-layer-flow.tsx) that contrasts a
buggy 4-table agent-authored SQL (chasm trap, LEFT-JOIN-in-WHERE,
hardcoded DATE_TRUNC) against the chasm-safe per-fact CTE SQL the
planner actually emits, including the outer GROUP BY over the requested
dimensions. Both lanes converge into a shared warehouse node and each
SQL card now has parallel bullet notes (failures on the left, KTX
behavior on the right).

Side fixes bundled in:
- include the /ktx basePath in the favicon metadata so the icon resolves
  under the production prefix
- migrate docs-site/middleware.ts to docs-site/proxy.ts (Next 16 rename)
- redirect / to /ktx/docs/getting-started/introduction so the apex docs
  URL works
- add tests covering the apex redirect, the favicon basePath, and the
  middleware-to-proxy rename
- propagate the Semantic Query terminology across the ktx-sl CLI
  reference, the context-layer concept page, and the agent-clients /
  primary-sources integration pages

* Fix CI dead-code failures

* docs-site: polish semantic-layer-internals code blocks and flow diagram

- Make CodeBlock a server component so children traverse synchronously
  under React 19 RSC streaming; previously extractText returned "" in
  dev SSR, leaving code blocks empty.
- Add custom JSON/YAML/SQL/code-like tokenizers with theme-aware token
  classes; drop the colored file-glyph dot and gradient tab-head.
- Tighten tab-head: subtle grey background, smaller monospace filename
  in muted grey, smaller rectangular language pill placed to the left
  of the filename.
- Polish the React Flow semantic-layer diagram (controls, fit-view
  padding, edge types).

* docs-site: annotate imperative SQL, add section anchor, drop ClickHouse

- Wire numbered red badges to each problematic span in the "Without KTX"
  SQL with hover sync between SQL gutter, lines, and the notes list.
- Add #imperative-vs-declarative anchor on the flow section header so
  the eyebrow link is shareable; reveals a # glyph on hover/focus.
- Align the compiled-SQL note dots to the first-line midpoint
  (mt-[6px] instead of mt-1) so 4px dots sit at y=8 in a 16px line.
- Remove all ClickHouse references from docs-site (primary-sources,
  quickstart, ktx-setup, contributing, agents-setup, mechanics test,
  warehouse drivers in the flow diagram).

* test: drop ClickHouse contributing-docs assertion

Align the workspace-package mirror test with the ClickHouse removal
from docs-site (75907eb). The connector-clickhouse package still
exists in packages/, but contributing.mdx no longer lists it, so the
test that mirrored docs against the workspace was failing.
2026-05-19 23:41:29 +02:00
Andrey Avtomonov
590dd5dddb
fix(cli): simplify setup flags and agents tty handling (#155)
* fix(cli): simplify setup flags and agents tty handling

* fix(context): update ingest setup guidance flag
2026-05-19 19:23:35 +02:00
Andrey Avtomonov
75bb4f9497
docs: align CLI install package manager guidance (#148)
* docs: align CLI install package manager guidance

* test: update CLI install docs expectation
2026-05-19 16:41:01 +02:00
Andrey Avtomonov
afcfecfcc9
docs: add ingestion flow diagram to readme (#144) 2026-05-19 16:02:20 +02:00
Andrey Avtomonov
e80bfbfda5
docs: add agent setup prompt endpoint (#142)
* docs: add agent setup prompt endpoint

* docs: wrap agent setup prompt

* docs: restyle agent setup note

* docs: make agent setup prompt copyable

* docs: update quickstart demo callout

* docs: align prompt copy button
2026-05-19 15:22:17 +02:00
Andrey Avtomonov
56f4f9c9e8
feat(cli): split Claude Desktop skills and polish setup-agents output (#141)
* fix(cli): package Claude Desktop skills in one zip

* Polish ktx setup-agents output: hint, summaries, outro

* test: update setup agents output polish assertion

* Add output-polish follow-up plan

* docs: align Claude Desktop split-ZIP wording

Update README and the agent-clients docs page to reflect that ktx setup
now produces one uploadable ZIP per Claude Desktop skill under
.ktx/agents/claude/ (ktx-analytics.zip and optionally ktx.zip) instead
of a single combined ktx-skills.zip.

* feat(cli): style next-actions note in TTY mode

Add createAgentNextActionsLineFormatter, an ANSI line transformer wired
into the "Required before using agents" Clack note. It activates only
when the target stream reports hasColors(), so non-TTY pipelines and
tests keep the existing plain-text output byte-identical.

Per-line rules: cyan-bold step numbers + bold titles; dim sub-prose
aligned under the title; dim-cyan bullet for .zip paths with HOME
shortened to ~; dim "›" replaces " > " breadcrumbs; RUN/PASTE/USE/OPEN
markers dimmed; already-styled lines pass through to avoid double-wrap.

* docs: move output polish specs out of ktx
2026-05-19 15:21:49 +02:00
Andrey Avtomonov
b42f418adc
fix: allow agent setup without context (#139)
* fix: allow agent setup without context

* docs: align readme command examples
2026-05-19 12:18:52 +02:00
Luca Martial
1331e573dd
Improve KTX agent setup guidance (#137)
* feat(cli): clarify MCP start output

* feat(cli): improve agent setup guidance

* docs: update agent client setup guidance
2026-05-18 18:54:20 -04:00
Andrey Avtomonov
b507ff171d
docs: revamp quickstart and tighten code-block styling (#135)
* docs: streamline quickstart

* feat(docs): simplify quickstart code-block styling

Remove the fake terminal chrome (traffic lights + zsh header) and language
pill from bash blocks, and the teal left-accent from output blocks. Bash
fences now render as minimal cards; text fences route to a muted "output"
preview. Make detectLanguage recursive and enable addLanguageClass in
source.config.ts so Shiki tokens carry through to the renderer. Switch
Shiki themes to min-light / github-dark and disable monospace ligatures so
flag pairs like --agents keep a visible space.

* fix(docs): restore quickstart CI snippets
2026-05-18 19:22:19 +02:00
Andrey Avtomonov
7d156d9a06
feat(docs): visualize KTX ingestion with ReactFlow diagram (#133)
* feat(docs): visualize KTX ingestion with ReactFlow diagram

Reframe the introduction around the two user-facing ingestion outputs (wiki
and executable semantic layer) and replace the static product-mechanics card
flow with a ReactFlow diagram: sources fan into a sequential ingest pipeline,
which forks into wiki and semantic-layer outputs connected by a bidirectional
"references" edge. Drop the .ktx/raw-sources internal-implementation rows from
the intro table and update the content test to guard the new copy.

* Improve KTX docs introduction

* feat(docs): animate ingestion flow with running dots

Replace static smoothstep edges in the introduction page's ingestion
diagram with a custom animated edge that runs glowing cyan dots along
each path, conveying the source → stage → output flow. Dot duration
scales with path length and is hidden under prefers-reduced-motion.

* feat(docs): route ingestion atoms through full source→output journey

Replace per-edge dots with full-journey particles: each atom is born at
a source, threads the entire stage chain, and lands at either the wiki
or semantic layer. Particles are tinted by their source's accent so
the origin is legible. Each source produces exactly 2 atoms (8 total)
to guarantee every input is visibly active, while the destination and
begin offsets are randomized per page load. Particles populate on
client mount to avoid hydration mismatch, and are hidden under
prefers-reduced-motion.
2026-05-18 17:41:37 +02:00
Luca Martial
611f830fe0
fix(docs): restore search overlay behavior (#134) 2026-05-18 10:50:34 -04:00
Luca Martial
6a60977eb5
fix(docs): replace broken page copy buttons with working single button (#132)
The three page-level buttons (Copy MD, View MD, Copy MDX) were broken
because the fetch URL missed the /ktx basePath. Replace with a single
"Copy as Markdown" button that strips frontmatter from the MDX source
already available client-side — no fetch needed. Drop the .md link
since agents discover markdown URLs through llms.txt and content
negotiation.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-18 09:59:01 -04:00
Luca Martial
c539433d66
docs: tighten guide copy (#131) 2026-05-18 09:57:27 -04:00
Luca Martial
d60d83e595
docs: remove section overview pages (#130) 2026-05-18 09:49:43 -04:00
Andrey Avtomonov
e64da5a85d
feat(ingest): default local ingest to isolated diffs (#128)
* docs: add isolated-diff ingestion design

* Refine isolated-diff ingestion design after adversarial review iteration 1

* Refine isolated-diff ingestion design after adversarial review iteration 2

* Refine isolated-diff ingestion design after adversarial review iteration 3

* feat: persist ingest trace events

* feat: add isolated ingest patch helpers

* feat: validate wiki body semantic references

* feat: add final ingest artifact gates

* feat: execute ingest work units in child worktrees

* feat: integrate isolated work unit patches

* feat: route selected ingest sources through isolated diffs

* test: cover isolated diff ingestion regressions

* feat: add isolated diff ingestion v1 core

* docs: document ingest trace inspection

* docs: add isolated diff ingestion v1 core plan

* fix(ingest): tighten final artifact gates

* fix(ingest): gate isolated final integration tree

* fix(ingest): persist postmortem failure traces

* fix(ingest): trace policy conflicts and cleanup child worktrees

* test(ingest): verify isolated diff postmortem coverage

* docs: add isolated diff ingestion gates and trace closure plan

* fix(ingest): gate provenance before isolated diff squash

* docs: add isolated diff ingestion provenance gate closure plan

* fix(ingest): gate final wiki references

* fix(ingest): enforce SL target connection scope

* fix(ingest): trace isolated SL target policy gates

* test(ingest): cover isolated diff reference and target gates

* chore(ingest): verify isolated diff gate closure

* docs: add isolated diff ingestion reference and target gate closure plan

* fix(ingest): gate global wiki references

* docs: add isolated diff ingestion global wiki reference gate closure plan

* fix(ingest): validate scan sources and wiki refs

* test(ingest): cover isolated diff textual conflict resolver

* test(ingest): cover isolated diff resolver integration

* feat(ingest): repair isolated diff textual conflicts

* feat(ingest): report isolated diff resolver outcomes

* test(ingest): verify isolated diff textual conflict repair

* test(ingest): align textual conflict failure coverage

* docs: add isolated diff textual conflict resolver plan

* test(ingest): cover isolated diff gate repair

* feat(ingest): add isolated diff gate repair agent

* feat(ingest): repair isolated diff semantic gate failures

* feat(ingest): wire isolated diff gate repair

* test(ingest): verify isolated diff final gate repair

* chore(ingest): verify isolated diff gate repair

* docs: add isolated diff gate repair plan

* Improve ingest progress updates

* feat(ingest): route direct-write connectors through isolated diffs

* test(ingest): cover non-metabase isolated diff routing

* feat(ingest): project metricflow semantic models before work units

* test(ingest): verify metricflow isolated projection path

* chore(ingest): verify isolated diff connector migration

* docs: add isolated diff connector migration plan

* feat(ingest): make isolated diff routing the private default

* feat(ingest): promote isolated diff to default runner path

* feat(ingest): default local ingest to isolated diffs

* chore(ingest): remove isolated diff allowlist references

* fix(ingest): preserve transient evidence for isolated work units

* docs: add isolated diff default promotion plan

* refactor(ingest): remove shared worktree WorkUnit path

* docs(ingest): align WorkUnit prompts with isolated diffs

* test(ingest): drop unused runner import

* docs: add isolated diff shared worktree removal plan

* docs: add isolated diff gate repair classification plan

* fix: restrict claude-code mcp servers

* docs: align ingest trace guidance with public CLI

---------

Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-18 13:38:06 +02:00
Andrey Avtomonov
d1c84e5564
fix: improve setup wizard behavior (#127)
* fix: improve setup wizard behavior

* fix: derive runtime versions from release metadata

* test: validate metabase source mapping requirements

* Fix boundary check release identifiers
2026-05-17 19:15:09 +02:00
Andrey Avtomonov
33a142f769
feat(cli): add read-only sql command (#126)
* feat(cli): add read-only sql command

* fix(cli): rename sql connection flag
2026-05-17 10:29:07 +02:00
Andrey Avtomonov
c89af7733a
fix: improve ingest runtime readiness (#124)
* fix: improve ingest runtime readiness

* fix(cli): mock runtime in slow setup tests

* test(cli): isolate setup runtime status
2026-05-17 10:27:29 +02:00
Andrey Avtomonov
74be832aea
feat(cli): improve search ranking output (#123) 2026-05-17 02:32:41 +02:00