Commit graph

52 commits

Author SHA1 Message Date
Andrey Avtomonov
05d666e75f Merge remote-tracking branch 'origin/main' into explore-research-agent-tools
# Conflicts:
#	packages/context/skills/metricflow_ingest/SKILL.md
2026-05-15 02:12:30 +02:00
Andrey Avtomonov
f8db99811a
feat(context): add driver-discriminated connection schemas (#96)
* refactor(context): export and describe mapping shape schemas

* feat(context): add driver-schemas module with warehouse drivers

* feat(context): add metabase, looker, lookml driver schemas with mappings

* feat(context): add notion, dbt, metricflow driver schemas

* refactor(context): make connectionSchema a driver-discriminated union

* chore(context): re-export KtxConnectionConfig from project package

* docs(context): add connection driver schema plan

* chore(secrets): allowlist example credentials in driver-schemas fixtures

* test(cli): align metabase fixtures with required api_url field

The driver-discriminated union added in this branch now requires api_url
for metabase connections and a known driver for warehouses. Update slow
CLI test fixtures and assertions so they exercise the new schema:
- ingest.test-utils.ts: add api_url to the prod-metabase fixture.
- setup.test.ts: switch metabase fixture from 'url' to 'api_url'.
- local-scan-connectors.test.ts: invalid-driver/missing-driver tests now
  expect the schema error from loadKtxProject (parse-time rejection).
2026-05-15 00:08:11 +02:00
Andrey Avtomonov
6c73029d0c Merge remote-tracking branch 'origin/main' into explore-research-agent-tools
# Conflicts:
#	packages/cli/src/print-command-tree.test.ts
#	packages/context/skills/sl_capture/SKILL.md
2026-05-14 22:05:00 +02:00
Andrey Avtomonov
8fd8ae6388 docs: add research-agent MCP ingest contract convergence plan 2026-05-14 19:27:38 +02:00
Andrey Avtomonov
6179667e45 docs: add research-agent MCP setup-agents plan 2026-05-14 19:11:30 +02:00
Andrey Avtomonov
d79c51abaa docs: add research-agent MCP http daemon plan 2026-05-14 18:55:22 +02:00
Luca Martial
372c90b533
Polish documentation copy (#98) 2026-05-14 12:43:14 -04:00
Andrey Avtomonov
e974f3e59f docs: add research-agent MCP discover_data plan 2026-05-14 18:38:18 +02:00
Andrey Avtomonov
f1c073b614 docs: add research-agent MCP dictionary_search plan 2026-05-14 18:24:48 +02:00
Andrey Avtomonov
ff3a8c5777 docs: add research-agent MCP entity_details plan 2026-05-14 18:14:17 +02:00
Andrey Avtomonov
371fcd47eb docs: add research-agent MCP sql execution foundation plan 2026-05-14 18:00:40 +02:00
Andrey Avtomonov
de9f4d97e7 Refine spec: drop connectionName compat carve-out and ground summary/snippet provenance per kind 2026-05-14 17:24:23 +02:00
Andrey Avtomonov
bef9d14b90 Refine research-agent MCP tools spec after adversarial review iteration 3 2026-05-14 17:06:35 +02:00
Andrey Avtomonov
1a79c0efc3 Refine research-agent MCP tools spec after adversarial review iteration 2 2026-05-14 16:58:28 +02:00
Andrey Avtomonov
9e6511182e Refine research-agent MCP tools spec after adversarial review iteration 1 2026-05-14 16:47:57 +02:00
Andrey Avtomonov
97ac2139bc docs(specs): design research-agent MCP tools and ktx mcp daemon
Adds the 2026-05-14 design spec for exposing four new MCP tools
(discover_data, entity_details, dictionary_search, sql_execution),
shipping a ktx-research skill, and introducing an HTTP-only ktx mcp
daemon so external agents can use KTX as a research-capable context
layer.
2026-05-14 16:42:40 +02:00
Andrey Avtomonov
b00c1a11a9
feat: merge ingest and scan
* docs: add CLI component reuse guidance

* docs: add unified ingest ux design

* Refine unified ingest UX design after adversarial review iteration 1

* Refine unified ingest UX design after adversarial review iteration 2

* Refine unified ingest UX design after adversarial review iteration 3

* feat(cli): route public connection ingest command

* feat(cli): hide standalone scan from public help

* feat(cli): plan public ingest depth and query history

* feat(cli): execute public database ingest facets

* feat(ingest): read connection query history config

* fix(cli): use public ingest wording

* fix(config): stop generating ingest adapter allow lists

* docs: document public ingest command

* test: align ingest surface expectations

* docs: add unified ingest public CLI surface plan

* feat(cli): preflight deep public ingest readiness

* feat(setup): store query history in connection context

* feat(setup): store database context depth

* feat(setup): verify context readiness by database depth

* fix(setup): keep context build foreground only

* fix(config): reject reserved ingest connection ids

* test: close unified ingest v1 expectations

* docs: add unified ingest v1 closure plan

* fix(ingest): bypass adapter allow-list for public source ingest

* fix(ingest): honor query history window intent

* fix(ingest): hide scan internals from public database ingest

* feat(ingest): use foreground view for interactive public ingest

* fix(setup): use schema context and query history wording

* test(cli): verify unified ingest public output

* docs: add unified ingest v1 public output closure plan

* fix(setup): forward query history flags

* fix(setup): prompt for postgres query history

* fix(status): report query history readiness

* fix(ingest): remove legacy public guidance

* fix(ingest): polish foreground retry copy

* docs(examples): use unified query history wording

* chore(ingest): finish public query history cleanup

* docs: add unified ingest v1 query history status cleanup plan

* test(docs): cover unified ingest public docs

* docs: align ingest CLI reference with unified UX

* docs: update context build guides for unified ingest

* docs: update setup and primary source ingest wording

* docs: stop advertising adapter-backed example ingest

* docs: close unified ingest public docs gaps

* docs: add unified ingest v1 docs site closure plan

* fix: render unified ingest foreground warnings

* fix: explain query history schema order

* fix: add public ingest retry guidance

* fix: align setup next steps with unified ingest

* fix: remove scan wording from demo progress

* test: verify unified ingest ux closure

* docs: add unified ingest v1 foreground and retry closure plan

* fix(cli): preserve query-history pull config in public ingest

* fix(cli): omit hidden commands from docs command tree

* test(cli): close unified ingest final public surface checks

* docs: add unified ingest v1 final public surface closure plan

* fix(cli): use public source labels in ingest reports

* fix(cli): suppress low-level public ingest output

* test(cli): verify unified ingest public plain output

* docs: add unified ingest v1 public plain output closure plan

* fix(cli): add public ingest copy sanitizers

* fix(cli): sanitize public ingest progress copy

* fix(cli): rename setup schema scope prompt

* docs(plan): add progress copy closure; test: align setup back-nav fixture

Adds the iter9 plan and updates the setup back-navigation test fixture
to pass disableQueryHistory plus listSchemas/listTables stubs that the
unified ingest setup step now requires.

* docs(plan): add final ux labels plan with narrowed label scans

* fix(cli): aggregate unsupported query-history warnings

* fix(cli): align setup database labels

* test(cli): fix setup database test type-check

* fix(cli): remove primary-source wording from setup output

* test(cli): verify unified ingest setup closure

* docs(plan): add unified ingest v1 verification copy closure plan

* fix(cli): remove top-level scan command

* fix(cli): remove legacy ingest and wiki commands

* Merge scan into ingest flow

* feat(cli): split ingest progress into per-phase rows, rename work units to tasks

Each database target in the unified ingest dashboard now renders one row per
real subprocess (Schema, then Query history when enabled) instead of a single
combined bar. Each phase has its own monotonic 0-100% bar so the progress
never snaps back to zero when historic-sql starts after scan completes.
Completed phases keep their final bar, summary, and elapsed time visible as
an inline audit trail; queued and skipped phases are shown explicitly.

Also rename user-facing "work units" / "Failed work units" to "tasks" /
"Failed tasks" in ingest output and parseIngestSummary. The parser still
accepts the legacy "Work units:" wording in captured output for backward
compat. Internal memory-flow event names and type fields are left alone.

* Fix test harness failures

* Fix CI smoke checks

---------

Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
Andrey Avtomonov
fa9237956e
ci: run pre-commit checks in CI (#74)
* ci: run pre-commit in CI

* test: update CI workflow guardrail
2026-05-13 19:49:25 +02:00
Andrey Avtomonov
c22248dabf
feat(context): add warehouse verification tools (#46)
* feat(context): add warehouse dialect dispatch

* feat(context): read warehouse scan catalog

* feat(context): add entity details verification tool

* feat(context): add ingest SQL verification tool

* feat(context): add raw warehouse discovery tool

* feat(context): expose warehouse verification tools to ingest

* docs(context): add ingest identifier verification protocol

* test(context): guard ingest identifier verification prompts

* chore(context): verify warehouse verification tools

* docs: add warehouse verification tools plan and spec

* fix(context): expose target warehouses to Notion ingest

* fix(context): update ingest prompts for warehouse verification tools

* fix(context): scope raw schema discovery to allowed connections

* fix(context): verify warehouse column display targets

* docs: add notion warehouse verification gap closure plan

* fix(context): include raw discovery connection names

* fix(context): expose warehouse targets for LookML and MetricFlow

* fix(context): pass connection config to ingest query executors

* fix(cli): enable read-only SQL probes for local ingest

* docs: add warehouse verification final v1 closure plan

* fix(context): align warehouse sql probe prompt shape

* docs: add warehouse verification prompt shape closure plan

* test(context): catch connectionless sql execution prompt examples

* fix(context): include connection name in sl capture sql example

* docs: add warehouse verification sql example closure plan

* fix(context): report structured entity detail misses

* docs: add warehouse verification structured target miss closure plan

* fix: report untracked squash merge conflicts

* feat: require ingest verification ledger

* fix: stabilize ingest wiki references
2026-05-13 13:43:23 +02:00
Andrey Avtomonov
7b52ff9bd0 docs: add cli command tree implementation plan 2026-05-13 00:36:19 +02:00
Luca Martial
950fa151ce
Merge pull request #26 from Kaelio/luca-martial/demo-data-wizard-ux
feat(cli): add demo guided tour with real Orbit project data
2026-05-12 01:06:45 -04:00
Luca Martial
b440c75092 docs: plan agent-friendly docs site 2026-05-11 16:38:15 -07:00
Luca Martial
8fdf578148 docs: design agent-friendly docs site 2026-05-11 16:30:47 -07:00
Luca Martial
e52713ca1e docs: add demo guided tour implementation plan
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-11 15:26:26 -07:00
Luca Martial
6e2a6b7611 docs: add demo guided tour design spec
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-11 14:34:07 -07:00
Andrey Avtomonov
1bd29c7eb1 Fix historic SQL ingest setup and progress 2026-05-11 22:35:07 +02:00
Andrey Avtomonov
f3f6b36551 Merge remote-tracking branch 'origin/main' into andreybavt/historic-sql-redesign 2026-05-11 20:52:19 +02:00
Andrey Avtomonov
fed5e6f89f docs(plans): add historic-sql pattern shard smoke/docs plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 20:29:08 +02:00
Andrey Avtomonov
1ba7be8f91 docs(plans): add historic-sql pattern workunit sharding plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 20:18:17 +02:00
Andrey Avtomonov
cb55a0d097 docs(plans): add historic-sql redaction hardening plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 20:08:36 +02:00
Andrey Avtomonov
f8312cf79a docs(plans): add historic-sql end-to-end retrieval acceptance plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 19:58:23 +02:00
Andrey Avtomonov
20061652df docs(plans): add historic-sql projection archive hardening plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 19:48:33 +02:00
Andrey Avtomonov
78e88a6aeb docs: refresh historic sql setup and smoke docs 2026-05-11 19:41:21 +02:00
Andrey Avtomonov
fc1fdb6bc9 docs(plans): add historic-sql docs/smoke/config-cleanup plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 19:37:32 +02:00
Andrey Avtomonov
83b6fd5bac docs(plans): add historic-sql cross-dialect readiness plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 19:22:58 +02:00
Andrey Avtomonov
f99f85361a docs(plans): add historic-sql skills/projection/cutover plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 18:39:18 +02:00
Andrey Avtomonov
3ada4601d1 docs(plans): add historic-sql unified hot-path plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 17:55:42 +02:00
Andrey Avtomonov
a1465502d4 docs(plans): add historic-sql search enrichment plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 17:20:48 +02:00
Andrey Avtomonov
a46563bb01 chore: move docs site workspace 2026-05-11 16:53:42 +02:00
Andrey Avtomonov
d73a54e8c7 docs(plans): add historic-sql foundations plan
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 16:40:07 +02:00
Andrey Avtomonov
2e10a2d0be
Draft historic-SQL ingestion redesign spec (#8)
Captures the design for replacing the current per-query, per-dialect
historic-SQL pipeline with a unified, LLM-driven ingest that feeds both
ktx wiki search (pattern pages) and ktx sl search (per-table usage in
_schema shards). Spec covers motivation, principles, architecture, hot
and cold paths, search-surface plumbing, schemas, configuration, cutover
plan, and rejected alternatives.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 16:14:14 +02:00
Andrey Avtomonov
9dad936ac7
feat: npm-managed Python runtime for @kaelio/ktx (#7)
* docs: add npm managed python runtime design

* build: add bundled python runtime wheel builder

* build: make local embedding dependencies optional

* build: bundle python runtime wheel in cli artifacts

* build: track bundled python runtime release artifact

* test: verify bundled python runtime wheel

* docs: add plan for bundled python runtime wheel

* test: cover managed python runtime lifecycle

* feat: add managed python runtime installer

* feat: add runtime command runner

* feat: expose runtime management commands

* test: verify managed python runtime commands

* docs: add plan for managed python runtime installer

* feat: add managed python command helper

* feat: use managed runtime for sl query compute

* feat: route sl query managed runtime policy

* docs: add plan for managed runtime sl query integration

* feat: add managed runtime daemon metadata

* feat: manage python daemon lifecycle

* feat: add runtime daemon start stop commands

* fix: verify managed runtime daemon lifecycle

* docs: add plan for managed runtime daemon lifecycle

* feat: add managed local embeddings config marker

* feat: add managed local embeddings daemon helper

* feat: use managed runtime for local embedding setup

* feat: pass managed runtime policy through setup

* docs: add plan for managed local embeddings runtime

* feat: read CLI package metadata dynamically

* feat: assemble public kaelio ktx npm package

* feat: release one public kaelio ktx npm artifact

* test: cover public kaelio ktx package invocations

* chore: verify public kaelio ktx package artifacts

* docs: add plan for public kaelio ktx npm package

* test: verify managed runtime in public package smoke

* test: finalize managed runtime release smoke

* docs: add plan for managed runtime release smoke

* test: specify local embeddings release smoke

* feat: add local embeddings runtime smoke

* chore: register local embeddings smoke

* fix: verify local embeddings smoke

* fix: restore artifact smoke python env helper

* docs: add plan for managed local embeddings release smoke

* refactor: share managed runtime install policy parsing

* feat: use managed runtime for agent semantic queries

* feat: use managed runtime for MCP semantic compute

* docs: add plan for managed agent and MCP semantic runtime

* feat(cli): add managed daemon HTTP helpers

* feat(cli): route local adapters through managed daemon

* feat(cli): use managed daemon for ingest helpers

* feat(cli): pass managed daemon options to scan

* feat(context): pass MCP ingest pull config options

* feat(cli): pass managed daemon options to serve ingest

* test: verify managed local ingest daemon runtime

* docs: add plan for managed local ingest daemon runtime

* docs: align managed runtime examples

* docs: add plan for managed runtime docs cleanup

* test: cover published package runtime smoke commands

* test: validate published package smoke outputs

* docs: add plan for published package runtime smoke

* build: stamp public npm package version

* release: add npm public release policy

* release: add guarded npm publish script

* release: document public npm release handoff

* docs: add plan for public npm release handoff

* test: cover managed runtime prune in package smoke

* docs: document managed runtime prune

* docs: add plan for managed runtime prune smoke and docs

* chore: encode uv runtime prerequisite policy

* fix: clarify missing uv runtime error

* docs: document uv runtime prerequisite

* docs: add plan for uv runtime prerequisite contract

* refactor: limit release artifacts to public package runtime

* chore: align release policy with bundled runtime wheel

* docs: describe single public runtime artifact surface

* test: verify single public runtime artifact contract

* docs: add plan for single public runtime artifact cleanup

* fix: align local embeddings smoke with public version

* docs: add plan for local embeddings smoke public version

* release: soft-launch as @kaelio/ktx@0.1.0-rc.0 on next tag

Publish target moves to the pre-release version 0.1.0-rc.0 under the next
dist-tag so npm install @kaelio/ktx (which resolves to latest) does not
pick up the soft-launch build. Users opt in via @kaelio/ktx@next.

* Fix release script boundary checks

* Remove PostHog from public package bundle
2026-05-11 15:50:34 +02:00
Luca Martial
572d515db0 feat(docs): add Fumadocs site workspace 2026-05-11 01:08:31 -07:00
Luca Martial
cb9ab25456 docs: add title attributes to config code blocks
Triggers VS Code tab styling on the docs' YAML/config blocks
where surrounding prose names the file. Mechanical, ~1 edit
across the guides section (building-context.mdx); primary-sources.mdx
and context-sources.mdx already had title= on their Connection config
blocks, so only the prose-named semantic source output example needed
updating.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 00:45:43 -07:00
Luca Martial
c54b5ca4cd feat(docs): style Mode B (filename tab) and Mode C (minimal)
Mode B kicks in when a code block has a title= attribute.
Mode C is the default for unlabeled non-shell blocks: hover
reveals the language label and copy button.

Reduced-motion media query extended to cover the new transitions
and the sidebar chevron rotation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 00:42:31 -07:00
Luca Martial
86fc6aa463 feat(docs): style Mode A (terminal) code blocks
Dark frame in both themes, traffic-light dots, zsh/title label,
copy button in the header. Wizard-glyph content (◆ ◇) also routes
through this mode.
2026-05-11 00:39:14 -07:00
Luca Martial
cc033f1ce7 feat(docs): wire CodeBlock into MDX components map
All <pre> elements in docs MDX now render through CodeBlock,
which selects one of three modes based on language and title.
CSS styling for each mode follows.
2026-05-11 00:37:12 -07:00
Luca Martial
3a3dfeeeba feat(docs): add CodeBlock with three context-aware modes
- Mode A (Terminal): bash/sh languages or wizard-glyph content
- Mode B (VS Code tab): blocks with a title attribute
- Mode C (Minimal): everything else

Component renders only structural markup; CSS for each mode is
added in the following commits.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 00:35:14 -07:00
Luca Martial
7607e9fd40 feat(docs): add CopyButton component
Reusable copy-to-clipboard button with animated check on success.
Used by CodeBlock for all three modes.
2026-05-11 00:33:10 -07:00
Luca Martial
ea2d002a69 fix(docs): tighten sidebar selectors + add focus-visible
- Scope section trigger rules to div[data-state]:not([class]) so they
  don't bleed onto other Radix collapsibles inside #nd-sidebar.
  Fumadocs 15 section wrapper divs are bare (no class, no id); content
  panels and popovers always carry a class attribute.
- Add focus-visible rings for section buttons and page links
  (a11y gap from code review).
- Add !important to margin-top for consistency with the
  surrounding padding overrides.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 00:31:58 -07:00