scan_enabled:false promised the connection is 'never used as a scan/ingest
target,' but the predicate only gated automatic selection — explicit
ktx scan <id> / ktx ingest <id> still resolved the connection id and reached the
live-database introspection path, so an execute-only connection could still be
scanned or ingested.
Guard runKtxScan and runKtxIngest at entry: if the target connection is
execute-only, refuse with an actionable error (remove the flag to scan, or use
ktx sql to query) before doing any work. This makes the flag a single declaration
honored on every scan/ingest entry point, not just auto-selection.
A configured warehouse was always a scan/ingest target. The only way to use a
connection purely for SQL execution (ktx sql / sql_execution) was the leaky
workaround of an empty setup.database_connection_ids — which actually re-includes
every warehouse via the 'fall back to all' branch — so e.g. a BigQuery connection
meant only for read-only queries triggered a full-billing-project scan.
- Add a per-connection scan_enabled flag (default true) to warehouse connections.
scan_enabled: false registers the connection for execution only and never as a
scan target.
- Route every scan-target selection path through one predicate
(isScanTargetWarehouse): both ingest (primaryWarehouseConnectionIds, including
the all-warehouses fallback) and setup (configuredPrimaryConnectionIds) now
exclude execute-only connections. Setup validates the credential but skips
scope discovery and scan for them. Execution paths are untouched — the warehouse
descriptor still resolves, so ktx sql / sql_execution keep working.
- Scripted setup with no --database-schema no longer silently scopes the scan to
every discovered schema/dataset: it warns with the count and names how to narrow
(--database-schema) or opt out (scan_enabled: false).
Both documented flags were read only for status display; every ingest path
squash-committed to main unconditionally, so setting either to false was a
silent no-op (the reported symptom: 'Memory ingest (external_ingest): ...'
commits despite memory.auto_commit: false).
Gate the commit at the squash-merge onto main — the one point where ingest work
becomes a permanent commit (intermediate session-worktree commits must still
happen for the squash to collapse). When auto-commit is off, apply the squash to
main's working tree and leave it staged instead of committing, so the run is
never silently discarded:
- GitService.stageSquashMergeIntoMain: shares the merge core with
squashMergeIntoMain but stops before committing and returns the staged tree
SHA (a valid diff/read ref).
- memory.auto_commit gates MemoryAgentService (its DB writes are eager, so the
staged files stay consistent); the commit-message job is skipped.
- storage.git.auto_commit gates IngestBundleRunner; the wiki index is reconciled
from the staged tree via the existing syncFromCommit (git diff/show accept a
write-tree ref), and SL reindex already reads from files.
Config descriptions now state precisely what each flag gates and the staged
semantics when false.
A ktx project assumes its config dir is its own git working-tree root: writes,
session worktrees, squash-merges, and reindex scans all resolve relative to it.
GitService.initialize() gated on checkIsRepo() (IN_TREE), which is also
satisfied by an *enclosing* repository — so a project nested inside another git
working tree silently operated against the outer repo. Worktree/ingest writes
landed at the outer root (e.g. <outer>/wiki/global/) while reindex scanned
<projectDir>/wiki/global/, so the wiki was seeded but never indexed:
wiki_search returned nothing and knowledge_pages stayed empty, with no error.
Semantic-layer and raw-sources had the same divergence.
Gate initialization on checkIsRepo('root') instead: require the repo root to be
the config dir itself, and initialize a dedicated repository there when it is
not (logging clearly when nesting inside an existing repo). This restores the
one-repo-per-project invariant at the shared git layer, fixing all artifacts at
once, and keeps ktx's commits out of the enclosing repository.
Non-interactive setup (--no-input) silently defaulted the LLM backend to
anthropic, the one backend that cannot self-configure without an extra flag,
then failed with 'Missing Anthropic API key: pass --anthropic-api-key-env or
--anthropic-api-key-file.' — an error that never mentioned --llm-backend. A
user who passed --target claude-code had no way to discover the (hidden)
--llm-backend claude-code flag from the error.
- chooseBackend no longer silently picks anthropic in disabled mode; it fails
with a message naming --llm-backend, listing every backend, and noting that
claude-code/codex use local auth (no key).
- The anthropic and embedding credential errors now name --llm-backend /
--embedding-backend so a keyless backend is discoverable from the error.
- --llm-backend and --embedding-backend use .choices(), so invalid values
report the allowed set (and the bespoke parser fns are removed).
Only invocations that already failed change behavior; they now fail with an
actionable error instead of a cryptic one.
* feat(setup): write per-role llm model presets
* feat(setup): remove llm model setup flag
* chore(setup): update llm preset guidance
* docs(setup): document llm model presets
* chore(release): sync uv.lock to 0.9.0
* fix(cli): make sl query --execute work on secret-backed connections
sl query --execute used a parallel SQL executor (createDefaultLocalQueryExecutor)
that passed connection.url verbatim into pg, so file:/env: secret references
failed with "SASL: SCRAM-SERVER-FIRST-MESSAGE: client password must be a string".
Collapse onto the connector-based executor already used by MCP and ingest
(createKtxCliIngestQueryExecutor), which resolves secret references and supports
every driver. Delete the now-dead local/postgres/sqlite query executors, their
tests, and the orphaned hasLocalQueryExecutor driver flag.
* docs(agents): require one implementation per capability
Add a design-reasoning default and a matching self-check question telling agents
to route callers through a single shared implementation of a capability rather
than forking a parallel one, and to fix the shared layer rather than patch one
branch. Encodes the lesson from a divergent SQL-execution-path bug, stated
generally.
CLAUDE.md is a symlink to AGENTS.md, so both agent-instruction files are covered.
- Link the Y Combinator badge and the docs "by Kaelio" label
- Add a maintainer line to the README
- Set the npm author field on @kaelio/ktx
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cli): show cached update notices after commands
* docs(cli): describe update notices
* fix(cli): type update check environment
* fix(cli): decouple update notice display from refresh and harden suppression
Display a cached "update available" notice based solely on the lastNoticeAt
24h throttle, independent of checkedAt refresh freshness, matching the design's
independent display/refresh decisions. Suppress the check unconditionally under
--json, CI, and non-TTY before consulting output-mode preferences, so a
KTX_OUTPUT=pretty override can no longer make CI/non-TTY contexts phone npm.
* feat(docs): add serving-phase diagram to the introduction page
The introduction's "How ktx works" section described both the ingest and serve sides but only rendered the ingestion diagram. Add a live, theme-aware React Flow diagram for the serving phase (agent <-> ktx via MCP -> context layer + database) so both phases are shown, with a matching content test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(diagram-studio): relabel context edge and use right-angle routing
The hub->context edge searches and reads definitions, not just searches; relabel it "search + read". Route the serving search/read-only edges with smoothstep (right angles) to match the docs diagram. (The README PNG is a baked export and is unchanged until re-exported from the studio.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(docs): point product-mechanics assertions at the FlowCanvas wrapper
product-mechanics renders via the shared FlowCanvas wrapper, so the ReactFlow config (nodesDraggable, zoomOnScroll, etc.) lives there now. Update the stale assertions that still expected those literals inline, fixing a pre-existing test failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(serving-diagram): shrink the boxes and drop OpenCode from the agent list
Reduce node dimensions, font sizes, padding, and the canvas height so the serving diagram renders ~25% smaller and more compact. Remove OpenCode from the agent's listed clients.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
ktx.sh/ and docs.ktx.sh/ redirected to
https://docs.kaelio.com/ktx/ktx/docs/... (note the doubled /ktx) and 404'd.
The host-agnostic `source: "/"` redirect ran before the alias-host
canonicalizers, so it injected the /ktx basePath into the path on the alias
domains, which the alias catch-all then prepended a second time.
Reorder redirects() so alias-host canonicalization runs first, leaving the
generic root/docs rules for the local/canonical host only. The /stars
exclusion stays because redirects run before beforeFiles rewrites.
Add Host-spoofing regression tests (the prior tests only used localhost,
which never exercised the alias-host rules) and remove the vestigial
website/vercel.json, which the live ktx.sh routing already bypasses.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cli): add ingest rate limit governor
* feat(cli): wire ingest rate-limit config
* feat(cli): report provider rate-limit signals
* feat(cli): show ingest rate-limit waits
* fix(cli): complete rate-limit event coverage
* fix(cli): abort ingest provider calls cleanly
* fix(cli): propagate ingest cancellation
* fix(cli): reject pre-aborted ingest rate-limit waits
* fix(cli): honor Claude rate-limit reset waits
* fix(cli): retry thrown Codex rate-limit failures
* fix(cli): type Claude rate-limit result details
* fix(cli): emit ingest rate-limit countdowns from rejected signals
* fix(cli): report ai sdk rate-limit header utilization
* fix(cli): gate LLM rate-limit retries on the governor budget
The AI SDK and Codex runtimes retried 429 / opaque rate-limit failures up
to 6-7 times with no backoff when constructed without a RateLimitGovernor
(scan, memory, setup) or with pacing disabled, ignoring Retry-After and
worsening the limit. The outer retry loop only cooperates with the
governor's pause, so without active pacing there is no backoff to apply.
Route the retry bound through a single source: RateLimitGovernor
.maxRetryAttempts(), which returns retry.maxAttempts when enabled and 1
(no outer retry) when absent or disabled. All three runtimes (ai-sdk,
codex, claude-code) now use it, so ingest.rateLimit.retry.maxAttempts
genuinely controls attempts and the hard-coded 6 (plus Codex's off-by-one
extra attempt) is gone. Backend-native retry (e.g. the AI SDK's maxRetries)
still handles transient 429s.
Also correct the ktx.yaml docs for maxWaitMs (caps each wait, not the whole
run) and maxAttempts, and sync uv.lock ktx-sl/ktx-daemon to 0.9.0.
* feat(cli): block context build when a required connection fails its live test
A context build can take several minutes, so a connection that is
unreachable or misconfigured should stop the build up front instead of
failing partway through. Before the build starts, run a live connection
test for every primary- and context-source connection the build depends
on.
Each test's output is captured in a discarded buffer so raw error text
(and database paths) never reach the user; failures are surfaced only by
connection id and connector type, with a pointer to `ktx connection test
<id>` for the underlying error.
- Interactive setup lets the user fix the connection and retry without
restarting, re-resolving targets so an added/removed/reconfigured
connection is honored.
- `--no-input` exits non-zero and writes a failed context state with a
failureReason, so scripts stop early and setup never reads as ready.
Extract the buffered command IO helper out of setup-databases into
src/io/buffered-command-io.ts so both call sites share one implementation.
* feat(cli): use recovery primitive for database setup
* feat(cli): use recovery primitive for source setup
* docs: document setup connection recovery
* fix(cli): close database recovery gaps
* fix(cli): target failing project in gate hint and preserve missing-input
Address two review findings on the connection-recovery work:
- The connection-gate failure hint emitted `ktx connection test <id>` with no
--project-dir, so a setup run started with `--project-dir ./analytics` pointed
users at cwd/KTX_PROJECT_DIR instead of the project that just failed. Emit the
resolved project dir, matching the contextBuildCommands convention.
- The non-interactive database configure path returned `cancelled`, which the
recovery primitive collapses to `failed`. Sibling paths still report
`missing-input` for absent flags, so incomplete-flag runs were
indistinguishable from real connection failures. The database wrapper now
tracks the configure missing-input signal and restores the `missing-input`
step status; the shared primitive keeps its four outcomes.
Re-running setup was the dominant action for installs that completed setup but never ingested. Classify completion (incomplete | needs-context | needs-agents | ready) and drive one obvious next action per state: route a config-complete project straight to the build, point unbuilt-context users at `ktx ingest` instead of re-running setup or dropping to a bare shell, and confirm readiness for fully-set-up projects rather than reopening the edit menu.
Three reliability gaps surfaced while auditing why PostHog numbers were
untrustworthy:
1. Interrupted commands lost their events. capture() is fire-and-forget and the
only flush guarantee lived in a finally block, which SIGINT/SIGTERM skip — so
Ctrl-C'ing a long ingest or an MCP client killing 'ktx mcp stdio' dropped the
command event and any queued events. Add SIGINT/SIGTERM handlers (real-process
entry only; never under test/programmatic io) that mark the active command
span aborted, emit it, drain the emitter, then exit. Idempotent with the
normal finally path via the single-consume command span.
2. Headless-first installs were invisible. loadTelemetryIdentity refused to mint
an installId unless stdout was a TTY, so a machine whose first run was an
IDE-launched MCP server or a script emitted nothing, ever. Mint on first run
regardless of surface (still honoring CI/DO_NOT_TRACK/KTX_TELEMETRY_DISABLED),
writing the one-time notice to stderr — safe under the MCP stdio protocol,
which reserves stdout. Drop the now-unused stdoutIsTTY option.
3. No guard against silent emit regressions (the 0.7.0 scan_completed blackout).
Add tests: the shared executePublicIngestTarget chokepoint emits exactly one
ingest_completed on success and on the preflight-failure branch, and a
database target invokes the scan that emits scan_completed; plus coverage for
the aborted-flush helper.
Identity is unchanged otherwise: every event still attributes to the installId
in ~/.ktx/telemetry.json. No event/field changes, so Node<->Python schema parity
is untouched. Docs updated to reflect first-run-on-any-surface activation.
* feat: add codex sdk runner foundation
* feat: parse codex runtime events
* feat: expose codex runtime mcp tools
* feat: add codex llm runtime
* feat: wire codex llm backend
* test: avoid Array.fromAsync in codex runner test
* docs: document codex llm backend
* fix: tighten codex runtime config ownership
* fix: use codex sdk env and thread options
* fix: parse codex sdk event shapes
* test: add codex backend live smoke
* docs: clarify codex backend isolation
* fix: drive codex loop metrics from mcp events
* fix: enforce codex local step budget
* docs: disclose codex isolation limits
* fix: count all codex agent steps and stream step callbacks live
The agent-loop step budget only counted completed mcp_tool_call items, so
built-in command_execution steps (which the public Codex SDK/CLI surface can
still expose) never decremented the budget, letting ingest/reconciliation run
past stepBudget until Codex stopped on its own. onStepFinish was also replayed
only after the whole stream drained, so live work_unit_step / reconciliation
progress appeared stuck until the Codex process exited.
collectEvents is now the single live step accumulator: it counts every
completed agent-action item via a shared isCompletedAgentStep predicate
(command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish
as each step completes, and enforces the budget on that broader count. A
no-tool turn still counts as one step. toolFailures stays MCP-specific, since a
non-zero command exit is normal agent exploration, not a loop failure.
* test: align ingest llm-guard assertions with codex backend
The skip-llm ingest guard message now lists codex as a valid backend and
mentions a Claude Code/Codex session plus a codex setup hint, but this slow
suite test still asserted the pre-codex wording. Update it to match the
production message (already covered by the local-bundle-runtime unit test) and
add the codex setup-line assertion.
* fix: treat codex error:null tool calls as success
The Codex SDK serializes error: null on successful mcp_tool_call items, so
the failure check (item.error !== undefined) flagged every successful tool
call as failed with the empty-payload default "Codex turn failed". This
killed every ingest work unit under the codex backend before it could
produce a patch.
Key on status === 'failed' (authoritative, always set) and only treat a
populated error object as a failure. Add a regression test built from a
verbatim real-SDK event capture.
* fix: default codex backend to gpt-5.5 and report real probe errors
The previous default gpt-5.3-codex is an API-key-only model that the OpenAI
API rejects under ChatGPT-account (subscription) auth, so codex status/setup
failed with a misleading "authentication is not usable" message even though
auth was fine.
- Default codex model is now gpt-5.5 (works on both subscription and API-key
auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and
keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark).
- runCodexAuthProbe now distinguishes "model not available" from an auth
failure and surfaces the real API error: collectEvents retains stream
events when the SDK throws on a non-zero exit, and the API error JSON
envelope is unwrapped to its human-readable message.
- The Codex isolation warning now renders inside the clack setup frame.
- Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth.
* fix: require llm.models.default in status and match codex probe remediation
Status reported a project ready when a non-none LLM backend was configured
without llm.models.default, but the runtime (resolveModelSlots) hard-requires
it, so ingest/scan/memory threw after `ktx status` said the project was usable.
buildLlmStatus now fails for any non-none backend missing models.default and no
longer invents a fallback model for claude-code/codex.
Codex probe failures now carry a category-matched fix: a model-access failure
steers the user at llm.models.default instead of the auth/install remediation.
runCodexAuthProbe returns the fix and status consumes it; the message stays
self-sufficient so setup output is unchanged.
Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx
states --llm-model only accepts codex/default or gpt-*/codex-* ids.
Repaired four doctor fixtures that configured a backend without models.default
(the now-correctly-blocked config) and added coverage for the new behavior.
The GitHub repo was renamed back from Kaelio/ktx-ai-data-agents-context to Kaelio/ktx, reverting the URL changes from #250 across package metadata, CI (codecov + star-history slugs), issue/security templates, the release runbook, and docs/install commands.
Also removes the rename-resilience machinery #250 added: semantic-release now reads the repository URL straight from package.json (Kaelio/ktx) again, so the repositoryUrl() derivation in scripts/semantic-release-config.cjs, its tests, and the rename note in docs/release.md are no longer needed.
* fix(release): point repository URLs at renamed GitHub repo
The GitHub repo was renamed from Kaelio/ktx to
Kaelio/ktx-ai-data-agents-context. semantic-release reads repositoryUrl
from package.json's repository field and the @semantic-release/github
plugin failed verifyConditions with EMISMATCHGITHUBURL because it no
longer matched the live clone URL.
Update every Kaelio/ktx reference to the renamed repo: package metadata
(root + CLI repository/bugs/homepage), the codecov upload slugs and
star-history slug in CI, the issue-template and security-advisory links,
the release runbook, and all docs/install commands.
* fix(release): derive semantic-release repositoryUrl from the CI repo
@semantic-release/github exact-matches repositoryUrl against the live
GitHub clone_url (no redirect following), so any repo rename re-breaks the
release when repositoryUrl is the static package.json value.
Derive repositoryUrl from the runner's GITHUB_REPOSITORY/GITHUB_SERVER_URL
so it always tracks the current repo name. A future rename (including back
to Kaelio/ktx) now resolves with no code change. Outside CI the option is
omitted, so semantic-release falls back to package.json as documented.
The package.json repository field stays ktx-ai-data-agents-context as
npm-display metadata, decoupled from the release-time match.
* feat(cli): profile ingest runs to find where wall-clock time goes
Add opt-in profiling for `ktx ingest`. Each timed phase, work unit, and
agent loop now records durationMs / step count / token usage in the
trace, and a post-run aggregator rolls them up into a "where did the
time go" report printed to stderr.
Enable per run with KTX_PROFILE_INGEST (1/true -> human table, json ->
raw structured profile) or persistently via `ingest.profile` in
ktx.yaml. The json form emits raw milliseconds, token counts, and a
summary.headline one-line diagnosis so coding agents can parse it
directly; json wins when both env and config request profiling.
- runtime-port: RunLoopMetrics (totalMs, usage, stepCount,
stepBoundariesMs) plus onMetrics callbacks on text/object generation
- ai-sdk + claude-code runtimes: capture per-loop timing and token usage
- work-unit-executor and stages 3/4: thread metrics into trace events
- ingest-bundle.runner: time worktree / triage / clustering / index /
reconcile / squash phases and emit the profile in a finally block
(best-effort; never affects the run outcome)
- ingest-profile: new trace+transcript aggregator with table/json formatters
- config: ingest.profile flag; docs: profiling section in ktx-ingest.mdx
* fix(cli): flush tool-call logs before reading ingest profile
Tool transcripts are appended fire-and-forget so the agent hot path never
blocks on logging. The ingest profiler read them before the writes settled,
so per-work-unit toolMs (and the model-vs-tool split derived from it) could
be incomplete. Track in-flight appends and expose flushToolCallLogs() —
bounded by a timeout so it can never hang — and flush before the profiler
reads the transcript.
Replace the tall portrait README ingestion SVG with two landscape
diagrams — "1 · Ingestion" (build the context layer) and "2 · Serving"
(agents query it through MCP) — wired in as transparent 2x PNGs that
read on GitHub light and dark.
Add docs-site/diagram-studio: a static React Flow page with custom
themed nodes and the inlined ktx mascot that renders both diagrams and
exports them to PNG via html-to-image (the diagrams' reproducible
source). Remove the superseded ingestion-flow SVGs.
* feat(completion): complete known argument values
* fix(completion): hide Commander-hidden subcommands from completions
Replace the `__`-prefix name heuristic with Commander's `_hidden` flag so
internal subcommands registered with { hidden: true } (e.g. `mcp serve-internal`)
are excluded from completions, mirroring `ktx --help`.
* test: cover wiki and sl read command routing
* test: cover raw wiki and sl reads
* feat: add wiki read command
* feat: add sl read command
* feat: complete read command entity names
* docs: document wiki and sl read commands
* test: include read commands in command tree
* feat(sl): read and validate unique sources by name
* feat(sl): make read and validate connection id optional
* fix(completion): dedupe semantic source names
* docs(sl): document connection-optional read and validate
* fix(sl): require connection id for query command
* docs(sl): clarify query connection requirement
* fix(completion): don't resolve option values as subcommands
resolveCommand skipped flag tokens but not the value consumed by a
value-taking option in the `--flag value` form, so a connection id like
`query` was matched as the `sl query` subcommand and yielded no `sl`
completions. Track value-taking options and skip their consumed value
before matching subcommands.
* test(telemetry): assert first-run notice via TELEMETRY_NOTICE constant
CI (which tests this branch merged with main) failed because #243 changed
the first-run notice wording in identity.ts (dropped "anonymous") but left
this test grepping for the old literal 'ktx collects anonymous usage data',
so indexOf returned -1. Assert against the exported TELEMETRY_NOTICE
constant instead so the test tracks the source of truth and cannot drift
when the notice text changes again.
Set disableGeoip: false on the CLI telemetry client so events are enriched with approximate, IP-based location at ingest. Update the first-run notice, public telemetry docs, and the AGENTS telemetry policy to drop the prior "anonymous" wording to match.
Fast mode (the ktx ingest --fast/--deep database-ingest depth toggle) is removed.
ktx ingest now always builds the full enriched ("deep") context. There is no
structural fallback: a database connection without a configured model and
embeddings fails the enrichment-readiness preflight before any work runs, with
a 'Run ktx setup to configure a model and embeddings' hint.
- Remove --fast/--deep flags, the per-connection context.depth field, and the
ktx setup depth prompt (delete setup-database-context-depth.ts).
- Rename ingest-depth.ts -> connection-drivers.ts; ingest always requests scan
mode 'enriched'; readiness gate (enrichmentReadinessGaps) runs for every
database target.
- Drop the database-context-depth telemetry step (Node + Python schema mirrors
regenerated).
- Update CLI, setup, context-build view, docs, the public ktx skill, and the
release-smoke / artifacts scripts (now assert the no-LLM guard failure).
ktx status --fast (a separate network-probe flag) is unchanged.
Follow-ups: KLO-726 (live progress for ktx ingest --all), KLO-727 (restore
credentialed successful-ingest release smoke coverage).
Notion's setup path read --source-api-key-ref while writing the auth_token_ref
config field, so --source-auth-token-ref was silently dropped. Align Notion to
the flag=field convention every other connector follows: it now reads
--source-auth-token-ref, and --source-api-key-ref becomes Metabase-only.
Also add validation rejecting any credential-ref flag not applicable to the
chosen --source, with a pointer to the correct flag, closing the silent-drop
class for all connectors.
Update CLI-reference docs, the ktx skill Notion example, and tests.
Fixes KLO-724.
- skills/ktx/SKILL.md: add an "Add context sources" section with the generic
`ktx setup --source ...` flags per connector (dbt, Metabase, Notion, ...),
warehouse mapping, the --metabase-database-id discovery note, and the
`ktx ingest` follow-up. The skill previously only documented database setup
with --skip-sources, so agents couldn't wire up dbt/Metabase/Notion (KLO-723).
- docs-site quickstart: the kaelio.com/start callout now points at the
"copy agent setup" one-shot prompt that installs the full four-source demo.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geist Mono fuses `--` into an em-dash glyph that visually swallows the
adjacent space, so prompts like `npx skills add Kaelio/ktx --skill ktx`
rendered as `Kaelio/ktx--skill ktx` on the quickstart page. The existing
ligature-off rule only covered <code>/<pre> and the .ktx-code wrapper —
quickstart.mdx puts the prompt in a plain <div className="font-mono">,
so the rule didn't apply. Extend the selector to also match the
.font-mono Tailwind utility and any inline-style opt-in via the mono
font CSS variable.
Document the convention in AGENTS.md so future docs additions keep
ligatures off on any new monospace container.
Add a host-scoped redirect for /slack on ktx.sh before the existing
catch-all so the path resolves to the community invite link instead of
docs.kaelio.com/ktx/slack.
* feat(cli): define full warehouse dialect contract
* test(cli): keep dialect edge tests focused
* fix(cli): stabilize dialect contract foundation
* refactor(connectors): own read-only query preparation
* refactor(connectors): resolve dialects through registry
* refactor(connectors): keep concrete dialect classes internal
* chore(workspace): enforce dialect import boundary
* refactor(cli): resolve relationship dialect at scan boundary
* refactor(cli): use dialect display parsing for entity details
* refactor(cli): use dialect display parsing for warehouse catalog
* refactor(cli): use dialect SQL in relationship workflows
* test(cli): verify solid dialect scan workflow closure
* test: split cli tests from source tree
* refactor(cli): standardize BigQuery scope listing
* feat(sqlite): implement connector scope listing
* test(connectors): cover required table listing
* feat(cli): add warehouse driver registry
* refactor(setup): route scope discovery through driver registry
* refactor(cli): route local query execution through driver registry
* refactor(historic-sql): route dialect support through driver registry
* refactor(cli): test warehouse connections through driver registry
* fix(cli): close driver registry type export gaps
* Improve setup daemon diagnostics
* refactor(setup): centralize rail-prefixed diagnostics + query-history fallback
Extract errorMessage, writePrefixedLines, and flushPrefixedBufferedCommandOutput
into clack.ts so the setup wizard, managed daemons, and embedding/agent steps
share one rail-formatted writer. setup-databases.ts also adds a
"disable query history and retry" option when the schema-context build fails
and query history is the likely culprit, surfaced via a new
failed-query-history-unavailable status.
* fix(cli): carry catalog through the picker so BigQuery/Snowflake/SQL Server scope filters match
The setup picker's KtxTableListEntry was a 2-level { schema, name }, so
qualifiedTableId always wrote db.name into enabled_tables. When BigQuery,
Snowflake, or SQL Server later ran fast ingest, their introspect step filtered
the scope set with scopedTableNames(scope, { catalog: projectId|database, db })
— catalog was non-null on the introspect side but null in the scope refs, so
every entry was rejected, the live-database adapter staged zero table files,
and detect() failed with 'Adapter "live-database" did not recognize fetched
source output'.
Align the picker boundary with the canonical 3-level KtxTableRef:
- Add catalog: string | null to KtxTableListEntry.
- BigQuery/Snowflake/SQL Server listTables populate catalog from the
resolved projectId / database; Postgres/MySQL/ClickHouse/SQLite set null.
- qualifiedTableId emits catalog.schema.name when catalog is non-null
(resolveEnabledTables already accepts the 3-part shape) and
schemasFromEnabledTables now goes through parseDottedTableEntry so it
recovers the schema correctly from both 2-part and 3-part entries.
- Export parseDottedTableEntry from enabled-tables.ts (@internal) for picker
reuse.
Update listTables expectations in all seven connector tests and the setup /
picker test fixtures. Add a picker regression test that covers the
catalog-bearing round-trip (save + refine).
* fix(cli): allow debug telemetry under opt-out env
* feat(cli): add 'skip context sources' option to database setup menu
After databases are configured, the post-setup menu now offers a 'Skip
context sources' choice equivalent to passing --skip-sources, which
plumbs through KtxSetupDatabasesResult.skipSources to bypass the
context-source step in the same run.
* feat(cli): standardize tree picker UX after clack autocomplete-multiselect
Search is always on (no '/' to enter): typed printable chars feed the
query, Tab toggles selection on the focused node without leaving the
search bar, and Space toggles only after arrow-key navigation
(isNavigating); otherwise it is appended to the query. Esc clears a
non-empty query before quitting, Ctrl+A and Ctrl+N replace bare-letter
bulk bindings, and the cursor refocuses on the first match when the
query change would hide it.
* feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure
Snowflake setup previously asked for a single schema as free text, then
ran a multiselect against the discovered schemas — two schema questions
back-to-back, with the first being only a session bootstrap. The SDK's
`schema` is optional, so the bootstrap step is unnecessary.
- Remove the free-text Snowflake schema prompt; only pass `schema` to
snowflake-sdk when one is configured.
- When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the
user for a comma-separated list, persist it as `schema_names`, and use
it as both the table-list filter and the multiselect default. Applies
to every driver with a scope-discovery spec, not just Snowflake.
- Update docs to lead with `schema_names`; keep `schema_name` as a
documented single-schema shorthand.
* fix(snowflake): keep introspecting when primary-key discovery is denied
The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and
INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the
connection role may not have. Previously a 'SQL compilation error:
Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist
or not authorized' aborted the entire introspect — schemas, columns,
and row counts were all discarded over a missing nice-to-have.
Wrap the constraint query in try/catch, log a one-line warning per
schema, and return an empty PK map. Columns end up with
primaryKey=false; relationship inference still has FK and profiling
to fall back on.
* fix(scan): unblock relationship discovery on Snowflake
Two adjacent bugs prevented the scan's relationship pipeline from producing
any joins on a Snowflake warehouse:
- relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch
for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table
profile query failed with "Unknown function GROUP_CONCAT". Add an explicit
Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter
(Snowflake requires the delimiter to be a constant, so CHR(31) is rejected).
- description-generation.ts destructured `connector.sampleTable` and
`connector.sampleColumn` into bare locals, losing the `this` binding when
the class-method connectors (Snowflake, Postgres, MySQL) were invoked.
Every sample call threw "Cannot read properties of undefined (reading
'assertConnection')" and degraded LLM descriptions to metadata-only
prompts. Call the methods through the connector instead.
Without these, even after the primary-key probe is allowed to fail softly,
the scan ends up with 0 validated relationships and an empty `joins:` block
in every shard YAML.
* test(scan): cover table-ref helpers
* feat(scan): plumb tableScope through live-database introspection port
* feat(scan): apply tableScope during metadata fetch
* feat(scan): enforce table scope at fetch boundary
* feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206)
* feat(cli): add RSA key-pair auth option to Snowflake setup wizard
Extends the interactive Snowflake setup flow with an authentication-method
prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key
path (env/file/absolute) and an optional passphrase; the resulting connection
config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead
of `password`.
* feat(scan): pool Snowflake sessions
* fix(scan): reuse structural snapshots and cleanup connectors
* feat(scan): parallelize relationship profiling
* feat(scan): batch table description generation
* docs: document Snowflake ingest concurrency knobs
* fix(scan): close Snowflake ingest perf verification gaps
* fix(scan): keep batched description failure bounded
* feat(scan): dispatch query-history probes by connection driver
Extract historic-sql dialect resolution into a shared helper so the
status-project readiness check and the local ingest factory agree on
which connections enable query history and which probe to run. The
status command now picks the postgres/snowflake/bigquery probe based on
the connection's driver instead of always reporting against postgres,
which previously caused snowflake connections with queryHistory.enabled
to surface a misleading "driver is snowflake" failure.
Also drops a noisy console.warn from Snowflake primary-key discovery —
INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only
roles and the FK + profiling paths handle the empty PK map already.
* fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject
The Claude Code agent SDK announces an internal pseudo-tool named
StructuredOutput in the system/init message whenever outputFormat is set
to { type: 'json_schema' }. The runtime's isolation check built its
allowedToolIds set only from MCP tool ids and treated StructuredOutput
as an unexpected host-injected tool, so every generateObject call threw
"Claude Code runtime isolation failed: tools=StructuredOutput ..." and
the table-descriptions and relationship-LLM-proposal enrichment stages
recorded null output across the board.
Whitelist StructuredOutput specifically in generateObject's
allowedToolIds — the check also enforces missing_tools symmetry, so
generateText and runAgentLoop, which do not see StructuredOutput, must
not require it.
generateObject also ran with maxTurns: 1, which the model intermittently
breached when it emitted thinking text before the structured response.
Raised to 5 to give the schema-bound call enough headroom without
allowing unbounded loops. The existing tests now exercise the path with
an init message that announces StructuredOutput so the regression cannot
slip back in.
* chore(scripts): add ktx-reset.sh project-cleanup helper
Convenience script for repeatable ingest testing: takes a project
directory and prunes everything except ktx.yaml and .ktx/secrets/, so
the next ktx setup or ktx ingest run starts from a known-clean state.
* docs(quickstart): redesign demo-warehouse callout with sticker icons
Replaces the plain warning-style callout with a two-column layout: text
and a pill-shaped CTA on the left, a 2x2 cluster of rotated Postgres,
Metabase, dbt, and Notion sticker tiles on the right. Adds the four
connector SVGs under docs-site/public/icons/ to support it.
* chore(docs-site): refresh auto-generated next-env.d.ts
* docs: rewrite context-as-code as reviewing-context guide
Move the page from Concepts to Guides and rebuild around an interactive
review-loop diagram. Extract pan/zoom + fit-view controls into a shared
FlowCanvas wrapper and adopt it across all three docs diagrams.
* test: point examples-docs assertion at reviewing-context
Update the doc smoke test that read context-as-code.mdx to read the new
guides/reviewing-context.mdx path. The `ktx ingest --all --no-input`
assertion still holds; the rename was the only break.
Adds a new Configuration section to the docs with a reference page that
covers every top-level block of ktx.yaml: connections, setup, storage,
llm, ingest, scan, agent, and memory. Each block lists fields, defaults,
accepted values, and a short YAML example, with a leading schematic that
groups blocks into inputs, compute, and persistence.
