ktx/packages/cli/src/status-project.ts

1345 lines
46 KiB
TypeScript
Raw Normal View History

import { stat as statAsync, readdir as readdirAsync } from 'node:fs/promises';
import { basename, join } from 'node:path';
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
import { runClaudeCodeAuthProbe } from './context/llm/claude-code-runtime.js';
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
import {
CODEX_ISOLATION_WARNING,
CODEX_ISOLATION_WARNING_FIX,
} from './context/llm/codex-isolation.js';
import { runCodexAuthProbe } from './context/llm/codex-runtime.js';
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
import type { KtxConfigIssue, KtxProjectConfig, KtxProjectConnectionConfig, KtxProjectEmbeddingConfig, KtxProjectLlmConfig } from './context/project/config.js';
import type { KtxLocalProject } from './context/project/project.js';
import { ktxLocalStateDbPath } from './context/project/local-state-db.js';
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
import { listReferencedConnectionIds } from './context/wiki/local-knowledge.js';
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
import {
isQueryHistoryEnabled,
queryHistoryDialectForConnection,
} from './context/ingest/adapters/historic-sql/connection-dialect.js';
import {
historicSqlProbeCatalogName,
runHistoricSqlReadinessProbe,
type HistoricSqlReadinessProbe,
} from './context/ingest/historic-sql-probes.js';
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
import {
formatClaudeCodePromptCachingFix,
formatClaudeCodePromptCachingWarning,
ignoredClaudeCodePromptCachingFields,
} from './claude-code-prompt-caching.js';
import type { DoctorCheck } from './doctor.js';
import {
bold as _bold,
dim as _dim,
green,
red,
yellow,
} from './io/symbols.js';
import { KTX_NEXT_STEP_DIRECT_COMMANDS } from './next-steps.js';
type ProjectStatusLevel = 'ok' | 'warn' | 'fail' | 'skipped';
type ProjectVerdict = 'ready' | 'partial' | 'blocked';
interface ProjectStatusLine {
status: ProjectStatusLevel;
detail: string;
fix?: string;
}
interface LlmStatus extends ProjectStatusLine {
backend: string;
model?: string;
}
interface EmbeddingsStatus extends ProjectStatusLine {
backend: string;
model?: string;
dimensions?: number;
}
interface ConnectionStatus extends ProjectStatusLine {
name: string;
driver: string;
}
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
interface QueryHistoryStatus extends ProjectStatusLine {
connection: string;
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
driver: string;
dialect: string;
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
}
interface PipelineStatus {
adapters: string[];
enrichmentMode: string;
relationshipsEnabled: boolean;
relationshipsLlmProposals: boolean;
relationshipsValidationRequired: boolean;
agentEnabled: boolean;
agentTools: string[];
agentMaxIterations: number;
}
interface StorageStatus {
state: string;
search: string;
gitAuthor: string;
}
interface ConfigStatus {
status: ProjectStatusLevel;
detail: string;
issues: KtxConfigIssue[];
}
interface WarningItem {
message: string;
fix?: string;
}
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
type ClaudeCodeAuthProbe = (input: {
projectDir: string;
model: string;
env?: NodeJS.ProcessEnv;
}) => Promise<{ ok: true } | { ok: false; message: string }>;
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
type CodexAuthProbe = (input: {
projectDir: string;
model: string;
}) => Promise<{ ok: true } | { ok: false; message: string; fix: string }>;
const PROJECT_READY_COMMANDS = KTX_NEXT_STEP_DIRECT_COMMANDS.map((step) => step.command);
interface LocalStatsIngestPerConnection {
connectionId: string;
adapter: string;
lastCompletedAt: string;
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
skippedObjects: Array<{ name: string; reason: string }>;
}
interface LocalStatsSemanticLayerEntry {
connectionId: string;
sourceCount: number;
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
embeddedSourceCount: number;
dictionaryValueCount: number;
}
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
interface LocalStatsWikiEntry {
scope: string;
count: number;
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
embeddedCount: number;
}
interface LocalStatsProjectDir {
dbSqliteBytes: number | null;
ktxCacheBytes: number;
rawSources: { fileCount: number; bytes: number };
}
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
/** @internal */
export interface LocalStatsStatus {
ingest: {
totalCompletedRuns: number;
perConnection: LocalStatsIngestPerConnection[];
};
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
wikiPages: LocalStatsWikiEntry[];
semanticLayer: LocalStatsSemanticLayerEntry[];
projectDir: LocalStatsProjectDir;
unavailable?: string;
}
export interface ProjectStatus {
projectName: string;
projectDir: string;
config: ConfigStatus;
llm: LlmStatus;
embeddings: EmbeddingsStatus;
storage: StorageStatus;
connections: ConnectionStatus[];
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
queryHistory: QueryHistoryStatus[];
pipeline: PipelineStatus;
warnings: WarningItem[];
localStats: LocalStatsStatus;
verdict: ProjectVerdict;
verdictReason: string;
nextActions: string[];
promptCaching?: { enabled: boolean; systemTtl?: string; toolsTtl?: string; historyTtl?: string };
workUnits?: { stepBudget: number; maxConcurrency: number; failureMode: string };
relationshipsDetail?: {
acceptThreshold: number;
reviewThreshold: number;
maxLlmTablesPerBatch: number;
validationConcurrency: number;
};
}
function resolveRef(value: unknown, env: NodeJS.ProcessEnv): { resolved: string; via: 'literal' | 'env' | 'file' | 'missing' } {
if (typeof value !== 'string') return { resolved: '', via: 'missing' };
const trimmed = value.trim();
if (trimmed.length === 0) return { resolved: '', via: 'missing' };
if (trimmed.startsWith('env:')) {
const name = trimmed.slice(4).trim();
const v = env[name];
return v && v.trim().length > 0 ? { resolved: v, via: 'env' } : { resolved: '', via: 'missing' };
}
if (trimmed.startsWith('file:')) {
return { resolved: trimmed.slice(5), via: 'file' };
}
return { resolved: trimmed, via: 'literal' };
}
function failureDetail(error: unknown): string {
if (error instanceof Error && error.message.trim().length > 0) {
return error.message.trim().split('\n')[0] ?? error.message.trim();
}
return String(error);
}
function envHint(value: unknown): string | undefined {
if (typeof value === 'string' && value.trim().startsWith('env:')) {
return value.trim().slice(4).trim();
}
return undefined;
}
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
async function buildLlmStatus(
config: KtxProjectLlmConfig,
options: {
projectDir: string;
env: NodeJS.ProcessEnv;
claudeCodeAuthProbe?: ClaudeCodeAuthProbe;
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
codexAuthProbe?: CodexAuthProbe;
fast?: boolean;
useSpinner?: boolean;
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
},
): Promise<LlmStatus> {
const env = options.env;
const backend = config.provider.backend;
const model = config.models?.default;
if (backend === 'none') {
return {
backend,
model,
status: 'fail',
detail: 'no LLM configured; research agent will not run',
fix: 'Run: ktx setup (choose an LLM provider)',
};
}
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
// The runtime (resolveModelSlots) hard-requires llm.models.default for every
// non-none backend; without it ingest/scan/memory throw. Report that here so
// status never marks a project ready that the runtime would refuse to run.
if (!model || model.trim().length === 0) {
return {
backend,
model,
status: 'fail',
detail: `llm.models.default is required for backend "${backend}"`,
fix: 'Set llm.models.default in ktx.yaml, then rerun `ktx status` (or rerun `ktx setup`).',
};
}
if (backend === 'anthropic') {
const ref = config.provider.anthropic?.api_key;
const resolved = resolveRef(ref, env);
if (resolved.resolved.length > 0) {
return { backend, model, status: 'ok', detail: `key set${resolved.via === 'env' ? ` (env)` : ''}` };
}
if (env.ANTHROPIC_API_KEY && env.ANTHROPIC_API_KEY.trim().length > 0) {
return { backend, model, status: 'ok', detail: 'key set (env: ANTHROPIC_API_KEY)' };
}
const hint = envHint(ref);
return {
backend,
model,
status: 'warn',
detail: hint ? `key missing (env: ${hint})` : 'key missing',
fix: hint ? `Set ${hint}` : 'Set ANTHROPIC_API_KEY or rerun `ktx setup`',
};
}
if (backend === 'vertex') {
const project = config.provider.vertex?.project;
if (project && project.length > 0) {
return { backend, model, status: 'ok', detail: `project=${project}` };
}
return { backend, model, status: 'warn', detail: 'vertex project not configured', fix: 'Rerun `ktx setup`' };
}
if (backend === 'gateway') {
const ref = config.provider.gateway?.api_key;
const resolved = resolveRef(ref, env);
if (resolved.resolved.length > 0) {
return { backend, model, status: 'ok', detail: 'key set' };
}
const hint = envHint(ref);
return {
backend,
model,
status: 'warn',
detail: hint ? `key missing (env: ${hint})` : 'key missing',
fix: hint ? `Set ${hint}` : 'Set the gateway api_key or rerun `ktx setup`',
};
}
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
if (backend === 'claude-code') {
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
const modelName = model;
if (options.fast === true) {
return {
backend,
model: modelName,
status: 'skipped',
detail: 'auth probe skipped (--fast)',
};
}
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
const probe = options.claudeCodeAuthProbe ?? runClaudeCodeAuthProbe;
const auth = await withSpinner(options.useSpinner === true, 'Probing Claude Code authentication', () =>
probe({ projectDir: options.projectDir, model: modelName, env }),
);
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
if (auth.ok) {
return {
backend,
model: modelName,
status: 'ok',
detail: 'local Claude Code session authenticated',
};
}
return {
backend,
model: modelName,
status: 'fail',
detail: auth.message,
fix: 'Authenticate Claude Code locally with the Claude Code CLI, then rerun `ktx status`.',
};
}
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
if (backend === 'codex') {
const modelName = model;
if (options.fast === true) {
return {
backend,
model: modelName,
status: 'skipped',
detail: 'auth probe skipped (--fast)',
};
}
const probe = options.codexAuthProbe ?? runCodexAuthProbe;
const auth = await withSpinner(options.useSpinner === true, 'Probing Codex authentication', () =>
probe({ projectDir: options.projectDir, model: modelName }),
);
if (auth.ok) {
return {
backend,
model: modelName,
status: 'ok',
detail: 'local Codex session authenticated',
};
}
return {
backend,
model: modelName,
status: 'fail',
detail: auth.message,
fix: auth.fix,
};
}
return { backend, model, status: 'warn', detail: 'unknown LLM backend' };
}
function buildEmbeddingsStatus(config: KtxProjectEmbeddingConfig, env: NodeJS.ProcessEnv): EmbeddingsStatus {
const backend = config.backend;
const model = config.model;
const dimensions = config.dimensions;
if (backend === 'none') {
return {
backend,
model,
dimensions,
status: 'warn',
detail: 'disabled — semantic search will be skipped',
};
}
if (backend === 'openai') {
const ref = config.openai?.api_key;
const resolved = resolveRef(ref, env);
if (resolved.resolved.length > 0 || (env.OPENAI_API_KEY && env.OPENAI_API_KEY.trim().length > 0)) {
return { backend, model, dimensions, status: 'ok', detail: 'key set' };
}
const hint = envHint(ref);
return {
backend,
model,
dimensions,
status: 'warn',
detail: hint ? `key missing (env: ${hint})` : 'key missing',
fix: hint ? `Set ${hint}` : 'Set OPENAI_API_KEY or rerun `ktx setup`',
};
}
if (backend === 'sentence-transformers') {
const url = config.sentenceTransformers?.base_url;
if (typeof url === 'string' && url.length > 0) {
return { backend, model, dimensions, status: 'ok', detail: `service: ${url}` };
}
return {
backend,
model,
dimensions,
status: 'ok',
detail: 'managed local embeddings daemon',
};
}
return { backend, model, dimensions, status: 'warn', detail: 'unknown embedding backend' };
}
function buildConnectionStatus(
name: string,
conn: KtxProjectConnectionConfig,
env: NodeJS.ProcessEnv,
): ConnectionStatus {
const driver = (conn.driver ?? 'unknown').toLowerCase();
const ok = (detail: string): ConnectionStatus => ({ name, driver, status: 'ok', detail });
const warn = (detail: string, fix?: string): ConnectionStatus => ({ name, driver, status: 'warn', detail, fix });
switch (driver) {
case 'postgres':
case 'mysql':
case 'clickhouse':
case 'sqlserver': {
const urlRef = resolveRef(conn.url, env);
if (urlRef.resolved.length > 0) return ok(`url configured`);
if (typeof (conn as Record<string, unknown>).host === 'string') return ok('host configured');
const hint = envHint(conn.url);
return warn(hint ? `url missing (env: ${hint})` : 'url not set', hint ? `Set ${hint}` : 'Rerun `ktx setup`');
}
case 'snowflake': {
const account = (conn as Record<string, unknown>).account;
if (typeof account === 'string' && account.length > 0) return ok(`account: ${account}`);
return warn('account not set', 'Rerun `ktx setup`');
}
case 'bigquery': {
const cred = resolveRef((conn as Record<string, unknown>).credentials_json, env);
if (cred.resolved.length > 0) return ok('credentials configured');
const hint = envHint((conn as Record<string, unknown>).credentials_json);
return warn(hint ? `credentials missing (env: ${hint})` : 'credentials not set', hint ? `Set ${hint}` : 'Rerun `ktx setup`');
}
case 'sqlite': {
const path = (conn as Record<string, unknown>).path;
if (typeof path === 'string' && path.length > 0) return ok(`path: ${path}`);
return warn('path not set', 'Rerun `ktx setup`');
}
case 'notion': {
const tokenRef =
(conn as Record<string, unknown>).auth_token_ref ??
(conn as Record<string, unknown>).auth_token;
const resolved = resolveRef(tokenRef, env);
if (resolved.resolved.length > 0) return ok('auth token configured');
const hint = envHint(tokenRef);
return warn(hint ? `auth token missing (env: ${hint})` : 'auth token not set', hint ? `Set ${hint}` : 'Rerun `ktx setup`');
}
case 'dbt':
case 'dbt-core':
case 'dbt-cloud': {
const repoUrl =
(conn as Record<string, unknown>).repoUrl ??
(conn as Record<string, unknown>).repo_url;
if (typeof repoUrl === 'string' && repoUrl.length > 0) return ok(`repo: ${repoUrl}`);
return warn('repoUrl not set', 'Rerun `ktx setup`');
}
case 'metabase': {
const url = (conn as Record<string, unknown>).api_url;
if (typeof url === 'string' && url.length > 0) return ok(`url: ${url}`);
return warn('api_url not set', 'Rerun `ktx setup`');
}
case 'looker':
case 'lookml': {
const url = (conn as Record<string, unknown>).base_url ?? (conn as Record<string, unknown>).url;
if (typeof url === 'string' && url.length > 0) return ok(`url: ${url}`);
return warn('base_url not set', 'Rerun `ktx setup`');
}
case 'metricflow': {
const repoUrl = (conn as Record<string, unknown>).repoUrl ?? (conn as Record<string, unknown>).repo_url;
if (typeof repoUrl === 'string' && repoUrl.length > 0) return ok(`repo: ${repoUrl}`);
return warn('repoUrl not set', 'Rerun `ktx setup`');
}
default:
return { name, driver, status: 'ok', detail: 'configured' };
}
}
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
async function buildQueryHistoryStatus(
project: KtxLocalProject,
options: BuildProjectStatusOptions,
): Promise<QueryHistoryStatus[]> {
const targets = Object.entries(project.config.connections)
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
.filter(([, connection]) => isQueryHistoryEnabled(connection))
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
.sort(([left], [right]) => left.localeCompare(right));
const probe = options.queryHistoryReadinessProbe ?? runHistoricSqlReadinessProbe;
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
const env = options.env ?? process.env;
const statuses: QueryHistoryStatus[] = [];
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
for (const [connectionId, connection] of targets) {
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
const driver = String(connection.driver ?? 'unknown').toLowerCase();
const dialect = queryHistoryDialectForConnection(connection);
if (!dialect) {
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
statuses.push({
connection: connectionId,
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
driver,
dialect: driver,
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
status: 'fail',
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
detail: `query history is not supported for driver "${driver}"`,
fix: `Disable connections.${connectionId}.context.queryHistory, or use a postgres, snowflake, or bigquery connection`,
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
});
continue;
}
const catalogName = historicSqlProbeCatalogName(dialect);
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
if (options.fast === true) {
statuses.push({
connection: connectionId,
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
driver,
dialect,
status: 'skipped',
detail: `${catalogName} probe skipped (--fast)`,
});
continue;
}
const outcome = await withSpinner(
options.useSpinner === true,
`Probing ${catalogName} on ${connectionId}`,
() =>
probe({
projectDir: project.projectDir,
connectionId,
connection,
env,
}),
);
if (!outcome) {
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
statuses.push({
connection: connectionId,
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
driver,
dialect: driver,
status: 'fail',
detail: `query history is not supported for driver "${driver}"`,
fix: `Disable connections.${connectionId}.context.queryHistory, or use a postgres, snowflake, or bigquery connection`,
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
});
continue;
}
if (outcome.ok) {
const { detail, warnings } = outcome.runner.formatSuccessDetail(outcome.result);
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
statuses.push({
connection: connectionId,
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204) * feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
2026-05-23 10:41:30 +02:00
driver,
dialect,
status: warnings.length > 0 ? 'warn' : 'ok',
detail,
...(dialect === 'postgres' && warnings.length > 0
? {
fix: `Update the Postgres parameter group or config, then rerun \`ktx status --project-dir ${project.projectDir}\``,
}
: {}),
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
});
continue;
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
}
const advice = outcome.runner.fixAdvice(outcome.error);
statuses.push({
connection: connectionId,
driver,
dialect,
status: 'fail',
detail: advice.failHeadline,
fix: advice.remediation,
});
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
}
return statuses;
}
const ADAPTER_DRIVER_REQUIREMENT: Record<string, string[]> = {
'live-database': ['postgres', 'mysql', 'snowflake', 'bigquery', 'clickhouse', 'sqlite', 'sqlserver'],
dbt: ['dbt', 'dbt-core', 'dbt-cloud'],
notion: ['notion'],
metabase: ['metabase'],
looker: ['looker', 'lookml'],
lookml: ['looker', 'lookml'],
metricflow: ['metricflow'],
};
function buildPipelineStatus(config: KtxProjectConfig): PipelineStatus {
return {
adapters: config.ingest.adapters,
enrichmentMode: config.scan.enrichment.mode,
relationshipsEnabled: config.scan.relationships.enabled,
relationshipsLlmProposals: config.scan.relationships.llmProposals,
relationshipsValidationRequired: config.scan.relationships.validationRequiredForManifest,
agentEnabled: config.agent.run_research.enabled,
agentTools: config.agent.run_research.default_toolset,
agentMaxIterations: config.agent.run_research.max_iterations,
};
}
function buildStorageStatus(config: KtxProjectConfig): StorageStatus {
return {
state: config.storage.state,
search: config.storage.search,
gitAuthor: config.storage.git.author,
};
}
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
/**
* Warn (never fail) when stored wiki pages reference connection ids that are no
* longer in `ktx.yaml`. Config and page content evolve independently, so a
* dangling reference is a soft condition the pages still load, search, and
* read; it just signals a typo or a removed connection.
*/
async function buildUnknownConnectionWarning(project: KtxLocalProject): Promise<WarningItem | null> {
let referenced: string[];
try {
referenced = await listReferencedConnectionIds(project);
} catch {
return null;
}
const unknown = referenced.filter((id) => !Object.hasOwn(project.config.connections, id));
if (unknown.length === 0) {
return null;
}
return {
message: `Wiki pages reference connection id(s) not in ktx.yaml: ${unknown.join(', ')}. Those pages still load and search.`,
fix: 'Add the connection(s) via `ktx setup`, or update the pages `connections` frontmatter.',
};
}
function buildWarnings(
config: KtxProjectConfig,
connections: ConnectionStatus[],
llm: LlmStatus,
embeddings: EmbeddingsStatus,
): WarningItem[] {
const warnings: WarningItem[] = [];
for (const adapter of config.ingest.adapters) {
const requiredDrivers = ADAPTER_DRIVER_REQUIREMENT[adapter];
if (!requiredDrivers) continue;
const hasMatching = connections.some((c) => requiredDrivers.includes(c.driver));
if (!hasMatching) {
warnings.push({
message: `Adapter "${adapter}" is enabled but no connection of type ${requiredDrivers.slice(0, 2).join('/')} is configured.`,
fix: 'Rerun `ktx setup` to add a connection, or remove the adapter from ingest.adapters.',
});
}
}
if (config.agent.run_research.enabled && llm.backend === 'none') {
warnings.push({
message: 'Research agent is enabled but LLM is not configured.',
fix: 'Set up an LLM provider via `ktx setup` or disable agent.run_research.enabled.',
});
}
if (embeddings.backend === 'none' && config.ingest.adapters.includes('live-database')) {
warnings.push({
message: 'Semantic search is off (embeddings backend = none). Lexical/dictionary lanes still work.',
});
}
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
const warning = formatClaudeCodePromptCachingWarning(ignoredClaudeCodePromptCachingFields(config.llm));
if (warning) {
warnings.push({
message: warning,
fix: formatClaudeCodePromptCachingFix(),
});
}
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
if (llm.backend === 'codex') {
warnings.push({
message: CODEX_ISOLATION_WARNING,
fix: CODEX_ISOLATION_WARNING_FIX,
});
}
return warnings;
}
function buildVerdict(
llm: LlmStatus,
embeddings: EmbeddingsStatus,
connections: ConnectionStatus[],
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
queryHistory: QueryHistoryStatus[],
warnings: WarningItem[],
): { verdict: ProjectVerdict; reason: string; nextActions: string[] } {
if (llm.status === 'fail') {
return {
verdict: 'blocked',
reason: 'LLM not configured; research agent will not run.',
nextActions: ['ktx setup'],
};
}
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
const failedQueryHistory = queryHistory.filter((entry) => entry.status === 'fail').length;
if (failedQueryHistory > 0) {
return {
verdict: 'blocked',
reason: `Query history readiness failed for ${failedQueryHistory} connection${failedQueryHistory === 1 ? '' : 's'}.`,
nextActions: ['ktx status --verbose'],
};
}
const reasons: string[] = [];
if (llm.status === 'warn') reasons.push('LLM credentials missing');
if (embeddings.status === 'warn') {
if (embeddings.backend === 'none') {
reasons.push('semantic search disabled');
} else {
reasons.push('embedding credentials missing');
}
}
const missing = connections.filter((c) => c.status !== 'ok' && c.status !== 'skipped').length;
if (missing > 0) reasons.push(`${missing} connection${missing === 1 ? '' : 's'} need configuration`);
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
const queryHistoryWarnings = queryHistory.filter((entry) => entry.status === 'warn').length;
if (queryHistoryWarnings > 0) {
reasons.push(`${queryHistoryWarnings} query history warning${queryHistoryWarnings === 1 ? '' : 's'}`);
}
if (warnings.length > 0) reasons.push(`${warnings.length} config warning${warnings.length === 1 ? '' : 's'}`);
if (reasons.length === 0) {
return {
verdict: 'ready',
reason: 'Ready.',
nextActions: [...PROJECT_READY_COMMANDS],
};
}
return {
verdict: 'partial',
reason: `Partially ready — ${reasons.join('; ')}.`,
nextActions: ['ktx setup'],
};
}
export interface BuildProjectStatusOptions {
env?: NodeJS.ProcessEnv;
queryHistoryReadinessProbe?: HistoricSqlReadinessProbe;
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
claudeCodeAuthProbe?: ClaudeCodeAuthProbe;
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
codexAuthProbe?: CodexAuthProbe;
configIssues?: KtxConfigIssue[];
fast?: boolean;
useSpinner?: boolean;
}
async function withSpinner<T>(
useSpinner: boolean,
label: string,
run: () => Promise<T>,
): Promise<T> {
if (!useSpinner) return run();
const { spinner } = await import('@clack/prompts');
const s = spinner();
s.start(label);
try {
const result = await run();
s.stop(label);
return result;
} catch (error) {
s.stop(`${label} — failed`);
throw error;
}
}
function buildConfigStatus(issues: KtxConfigIssue[] | undefined): ConfigStatus {
const list = issues ?? [];
if (list.length === 0) {
return { status: 'ok', detail: 'ktx.yaml schema valid', issues: [] };
}
// Error-severity issues never reach here — the doctor exits on them first.
return {
status: 'warn',
detail: `ktx.yaml schema valid · ${list.length} ignored field${list.length === 1 ? '' : 's'}`,
issues: list,
};
}
interface DirSummary {
fileCount: number;
bytes: number;
}
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
async function summarizeDir(dir: string, maxDepth = 10): Promise<DirSummary> {
let fileCount = 0;
let bytes = 0;
const walk = async (current: string, depth: number): Promise<void> => {
if (depth > maxDepth) return;
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
let entries;
try {
entries = await readdirAsync(current, { withFileTypes: true });
} catch {
return;
}
for (const entry of entries) {
const full = join(current, entry.name);
if (entry.isDirectory()) {
await walk(full, depth + 1);
continue;
}
if (!entry.isFile()) continue;
try {
const s = await statAsync(full);
fileCount += 1;
bytes += s.size;
} catch {
// skip individual stat failures
}
}
};
await walk(dir, 0);
return { fileCount, bytes };
}
async function fileSizeOrNull(filePath: string): Promise<number | null> {
try {
const s = await statAsync(filePath);
return s.isFile() ? s.size : null;
} catch {
return null;
}
}
function tryQuery<T>(run: () => T, fallback: T): T {
try {
return run();
} catch {
return fallback;
}
}
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
function skippedObjectsFromReportBody(bodyJson: string): Array<{ name: string; reason: string }> {
try {
const body = JSON.parse(bodyJson) as { fetch?: { skipped?: Array<{ entityId?: unknown; message?: unknown }> } };
const skipped = body.fetch?.skipped;
if (!Array.isArray(skipped)) return [];
return skipped.map((issue) => ({
name: typeof issue.entityId === 'string' && issue.entityId.length > 0 ? issue.entityId : 'object',
reason: typeof issue.message === 'string' ? issue.message : 'introspection failed',
}));
} catch {
return [];
}
}
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
/** @internal */
export async function buildLocalStatsStatus(project: KtxLocalProject): Promise<LocalStatsStatus> {
const dbPath = ktxLocalStateDbPath(project);
const dbSqliteBytes = await fileSizeOrNull(dbPath);
const projectDirSummary: LocalStatsProjectDir = {
dbSqliteBytes,
ktxCacheBytes: (await summarizeDir(join(project.projectDir, '.ktx', 'cache'))).bytes,
rawSources: await summarizeDir(join(project.projectDir, 'raw-sources')),
};
if (dbSqliteBytes === null) {
return {
ingest: { totalCompletedRuns: 0, perConnection: [] },
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
wikiPages: [],
semanticLayer: [],
projectDir: projectDirSummary,
unavailable: 'no .ktx/db.sqlite yet',
};
}
let database: import('better-sqlite3').Database | null = null;
try {
const { default: Database } = await import('better-sqlite3');
database = new Database(dbPath, { readonly: true, fileMustExist: true });
const db = database;
const totalCompletedRuns = tryQuery(
() =>
(
db
.prepare(`SELECT COUNT(*) AS n FROM local_ingest_reports WHERE status = 'done'`)
.get() as { n: number } | undefined
)?.n ?? 0,
0,
);
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
type IngestStatsRow = { connection_id: string; adapter: string; last_completed_at: string; body_json: string };
const ingestRows = tryQuery(
() =>
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
// SQLite returns body_json from the MAX(completed_at) row for each group.
db
.prepare(
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
`SELECT connection_id, adapter, MAX(completed_at) AS last_completed_at, body_json
FROM local_ingest_reports
WHERE status = 'done'
GROUP BY connection_id, adapter`,
)
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
.all() as IngestStatsRow[],
[] as IngestStatsRow[],
);
const perConnectionMap = new Map<string, LocalStatsIngestPerConnection>();
for (const row of ingestRows) {
const existing = perConnectionMap.get(row.connection_id);
if (!existing || row.last_completed_at > existing.lastCompletedAt) {
perConnectionMap.set(row.connection_id, {
connectionId: row.connection_id,
adapter: row.adapter,
lastCompletedAt: row.last_completed_at,
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
skippedObjects: skippedObjectsFromReportBody(row.body_json),
});
}
}
const perConnection = [...perConnectionMap.values()].sort((left, right) =>
left.connectionId.localeCompare(right.connectionId),
);
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
const wikiRows = tryQuery(
() =>
db
.prepare(
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
`SELECT scope, COUNT(*) AS n, SUM(CASE WHEN embedding_json IS NOT NULL THEN 1 ELSE 0 END) AS embedded
FROM knowledge_pages
GROUP BY scope
ORDER BY scope`,
)
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
.all() as Array<{ scope: string; n: number; embedded: number | null }>,
[] as Array<{ scope: string; n: number; embedded: number | null }>,
);
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
const wikiPages: LocalStatsWikiEntry[] = wikiRows.map((row) => ({
scope: row.scope,
count: row.n,
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
embeddedCount: row.embedded ?? 0,
}));
const sourceRows = tryQuery(
() =>
db
.prepare(
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
`SELECT connection_id, COUNT(*) AS n, SUM(CASE WHEN embedding_json IS NOT NULL THEN 1 ELSE 0 END) AS embedded
FROM local_sl_sources
GROUP BY connection_id`,
)
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
.all() as Array<{ connection_id: string; n: number; embedded: number | null }>,
[] as Array<{ connection_id: string; n: number; embedded: number | null }>,
);
const dictionaryRows = tryQuery(
() =>
db
.prepare(
`SELECT connection_id, COUNT(*) AS n FROM local_sl_dictionary_values GROUP BY connection_id`,
)
.all() as Array<{ connection_id: string; n: number }>,
[] as Array<{ connection_id: string; n: number }>,
);
const slMap = new Map<string, LocalStatsSemanticLayerEntry>();
for (const row of sourceRows) {
slMap.set(row.connection_id, {
connectionId: row.connection_id,
sourceCount: row.n,
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
embeddedSourceCount: row.embedded ?? 0,
dictionaryValueCount: 0,
});
}
for (const row of dictionaryRows) {
const existing = slMap.get(row.connection_id) ?? {
connectionId: row.connection_id,
sourceCount: 0,
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
embeddedSourceCount: 0,
dictionaryValueCount: 0,
};
existing.dictionaryValueCount = row.n;
slMap.set(row.connection_id, existing);
}
const semanticLayer = [...slMap.values()].sort((left, right) =>
left.connectionId.localeCompare(right.connectionId),
);
return {
ingest: { totalCompletedRuns, perConnection },
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
wikiPages,
semanticLayer,
projectDir: projectDirSummary,
};
} catch (error) {
return {
ingest: { totalCompletedRuns: 0, perConnection: [] },
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
wikiPages: [],
semanticLayer: [],
projectDir: projectDirSummary,
unavailable: failureDetail(error),
};
} finally {
if (database) {
try {
database.close();
} catch {
// ignore close failures
}
}
}
}
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
export async function buildProjectStatus(project: KtxLocalProject, options: BuildProjectStatusOptions = {}): Promise<ProjectStatus> {
const env = options.env ?? process.env;
const config = project.config;
const configStatus = buildConfigStatus(options.configIssues);
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
const llm = await buildLlmStatus(config.llm, {
projectDir: project.projectDir,
env,
claudeCodeAuthProbe: options.claudeCodeAuthProbe,
feat: add codex llm backend for ktx runtime work (#253) * feat: add codex sdk runner foundation * feat: parse codex runtime events * feat: expose codex runtime mcp tools * feat: add codex llm runtime * feat: wire codex llm backend * test: avoid Array.fromAsync in codex runner test * docs: document codex llm backend * fix: tighten codex runtime config ownership * fix: use codex sdk env and thread options * fix: parse codex sdk event shapes * test: add codex backend live smoke * docs: clarify codex backend isolation * fix: drive codex loop metrics from mcp events * fix: enforce codex local step budget * docs: disclose codex isolation limits * fix: count all codex agent steps and stream step callbacks live The agent-loop step budget only counted completed mcp_tool_call items, so built-in command_execution steps (which the public Codex SDK/CLI surface can still expose) never decremented the budget, letting ingest/reconciliation run past stepBudget until Codex stopped on its own. onStepFinish was also replayed only after the whole stream drained, so live work_unit_step / reconciliation progress appeared stuck until the Codex process exited. collectEvents is now the single live step accumulator: it counts every completed agent-action item via a shared isCompletedAgentStep predicate (command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish as each step completes, and enforces the budget on that broader count. A no-tool turn still counts as one step. toolFailures stays MCP-specific, since a non-zero command exit is normal agent exploration, not a loop failure. * test: align ingest llm-guard assertions with codex backend The skip-llm ingest guard message now lists codex as a valid backend and mentions a Claude Code/Codex session plus a codex setup hint, but this slow suite test still asserted the pre-codex wording. Update it to match the production message (already covered by the local-bundle-runtime unit test) and add the codex setup-line assertion. * fix: treat codex error:null tool calls as success The Codex SDK serializes error: null on successful mcp_tool_call items, so the failure check (item.error !== undefined) flagged every successful tool call as failed with the empty-payload default "Codex turn failed". This killed every ingest work unit under the codex backend before it could produce a patch. Key on status === 'failed' (authoritative, always set) and only treat a populated error object as a failure. Add a regression test built from a verbatim real-SDK event capture. * fix: default codex backend to gpt-5.5 and report real probe errors The previous default gpt-5.3-codex is an API-key-only model that the OpenAI API rejects under ChatGPT-account (subscription) auth, so codex status/setup failed with a misleading "authentication is not usable" message even though auth was fine. - Default codex model is now gpt-5.5 (works on both subscription and API-key auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark). - runCodexAuthProbe now distinguishes "model not available" from an auth failure and surfaces the real API error: collectEvents retains stream events when the SDK throws on a non-zero exit, and the API error JSON envelope is unwrapped to its human-readable message. - The Codex isolation warning now renders inside the clack setup frame. - Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth. * fix: require llm.models.default in status and match codex probe remediation Status reported a project ready when a non-none LLM backend was configured without llm.models.default, but the runtime (resolveModelSlots) hard-requires it, so ingest/scan/memory threw after `ktx status` said the project was usable. buildLlmStatus now fails for any non-none backend missing models.default and no longer invents a fallback model for claude-code/codex. Codex probe failures now carry a category-matched fix: a model-access failure steers the user at llm.models.default instead of the auth/install remediation. runCodexAuthProbe returns the fix and status consumes it; the message stays self-sufficient so setup output is unchanged. Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx states --llm-model only accepts codex/default or gpt-*/codex-* ids. Repaired four doctor fixtures that configured a backend without models.default (the now-correctly-blocked config) and added coverage for the new behavior.
2026-06-02 13:57:11 +02:00
codexAuthProbe: options.codexAuthProbe,
fast: options.fast,
useSpinner: options.useSpinner,
feat: add claude-code llm backend with runtime port (#115) * docs: revise claude-code ingest backend spec * docs: keep claude-code spec focused on ingest * docs: expand claude-code spec to full llm parity * Refine claude-code backend spec after adversarial review iteration 1 * Refine claude-code backend spec after adversarial review iteration 2 * Refine claude-code backend spec after adversarial review iteration 3 * feat: recognize claude-code llm backend * feat: add ktx llm runtime port * feat: add claude-code llm runtime * feat: route non-agent llm calls through runtime * feat: run ingest agents through llm runtime * feat: support claude-code setup and status * test: verify claude-code backend runtime * docs: add claude-code backend v1 runtime plan * fix: close claude-code runtime isolation checks * fix: warn on claude-code prompt caching during setup * chore: verify claude-code v1 closure * docs: add claude-code backend v1 isolation closure plan * fix: update claude-code ingest setup guidance * docs: add claude-code backend v1 ingest guidance closure plan * docs: align claude-code isolation spec with sdk metadata * test: cover claude-code host discovery metadata * fix: tolerate claude-code host discovery metadata * docs: clarify claude-code host discovery metadata * docs: add claude-code auth-probe isolation fix plan * chore: prepare kaelio ktx rc1 release * chore: add semantic release workflow * fix: unblock ci checks * chore(release): 0.1.0-rc.1 * feat: add Claude Code model selection to setup * fix: keep git maintenance attached in local repos
2026-05-16 12:06:34 +02:00
});
const embeddings = buildEmbeddingsStatus(config.ingest.embeddings, env);
const storage = buildStorageStatus(config);
const connections = Object.entries(config.connections).map(([name, conn]) =>
buildConnectionStatus(name, conn, env),
);
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
const queryHistory = await buildQueryHistoryStatus(project, options);
const pipeline = buildPipelineStatus(config);
const warnings = buildWarnings(config, connections, llm, embeddings);
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
const unknownConnectionWarning = await buildUnknownConnectionWarning(project);
if (unknownConnectionWarning) {
warnings.push(unknownConnectionWarning);
}
const localStats = await buildLocalStatsStatus(project);
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
const { verdict, reason, nextActions } = buildVerdict(llm, embeddings, connections, queryHistory, warnings);
return {
projectName: basename(project.projectDir) || project.projectDir,
projectDir: project.projectDir,
config: configStatus,
llm,
embeddings,
storage,
connections,
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
queryHistory,
pipeline,
warnings,
localStats,
verdict,
verdictReason: reason,
nextActions,
promptCaching: config.llm.promptCaching
? {
enabled: config.llm.promptCaching.enabled ?? false,
systemTtl: config.llm.promptCaching.systemTtl,
toolsTtl: config.llm.promptCaching.toolsTtl,
historyTtl: config.llm.promptCaching.historyTtl,
}
: undefined,
workUnits: {
stepBudget: config.ingest.workUnits.stepBudget,
maxConcurrency: config.ingest.workUnits.maxConcurrency,
failureMode: config.ingest.workUnits.failureMode,
},
relationshipsDetail: {
acceptThreshold: config.scan.relationships.acceptThreshold,
reviewThreshold: config.scan.relationships.reviewThreshold,
maxLlmTablesPerBatch: config.scan.relationships.maxLlmTablesPerBatch,
validationConcurrency: config.scan.relationships.validationConcurrency,
},
};
}
// ─── Rendering ──────────────────────────────────────────────────────────────
const SYMBOL: Record<ProjectStatusLevel, string> = { ok: '✓', warn: '⚠', fail: '✗', skipped: '-' };
function colorForLevel(useColor: boolean, level: ProjectStatusLevel, text: string): string {
if (!useColor) return text;
if (level === 'ok') return green(text);
if (level === 'warn') return yellow(text);
if (level === 'fail') return red(text);
return _dim(text);
}
function formatBytes(bytes: number): string {
if (bytes < 1024) return `${bytes} B`;
const units = ['KiB', 'MiB', 'GiB', 'TiB'];
let value = bytes / 1024;
let unitIndex = 0;
while (value >= 1024 && unitIndex < units.length - 1) {
value /= 1024;
unitIndex += 1;
}
const precision = value >= 100 ? 0 : value >= 10 ? 1 : 2;
return `${value.toFixed(precision)} ${units[unitIndex]}`;
}
const RELATIVE_TIME_DIVISIONS: Array<{ amount: number; name: Intl.RelativeTimeFormatUnit }> = [
{ amount: 60, name: 'second' },
{ amount: 60, name: 'minute' },
{ amount: 24, name: 'hour' },
{ amount: 7, name: 'day' },
{ amount: 4.34524, name: 'week' },
{ amount: 12, name: 'month' },
{ amount: Number.POSITIVE_INFINITY, name: 'year' },
];
function formatRelativeFromNow(iso: string): string {
const parsed = Date.parse(iso);
if (Number.isNaN(parsed)) return iso;
const formatter = new Intl.RelativeTimeFormat('en', { numeric: 'auto' });
let duration = (parsed - Date.now()) / 1000;
for (const division of RELATIVE_TIME_DIVISIONS) {
if (Math.abs(duration) < division.amount) {
return formatter.format(Math.round(duration), division.name);
}
duration /= division.amount;
}
return iso;
}
function abbreviateHome(filePath: string, env: NodeJS.ProcessEnv): string {
const home = env.HOME;
if (home && (filePath === home || filePath.startsWith(`${home}/`))) {
return filePath === home ? '~' : `~${filePath.slice(home.length)}`;
}
return filePath;
}
function renderLocalStats(
lines: string[],
stats: LocalStatsStatus,
dim: (text: string) => string,
bold: (text: string) => string,
): void {
lines.push(` ${bold('Local data')}`);
if (stats.unavailable) {
lines.push(` ${dim(`(—) ${stats.unavailable}`)}`);
lines.push('');
return;
}
const localLabelWidth = Math.max(
'Ingest'.length,
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
'Wiki'.length,
'Semantic layer'.length,
'Disk'.length,
);
const lLabel = (text: string) => text.padEnd(localLabelWidth);
const ingest = stats.ingest;
const ingestSummary =
ingest.totalCompletedRuns === 0
? dim('no completed runs yet')
: `${ingest.totalCompletedRuns} completed run${ingest.totalCompletedRuns === 1 ? '' : 's'}`;
lines.push(` ${lLabel('Ingest')} ${ingestSummary}`);
if (ingest.perConnection.length > 0) {
const nameWidth = Math.max(...ingest.perConnection.map((entry) => entry.connectionId.length));
const adapterWidth = Math.max(...ingest.perConnection.map((entry) => entry.adapter.length));
for (const entry of ingest.perConnection) {
lines.push(
` ${entry.connectionId.padEnd(nameWidth)} ${dim(entry.adapter.padEnd(adapterWidth))} ${dim(`last ${formatRelativeFromNow(entry.lastCompletedAt)}`)}`,
);
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312) * docs: add spider2-specs handoff directory for benchmark-driven feature specs * feat(cli): connection-scoped wiki pages Add an optional `connections` frontmatter field so database-specific wiki knowledge can be scoped to a connection without polluting searches about other databases, while page keys stay a flat, globally-unique namespace. - connections: single string or list; absent/empty ⇒ unscoped (applies to all) - wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching pages, filtered at the disk-load seam so all three search lanes draw their candidate pool from the already-scoped set (not a post-filter) - wiki_write accepts connections with REPLACE semantics and rejects a connection-scoped write whose key collides with a disjoint-connection page (data-loss guard; hard error, no silent clobber) - explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are validated against ktx.yaml via a shared assertConfiguredConnectionId, which also closes the prior gap where memory_ingest's connectionId was unvalidated; persisted ids absent from config warn (not fail) in `ktx status` - prompt guidance in the wiki_capture skill and external-ingest prompt; the session connectionId is surfaced to the memory agent and ingest work units Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved to spider2-specs/done/. * docs(spider2-specs): add specs/ refinement stage and composite-key join spec Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are the durable artifact; intake drafts move to done/ on ship) and add a MEDIUM-priority spec for multi-column composite-key join detection found during the first sqlite smoke test. * feat(cli): add --verbatim ingest mode for authoritative documents Store each --text/--file document body unchanged as a GLOBAL wiki page instead of routing it through the memory agent, which may rewrite, condense, or re-title it. The LLM derives only metadata (summary, tags, sl_refs) and only for frontmatter fields the document does not already set; the stored body is written by code and never edited. - Deterministic page key: files derive it from the filename, inline text from its leading Markdown heading (headless inline text is rejected — pass it as --file instead). - Idempotent: re-running the same body is a no-op; a different body at the same key fails loudly rather than overwriting. - Works with llm.provider.backend: none, deriving a degraded summary from the heading or first sentence. - Existing frontmatter (including unmodeled fields like effective_date) passes through untouched; --connection-id scopes the page. * feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged. Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool. Both spider2-specs intake drafts move to done/ with implementation notes. * feat(cli): tolerate objects that fail introspection during scan Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests. * docs(spider2-specs): add 06-scan-tolerate-broken-objects spec * feat(cli): generalize analytics fan-out rule to multi-hop join chains The ktx-analytics skill's fan-out rule only reliably caught single-hop inflation; agents still silently fanned out on multi-hop chains where the offending one-to-many join sits several hops below the SUM/COUNT and is easy to miss. Rewrite the Composition rule so the danger reads as cumulative across the whole chain (pre-aggregate per measure-owning table), add an affirmative grain-verification habit (default: pre-aggregate to grain; escape hatch: COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must pre-aggregate), and add one generic wrong-vs-right worked example. Content-only and dialect-agnostic; no new tool, flag, or config. Implements spider2-specs/specs/09 and annotates spec 07's one-example constraint as superseded. * feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft Extend the analytics skill's <sql_craft> with three correctness habits and route the dialect-specific halves through sql_dialect_notes: - Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for "each/every/all/per" questions, defaulted by measure additivity. - Time-series windows (spec 11): explicit cumulative frames, calendar-range rolling windows with minimum-periods guards, and period-over-period via LAG. - Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in one early CTE, and confirm coverage with a failure-detecting cast. Add per-dialect Series, Rolling window, and Safe cast notes to all seven dialect files so the skill stays dialect-agnostic while the engine-specific syntax lives in sql_dialect_notes. Tests updated and passing (19). * docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions Refined specs and completion records for the panel-completeness spine (10), time-series window recipes (11), and text-encoded numeric parsing (12) implemented in the preceding commit. * docs(spider2-specs): add backlog intake drafts 13-14 - 13: canonical authoritative-source measures - 14: output-completeness final check * skill(analytics): spec 14 output-completeness + iter1 (active column planning) Bundles two changes (entangled in SKILL.md; future spider2 iterations land as separate commits): - spec 14 (output-completeness): multi-part "answer every requested output" rule + a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics skill-content test updated; intake draft -> done/, refined spec added. - iter1 experiment: spec 14's passive end-check did not change behavior on the benchmark's output-completeness failures, so (a) the Plan step now writes the exact output-column list UP FRONT as a contract the final SELECT must match, and (b) "expose identity" -> "project BOTH the entity id and its name" (covers both omission directions). All generic craft. Driven by the Spider 2.0-Lite failure analysis (incomplete output was the largest failure bucket); benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): iter2 — deterministic order in string/array aggregation GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft. Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the within-string element order (right elements, wrong order); benchmark as motivation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(mcp): structured, leveled logging for the MCP server Add one synchronous pino logger per MCP server process, written through the io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty (sync, in-process) when it is. Every tool call logs tool.start with its raw params BEFORE the handler runs and tool.end after (info / warn past KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a runaway sql_execution leaves a recoverable start line with its exact SQL and no matching end. HTTP logs session.open/close and wires the previously-dead transport.onerror to transport.error; stdio routes its transport error through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing mcp_request_completed telemetry and registerParsedTool are unchanged; no worker/async transport and no redaction in v1 (logs are local-only). Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves the intake draft to done/. * feat(mcp): report uptimeMs in MCP server /health The /health endpoint now includes uptimeMs (monotonic elapsed time since the server started), mirroring the Python daemon's uptime_ms telemetry field. * feat(cli): bound read-query execution with a per-connection deadline Enforce one shared query deadline (default 30s, overridable per connection via query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of hanging the MCP server. - New shared contract context/connections/query-deadline.ts (resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to the shared warehouse schema; BigQuery's job_timeout_ms removed. - SQLite runs the read query in a short-lived forked child process and enforces the deadline with SIGKILL. worker_threads + terminate() was tried first but cannot interrupt a synchronous better-sqlite3 scan (the native loop never yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free. - Remote connectors apply a real server-side statement timeout and re-wrap their own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/ ETIMEOUT, BigQuery jobTimeoutMs. - Relationship validation skips a candidate to review on a deadline timeout instead of aborting the pass; the deadline surfaces through the existing MCP pino logger as a matched tool.start/tool.end(error) pair (no new logging code). Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts that was breaking tsc -p tsconfig.test.json. * docs(spider2-specs): mark spec 16 (bounded query execution) done Append Implementation notes to the refined spec (what shipped, where, and the worker-thread -> child-process+SIGKILL deviation with its evidence) and move the intake draft from todo/ to done/. * skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career Three generic interpretation rules: a named business measure (sales/revenue/spend) means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between events not a magnitude column; "highest across several achievements" aggregates per metric over the whole history. All three demonstrably FIRE (verified on local008/003/152 SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different axis (source-column / grouping). Generic craft; benchmark only as motivation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set Two generic answer-completeness refinements: - Selecting the extreme group (lowest/highest count over a period/category domain) must rank over the COMPLETE spine, not only groups with fact rows — an empty period is a genuine 0 and often the true minimum. - An aggregate scoped to a per-entity selected set ('avg revenue per actor in those top-3 films') is computed ACROSS that set, distinct from the per-item value; project both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure - spine-for-extreme: concrete cue that a zero-row period never appears in a GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank. - aggregate-over-selected-set: top-N selection ranks by the named ranking measure (the item's own revenue), independent of the per-item share that feeds the aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter3 — comparison-between-two-extremes is one wide row Distinguishes a cross-item comparison ('the difference between the highest and lowest month' -> single wide row, both extremes side by side + the comparison column) from 'report a metric for each group' (-> stays long). Generic, question- derived; targets the wide-vs-long shape gap without affecting per-group long output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter4 — anchor a period bucket to the named lifecycle event When a record carries multiple lifecycle timestamps (created/placed, approved, shipped, delivered, completed, settled) and the question counts/measures records in a named *completed state* by period ("delivered orders by month", "shipped items per week"), bucket the period by that named event's own timestamp, not the record-creation timestamp; the state value is the qualifying filter, the matching timestamp is the time anchor. Wording priority is explicit — purchased/placed/ created/submitted/ordered keep the start-event timestamp — and a non-temporal state filter (counts by customer/city/seller with no period) introduces no anchor. Generic analytics craft: counting completed-state records by their creation date silently answers "records that later reached that state, grouped by when they started" instead of the question asked. Surfaced via the spider2-autofix loop; FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/ semantic-layer lifecycle descriptions, no gold dependency). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis When a question groups/filters/sequences web pages by a path/url column, sample its distinct values; if the data itself shows /route and /route/ variants for the same page context, canonicalize in an early CTE (preserve / as root, strip trailing slashes from non-root paths, map an observed empty path to / only when the column is a URL path with blank root-page events) and use the canonical path everywhere above. Explicitly forbids inventing aliases the data doesn't show: no merging different route names, no stripping query/fragment/host/scheme, no lowercasing, and no canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs. Generic web-analytics craft: raw request logs routinely store the same user-visible page with and without a trailing slash, so grouping raw labels silently splits one page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2); FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain question wording + solver-observed distinct values, no gold dependency). The rule fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts), local331 residual is a separate sequence-semantics axis beyond canonicalization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate When a question first selects a group of entities ("the top 5 actors", "these products") and then asks what count/share/percentage of a DIFFERENT subject domain relates to *these* selected entities ("what % of customers rented films featuring these actors"), the subject set is the UNION across the whole group: count DISTINCT subject ids once across the selected entities and return one collective value at the subject-domain grain — not one row per selected entity (which double-counts subjects related to more than one entity and answers a different question). Narrowly guarded: emit one row per entity only when the wording says "for each / per / by / list" or asks for each entity's own metric ("top 5 players and their batting averages"). The collective-coverage cousin of the existing per-entity selected-set rule. Generic analytics craft (per-entity metric vs set-level coverage). Surfaced via the spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened, restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned (union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing per-entity top-N guards (local023/024/029/212/221 stayed long). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a dimension table used only to attach a display attribute silently discards every fact row whose key has no parent when the dimension is incomplete (trimmed catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe over which shares/averages/medians are computed. Guidance: LEFT JOIN pure enrichment; inner-join a dimension only when intended as a filter; key the aggregate/GROUP BY on the fact column, not the dimension column. Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS, official scorer) — solver dropped the gratuitous products inner-join and recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but remain gold-convention-blocked. Guards local061/067 held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer) Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the delivery timestamp) during enrichment so the correct time anchor is the default for any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): accept leading underscore in connection/identifier ids The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an underscore everywhere except the first character, so a connection id / database name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES) could never be ingested or queried. Allow a leading underscore across all 16 duplicated validators (connection ids, source ids, page/wiki keys, warehouse- verification tool schemas). Path-safety is unaffected — '.' and '/' remain excluded, and assertSafePathToken still blocks traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): generic geospatial query guidance Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/ ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the entity geometry, build an area-of-interest polygon, test with the engine's containment/proximity/overlap predicate; mind lon/lat argument order). Steers the solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted geospatial predicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): parse code/dependency text by language grammar Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by the language or manifest format (Java import keep-package-path allowing underscores/ mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse JSON cell source before language rules; JSON manifests flatten the dependency object keys), stripping comments/prose and splitting multi-import lines; (2) on a de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the weight column from the population the question names, not silently. Steers off one broad regex that drops valid identifiers and matches prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): source filters/dates/measures from the owning fact grain Add a <sql_craft> rule for joined fact tables at different grains (parent order vs child line item): read each predicate, calendar bucket, and measure from the table whose grain the question names, not whichever is in scope post-join. An order-grain filter ("orders that are Complete", "the order's creation date") must come from the parent even though the child carries its own status/created_at; line price/cost come from the child. Mirror at metric grain: don't combine a parent-grain count with child rows (num_of_item * SUM(line_price) per line) — aggregate each measure at its own grain before combining. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration When an entity carries a multi-valued classification array (IPC/CPC codes, tags) and the methodology counts entities-per-class or a concentration/diversity metric (HHI, originality, share), pick ONE representative per entity first (the array's main/primary/first flag, else a defined fallback like most-frequent), then aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a count of entities. Unnesting the array otherwise multiplies an entity's weight by its code count, inflating per-class frequencies and skewing the ranking/score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): introspect BigQuery datasets hosted in foreign projects A dataset_ids/dataset_id entry may now be written `project.dataset` to introspect a dataset hosted in another project while query jobs still bill to credentials.project_id. Entries are parsed once at the config boundary into canonical {project, dataset} pairs; introspection, primary-key discovery, testConnection, getTableRowCount, and listTables (grouped per project) all resolve in the dataset's own project, and scanned tables are labeled with that project so sampling, distinct-value, and read queries resolve. Bare entries are unchanged. Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md. * feat(scan): durable, resumable, bounded relationship detection during enrichment Move the enrichment persistence boundary to the cost boundary and bound the open-ended relationship stage (spec 19). - Checkpoint descriptions + embeddings into the queryable `_schema` manifest (and the raw enrichment artifacts) before relationship detection runs, via a new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted, budget-truncated, or failed relationship stage now degrades to "no joins", never "no descriptions". - Resume the enrichment cache by content identity: re-key the SQLite stage store on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes finished descriptions/embeddings instead of re-paying for LLM work. The disposable cache recreates its table if the on-disk key shape differs. - Make the relationship stage observable and bounded: a sticky wall-clock budget (`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress + honored `ctx.signal`, threaded through profiling, validation, and composite detection. On exhaustion/abort it stops scheduling, finalizes, and returns a partial result instead of throwing or hanging. - Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason` + recoverable `relationship_detection_partial` warning). A graceful partial saves as a completed stage and resumes cheaply; raising the budget changes inputHash and forces a fresh, fuller run. A process killed mid-stage saves nothing. Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation notes to specs/19 and move the intake draft to done/. Also carries the in-tree per-table enrichment LLM timeout work it builds on (`description-generation.ts` + the `enrichment_timeout` warning code), which is intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a separately-building commit. * feat(scan): bound + retry the per-table enrichment LLM call The batched table-description call had no retry (sampleTable retried 3x, this did not), so a single transient backend error (e.g. an overloaded/burst rejection when many tables enrich concurrently) silently nulled a whole table's descriptions — observed dropping ~70% of a db's tables during a bad window despite ample quota. - Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS). - Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried (one wedge stays one timeout, not 3x). - Granular per-table progress + start/done/retry/timeout logging. Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable; this makes more of them complete. * feat(scan): survive a hung LLM enrichment backend and resume descriptions Two compounding failure modes on the per-table description-enrichment path (spec 20): Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout. Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output). Spec 20 refined + intake draft moved to done/. * feat(scan): selective enrichment stages (--stages) + per-stage cache keys Split the single coarse enrichment cache key into per-stage hashes (descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding identity + description digest; relationships <- snapshot + relationship settings + LLM identity), so changing one stage's inputs invalidates only that stage and never throws away the expensive per-table descriptions on an unrelated edit. Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an already-ingested connection: a named stage bypasses the completed-stage short-circuit while the per-table descriptions resume record still skips already-enriched tables, and unselected stages are left untouched on disk. Feed embeddings + relationships their description context from the on-disk _schema when descriptions do not run this invocation, and carry descriptions into the llmProposals evidence packet (closing a latent gap on the full-run path too). Surface an enrichment_stage_stale warning when an unselected stage's inputs have drifted, rather than silently cascading the work. Implements spider2-specs/specs/21-selective-enrichment-stages.md. * test(analytics): realign SKILL.md acceptance test with the evolved skill Three assertions in analytics-skill-content.test.ts drifted from the analytics SKILL.md as later iterations edited the skill without updating the test: - the sub-heading was renamed Window functions -> Ordering & aggregation determinism (iter2), so follow the source name; - the rule "Expose identity, not just the label" was renamed to "Project BOTH identity and label" (spec 14), so match the new wording; - the dialect-FQTN guard false-positived on the Java package example com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the guard stays at full strength without weakening it. * fix(scan): --stages subset must not delete unselected stages' on-disk artifacts A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions from the written _schema. runLocalScan writes the structural manifest shard from the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as scan-managed and overwrites them with whatever the run emits — none, on a subset that skips descriptions. Enrichment then read the already-wiped shard via loadPriorDescriptions and had nothing to restore. runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run if descriptions ran, else loaded from the on-disk _schema) instead of [], and runLocalScan captures the prior descriptions before the structural write and feeds them to both the structural write and enrichment, so an unselected stage's artifacts survive. Joins were already preserved for --stages descriptions via the manual/inferred preservedJoins path. Tests: a full runLocalScan --stages relationships path test (RED without the fix, GREEN with it — the earlier unit test missed the structural-pre-write ordering), plus enrichment-layer contract tests for both directions. Validated live on northwind: --stages relationships keeps all 110 descriptions + 22 joins (was wiping to 0); --stages descriptions restores descriptions from the spec-20 resume record (no LLM calls) while keeping joins. * feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md): - Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params), dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard. - Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection predicates, areal allocation via ST_AREA(ST_INTERSECTION()). - SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note. Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits). * feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND) SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a dialect note: nudge by a tiny epsilon (1e-9) below display precision before rounding for deterministic half-up, leaving non-boundary values unchanged. Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a displayed average/rate/price benefits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns Add SKILL.md guidance to emit list-valued answer cells as delimited STRING (not ARRAY/repeated column), answer the literal ask without unrequested transformations (HAVING for aggregate bounds), and avoid projecting unrequested free-text columns that corrupt row-delimited output. * fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout - gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a version-controlled `.ktx/`, and snowflake.log already sat there unprotected. - gate the LLM relationship proposal on the detection budget/abort signal so an exhausted or aborted stage cannot start a fresh LLM call; document the boundary. - validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts, so a bad value no longer times out every table immediately. - daemon introspection now warns on malformed column/FK rows instead of dropping them silently, matching the table-row path and the "surface broken objects" goal. - docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema doc (forked-subprocess SIGKILL, not worker-thread termination). * fix(scan,wiki,mcp): address PR #312 review findings - scan: key the description pipeline (resume map, enriched-schema and embedding-text lookups, manifest write/read) by full table identity via tableRefKey/buildTableRef, so two same-named tables in different schemas no longer cross-assign descriptions or skip a sibling on resume - scan: re-throw a genuine context cancel during the batched description LLM call so Ctrl-C resumes the stage instead of nulling tables and recording it completed; per-table timeouts still degrade (context.signal not aborted) - scan: report statisticalValidation 'skipped' (not 'completed') when a budget/abort stop leaves relationship profiling partial - wiki: sync the full page corpus into the sqlite index and filter only the candidate/result set, so a connection-scoped search no longer prunes other connections' pages and cached embeddings from the shared index - wiki: route verbatim ingest through the canonical writePageAndSync so contentHash is set and later syncs can short-circuit - mcp: drop the as-unknown-as cast in serializeMcpError - dialects/analytics: document the integer-division trap on postgres/sqlite/tsql Adds regression tests for each behavior change. * fix(wiki): scope connection filter before SQLite lane limit Connection-scoped wiki search applied the connectionId allowlist after the lexical/semantic lanes had already truncated to laneCandidatePoolLimit over the full (connection-agnostic) corpus. When the requested connection was a minority of a large corpus, its pages were crowded out of the candidate pool before filtering, so a semantic-only match could be missed outright and lexical hits under-ranked. Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates so LIMIT applies to in-scope rows, matching what the token lane already did, and drop the now-redundant post-limit JS filters. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:57 +02:00
if (entry.skippedObjects.length > 0) {
const first = entry.skippedObjects[0]!;
const extra = entry.skippedObjects.length - 1;
const detail = `${first.name}: ${first.reason}${extra > 0 ? ` (+${extra} more)` : ''}`;
lines.push(
` ${' '.repeat(nameWidth)} ${dim(`${entry.skippedObjects.length} object${entry.skippedObjects.length === 1 ? '' : 's'} skipped — ${detail}`)}`,
);
}
}
}
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
if (stats.wikiPages.length === 0) {
lines.push(` ${lLabel('Wiki')} ${dim('no pages yet')}`);
} else {
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
const wikiText = stats.wikiPages
.map((entry) => `${entry.scope}=${entry.count} ${dim(`(${entry.embeddedCount} embedded)`)}`)
.join(` ${dim('·')} `);
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
lines.push(` ${lLabel('Wiki')} ${wikiText}`);
}
if (stats.semanticLayer.length === 0) {
lines.push(` ${lLabel('Semantic layer')} ${dim('no indexed sources yet')}`);
} else {
const nameWidth = Math.max(...stats.semanticLayer.map((entry) => entry.connectionId.length));
let firstLine = true;
for (const entry of stats.semanticLayer) {
const prefix = firstLine ? lLabel('Semantic layer') : ' '.repeat(localLabelWidth);
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
const sourcesText = `${entry.sourceCount} source${entry.sourceCount === 1 ? '' : 's'} (${entry.embeddedSourceCount} embedded)`;
const dictText = `${entry.dictionaryValueCount} dictionary value${entry.dictionaryValueCount === 1 ? '' : 's'}`;
lines.push(
chore(workspace): gate dead-code with knip production mode (#196) * refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00
` ${prefix} ${entry.connectionId.padEnd(nameWidth)} ${dim(`${sourcesText} · ${dictText}`)}`,
);
firstLine = false;
}
}
const disk = stats.projectDir;
const diskBits: string[] = [];
diskBits.push(`db=${disk.dbSqliteBytes === null ? '' : formatBytes(disk.dbSqliteBytes)}`);
diskBits.push(`cache=${formatBytes(disk.ktxCacheBytes)}`);
diskBits.push(
`raw-sources=${disk.rawSources.fileCount} file${disk.rawSources.fileCount === 1 ? '' : 's'} (${formatBytes(disk.rawSources.bytes)})`,
);
lines.push(` ${lLabel('Disk')} ${dim(diskBits.join(` ${dim('·')} `))}`);
lines.push('');
}
export interface RenderProjectStatusOptions {
verbose?: boolean;
useColor?: boolean;
durationMs?: number;
toolchainChecks?: DoctorCheck[];
env?: NodeJS.ProcessEnv;
}
export function renderProjectStatus(status: ProjectStatus, options: RenderProjectStatusOptions = {}): string {
const verbose = options.verbose ?? false;
const useColor = options.useColor ?? false;
const env = options.env ?? process.env;
const dim = useColor ? _dim : (s: string) => s;
const bold = useColor ? _bold : (s: string) => s;
const color = (level: ProjectStatusLevel, s: string) => colorForLevel(useColor, level, s);
const sym = (level: ProjectStatusLevel) => color(level, SYMBOL[level]);
const lines: string[] = [];
const dirStr = abbreviateHome(status.projectDir, env);
lines.push(`${bold('ktx status')} ${dim('·')} ${status.projectName} ${dim(`(${dirStr})`)}`);
lines.push('');
const labelPad = 'Connections'.length;
const label = (text: string) => text.padEnd(labelPad);
// Core readiness rows
const llmDetail = [status.llm.backend, status.llm.model].filter(Boolean).join(` ${dim('·')} `);
lines.push(` ${label('LLM')} ${llmDetail} ${sym(status.llm.status)} ${dim(status.llm.detail)}`);
const embedParts = [status.embeddings.backend];
if (status.embeddings.model) embedParts.push(status.embeddings.model);
const embedDim = status.embeddings.dimensions ? `(${status.embeddings.dimensions}d)` : '';
const embedDetail = `${embedParts.join(` ${dim('·')} `)}${embedDim ? ` ${embedDim}` : ''}`;
lines.push(` ${label('Embeddings')} ${embedDetail} ${sym(status.embeddings.status)} ${dim(status.embeddings.detail)}`);
lines.push(` ${label('Storage')} ${dim(`${status.storage.state} (state) · ${status.storage.search} (search)`)}`);
lines.push(` ${label('Config')} ${sym(status.config.status)} ${dim(status.config.detail)}`);
if (status.config.issues.length > 0) {
for (const issue of status.config.issues) {
lines.push(` ${color('warn', SYMBOL.warn)} ${issue.message}`);
if (issue.fix) lines.push(` ${dim(`${issue.fix}`)}`);
}
}
lines.push('');
// Connections
if (status.connections.length === 0) {
lines.push(` ${bold('Connections')} ${dim('(none)')}`);
lines.push(` ${dim('No connections configured. Run `ktx setup` to add one.')}`);
} else {
lines.push(` ${bold('Connections')} ${dim(`(${status.connections.length})`)}`);
const nameWidth = Math.max(...status.connections.map((c) => c.name.length));
const driverWidth = Math.max(...status.connections.map((c) => c.driver.length));
for (const conn of status.connections) {
lines.push(
` ${sym(conn.status)} ${conn.name.padEnd(nameWidth)} ${dim(conn.driver.padEnd(driverWidth))} ${conn.detail}`,
);
if (conn.fix && conn.status !== 'ok') {
const indent = 6 + nameWidth + 3 + driverWidth + 3;
lines.push(`${' '.repeat(indent)}${dim(`${conn.fix}`)}`);
}
}
}
lines.push('');
feat: merge ingest and scan * docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
2026-05-14 01:43:06 +02:00
if (status.queryHistory.length > 0) {
lines.push(` ${bold('Query history')}`);
const connectionWidth = Math.max(...status.queryHistory.map((entry) => entry.connection.length));
for (const entry of status.queryHistory) {
lines.push(
` ${sym(entry.status)} ${entry.connection.padEnd(connectionWidth)} ${dim(entry.dialect)} ${entry.detail}`,
);
if (entry.fix && entry.status !== 'ok') {
const indent = 6 + connectionWidth + 3 + entry.dialect.length + 3;
lines.push(`${' '.repeat(indent)}${dim(`${entry.fix}`)}`);
}
}
lines.push('');
}
// Pipeline
lines.push(` ${bold('Pipeline')}`);
const pipelineLabelWidth = Math.max('Adapters'.length, 'Enrichment'.length, 'Research agent'.length);
const pLabel = (text: string) => text.padEnd(pipelineLabelWidth);
lines.push(` ${pLabel('Adapters')} ${status.pipeline.adapters.length > 0 ? status.pipeline.adapters.join(', ') : dim('(none)')}`);
const enrichmentDetail = [`${status.pipeline.enrichmentMode} mode`];
if (status.pipeline.relationshipsEnabled) {
const bits = ['relationships on'];
if (status.pipeline.relationshipsLlmProposals) bits.push('LLM proposals');
if (status.pipeline.relationshipsValidationRequired) bits.push('validation required');
enrichmentDetail.push(bits.join(', '));
} else {
enrichmentDetail.push('relationships off');
}
lines.push(` ${pLabel('Enrichment')} ${enrichmentDetail.join(` ${dim('·')} `)}`);
const agentDetail = status.pipeline.agentEnabled
? `enabled ${dim(`(${status.pipeline.agentTools.length} tool${status.pipeline.agentTools.length === 1 ? '' : 's'})`)}`
: dim('disabled');
lines.push(` ${pLabel('Research agent')} ${agentDetail}`);
lines.push('');
// Local data
renderLocalStats(lines, status.localStats, dim, bold);
// Warnings
if (status.warnings.length > 0) {
lines.push(` ${bold('Warnings')}`);
for (const w of status.warnings) {
lines.push(` ${color('warn', SYMBOL.warn)} ${w.message}`);
if (w.fix) lines.push(` ${dim(`${w.fix}`)}`);
}
lines.push('');
}
// Verbose extras
if (verbose) {
if (options.toolchainChecks && options.toolchainChecks.length > 0) {
lines.push(` ${bold('Toolchain')}`);
for (const check of options.toolchainChecks) {
const lv: ProjectStatusLevel = check.status === 'pass' ? 'ok' : check.status === 'warn' ? 'warn' : 'fail';
lines.push(` ${sym(lv)} ${check.label}: ${check.detail}`);
if (check.fix && lv !== 'ok') lines.push(` ${dim(`${check.fix}`)}`);
}
lines.push('');
}
if (status.promptCaching) {
const pc = status.promptCaching;
const bits = [`enabled=${pc.enabled}`];
if (pc.systemTtl) bits.push(`system=${pc.systemTtl}`);
if (pc.toolsTtl) bits.push(`tools=${pc.toolsTtl}`);
if (pc.historyTtl) bits.push(`history=${pc.historyTtl}`);
lines.push(` ${bold('Prompt caching')} ${dim(bits.join(', '))}`);
}
if (status.workUnits) {
const wu = status.workUnits;
lines.push(` ${bold('Work units')} ${dim(`stepBudget=${wu.stepBudget}, maxConcurrency=${wu.maxConcurrency}, failureMode=${wu.failureMode}`)}`);
}
if (status.relationshipsDetail) {
const r = status.relationshipsDetail;
lines.push(
` ${bold('Relationships')} ${dim(`accept=${r.acceptThreshold}, review=${r.reviewThreshold}, maxLlmTables=${r.maxLlmTablesPerBatch}, concurrency=${r.validationConcurrency}`)}`,
);
}
lines.push(
` ${bold('Agent')} ${dim(`max_iterations=${status.pipeline.agentMaxIterations}, tools=${status.pipeline.agentTools.join(', ') || '(none)'}`)}`,
);
fix(cli): isolate ktx-owned project repositories (#283) * fix(cli): isolate ktx project git repos * fix(cli): remove inert auto commit config * test(cli): drop stale auto commit fixtures * docs: document isolated ktx project repos * test(cli): keep stale config grep clean * fix(cli): guide setup away from foreign repos at the project dir ktx owns the git repo rooted at the project dir and refuses to adopt one it did not create (the Finding 3 isolation invariant). But setup steered users straight into that failure: the interactive menu offers "Current directory" first, and `--no-input --yes --project-dir <repo-root>` created directly in place — both then threw a generic "Failed to initialize git repository:" wrapper from deep in GitService.initialize(). Extract the ownership rule into a shared `classifyKtxRepoOwnership(dir)` used by both GitService.initialize() (the invariant) and the setup wizard (pre-flight guidance), so the decision derives from one rule. Setup now detects a foreign repo before constructing GitService and: interactively re-prompts (the user picks the existing `ktx-project` subfolder), or non-interactively returns a clean missing-input with the actionable message. The typed foreign-repo error is also surfaced verbatim instead of being buried under the generic wrapper. Empty/non-repo current directories still work — only foreign repos are blocked. * fix(cli): keep classifyKtxRepoOwnership total for non-directory paths The setup ownership guard runs before the existing not-a-directory check, so pointing a custom/--project-dir path at a file made classifyKtxRepoOwnership lstat `<file>/.git`, hit ENOTDIR, and throw — crashing the setup step instead of returning the friendly "path exists and is not a directory" result. A path that is a file (or missing) holds no git repo for ktx to avoid, so treat ENOTDIR like ENOENT and return 'unowned'. The downstream existingFolderState check still rejects a non-directory with its friendly message, and the classifier no longer throws raw errno for any caller.
2026-06-10 14:12:25 +02:00
lines.push(` ${bold('Git')} ${dim(`author=${status.storage.gitAuthor}`)}`);
lines.push('');
}
// Verdict + next steps
const verdictLevel: ProjectStatusLevel =
status.verdict === 'ready' ? 'ok' : status.verdict === 'partial' ? 'warn' : 'fail';
const duration = options.durationMs !== undefined ? ` ${dim(`(${(options.durationMs / 1000).toFixed(2)}s)`)}` : '';
if (status.verdict === 'ready') {
const hint = ` ${dim('Try:')} ${status.nextActions.join(dim(' · '))}`;
lines.push(`${color(verdictLevel, status.verdictReason)}${hint}${duration}`);
} else {
const hint = status.nextActions.length > 0 ? ` ${dim('Next:')} ${status.nextActions.join(dim(' · '))}` : '';
lines.push(`${color(verdictLevel, status.verdictReason)}${hint}${duration}`);
}
lines.push('');
return lines.join('\n');
}