mirror of
https://github.com/Kaelio/ktx.git
synced 2026-07-01 08:59:39 +02:00
308 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
semantic-release-bot
|
f21594c42a |
chore(release): 0.15.0 [skip ci]
## [0.15.0](https://github.com/Kaelio/ktx/compare/v0.14.0...v0.15.0) (2026-06-30) ### Features * **sigma:** add Sigma Computing context-source adapter ([#316](https://github.com/Kaelio/ktx/issues/316)) ([ |
||
|
Matt Senick (Sigma)
|
acd20ac248
|
feat(sigma): add Sigma Computing context-source adapter (#316)
* feat(sigma): add Sigma Computing context-source adapter Closes #168 Adds a full ingest adapter for Sigma Computing so `ktx ingest` can pull data model specs and workbook summaries into the ktx context layer. The implementation follows the same fetch → chunk → project → LLM pattern used by the Looker, Metabase, and MetricFlow adapters. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(sigma): address PR review comments - Remove manifest from rawFiles; moves to peerFileIndex so fetchedAt changes don't mark all work units dirty every run - Fix workbookFilter.updatedSince eviction bug: fetch full universe first, apply filter client-side, evict only on archived/deleted - Remove measure projection entirely; project() writes measures: [] and the sigma_ingest skill surfaces Lookup/aggregation formulas as wiki prose - Remove joins projection (v1 limitation); project() writes joins: [] and Lookup relationships are described in wiki prose instead - Remove write-back dead code: createDataModel, updateDataModel, SigmaDataModelPushResult, mutate/post/put - Fix emitBatches notes pluralization bug ('2 data modelss' → '2 data models') - Add tokenInflight dedup on ensureToken to coalesce concurrent auth requests - Retry spec fetch when existing staged spec is null (transient failure cache) - Drop unused WorkbookFilter import from client-port.ts - Note in docs that joins are not projected from Sigma data models in this release Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * updates * fix(sigma): restore sigma in local adapter test + small cleanups The gdrive↔sigma merge dropped 'sigma' from the expected adapter source list in local-adapters.test.ts while keeping gdrive, so the slow TS suite failed even though the source registers both. Add 'sigma' back at its registration position (after metabase, before gdrive). Also: - Move the orphaned SigmaPullConfig docstring onto the schema it documents and drop the stale BullMQ reference (standalone ktx has no BullMQ; the config lives in the ingest job's bundleRef.config). - Drop an O(n^2) find() round-trip in fetch() when building the active data-model list; filter once and reuse for the eviction id set. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com> Co-authored-by: Luca Martial <48870843+luca-martial@users.noreply.github.com> |
||
|
semantic-release-bot
|
1ec29e82f7 |
chore(release): 0.14.0 [skip ci]
## [0.14.0](https://github.com/Kaelio/ktx/compare/v0.13.1...v0.14.0) (2026-06-30) ### Features * **connectors:** add MongoDB connector ([#305](https://github.com/Kaelio/ktx/issues/305)) ([#310](https://github.com/Kaelio/ktx/issues/310)) ([ |
||
|
Andrey Avtomonov
|
67a69dba8b
|
Resumable and fault-tolerant source ingest (spec #22) (#315)
* docs: add spider2-specs handoff directory for benchmark-driven feature specs
* feat(cli): connection-scoped wiki pages
Add an optional `connections` frontmatter field so database-specific wiki
knowledge can be scoped to a connection without polluting searches about other
databases, while page keys stay a flat, globally-unique namespace.
- connections: single string or list; absent/empty ⇒ unscoped (applies to all)
- wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching
pages, filtered at the disk-load seam so all three search lanes draw their
candidate pool from the already-scoped set (not a post-filter)
- wiki_write accepts connections with REPLACE semantics and rejects a
connection-scoped write whose key collides with a disjoint-connection page
(data-loss guard; hard error, no silent clobber)
- explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are
validated against ktx.yaml via a shared assertConfiguredConnectionId, which
also closes the prior gap where memory_ingest's connectionId was unvalidated;
persisted ids absent from config warn (not fail) in `ktx status`
- prompt guidance in the wiki_capture skill and external-ingest prompt; the
session connectionId is surfaced to the memory agent and ingest work units
Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved
to spider2-specs/done/.
* docs(spider2-specs): add specs/ refinement stage and composite-key join spec
Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are
the durable artifact; intake drafts move to done/ on ship) and add a
MEDIUM-priority spec for multi-column composite-key join detection found during
the first sqlite smoke test.
* feat(cli): add --verbatim ingest mode for authoritative documents
Store each --text/--file document body unchanged as a GLOBAL wiki page
instead of routing it through the memory agent, which may rewrite,
condense, or re-title it. The LLM derives only metadata (summary, tags,
sl_refs) and only for frontmatter fields the document does not already
set; the stored body is written by code and never edited.
- Deterministic page key: files derive it from the filename, inline
text from its leading Markdown heading (headless inline text is
rejected — pass it as --file instead).
- Idempotent: re-running the same body is a no-op; a different body at
the same key fails loudly rather than overwriting.
- Works with llm.provider.backend: none, deriving a degraded summary
from the heading or first sentence.
- Existing frontmatter (including unmodeled fields like effective_date)
passes through untouched; --connection-id scopes the page.
* feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill
Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged.
Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool.
Both spider2-specs intake drafts move to done/ with implementation notes.
* feat(cli): tolerate objects that fail introspection during scan
Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests.
* docs(spider2-specs): add 06-scan-tolerate-broken-objects spec
* feat(cli): generalize analytics fan-out rule to multi-hop join chains
The ktx-analytics skill's fan-out rule only reliably caught single-hop
inflation; agents still silently fanned out on multi-hop chains where the
offending one-to-many join sits several hops below the SUM/COUNT and is easy
to miss.
Rewrite the Composition rule so the danger reads as cumulative across the whole
chain (pre-aggregate per measure-owning table), add an affirmative
grain-verification habit (default: pre-aggregate to grain; escape hatch:
COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must
pre-aggregate), and add one generic wrong-vs-right worked example. Content-only
and dialect-agnostic; no new tool, flag, or config.
Implements spider2-specs/specs/09 and annotates spec 07's one-example
constraint as superseded.
* feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft
Extend the analytics skill's <sql_craft> with three correctness habits and
route the dialect-specific halves through sql_dialect_notes:
- Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for
"each/every/all/per" questions, defaulted by measure additivity.
- Time-series windows (spec 11): explicit cumulative frames, calendar-range
rolling windows with minimum-periods guards, and period-over-period via LAG.
- Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in
one early CTE, and confirm coverage with a failure-detecting cast.
Add per-dialect Series, Rolling window, and Safe cast notes to all seven
dialect files so the skill stays dialect-agnostic while the engine-specific
syntax lives in sql_dialect_notes. Tests updated and passing (19).
* docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions
Refined specs and completion records for the panel-completeness spine (10),
time-series window recipes (11), and text-encoded numeric parsing (12)
implemented in the preceding commit.
* docs(spider2-specs): add backlog intake drafts 13-14
- 13: canonical authoritative-source measures
- 14: output-completeness final check
* skill(analytics): spec 14 output-completeness + iter1 (active column planning)
Bundles two changes (entangled in SKILL.md; future spider2 iterations land as
separate commits):
- spec 14 (output-completeness): multi-part "answer every requested output" rule
+ a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics
skill-content test updated; intake draft -> done/, refined spec added.
- iter1 experiment: spec 14's passive end-check did not change behavior on the
benchmark's output-completeness failures, so (a) the Plan step now writes the
exact output-column list UP FRONT as a contract the final SELECT must match,
and (b) "expose identity" -> "project BOTH the entity id and its name" (covers
both omission directions). All generic craft.
Driven by the Spider 2.0-Lite failure analysis (incomplete output was the
largest failure bucket); benchmark only as motivation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* skill(analytics): iter2 — deterministic order in string/array aggregation
GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit
ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase
before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft.
Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the
within-string element order (right elements, wrong order); benchmark as motivation only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(mcp): structured, leveled logging for the MCP server
Add one synchronous pino logger per MCP server process, written through the
io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty
(sync, in-process) when it is. Every tool call logs tool.start with its raw
params BEFORE the handler runs and tool.end after (info / warn past
KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a
runaway sql_execution leaves a recoverable start line with its exact SQL and
no matching end. HTTP logs session.open/close and wires the previously-dead
transport.onerror to transport.error; stdio routes its transport error
through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing
mcp_request_completed telemetry and registerParsedTool are unchanged; no
worker/async transport and no redaction in v1 (logs are local-only).
Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves
the intake draft to done/.
* feat(mcp): report uptimeMs in MCP server /health
The /health endpoint now includes uptimeMs (monotonic elapsed time since
the server started), mirroring the Python daemon's uptime_ms telemetry
field.
* feat(cli): bound read-query execution with a per-connection deadline
Enforce one shared query deadline (default 30s, overridable per connection via
query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive
LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of
hanging the MCP server.
- New shared contract context/connections/query-deadline.ts
(resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to
the shared warehouse schema; BigQuery's job_timeout_ms removed.
- SQLite runs the read query in a short-lived forked child process and enforces
the deadline with SIGKILL. worker_threads + terminate() was tried first but
cannot interrupt a synchronous better-sqlite3 scan (the native loop never
yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free.
- Remote connectors apply a real server-side statement timeout and re-wrap their
own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL
max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse
max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/
ETIMEOUT, BigQuery jobTimeoutMs.
- Relationship validation skips a candidate to review on a deadline timeout
instead of aborting the pass; the deadline surfaces through the existing MCP
pino logger as a matched tool.start/tool.end(error) pair (no new logging code).
Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts
that was breaking tsc -p tsconfig.test.json.
* docs(spider2-specs): mark spec 16 (bounded query execution) done
Append Implementation notes to the refined spec (what shipped, where, and the
worker-thread -> child-process+SIGKILL deviation with its evidence) and move the
intake draft from todo/ to done/.
* skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career
Three generic interpretation rules: a named business measure (sales/revenue/spend)
means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between
events not a magnitude column; "highest across several achievements" aggregates per
metric over the whole history. All three demonstrably FIRE (verified on local008/003/152
SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different
axis (source-column / grouping). Generic craft; benchmark only as motivation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set
Two generic answer-completeness refinements:
- Selecting the extreme group (lowest/highest count over a period/category
domain) must rank over the COMPLETE spine, not only groups with fact rows —
an empty period is a genuine 0 and often the true minimum.
- An aggregate scoped to a per-entity selected set ('avg revenue per actor in
those top-3 films') is computed ACROSS that set, distinct from the per-item
value; project both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure
- spine-for-extreme: concrete cue that a zero-row period never appears in a
GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank.
- aggregate-over-selected-set: top-N selection ranks by the named ranking measure
(the item's own revenue), independent of the per-item share that feeds the aggregate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter3 — comparison-between-two-extremes is one wide row
Distinguishes a cross-item comparison ('the difference between the highest and
lowest month' -> single wide row, both extremes side by side + the comparison
column) from 'report a metric for each group' (-> stays long). Generic, question-
derived; targets the wide-vs-long shape gap without affecting per-group long output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter4 — anchor a period bucket to the named lifecycle event
When a record carries multiple lifecycle timestamps (created/placed, approved,
shipped, delivered, completed, settled) and the question counts/measures records
in a named *completed state* by period ("delivered orders by month", "shipped
items per week"), bucket the period by that named event's own timestamp, not the
record-creation timestamp; the state value is the qualifying filter, the matching
timestamp is the time anchor. Wording priority is explicit — purchased/placed/
created/submitted/ordered keep the start-event timestamp — and a non-temporal
state filter (counts by customer/city/seller with no period) introduces no anchor.
Generic analytics craft: counting completed-state records by their creation date
silently answers "records that later reached that state, grouped by when they
started" instead of the question asked. Surfaced via the spider2-autofix loop;
FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/
semantic-layer lifecycle descriptions, no gold dependency).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis
When a question groups/filters/sequences web pages by a path/url column, sample
its distinct values; if the data itself shows /route and /route/ variants for the
same page context, canonicalize in an early CTE (preserve / as root, strip trailing
slashes from non-root paths, map an observed empty path to / only when the column is
a URL path with blank root-page events) and use the canonical path everywhere above.
Explicitly forbids inventing aliases the data doesn't show: no merging different
route names, no stripping query/fragment/host/scheme, no lowercasing, and no
canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs.
Generic web-analytics craft: raw request logs routinely store the same user-visible
page with and without a trailing slash, so grouping raw labels silently splits one
page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2);
FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain
question wording + solver-observed distinct values, no gold dependency). The rule
fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts),
local331 residual is a separate sequence-semantics axis beyond canonicalization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate
When a question first selects a group of entities ("the top 5 actors", "these
products") and then asks what count/share/percentage of a DIFFERENT subject domain
relates to *these* selected entities ("what % of customers rented films featuring
these actors"), the subject set is the UNION across the whole group: count DISTINCT
subject ids once across the selected entities and return one collective value at the
subject-domain grain — not one row per selected entity (which double-counts subjects
related to more than one entity and answers a different question). Narrowly guarded:
emit one row per entity only when the wording says "for each / per / by / list" or
asks for each entity's own metric ("top 5 players and their batting averages").
The collective-coverage cousin of the existing per-entity selected-set rule. Generic
analytics craft (per-entity metric vs set-level coverage). Surfaced via the
spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened,
restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned
(union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing
per-entity top-N guards (local023/024/029/212/221 stayed long).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows
Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a
dimension table used only to attach a display attribute silently discards every
fact row whose key has no parent when the dimension is incomplete (trimmed
catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe
over which shares/averages/medians are computed. Guidance: LEFT JOIN pure
enrichment; inner-join a dimension only when intended as a filter; key the
aggregate/GROUP BY on the fact column, not the dimension column.
Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS,
official scorer) — solver dropped the gratuitous products inner-join and
recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but
remain gold-convention-blocked. Guards local061/067 held.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer)
Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer
form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state
lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the
delivery timestamp) during enrichment so the correct time anchor is the default for
any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(connectors): accept leading underscore in connection/identifier ids
The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an
underscore everywhere except the first character, so a connection id / database
name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES)
could never be ingested or queried. Allow a leading underscore across all 16
duplicated validators (connection ids, source ids, page/wiki keys, warehouse-
verification tool schemas). Path-safety is unaffected — '.' and '/' remain
excluded, and assertSafePathToken still blocks traversal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): generic geospatial query guidance
Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/
ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a
dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the
entity geometry, build an area-of-interest polygon, test with the engine's
containment/proximity/overlap predicate; mind lon/lat argument order). Steers the
solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted
geospatial predicates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): parse code/dependency text by language grammar
Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by
the language or manifest format (Java import keep-package-path allowing underscores/
mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse
JSON cell source before language rules; JSON manifests flatten the dependency object
keys), stripping comments/prose and splitting multi-import lines; (2) on a
de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the
weight column from the population the question names, not silently. Steers off one
broad regex that drops valid identifiers and matches prose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): source filters/dates/measures from the owning fact grain
Add a <sql_craft> rule for joined fact tables at different grains (parent order
vs child line item): read each predicate, calendar bucket, and measure from the
table whose grain the question names, not whichever is in scope post-join. An
order-grain filter ("orders that are Complete", "the order's creation date")
must come from the parent even though the child carries its own status/created_at;
line price/cost come from the child. Mirror at metric grain: don't combine a
parent-grain count with child rows (num_of_item * SUM(line_price) per line) —
aggregate each measure at its own grain before combining.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration
When an entity carries a multi-valued classification array (IPC/CPC codes, tags)
and the methodology counts entities-per-class or a concentration/diversity metric
(HHI, originality, share), pick ONE representative per entity first (the array's
main/primary/first flag, else a defined fallback like most-frequent), then
aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a
count of entities. Unnesting the array otherwise multiplies an entity's weight by
its code count, inflating per-class frequencies and skewing the ranking/score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(connectors): introspect BigQuery datasets hosted in foreign projects
A dataset_ids/dataset_id entry may now be written `project.dataset` to
introspect a dataset hosted in another project while query jobs still bill to
credentials.project_id. Entries are parsed once at the config boundary into
canonical {project, dataset} pairs; introspection, primary-key discovery,
testConnection, getTableRowCount, and listTables (grouped per project) all
resolve in the dataset's own project, and scanned tables are labeled with that
project so sampling, distinct-value, and read queries resolve. Bare entries are
unchanged.
Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md.
* feat(scan): durable, resumable, bounded relationship detection during enrichment
Move the enrichment persistence boundary to the cost boundary and bound the
open-ended relationship stage (spec 19).
- Checkpoint descriptions + embeddings into the queryable `_schema` manifest
(and the raw enrichment artifacts) before relationship detection runs, via a
new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted,
budget-truncated, or failed relationship stage now degrades to "no joins",
never "no descriptions".
- Resume the enrichment cache by content identity: re-key the SQLite stage store
on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes
finished descriptions/embeddings instead of re-paying for LLM work. The
disposable cache recreates its table if the on-disk key shape differs.
- Make the relationship stage observable and bounded: a sticky wall-clock budget
(`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress
+ honored `ctx.signal`, threaded through profiling, validation, and composite
detection. On exhaustion/abort it stops scheduling, finalizes, and returns a
partial result instead of throwing or hanging.
- Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason`
+ recoverable `relationship_detection_partial` warning). A graceful partial saves
as a completed stage and resumes cheaply; raising the budget changes inputHash
and forces a fresh, fuller run. A process killed mid-stage saves nothing.
Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation
notes to specs/19 and move the intake draft to done/.
Also carries the in-tree per-table enrichment LLM timeout work it builds on
(`description-generation.ts` + the `enrichment_timeout` warning code), which is
intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a
separately-building commit.
* feat(scan): bound + retry the per-table enrichment LLM call
The batched table-description call had no retry (sampleTable retried 3x, this did
not), so a single transient backend error (e.g. an overloaded/burst rejection when
many tables enrich concurrently) silently nulled a whole table's descriptions —
observed dropping ~70% of a db's tables during a bad window despite ample quota.
- Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS).
- Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a
wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried
(one wedge stays one timeout, not 3x).
- Granular per-table progress + start/done/retry/timeout logging.
Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable;
this makes more of them complete.
* feat(scan): survive a hung LLM enrichment backend and resume descriptions
Two compounding failure modes on the per-table description-enrichment path (spec 20):
Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout.
Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output).
Spec 20 refined + intake draft moved to done/.
* feat(scan): selective enrichment stages (--stages) + per-stage cache keys
Split the single coarse enrichment cache key into per-stage hashes
(descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding
identity + description digest; relationships <- snapshot + relationship settings
+ LLM identity), so changing one stage's inputs invalidates only that stage and
never throws away the expensive per-table descriptions on an unrelated edit.
Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an
already-ingested connection: a named stage bypasses the completed-stage
short-circuit while the per-table descriptions resume record still skips
already-enriched tables, and unselected stages are left untouched on disk. Feed
embeddings + relationships their description context from the on-disk _schema
when descriptions do not run this invocation, and carry descriptions into the
llmProposals evidence packet (closing a latent gap on the full-run path too).
Surface an enrichment_stage_stale warning when an unselected stage's inputs have
drifted, rather than silently cascading the work.
Implements spider2-specs/specs/21-selective-enrichment-stages.md.
* test(analytics): realign SKILL.md acceptance test with the evolved skill
Three assertions in analytics-skill-content.test.ts drifted from the analytics
SKILL.md as later iterations edited the skill without updating the test:
- the sub-heading was renamed Window functions -> Ordering & aggregation
determinism (iter2), so follow the source name;
- the rule "Expose identity, not just the label" was renamed to "Project BOTH
identity and label" (spec 14), so match the new wording;
- the dialect-FQTN guard false-positived on the Java package example
com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read
as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the
guard stays at full strength without weakening it.
* fix(scan): --stages subset must not delete unselected stages' on-disk artifacts
A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions
from the written _schema. runLocalScan writes the structural manifest shard from
the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as
scan-managed and overwrites them with whatever the run emits — none, on a subset
that skips descriptions. Enrichment then read the already-wiped shard via
loadPriorDescriptions and had nothing to restore.
runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run
if descriptions ran, else loaded from the on-disk _schema) instead of [], and
runLocalScan captures the prior descriptions before the structural write and feeds
them to both the structural write and enrichment, so an unselected stage's
artifacts survive. Joins were already preserved for --stages descriptions via the
manual/inferred preservedJoins path.
Tests: a full runLocalScan --stages relationships path test (RED without the fix,
GREEN with it — the earlier unit test missed the structural-pre-write ordering),
plus enrichment-layer contract tests for both directions. Validated live on
northwind: --stages relationships keeps all 110 descriptions + 22 joins (was
wiping to 0); --stages descriptions restores descriptions from the spec-20 resume
record (no LLM calls) while keeping joins.
* feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE
bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md):
- Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params),
dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard.
- Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection
predicates, areal allocation via ST_AREA(ST_INTERSECTION()).
- SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note.
Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits).
* spec(ingest): resumable + fault-tolerant source ingest (#22)
Refined spec for two source-ingest durability gaps surfaced by a real
user report on a ~2-day dbt ingest: (1) interrupted runs restart every
work unit from scratch (no cross-run reuse), and (2) the final
integration gate is all-or-nothing — one unfixable artifact discards the
whole run.
Design: automatic content-keyed work-unit resume reusing the scan
durability primitive (specs 19/20), plus a deterministic dangling-edge
prune that replaces the fatal final-gate throw so a single bad model
costs only that model, not the run. Prune operates on the integrated
tree and never poisons the cache, so resume and prune self-heal.
* refactor(scan): route enrichment resume through shared cache
* feat(ingest): replay cached work unit patches
* refactor(ingest): return structured final gate findings
* feat(ingest): prune final gates without LLM repair
* docs(ingest): document final gate pruning
* test(ingest): cover stale work unit cache recompute
* fix(ingest): refresh stale cache recompute metadata
* test(ingest): cover missing-target prune and self-heal
* fix(ingest): defer pruneable final gate findings
* fix(ingest): replay pruned cached work unit intent
* chore(ingest): verify resumable source ingest self-heal
* test(ingest): cover final gate prune source path resolution
* fix(ingest): resolve final gate prune sources canonically
* fix: defer wiki ref cleanup out of stage 3
* test: cover non-cascading final gate join pruning
* test: cover intrinsic final gate source drops
* docs(spec): record implementation notes for resumable source ingest (#22)
* fix(ingest): prune dangling joins on untouched sources and stop storing cache patch text
- final gate: drop a dropped source's dangling join edges from every owner on
the connection, including untouched siblings the touched-scoped gate never
revisits, so a committed orphan join can't break SL queries
- work-unit cache: drop the stored patch text; replay re-derives the diff from
the before/after artifact snapshots, carrying each touched file only once
- scan enrichment: checkpoint recomputed embeddings before the kill-prone
relationship stage even when descriptions load from disk, using the
best-available description set so the manifest merge can't delete them
- sl: extract listSlSourceFiles so the final gate and resolveSlSourceFile
share one listing path
* fix(scan): accept relationships mode in enrichment state metadata
Listing run stages after a relationships-mode scan threw "Invalid scan
enrichment cache metadata" because the parser hand-enumerated only the
structural/enriched modes while a relationships scan persists its stage with
mode 'relationships'. Derive the mode and stage allowlists from the canonical
KTX_SCAN_MODES and KTX_SCAN_ENRICHMENT_STAGES registries so the runtime check
cannot drift from the type again.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Andrey Avtomonov
|
f65a5b0e2e
|
feat: ktx batch — scan resilience, analytics SQL craft, connector hardening (#312)
* docs: add spider2-specs handoff directory for benchmark-driven feature specs
* feat(cli): connection-scoped wiki pages
Add an optional `connections` frontmatter field so database-specific wiki
knowledge can be scoped to a connection without polluting searches about other
databases, while page keys stay a flat, globally-unique namespace.
- connections: single string or list; absent/empty ⇒ unscoped (applies to all)
- wiki_search (MCP) and `ktx wiki --connection` return unscoped ∪ matching
pages, filtered at the disk-load seam so all three search lanes draw their
candidate pool from the already-scoped set (not a post-filter)
- wiki_write accepts connections with REPLACE semantics and rejects a
connection-scoped write whose key collides with a disjoint-connection page
(data-loss guard; hard error, no silent clobber)
- explicit connection-id args (wiki_search, memory_ingest, ktx wiki) are
validated against ktx.yaml via a shared assertConfiguredConnectionId, which
also closes the prior gap where memory_ingest's connectionId was unvalidated;
persisted ids absent from config warn (not fail) in `ktx status`
- prompt guidance in the wiki_capture skill and external-ingest prompt; the
session connectionId is surfaced to the memory agent and ingest work units
Implements spider2-specs/specs/01-connection-scoped-wiki.md; intake draft moved
to spider2-specs/done/.
* docs(spider2-specs): add specs/ refinement stage and composite-key join spec
Describe the todo/ → specs/ → done/ pipeline in the README (refined specs are
the durable artifact; intake drafts move to done/ on ship) and add a
MEDIUM-priority spec for multi-column composite-key join detection found during
the first sqlite smoke test.
* feat(cli): add --verbatim ingest mode for authoritative documents
Store each --text/--file document body unchanged as a GLOBAL wiki page
instead of routing it through the memory agent, which may rewrite,
condense, or re-title it. The LLM derives only metadata (summary, tags,
sl_refs) and only for frontmatter fields the document does not already
set; the stored body is written by code and never edited.
- Deterministic page key: files derive it from the filename, inline
text from its leading Markdown heading (headless inline text is
rejected — pass it as --file instead).
- Idempotent: re-running the same body is a no-op; a different body at
the same key fails loudly rather than overwriting.
- Works with llm.provider.backend: none, deriving a degraded summary
from the heading or first sentence.
- Existing frontmatter (including unmodeled fields like effective_date)
passes through untouched; --connection-id scopes the page.
* feat(cli): SQL-authoring craft and per-dialect notes tool for the analytics skill
Spec 07: add a dialect-agnostic <sql_craft> block to the ktx-analytics skill (schema discovery, composition, window-function correctness, numeric precision, answer completeness) with one worked window-then-filter example. Workflow steps gain pointers into it; existing guidance is unchanged.
Spec 08: add a read-only sql_dialect_notes MCP tool returning a connection's engine SQL conventions (FQTN form, identifier quoting/case, date/time, top-N idiom, JSON access), resolved through the existing sqlAnalysisDialectForDriver path. Notes are per-dialect markdown files under context/sql-analysis/dialects, served by the tool and copied to dist (package-internal, never installed). Non-SQL connections return a clear KtxExpectedError. The flat skill gains a one-line pointer to the tool.
Both spider2-specs intake drafts move to done/ with implementation notes.
* feat(cli): tolerate objects that fail introspection during scan
Isolate per-object introspection failures so one broken or inaccessible object no longer zeroes out a connection's whole semantic layer: the sqlite and bigquery connectors introspect each object defensively (tryIntrospectObject), the live-database adapter records a scan outcome and fetch report, and enabled_tables accepts catalog.db.name, db.name, or bare names with a clear no-match error. Includes matching ktx-daemon introspection changes, docs, and tests.
* docs(spider2-specs): add 06-scan-tolerate-broken-objects spec
* feat(cli): generalize analytics fan-out rule to multi-hop join chains
The ktx-analytics skill's fan-out rule only reliably caught single-hop
inflation; agents still silently fanned out on multi-hop chains where the
offending one-to-many join sits several hops below the SUM/COUNT and is easy
to miss.
Rewrite the Composition rule so the danger reads as cumulative across the whole
chain (pre-aggregate per measure-owning table), add an affirmative
grain-verification habit (default: pre-aggregate to grain; escape hatch:
COUNT(DISTINCT key) for pure counts only; SUM/AVG of a fanned-out measure must
pre-aggregate), and add one generic wrong-vs-right worked example. Content-only
and dialect-agnostic; no new tool, flag, or config.
Implements spider2-specs/specs/09 and annotates spec 07's one-example
constraint as superseded.
* feat(cli): add panel-completeness, time-series window, and text-encoded numeric SQL craft
Extend the analytics skill's <sql_craft> with three correctness habits and
route the dialect-specific halves through sql_dialect_notes:
- Panel completeness (spec 10): full-domain spine -> LEFT JOIN -> COALESCE for
"each/every/all/per" questions, defaulted by measure additivity.
- Time-series windows (spec 11): explicit cumulative frames, calendar-range
rolling windows with minimum-periods guards, and period-over-period via LAG.
- Text-encoded numerics (spec 12): sample distinct values, strip/scale/cast in
one early CTE, and confirm coverage with a failure-detecting cast.
Add per-dialect Series, Rolling window, and Safe cast notes to all seven
dialect files so the skill stays dialect-agnostic while the engine-specific
syntax lives in sql_dialect_notes. Tests updated and passing (19).
* docs(spider2-specs): add specs 10-12 for analytics SQL-craft additions
Refined specs and completion records for the panel-completeness spine (10),
time-series window recipes (11), and text-encoded numeric parsing (12)
implemented in the preceding commit.
* docs(spider2-specs): add backlog intake drafts 13-14
- 13: canonical authoritative-source measures
- 14: output-completeness final check
* skill(analytics): spec 14 output-completeness + iter1 (active column planning)
Bundles two changes (entangled in SKILL.md; future spider2 iterations land as
separate commits):
- spec 14 (output-completeness): multi-part "answer every requested output" rule
+ a "Final completeness check" in workflow Step 6 and <sql_craft>; analytics
skill-content test updated; intake draft -> done/, refined spec added.
- iter1 experiment: spec 14's passive end-check did not change behavior on the
benchmark's output-completeness failures, so (a) the Plan step now writes the
exact output-column list UP FRONT as a contract the final SELECT must match,
and (b) "expose identity" -> "project BOTH the entity id and its name" (covers
both omission directions). All generic craft.
Driven by the Spider 2.0-Lite failure analysis (incomplete output was the
largest failure bucket); benchmark only as motivation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* skill(analytics): iter2 — deterministic order in string/array aggregation
GROUP_CONCAT/string_agg/array_agg element order is undefined without an explicit
ORDER BY; also note SQLite's default text sort is binary/case-sensitive (uppercase
before lowercase) vs case-insensitive (COLLATE NOCASE). Generic SQLite craft.
Spider 2.0-Lite motivation: an ordered-ingredient-list question failed only on the
within-string element order (right elements, wrong order); benchmark as motivation only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(mcp): structured, leveled logging for the MCP server
Add one synchronous pino logger per MCP server process, written through the
io.stderr sink: plain JSON when stderr is not a TTY, colorized pino-pretty
(sync, in-process) when it is. Every tool call logs tool.start with its raw
params BEFORE the handler runs and tool.end after (info / warn past
KTX_MCP_SLOW_TOOL_MS / error), correlated by callId plus sessionId, so a
runaway sql_execution leaves a recoverable start line with its exact SQL and
no matching end. HTTP logs session.open/close and wires the previously-dead
transport.onerror to transport.error; stdio routes its transport error
through the logger. Level via KTX_MCP_LOG_LEVEL (default info). Existing
mcp_request_completed telemetry and registerParsedTool are unchanged; no
worker/async transport and no redaction in v1 (logs are local-only).
Implements spider2-specs/specs/15-mcp-server-structured-logging.md and moves
the intake draft to done/.
* feat(mcp): report uptimeMs in MCP server /health
The /health endpoint now includes uptimeMs (monotonic elapsed time since
the server started), mirroring the Python daemon's uptime_ms telemetry
field.
* feat(cli): bound read-query execution with a per-connection deadline
Enforce one shared query deadline (default 30s, overridable per connection via
query_timeout_ms) on every executeReadOnly path, so an accidentally-expensive
LLM-authored query returns a fast "query exceeded Ns" KtxQueryError instead of
hanging the MCP server.
- New shared contract context/connections/query-deadline.ts
(resolveQueryDeadlineMs, queryDeadlineExceededError); query_timeout_ms added to
the shared warehouse schema; BigQuery's job_timeout_ms removed.
- SQLite runs the read query in a short-lived forked child process and enforces
the deadline with SIGKILL. worker_threads + terminate() was tried first but
cannot interrupt a synchronous better-sqlite3 scan (the native loop never
yields); SIGKILL reclaims the process in ~2ms and keeps the event loop free.
- Remote connectors apply a real server-side statement timeout and re-wrap their
own timeout signal as KtxQueryError: Postgres statement_timeout/57014, MySQL
max_execution_time/3024, Snowflake STATEMENT_TIMEOUT_IN_SECONDS/604, ClickHouse
max_execution_time + aligned request_timeout/159, SQL Server requestTimeout/
ETIMEOUT, BigQuery jobTimeoutMs.
- Relationship validation skips a candidate to review on a deadline timeout
instead of aborting the pass; the deadline surfaces through the existing MCP
pino logger as a matched tool.start/tool.end(error) pair (no new logging code).
Also fixes a pre-existing, unrelated invalid cast in mcp-server-factory.test.ts
that was breaking tsc -p tsconfig.test.json.
* docs(spider2-specs): mark spec 16 (bounded query execution) done
Append Implementation notes to the refined spec (what shipped, where, and the
worker-thread -> child-process+SIGKILL deviation with its evidence) and move the
intake draft from todo/ to done/.
* skill(analytics): iter3 — measure-as-amount, inter-event gap, top-per-metric career
Three generic interpretation rules: a named business measure (sales/revenue/spend)
means its amount not a row count; "inter-event duration/gap" is LAG/LEAD time-between
events not a magnitude column; "highest across several achievements" aggregates per
metric over the whole history. All three demonstrably FIRE (verified on local008/003/152
SQL). local008 flips to correct (mechanism-aligned). 003/152 still fail on a different
axis (source-column / grouping). Generic craft; benchmark only as motivation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* skill(analytics): spine-for-extreme-selection + aggregate-over-selected-set
Two generic answer-completeness refinements:
- Selecting the extreme group (lowest/highest count over a period/category
domain) must rank over the COMPLETE spine, not only groups with fact rows —
an empty period is a genuine 0 and often the true minimum.
- An aggregate scoped to a per-entity selected set ('avg revenue per actor in
those top-3 films') is computed ACROSS that set, distinct from the per-item
value; project both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter2 — sharpen extreme-selection spine + top-N ranking-measure
- spine-for-extreme: concrete cue that a zero-row period never appears in a
GROUP BY of the facts; generate the full calendar, LEFT JOIN, COALESCE, then rank.
- aggregate-over-selected-set: top-N selection ranks by the named ranking measure
(the item's own revenue), independent of the per-item share that feeds the aggregate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter3 — comparison-between-two-extremes is one wide row
Distinguishes a cross-item comparison ('the difference between the highest and
lowest month' -> single wide row, both extremes side by side + the comparison
column) from 'report a metric for each group' (-> stays long). Generic, question-
derived; targets the wide-vs-long shape gap without affecting per-group long output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter4 — anchor a period bucket to the named lifecycle event
When a record carries multiple lifecycle timestamps (created/placed, approved,
shipped, delivered, completed, settled) and the question counts/measures records
in a named *completed state* by period ("delivered orders by month", "shipped
items per week"), bucket the period by that named event's own timestamp, not the
record-creation timestamp; the state value is the qualifying filter, the matching
timestamp is the time anchor. Wording priority is explicit — purchased/placed/
created/submitted/ordered keep the start-event timestamp — and a non-temporal
state filter (counts by customer/city/seller with no period) introduces no anchor.
Generic analytics craft: counting completed-state records by their creation date
silently answers "records that later reached that state, grouped by when they
started" instead of the question asked. Surfaced via the spider2-autofix loop;
FAIR_PRODUCT (adversary-screened, restatable from question wording + schema/
semantic-layer lifecycle descriptions, no gold dependency).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter5 — canonicalize observed URL-path variants before page-level analysis
When a question groups/filters/sequences web pages by a path/url column, sample
its distinct values; if the data itself shows /route and /route/ variants for the
same page context, canonicalize in an early CTE (preserve / as root, strip trailing
slashes from non-root paths, map an observed empty path to / only when the column is
a URL path with blank root-page events) and use the canonical path everywhere above.
Explicitly forbids inventing aliases the data doesn't show: no merging different
route names, no stripping query/fragment/host/scheme, no lowercasing, and no
canonicalization when the question asks for raw URL/path or slash-vs-no-slash diffs.
Generic web-analytics craft: raw request logs routinely store the same user-visible
page with and without a trailing slash, so grouping raw labels silently splits one
page into several. Surfaced via the spider2-autofix loop (Codex runner, round r2);
FAIR_PRODUCT (adversary-screened, restatable from URL-path semantics + page-grain
question wording + solver-observed distinct values, no gold dependency). The rule
fired mechanism-aligned on both targets; flipped local330 (landing/exit page counts),
local331 residual is a separate sequence-semantics axis beyond canonicalization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): iter6 — coverage over a selected group is a set-membership aggregate
When a question first selects a group of entities ("the top 5 actors", "these
products") and then asks what count/share/percentage of a DIFFERENT subject domain
relates to *these* selected entities ("what % of customers rented films featuring
these actors"), the subject set is the UNION across the whole group: count DISTINCT
subject ids once across the selected entities and return one collective value at the
subject-domain grain — not one row per selected entity (which double-counts subjects
related to more than one entity and answers a different question). Narrowly guarded:
emit one row per entity only when the wording says "for each / per / by / list" or
asks for each entity's own metric ("top 5 players and their batting averages").
The collective-coverage cousin of the existing per-entity selected-set rule. Generic
analytics craft (per-entity metric vs set-level coverage). Surfaced via the
spider2-autofix loop (Codex runner, round r3); FAIR_PRODUCT (adversary-screened,
restatable from wording alone, no gold dependency). Flipped local195 mechanism-aligned
(union COUNT(DISTINCT customer)/total, one scalar); 0 regression across 5 passing
per-entity top-N guards (local023/024/029/212/221 stayed long).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* skill(analytics): label-only joins must LEFT JOIN — incomplete dims silently drop fact rows
Mirror of the existing fan-out rule for the DROP direction: an inner JOIN to a
dimension table used only to attach a display attribute silently discards every
fact row whose key has no parent when the dimension is incomplete (trimmed
catalogs, late-arriving / SCD-gap rows), shrinking counts/sums and the universe
over which shares/averages/medians are computed. Guidance: LEFT JOIN pure
enrichment; inner-join a dimension only when intended as a filter; key the
aggregate/GROUP BY on the fact column, not the dimension column.
Spider2 autofix round 'joindim': flips complex_oracle local050 (FAIL->PASS,
official scorer) — solver dropped the gratuitous products inner-join and
recovered the exact gold. local060/063 also adopt LEFT JOIN (rule fires) but
remain gold-convention-blocked. Guards local061/067 held.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(spider2-specs): add todo/17 — lifecycle-event metrics (semantic-layer)
Draft intake spec surfaced by the spider2-autofix loop (round r1): the model-layer
form of the shipped iter4 lifecycle-date-anchoring skill rule — infer per-state
lifecycle-event metrics (e.g. delivered_orders with defaultTimeDimension = the
delivery timestamp) during enrichment so the correct time anchor is the default for
any consumer, not only an agent that loaded the skill. Generic; FAIR_PRODUCT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(connectors): accept leading underscore in connection/identifier ids
The safe-identifier validator regex /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/ allowed an
underscore everywhere except the first character, so a connection id / database
name that legitimately starts with '_' (valid in Snowflake, e.g. _1000_GENOMES)
could never be ingested or queried. Allow a leading underscore across all 16
duplicated validators (connection ids, source ids, page/wiki keys, warehouse-
verification tool schemas). Path-safety is unaffected — '.' and '/' remain
excluded, and assertSafePathToken still blocks traversal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): generic geospatial query guidance
Add a Snowflake ST_* dialect note (ST_MAKEPOINT lon-first, ST_DWITHIN/ST_CONTAINS/
ST_WITHIN/ST_INTERSECTS, bbox->polygon via ST_MAKEPOLYGON/ST_MAKELINE) and a
dialect-agnostic 'Spatial predicates' recipe in the analytics skill (resolve the
entity geometry, build an area-of-interest polygon, test with the engine's
containment/proximity/overlap predicate; mind lon/lat argument order). Steers the
solver off hand-rolled lat/lon BETWEEN boxes toward correct, index-assisted
geospatial predicates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): parse code/dependency text by language grammar
Add two generic <sql_craft> rules: (1) parse imported/required/loaded packages by
the language or manifest format (Java import keep-package-path allowing underscores/
mixed-case; Python import/from + alias stripping; R library/require; .ipynb parse
JSON cell source before language rules; JSON manifests flatten the dependency object
keys), stripping comments/prose and splitting multi-import lines; (2) on a
de-duplicated table with a documented copy/occurrence count, choose COUNT(*) vs the
weight column from the population the question names, not silently. Steers off one
broad regex that drops valid identifiers and matches prose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): source filters/dates/measures from the owning fact grain
Add a <sql_craft> rule for joined fact tables at different grains (parent order
vs child line item): read each predicate, calendar bucket, and measure from the
table whose grain the question names, not whichever is in scope post-join. An
order-grain filter ("orders that are Complete", "the order's creation date")
must come from the parent even though the child carries its own status/created_at;
line price/cost come from the child. Mirror at metric grain: don't combine a
parent-grain count with child rows (num_of_item * SUM(line_price) per line) —
aggregate each measure at its own grain before combining.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(analytics): collapse multi-valued classes to one representative per entity before counting/concentration
When an entity carries a multi-valued classification array (IPC/CPC codes, tags)
and the methodology counts entities-per-class or a concentration/diversity metric
(HHI, originality, share), pick ONE representative per entity first (the array's
main/primary/first flag, else a defined fallback like most-frequent), then
aggregate; and use COUNT(DISTINCT entity) when the denominator is defined as a
count of entities. Unnesting the array otherwise multiplies an entity's weight by
its code count, inflating per-class frequencies and skewing the ranking/score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(connectors): introspect BigQuery datasets hosted in foreign projects
A dataset_ids/dataset_id entry may now be written `project.dataset` to
introspect a dataset hosted in another project while query jobs still bill to
credentials.project_id. Entries are parsed once at the config boundary into
canonical {project, dataset} pairs; introspection, primary-key discovery,
testConnection, getTableRowCount, and listTables (grouped per project) all
resolve in the dataset's own project, and scanned tables are labeled with that
project so sampling, distinct-value, and read queries resolve. Bare entries are
unchanged.
Implements spider2-specs/specs/18-bigquery-cross-project-datasets.md.
* feat(scan): durable, resumable, bounded relationship detection during enrichment
Move the enrichment persistence boundary to the cost boundary and bound the
open-ended relationship stage (spec 19).
- Checkpoint descriptions + embeddings into the queryable `_schema` manifest
(and the raw enrichment artifacts) before relationship detection runs, via a
new `onCheckpoint` hook + `writeLocalScanEnrichmentCheckpoint`. An interrupted,
budget-truncated, or failed relationship stage now degrades to "no joins",
never "no descriptions".
- Resume the enrichment cache by content identity: re-key the SQLite stage store
on `(connection_id, stage, input_hash)` so a re-run with a fresh runId resumes
finished descriptions/embeddings instead of re-paying for LLM work. The
disposable cache recreates its table if the on-disk key shape differs.
- Make the relationship stage observable and bounded: a sticky wall-clock budget
(`scan.relationships.detectionBudgetMs`, default 600000 ms) + per-unit progress
+ honored `ctx.signal`, threaded through profiling, validation, and composite
detection. On exhaustion/abort it stops scheduling, finalizes, and returns a
partial result instead of throwing or hanging.
- Mark a budget/abort-truncated result partial (diagnostics `partial`/`partialReason`
+ recoverable `relationship_detection_partial` warning). A graceful partial saves
as a completed stage and resumes cheaply; raising the budget changes inputHash
and forces a fresh, fuller run. A process killed mid-stage saves nothing.
Document `detectionBudgetMs` in the ktx.yaml reference. Append implementation
notes to specs/19 and move the intake draft to done/.
Also carries the in-tree per-table enrichment LLM timeout work it builds on
(`description-generation.ts` + the `enrichment_timeout` warning code), which is
intertwined in `local-enrichment.ts`/`types.ts` and cannot be split into a
separately-building commit.
* feat(scan): bound + retry the per-table enrichment LLM call
The batched table-description call had no retry (sampleTable retried 3x, this did
not), so a single transient backend error (e.g. an overloaded/burst rejection when
many tables enrich concurrently) silently nulled a whole table's descriptions —
observed dropping ~70% of a db's tables during a bad window despite ample quota.
- Wrap generateObject in retryAsync (3 attempts + backoff; KTX_ENRICH_LLM_ATTEMPTS).
- Fresh per-attempt timeout (KTX_ENRICH_LLM_TIMEOUT_MS, default 120s) still bounds a
wedged wide table; a timeout is surfaced as KtxAbortedError so it is NOT retried
(one wedge stays one timeout, not 3x).
- Granular per-table progress + start/done/retry/timeout logging.
Composes with spec 19 (its non-goal #1): spec 19 makes completed descriptions durable;
this makes more of them complete.
* feat(scan): survive a hung LLM enrichment backend and resume descriptions
Two compounding failure modes on the per-table description-enrichment path (spec 20):
Enforced per-table timeout for subprocess backends. The runtime declares whether it owns an SDK subprocess (subprocessForkSpec on KtxLlmRuntimePort); codex/claude-code calls run behind a ktx-owned detached child that is tree-killed (SIGKILL of the process group on POSIX, taskkill /T on Windows) on the deadline or ctx.signal, reaping the wedged model grandchild. HTTP backends keep native fetch abort. Default stays 120s, one-wedge-one-timeout.
Incremental, resumable descriptions persistence. generateDescriptions flushes enriched tables per batch to an inputHash-tagged durable record (at a stable, non-syncId path) plus only the changed manifest shards, skips already-enriched tables on resume, and never lets one table's failure discard the stage (a skipped table costs one missing description, not the whole stage's output).
Spec 20 refined + intake draft moved to done/.
* feat(scan): selective enrichment stages (--stages) + per-stage cache keys
Split the single coarse enrichment cache key into per-stage hashes
(descriptions <- snapshot + LLM identity; embeddings <- snapshot + embedding
identity + description digest; relationships <- snapshot + relationship settings
+ LLM identity), so changing one stage's inputs invalidates only that stage and
never throws away the expensive per-table descriptions on an unrelated edit.
Add `ktx ingest --stages <list>` to force-re-run a chosen subset on an
already-ingested connection: a named stage bypasses the completed-stage
short-circuit while the per-table descriptions resume record still skips
already-enriched tables, and unselected stages are left untouched on disk. Feed
embeddings + relationships their description context from the on-disk _schema
when descriptions do not run this invocation, and carry descriptions into the
llmProposals evidence packet (closing a latent gap on the full-run path too).
Surface an enrichment_stage_stale warning when an unselected stage's inputs have
drifted, rather than silently cascading the work.
Implements spider2-specs/specs/21-selective-enrichment-stages.md.
* test(analytics): realign SKILL.md acceptance test with the evolved skill
Three assertions in analytics-skill-content.test.ts drifted from the analytics
SKILL.md as later iterations edited the skill without updating the test:
- the sub-heading was renamed Window functions -> Ordering & aggregation
determinism (iter2), so follow the source name;
- the rule "Expose identity, not just the label" was renamed to "Project BOTH
identity and label" (spec 14), so match the new wording;
- the dialect-FQTN guard false-positived on the Java package example
com.planet_ink.coffee_mud, whose backticks made a 3-segment package path read
as a BigQuery/Snowflake `a.b.c` table reference. Drop the backticks so the
guard stays at full strength without weakening it.
* fix(scan): --stages subset must not delete unselected stages' on-disk artifacts
A --stages subset that omitted descriptions wiped all on-disk ai/db descriptions
from the written _schema. runLocalScan writes the structural manifest shard from
the bare snapshot BEFORE enrichment runs, and the shard merge treats ai/db as
scan-managed and overwrites them with whatever the run emits — none, on a subset
that skips descriptions. Enrichment then read the already-wiped shard via
loadPriorDescriptions and had nothing to restore.
runLocalScanEnrichment now returns the best-available descriptions (fresh-this-run
if descriptions ran, else loaded from the on-disk _schema) instead of [], and
runLocalScan captures the prior descriptions before the structural write and feeds
them to both the structural write and enrichment, so an unselected stage's
artifacts survive. Joins were already preserved for --stages descriptions via the
manual/inferred preservedJoins path.
Tests: a full runLocalScan --stages relationships path test (RED without the fix,
GREEN with it — the earlier unit test missed the structural-pre-write ordering),
plus enrichment-layer contract tests for both directions. Validated live on
northwind: --stages relationships keeps all 110 descriptions + 22 joins (was
wiping to 0); --stages descriptions restores descriptions from the spec-20 resume
record (no LLM calls) while keeping joins.
* feat(dialects): bigquery nested-data (ARRAY/STRUCT/UNNEST), geospatial (GEOGRAPHY), SAFE_DIVIDE
bigquery.md lacked the two sections that define BigQuery analytics (present in snowflake.md):
- Nested & repeated data: UNNEST to flatten arrays of STRUCTs (GA360 hits, GA4 event_params),
dot-notation field access, key-value param scalar-subquery extraction, fan-out/COUNT(DISTINCT) guard.
- Geospatial (GEOGRAPHY): ST_GEOGPOINT (lon-first), containment/proximity/distance/intersection
predicates, areal allocation via ST_AREA(ST_INTERSECTION()).
- SAFE_DIVIDE for zero-denominator-safe rates; sharded-table shard-presence note.
Generic BigQuery craft surfaced by sql_dialect_notes; product-completeness (any BQ analyst benefits).
* feat(dialects): sqlite ROUND half-up FP-underflow note (+1e-9 before ROUND)
SQLite ROUND(x,n) rounds half-away-from-zero, but binary FP stores an exact
half-way value just below it, so ROUND(6.475,2) returns 6.47 not 6.48. Add a
dialect note: nudge by a tiny epsilon (1e-9) below display precision before
rounding for deterministic half-up, leaving non-boundary values unchanged.
Generic SQLite craft surfaced by sql_dialect_notes (any analyst rounding a
displayed average/rate/price benefits).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(analytics): list-as-delimited-string, answer-literally, drop free-text columns
Add SKILL.md guidance to emit list-valued answer cells as delimited
STRING (not ARRAY/repeated column), answer the literal ask without
unrequested transformations (HAVING for aggregate bounds), and avoid
projecting unrequested free-text columns that corrupt row-delimited output.
* fix(scan,mcp): gitignore runtime logs, budget-guard LLM proposal, validate enrich timeout
- gitignore `.ktx/logs/` in both scaffold + setup-merge lists: the managed MCP
daemon writes raw tool params (SQL, memory_ingest content) to mcp.log under a
version-controlled `.ktx/`, and snowflake.log already sat there unprotected.
- gate the LLM relationship proposal on the detection budget/abort signal so an
exhausted or aborted stage cannot start a fresh LLM call; document the boundary.
- validate KTX_ENRICH_LLM_TIMEOUT_MS (NaN/0 → 120s default) like enrichAttempts,
so a bad value no longer times out every table immediately.
- daemon introspection now warns on malformed column/FK rows instead of dropping
them silently, matching the table-row path and the "surface broken objects" goal.
- docs: document `ktx wiki -c/--connection`; fix the SQLite query-deadline schema
doc (forked-subprocess SIGKILL, not worker-thread termination).
* fix(scan,wiki,mcp): address PR #312 review findings
- scan: key the description pipeline (resume map, enriched-schema and
embedding-text lookups, manifest write/read) by full table identity via
tableRefKey/buildTableRef, so two same-named tables in different schemas no
longer cross-assign descriptions or skip a sibling on resume
- scan: re-throw a genuine context cancel during the batched description LLM
call so Ctrl-C resumes the stage instead of nulling tables and recording it
completed; per-table timeouts still degrade (context.signal not aborted)
- scan: report statisticalValidation 'skipped' (not 'completed') when a
budget/abort stop leaves relationship profiling partial
- wiki: sync the full page corpus into the sqlite index and filter only the
candidate/result set, so a connection-scoped search no longer prunes other
connections' pages and cached embeddings from the shared index
- wiki: route verbatim ingest through the canonical writePageAndSync so
contentHash is set and later syncs can short-circuit
- mcp: drop the as-unknown-as cast in serializeMcpError
- dialects/analytics: document the integer-division trap on postgres/sqlite/tsql
Adds regression tests for each behavior change.
* fix(wiki): scope connection filter before SQLite lane limit
Connection-scoped wiki search applied the connectionId allowlist after
the lexical/semantic lanes had already truncated to laneCandidatePoolLimit
over the full (connection-agnostic) corpus. When the requested connection
was a minority of a large corpus, its pages were crowded out of the
candidate pool before filtering, so a semantic-only match could be missed
outright and lexical hits under-ranked.
Push the path allowlist into searchLexicalCandidates/searchSemanticCandidates
so LIMIT applies to in-scope rows, matching what the token lane already did,
and drop the now-redundant post-limit JS filters.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Pintouch
|
2afab61417
|
feat(connectors): add MongoDB connector (#305) (#310)
* refactor(connectors): split KtxDialect into core and KtxSqlDialect Separate the dialect contract into a driver-agnostic core (display/ref formatting and type mapping) and a SQL-only extension (query generators). The catalog and entity-details paths resolve the core dialect for any snapshot driver, so it must stay free of SQL generation; this is the prerequisite refactor for adding non-SQL primary sources. - KtxDialect keeps type, formatDisplayRef, parseDisplayRef, columnDisplayTablePartCount, mapDataType, mapToDimensionType - KtxSqlDialect extends it with quoteIdentifier, formatTableName, and the query/sample/statistics generators; the 7 SQL dialects implement it - add getSqlDialectForDriver for SQL drivers; the 7 connectors and the relationship-benchmark harness consume it - thread the relationship pipeline (profiling/validation/composite/ discovery) as KtxSqlDialect | null so a non-SQL source skips coverage SQL and its candidates stay in review; local-enrichment builds the SQL dialect only when the connector advertises readOnlySql Pure extraction: no behavior change for the existing 7 drivers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(connectors): add MongoDB connector for issue #305 Add a read-only MongoDB connector that treats a database as a primary context source: collections map to tables and inferred top-level fields to columns. MongoDB is the first non-SQL source (readOnlySql: false), so ktx sql and metric compilation do not apply, but its collections flow through ingest, descriptions, and relationship discovery. - schema-inference: infer a flat column schema from the most recent sample_size documents (by _id desc, or order_by for non-ObjectId keys). Union BSON types per field, mark multi-type fields mixed (string), keep sub-documents/arrays as a single opaque json column, derive nullability from presence, treat _id as the primary key - connector: KtxMongoDbScanConnector behind an injectable client seam; strictly read-only (find/listCollections/estimatedDocumentCount only), no executeReadOnly; resolves env:/file: via resolveKtxConfigReference - core-only KtxMongoDbDialect and a live-database introspection adapter - wire the mongodb driver: driver union, dialect registry, driver registration (scopeConfigKey databases), mongodbConnectionSchema, connection-drivers, normalizeDriver, the live-database route, and the ktx setup picker. ktx sql is refused by the read-only SQL capability gate - tests: schema inference, connector snapshot via a fake client, dialect, driver-schema parsing, and the ktx sql rejection Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(integrations): document the MongoDB primary source Add a MongoDB section to the primary-sources reference: connection config (url, databases, enabled_tables, sample_size, order_by), mongodb+srv/TLS/ Atlas notes, the schema-inference explainer, a features matrix, and the non-SQL caveat. Update the frontmatter and connection field reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(connectors): address review blockers on the MongoDB connector - introspect: skip estimatedDocumentCount for views. The count command is rejected on a MongoDB view (CommandNotSupportedOnView), so counting a view aborted introspect for the whole connection; compute estimatedRows only for real collections, as ClickHouse does. - sl: refuse a semantic-layer query against a non-SQL connection instead of defaulting it to the Postgres dialect. compileLocalSlQuery (the shared CLI + MCP path) now rejects a driver with no SQL dialect via the new isSqlQueryableDriver authority, keeping MongoDB context-only per issue #305. - tests: cover input.tableScope and the empty-scope skip for the Mongo connector (the scan layer does not post-filter), the view no-count path, and the ktx sl query refusal for a mongodb connection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * polish(mongodb): compute sampled nullCount and document sampling caveats Address the non-blocking review notes: - sampleColumn now counts null/absent values over the sampled window instead of returning nullCount: null, since the documents are already in hand - warn that a custom order_by must be indexed (an unindexed sort hits MongoDB's in-memory sort limit on large collections) in the connection schema and docs - note that sampled values for nested fields are stringified, not faithfully serialized, so the json opacity is deliberate Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): add a MongoDB connector example A manual, container-backed example mirroring examples/postgres-historic: - docker-compose.yml + init/seed.js seed a representative dataset (nested documents, arrays, a Decimal128, a mixed-type field, a nullable field, an ObjectId reference, and a view) on first container start - scripts/smoke.sh + introspect-smoke.mjs assert the connector's inferred schema with no LLM credentials — the same introspection entry point ktx ingest's database-schema stage uses, including the view-no-count path - README.md documents the smoke and a full keyless ktx ingest run (claude-code LLM + managed sentence-transformers embeddings) Works with Docker Compose or podman compose. Verified end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: ignore examples/** in knip to fix dead-code false positives The MongoDB connector example files (examples/mongodb/init/seed.js and examples/mongodb/scripts/introspect-smoke.mjs) are used at runtime but were flagged as unused by knip. Add examples/** to the ignore array, matching the existing .context/** entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0114qQV8fJ5a5ME3XbMVRzbL * fix(mongodb): refuse non-SQL connections before SQL analysis `ktx sql` and the MCP sql_execution tool resolved a SQL-analysis dialect (falling back to Postgres for a non-SQL driver) and ran read-only validation before the connector capability gate refused the connection. For a MongoDB connection that spun up the parser/daemon and produced Postgres parser diagnostics instead of a clean non-SQL refusal. Route both entry points through a shared assertSqlQueryableConnection guard before dialect selection, mirroring compileLocalSlQuery. The federated duckdb path has no driver and is exempted at each call site. Add CLI and MCP regression tests asserting validation/connector work never starts for a MongoDB connection. * fix(mongodb): pass CI gates (dialect boundary, secrets, setup test) Three latent failures in the connector surfaced once CI ran on the branch: - connector.ts imported the concrete KtxMongoDbDialect, which the connector dialect-import boundary forbids. Route it through getDialectForDriver('mongodb') and widen inferKtxMongoCollectionColumns to the base KtxDialect (it only uses mapDataType/mapToDimensionType). - detect-secrets flagged a test ObjectId hex and the mongodb+srv example URL; annotate both with allowlist pragmas. - the "shows every supported database" setup test omitted the new MongoDB option. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Luca Martial <48870843+luca-martial@users.noreply.github.com> Co-authored-by: Luca Martial <lucamrtl@gmail.com> Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com> |
||
|
Andrey Avtomonov
|
ca231df5fe
|
fix(gdrive): validate folder access, run config test, harden Drive API (#321)
* fix(gdrive): validate folder access, run config test, harden Drive API Connection test and setup validation now verify folder_id resolves to an accessible Drive folder before counting Docs, via a shared verifyGdriveFolderAndCountDocs helper, so a wrong or unshared folder fails instead of passing with 0 docs. Move gdrive-config.test.ts under test/ so Vitest's test/** glob actually runs it; escape folder_id in the Drive query; add retry/backoff on transient Google API responses; and record skipped non-Google-Doc files in the staged manifest. * chore: sync uv.lock to ktx-daemon/ktx-sl 0.13.1 |
||
|
ARYAN
|
5645dc4d28
|
Add gdrive context source adapter (#209)
* Add gdrive context source adapter * feat(gdrive): normalize internal doc links, tabs, and header/footer structure * fix(gdrive): reject generic source credential flags * test(gdrive): include local adapter in expected list * fix(gdrive): remove dead exports and silence false positive secret checks * fix(setup): restore notion source auth flow |
||
|
semantic-release-bot
|
2830cb5ac7 |
chore(release): 0.13.1 [skip ci]
## [0.13.1](https://github.com/Kaelio/ktx/compare/v0.13.0...v0.13.1) (2026-06-23) ### Bug Fixes * **sqlserver:** hoist leading CTEs out of row-limit derived-table wrap ([#311](https://github.com/Kaelio/ktx/issues/311)) ([ |
||
|
Andrey Avtomonov
|
c815e10fb3
|
fix(sqlserver): hoist leading CTEs out of row-limit derived-table wrap (#311)
* test(sql): cover leading CTE row-limit wrapping * fix(sql): hoist leading CTEs before generic row limits * fix(sqlserver): hoist leading CTEs before TOP row limits * test(scan): note relationship limiter coverage boundary * chore: sync uv.lock to ktx-daemon/ktx-sl 0.13.0 |
||
|
semantic-release-bot
|
d62dc46a86 |
chore(release): 0.13.0 [skip ci]
## [0.13.0](https://github.com/Kaelio/ktx/compare/v0.12.0...v0.13.0) (2026-06-19) ### Features * **cli:** let ktx setup --agents choose an install directory ([#298](https://github.com/Kaelio/ktx/issues/298)) ([ |
||
|
Andrey Avtomonov
|
fde9f9862d
|
test(cli): persist warehouse connection in sl query tests (#303)
#301 strengthened resolveLocalConnectionId to reject connections absent from ktx.yaml, but three sl query tests configured the warehouse connection only in memory while runKtxSl reloads the project from disk, so they returned exit code 1 once the check landed on main. Persist the connection to ktx.yaml so it survives the reload and the tests exercise the real on-disk load path. |
||
|
Kevin Messiaen
|
6c815ef529
|
feat(duckdb): cross-database federation via derived DuckDB connection (#295)
* feat(duckdb): add @duckdb/node-api dependency for federation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(connectors): extract resolveStringReference to shared module Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(connectors): route all identical connectors through shared resolveStringReference Collapse the 5 remaining private copies in bigquery, clickhouse, mysql, snowflake, and sqlserver into the shared module. Fix a latent bug in the shared module where `~/path` was incorrectly sliced (dropping only `~`, leaving the leading `/` and making resolve() ignore homedir). Add a tilde-expansion test that caught the bug and now covers that branch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sl): reserve _ktx_ connection-id prefix for virtual connections Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(connections): derive virtual federated connection from compatible members Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(duckdb): federated executor builds READ_ONLY attaches and runs SQL Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(duckdb): close federated DuckDB instance and escape quotes in attach url Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sl): union member source directories for _ktx_federated Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(query): route _ktx_federated through DuckDB executor Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sl): use duckdb dialect for federated query compilation Bypass assertSafeConnectionId for _ktx_federated in resolveLocalConnectionId and loadComputableSources, and resolve the compute dialect to 'duckdb' when connectionId is FEDERATED_CONNECTION_ID instead of falling through to the default postgres lookup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(duckdb): end-to-end cross-catalog federated join Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(duckdb): harden federated join test with multi-book join-key coverage Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(ingest): keep declared cross-DB joins to federated siblings Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(setup): surface federated connection availability after adding a member Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(setup): mark federationNoticeFor @internal for dead-code gate Also marks attachTypeForDriver, buildAttachStatements, and isReservedConnectionId @internal — all three are exported solely for unit-test access with no production cross-file consumer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(concepts): document cross-database federation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(concepts): correct sqlite two-part naming in federation doc Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(duckdb): quote federated catalog alias so hyphenated connection ids attach * refactor(duckdb): single-source federation driver list, dedup attach loads Collapse the parallel ATTACH_COMPATIBLE_DRIVERS set and ATTACH_TYPE_BY_DRIVER map into one map in federation.ts whose keys are the membership rule. Replace FederatedMember.config (read only via a type-erasing cast) with a typed url field extracted at derive time. Emit INSTALL/LOAD once per distinct driver type instead of once per member. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(duckdb): close federated DuckDB instance on connect failure; dedup id validation Wrap the federated DuckDB instance in its own try/finally so a failing connect() or a throwing connection.closeSync() no longer leaks the native instance. Route setup-sources connection-id validation through the canonical assertSafeConnectionId so the reserved _ktx_ prefix guard applies there too. Derive the federated dialect through sqlAnalysisDialectForDriver instead of a hardcoded literal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(federation): carry member connection config and projectDir on FederatedMember Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(federation): resolve per-member attach targets via canonical connector resolvers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): quote mysql attach-string values like postgres Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): resolve member attach targets via canonical resolvers, supporting sqlite path: Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(federation): thread projectDir through deriveFederatedConnection callers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(federation): add shared project read-only SQL executor that routes _ktx_federated Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(federation): exercise shared executor default federated path with real DuckDB Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(federation): route ingest query executor through shared executor Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): route MCP sql_execution _ktx_federated through shared executor Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): preserve cross-DB joins to federated siblings in manifest re-emit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): preserve declared cross-DB joins through scan re-ingest Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(federation): document sibling-ref invariant, drop unsafe casts in test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): namespace federated source names by member to avoid collisions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(federation): document member-namespaced federated source names Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): preserve member SSL/search_path in attach, classify federated MCP errors Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(federation): simplify federated dispatch and parallelize sibling reads Dedup the federated driver ternary in local-query, derive the prefixed source.name from the already-built name, drop the duplicated error in federatedAttachTarget's exhaustive switch, inline the one-line cleanupConnector wrapper, and parallelize federatedSiblingTargets' shard reads (was sequential await-in-for on the scan hot path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(federation): carry headerTypes through shared SQL executor Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(federation): add shared federated connection listing builder Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): route ktx sql through shared executor for _ktx_federated parity Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(federation): show _ktx_federated in ktx connection list Surfaces the virtual federated connection in the output of `ktx connection list` so agents and users can discover cross-database querying when 2+ attach-compatible connections are configured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(federation): surface _ktx_federated in MCP connection_list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(federation): ktx sql federated cross-file join end-to-end Drive runKtxSql with the real federated DuckDB executor against two on-disk sqlite files, stubbing only SQL validation. The test surfaced that the JSON output path could not serialize bigint values DuckDB returns for integer columns; printJson now coerces bigint to JSON numbers, matching the plain/pretty paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(federation): document direct _ktx_federated query surface Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): coerce DuckDB bigint to number in shared federated executor DuckDB returns integer columns as JS bigint, which JSON.stringify cannot serialize. The CLI --json path worked around this with a replacer, but the MCP sql_execution tool serializes via plain JSON.stringify and crashed on any federated query selecting an integer column. Coerce bigint to Number once in executeFederatedQuery so every consumer (CLI, MCP, ingest, SL) gets a JSON-safe result, and remove the now-redundant CLI replacer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(federation): simplify driver map and collapse forked MCP SQL path - Replace the identity-valued ATTACH_TYPE_BY_DRIVER record with a ATTACH_COMPATIBLE_DRIVERS Set; the driver name doubles as the attach type, so the map encoded nothing beyond membership. - Switch federatedAttachTarget directly on the driver with a default throw, dropping the unreachable post-switch throw and its comment. - Route the MCP sql_execution standard-connection case through the shared executeProjectReadOnlySql instead of reimplementing the connector create/capability-check/execute/cleanup ceremony, so federated and standard connections share one execution path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(federation): allowlist placeholder credentials for detect-secrets The federation doc example URL and the federated-attach test fixtures use literal placeholder credentials that trip detect-secrets. Mark them with line-scoped pragma allowlist comments so a real secret added later is still caught. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(federation): correct SL addressing, join pruning, and id-quoting guidance - Federated SL list/search records carry the virtual `_ktx_federated` connection id (member origin stays in the prefixed source name), so rows round-trip to `ktx sl -c _ktx_federated read` and the fts index no longer clobbers per-connection partitions. - Prune semantic-layer joins by membership in the connection's own source set instead of matching the target's first dotted segment against other connection ids; a same-connection join whose target name collides with a sibling connection id is preserved, and orphan targets that would poison the planner are dropped. - Document double-quoting for connection ids that are not bare SQL identifiers (e.g. "books-db".public.books) in the federated naming hint, the sl-query rejection error, and the federation docs. - Preserve exact federated BIGINT values beyond 2^53 as strings instead of rounding, and steer the setup federation notice to raw SQL against `_ktx_federated`. * fix(federation): carry ssl:true into postgres URL attach target A postgres member configured with `url` plus `ssl: true` resolved to both a connectionString and an ssl flag, but the federated attach builder early-returned the bare URL and dropped the ssl intent. DuckDB then handed libpq a URL with no sslmode, so the URL path silently diverged from the discrete-field path (which emits sslmode=require) and from the direct scan path (which enforces TLS). Append sslmode=require to the URL when the member sets ssl, unless the URL already pins a stronger sslmode. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com> |
||
|
Kevin Messiaen
|
b81391cd9f
|
fix(cli): stop framing Claude Code session limits as auth failures (KLO-734) (#300)
The Claude Code auth probe wrapped every error as "authentication is not usable", so a subscription session limit told users to re-authenticate — which cannot help, since auth already succeeded and only a reset clears it. Discriminate probe failures with describeClaudeProbeFailure: session-limit and rate-limit hits get their own messages (preserving the upstream reset text), and genuine auth errors keep the original guidance. Also add the session/usage-limit markers to the shared rate-limit classifier so the governor stops treating a cap as a generic error. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Andrey Avtomonov
|
7e29543398
|
fix: classify mcp query failures (#302) | ||
|
Andrey Avtomonov
|
8a50601582
|
fix(cli): make connection-not-configured errors actionable and expected (#301)
The MCP sql_execution/sl_query tools and the `ktx sql` CLI threw a plain Error naming no valid connection ids when an agent passed an unconfigured connectionId (or omitted it with multiple connections). The message reached the agent verbatim but gave it nothing to correct with, so it re-guessed for days, and each correct caller-driven rejection filed in PostHog Error Tracking as a ktx fault (issue 019eb10c, 8 occurrences on one install). Add a shared resolver (resolveConfiguredConnection / resolveRequiredConnectionId) that throws KtxExpectedError listing the configured connections, and route the three SQL-execution call sites through it. Expected-error classification keeps these out of Error Tracking while the actionable message lets agents self-correct. |
||
|
Kevin Messiaen
|
9587049283
|
fix(git): disable gpg signing for ktx's own commits (#299)
ktx commits under a synthetic identity (ktx <ktx@example.com>) that can never own a GPG secret key. On a machine with commit.gpgsign=true, git tried to sign every ktx commit and failed with "No secret key", breaking ingest, scan, wiki, memory, and bootstrap commits. Inject commit.gpgsign=false as a per-invocation -c override in the single core git client factory every ktx commit flows through. This honors the existing principle of not mutating the user's repo config, and is locale-independent (no error-message matching). Also harden the repo-isolation fixture helper to disable signing on its raw commits so the suite is deterministic regardless of the contributor's global git config. Fixes KLO-735. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Andrey Avtomonov
|
4e61020089
|
feat(cli): let ktx setup --agents choose an install directory (#298)
Split the fused directory concept into projectDir (what the agent config references) and installRoot (where project-scoped files are written), so users can install .claude/, .mcp.json, skills, and rules where they open their agent instead of only in the ktx project directory. - Add --install-dir <path> (resolved against cwd, created if missing, mutually exclusive with --global/--local, rejected for claude-desktop). - Add an interactive directory menu: ktx project dir / Current directory (hidden when it equals the project dir) / Custom directory… / Global scope (shown only when every target supports it). - Expand a leading ~ in typed/quoted paths so the ~/… menu hints round-trip. - Record installRoot in the install manifest and merge key; thread it through file planning, MCP config paths, summaries, and next actions. - Refresh uv.lock to 0.12.0 for the editable ktx-sl and ktx-daemon packages. |
||
|
semantic-release-bot
|
cf2871ec8b |
chore(release): 0.12.0 [skip ci]
## [0.12.0](https://github.com/Kaelio/ktx/compare/v0.11.0...v0.12.0) (2026-06-12) ### Features * **cli:** add ktx wordmark banner to setup intro ([#290](https://github.com/Kaelio/ktx/issues/290)) ([ |
||
|
Andrey Avtomonov
|
feb0818444
|
feat(cli): self-provision pinned uv and defer MCP Python runtime install (#297)
Fixes a production crash-loop (PostHog issue 019eb68e): ktx mcp start --foreground on a uv-less container eagerly installed the managed Python runtime at boot, failed, and was restarted by its supervisor every ~62s (122 exceptions from one install). - MCP server factory now wires a lazy semantic-layer compute port that defers the runtime install to the first call, mirroring the already-lazy SQL-analysis port; the server boots and serves non-Python tools without the runtime. - ktx no longer requires uv on PATH: it downloads its own pinned, sha256-verified uv build under the runtime root (KTX_RUNTIME_ROOT aware), always musl-static on Linux. PATH uv is never consulted. - uv is acquired before the version dir is wiped, so a failed download cannot destroy an existing runtime. - Acquisition failures (offline, intercepted download, unsupported platform) throw KtxExpectedError and stay out of Error Tracking; a missing binary inside a checksum-verified archive remains a plain Error. - scripts/refresh-uv-manifest.mjs regenerates the pinned manifest (packages/cli/src/managed-uv-release.ts) on uv bumps. - Setup consent prompt now discloses the uv download; docs updated. |
||
|
Andrey Avtomonov
|
663eaff940
|
feat(cli): setup progress spinners, Tab-to-select, and banner polish (#296)
* fix(cli): double the height of the setup banner t crossbar
* fix(cli): unify setup multi-select hints and make Tab the select key
The six interactive multi-select surfaces in `ktx setup` documented three
different hint voices, one had no hint at all, and they named two different
select keys (Space vs Tab). Tab is the only key that can toggle selection
without colliding with type-to-search input, so make it the single documented
select key everywhere and compose every hint from one shared fragment
vocabulary in prompt-navigation.ts.
- Register `updateSettings({ aliases: { tab: 'space' } })` so Tab toggles flat
multiselects; the alias applies only to non-text prompts, leaving typed
search input (schema/Notion) untouched.
- Add the missing hint to the agent-targets prompt and drop the stray
"Space to select … Esc …" info line plus the now-dead writeSetupInfo helper.
- Replace the schema-scope ad-hoc hint with the searchable-multiselect voice
and standardize "filter" -> "search" vocabulary.
- Delete DEFAULT_TREE_PICKER_HELP_TEXT and the unused TreePickerChrome.helpText
seam; render the shared tree hint instead.
* refactor(cli): show LLM check progress for every setup backend
Rename runLlmHealthCheckWithProgress to validateModelWithProgress and
wrap the Claude subscription and Codex auth probes in the same spinner
progress as the Anthropic API and Vertex backends, so each backend shows
consistent "Checking <provider> LLM" output during setup.
* feat(cli): add ktx-orange progress spinners to setup steps
Add a shared runWithCliSpinner helper and a TTY-aware createCliSpinner:
an animated clack spinner in a terminal, and a static stderr-only spinner
before raw-mode pickers (the table tree picker and demo tour), where the
animated spinner's stdin grab would otherwise corrupt the next prompt.
Wrap the slow setup waits in progress spinners: managed runtime install,
embedding daemon start + first-run model download, embeddings health
check, the connection-test gate, and source validation / dbt clone /
Metabase discovery. Recolor every spinner frame from clack's magenta to
the ktx mascot orange (#FF8A4C) via the static helper and clack's
styleFrame option.
|
||
|
Andrey Avtomonov
|
0689d709d2
|
fix(cli): survive ktx.yaml version skew and derive repo ownership from disk (#293)
* fix(cli): survive ktx.yaml version skew and derive repo ownership from disk Loading ktx.yaml is now tolerant of keys this ktx version does not recognize: they are stripped from the in-memory config (the file on disk is never rewritten) and reported by ktx status as non-blocking warnings, while invalid values on recognized fields still fail hard. Repo ownership is derived from observed state (a .git directory plus a root ktx.yaml) instead of a ktx.managed git-config marker, so projects created by any past or future ktx classify identically. initKtxProject now runs an explicit foreign-repo pre-check and writes ktx.yaml before initializing git, so an interrupted init leaves only recoverable residue instead of a bare .git misread as foreign. * style(cli): trim comment blocks to constraint-only notes * docs(agents): require constraint-only code comments |
||
|
Andrey Avtomonov
|
a278d2f7d0
|
fix(ingest): verify repair outcomes and reject dangling join targets (#292)
One ingest integration hiccup no longer discards a whole source: - Replace the duplicated gate-repair and textual-resolver loops with one shared constrained-repair loop whose success criterion is re-running the failed check (verify), not whether the agent edited files. Verify failures feed the retry prompt; maxAttempts is 2. - Let the resolver declare a conflicting patch redundant: a verified no-change resolution is accepted as subsumed instead of failing the source (duplicate wiki-page creation from parallel work units). - Carry per-source validation errors through validateWuTouchedSources into gate messages and work-unit failure reasons instead of discarding them. - Move join-neighbor expansion into the shared validation path so work-unit validation and integration gates check the same set. - Reject joins whose target resolves to no source, at sl_write time and in the gates, attributed to the declaring source. Resolution mirrors the Python engine exactly (case-sensitive name within the connection), with a case-mismatch hint for the writing agent. |
||
|
Andrey Avtomonov
|
00cdf2de90
|
refactor: enforce ktx naming and AGENTS.md compliance sweep (#289)
Align the tree with AGENTS.md/CLAUDE.md conventions: - Rewrite user-facing strings, docs, and tests to lowercase `ktx` (no bare uppercase `KTX` tokens remain outside literal identifiers). - Drop the legacy `historicSql` migration path and its now-unused helpers, per the no-backward-compat rule. - Remove `as unknown as` / `any` casts: narrow `BaseTool` generics to `z.ZodObject`, add a typed `createLookerClient`, and delete the dead `getParametersSchema`/`toAnthropicFormat` pre-AI-SDK helpers. - Use `InvalidArgumentError` for Commander parse failures. - Finish the adapter→connector prose conversion in the `ktx.yaml` docs while keeping the literal `adapters` config key. |
||
|
Andrey Avtomonov
|
28953eb616
|
feat(cli): add ktx wordmark banner to setup intro (#290)
Render a lowercase ktx half-block wordmark with the brand-orange gradient above the `ktx setup` intro on interactive TTYs. The banner degrades through truecolor, xterm-256, and monochrome, and is skipped on non-TTY, non-Unicode, or too-narrow terminals. Extract shared color/Unicode capability detection into io helpers (shouldUseColorOutput, colorDepthForOutput, unicodeSupported) so the banner and doctor report route through one implementation. |
||
|
Andrey Avtomonov
|
058051f1b9
|
test(ingest): supply explicit no-input schema scope in skip-llm setup (#287)
The "prints provider setup guidance when a skip-llm setup project runs ingest" test drives runKtxSetup in --no-input (inputMode: disabled) mode with a postgres warehouse but databaseSchemas: []. PR #286 changed the no-input contract to require an explicit database scope instead of auto-scanning, so setup now exits 1 and the test's first assertion (resolves.toBe(0)) fails. #286 updated setup-databases.test.ts but missed this runKtxSetup call in ingest.test.ts. Pass databaseSchemas: ['public'] to satisfy the new contract, matching #286's own test updates. |
||
|
Andrey Avtomonov
|
2877b85adc
|
fix(cli): isolate ktx-owned project repositories (#283)
* fix(cli): isolate ktx project git repos * fix(cli): remove inert auto commit config * test(cli): drop stale auto commit fixtures * docs: document isolated ktx project repos * test(cli): keep stale config grep clean * fix(cli): guide setup away from foreign repos at the project dir ktx owns the git repo rooted at the project dir and refuses to adopt one it did not create (the Finding 3 isolation invariant). But setup steered users straight into that failure: the interactive menu offers "Current directory" first, and `--no-input --yes --project-dir <repo-root>` created directly in place — both then threw a generic "Failed to initialize git repository:" wrapper from deep in GitService.initialize(). Extract the ownership rule into a shared `classifyKtxRepoOwnership(dir)` used by both GitService.initialize() (the invariant) and the setup wizard (pre-flight guidance), so the decision derives from one rule. Setup now detects a foreign repo before constructing GitService and: interactively re-prompts (the user picks the existing `ktx-project` subfolder), or non-interactively returns a clean missing-input with the actionable message. The typed foreign-repo error is also surfaced verbatim instead of being buried under the generic wrapper. Empty/non-repo current directories still work — only foreign repos are blocked. * fix(cli): keep classifyKtxRepoOwnership total for non-directory paths The setup ownership guard runs before the existing not-a-directory check, so pointing a custom/--project-dir path at a file made classifyKtxRepoOwnership lstat `<file>/.git`, hit ENOTDIR, and throw — crashing the setup step instead of returning the friendly "path exists and is not a directory" result. A path that is a file (or missing) holds no git repo for ktx to avoid, so treat ENOTDIR like ENOENT and return 'unowned'. The downstream existingFolderState check still rejects a non-directory with its friendly message, and the classifier no longer throws raw errno for any caller. |
||
|
Andrey Avtomonov
|
f3f893bf01
|
fix: read semantic sources safely (#284)
* fix: read semantic sources safely
* test: retarget reindex per-scope error case to a broken manifest
Reading a broken standalone source was made non-fatal in
|
||
|
Andrey Avtomonov
|
853f39a7c3
|
fix(setup): require explicit no-input database scope (#286)
* test(setup): supply explicit --no-input scope to disabled-mode database tests * fix(setup): require explicit database scope in --no-input instead of auto-scanning the warehouse * docs(setup): document --no-input database scope requirement |
||
|
Andrey Avtomonov
|
036a745fc1
|
fix: classify MCP SQL query errors as expected (#285) | ||
|
Andrey Avtomonov
|
fd18caa26a
|
fix(cli): own a dedicated git repo at the project dir when nested in an enclosing repo (#282)
GitService.initialize() used checkIsRepo(), which is true whenever the project dir sits anywhere inside a git working tree. So when a ktx project lived in a subdirectory of an enclosing repo, ktx skipped `git init` and silently adopted the enclosing repo as its store. Every ktx relative path assumes the project dir IS the working-tree root. During ingest, wiki/SL pages are written through a session worktree (whose root is the worktree dir, so the page is recorded at repo-relative `wiki/global/<key>.md`) and then squash-merged into the main worktree. With an adopted enclosing repo, the main worktree's root is the enclosing git root, so the merge wrote the page to `<gitRoot>/wiki/global/` — outside the project dir. reindex scans `<projectDir>/wiki/global/`, found nothing, and wiki_search silently returned empty (knowledge_pages = 0) even though ingest reported success. Detect the project dir's own root with checkIsRepo(IS_REPO_ROOT) and initialize a dedicated repo there unless the project dir is already a repo root. This keeps adopting a user-created repo when the project dir IS that repo's root, fixes the silent wiki/SL/memory divergence at its source for every writer, and stops ktx from committing its scaffold into the user's enclosing repo. Regression tests cover both layers: a project nested in an enclosing repo gets its own .git (and the enclosing repo stays untouched), and a wiki page written through a session worktree + squash-merge lands in the project dir and is discovered by reindex. |
||
|
Andrey Avtomonov
|
0425160857
|
fix(cli): clear error when ktx setup has no LLM backend under --no-input (#281)
* fix(cli): fail clearly when ktx setup has no LLM backend under --no-input Non-interactive `ktx setup` silently defaulted the LLM backend to `anthropic` and then failed with `Missing Anthropic API key: pass --anthropic-api-key-env or --anthropic-api-key-file` — confusing for users who selected a different provider (e.g. `--target claude-code`) and never asked for the Anthropic API backend. That silent default could never succeed: it was reached only when no backend, Anthropic key, or Vertex flag was supplied, and in exactly that case the Anthropic credential resolver always failed (no env fallback in disabled mode). Unlike embeddings, the LLM has no credential-free default (anthropic needs a key, vertex needs gcloud ADC, claude-code/codex need a logged-in local CLI), so there is nothing safe to assume. `chooseBackend` now fails clearly in disabled mode with no backend, naming the (hidden) `--llm-backend` flag and its choices and noting each backend's credential needs. `--llm-backend` stays hidden in `--help`, consistent with the rest of the documented automation surface; the error message is the discovery path. - Add a unit test (no backend, disabled -> clear message) and a CLI/integration test (`--target claude-code --no-input` -> exit 1, clear message, not the Anthropic red herring). - Document the no-default behavior and add a Common-errors row in docs-site ktx-setup.mdx. * refactor(cli): single source of truth for setup LLM backends The set of LLM backends a user can pick during `ktx setup` (claude-code, codex, anthropic, vertex) was hand-enumerated in five places: the `--llm-backend` arg parser, the `KtxSetupLlmBackend` union, the interactive prompt's narrowing, the prompt options, and the missing-backend error. Only some had TypeScript coverage, so adding a backend could silently drift (e.g. a valid value rejected by the parser, or routed to anthropic by the prompt's `? : 'anthropic'` fallback). Collapse them onto one `KTX_SETUP_LLM_BACKENDS` list: - `KtxSetupLlmBackend` is derived from it. - `isKtxSetupLlmBackend` is the shared validator; the arg parser and the prompt both route through it instead of re-listing literals. - The prompt options derive from the list, with a `Record<KtxSetupLlmBackend, string>` label map so a new backend fails to compile until it has a label. - The missing-backend error builds its choice list from the same source. Behavior-preserving: identical accepted values and parse error, identical prompt options (asserted by an existing test), and the prompt's unreachable fallback now cancels rather than silently assuming anthropic. |
||
|
semantic-release-bot
|
07ab275662 |
chore(release): 0.11.0 [skip ci]
## [0.11.0](https://github.com/Kaelio/ktx/compare/v0.10.0...v0.11.0) (2026-06-09) ### Features * **cli:** add Slack community CTA on errors, crashes, setup, and help ([#277](https://github.com/Kaelio/ktx/issues/277)) ([ |
||
|
Andrey Avtomonov
|
470802e58e
|
fix(cli): classify ktx setup abandonment as aborted, not a blank error (#278)
* fix(cli): classify ktx setup abandonment as aborted, not a blank error ktx setup returned a non-zero exit code without throwing when a user abandoned the interactive wizard, so the command telemetry recorded outcome=error with no errorClass/errorDetail — an unactionable blank in the errors dashboard, where most ktx setup "errors" were really people backing out of the wizard. Add annotateCommandOutcome() to the command span so the setup flow (the decision-maker) records the true outcome: genuine step failures and --no-input missing input become outcome=error with a self-diagnosing reason, while interactive abandonment and project cancellation become outcome=aborted and drop out of the error view. Unify the exit code and telemetry through setupTerminalOutcome() so they can never diverge: aborts now exit 0 (matching the entry-menu Exit, project cancel, and a confirmed Ctrl+C), while failures and automation errors still exit 1. * fix(cli): treat non-TTY setup missing-input as an error, not an abort setupTerminalOutcome classified `missing-input` by `args.inputMode`, but `auto` only means "interactive if a TTY is attached". A piped/CI `ktx setup` without `--no-input` and without `--yes` is still `auto`, yet the project and agents steps return `missing-input` there without ever prompting (e.g. "pass --yes to create a project outside an interactive terminal"). Classifying that as `aborted` made a broken automation run exit 0 — a silent failure. Key the classification off actual interactivity instead: input enabled AND `io.stdout.isTTY === true`. Non-interactive missing-input now exits 1 with a `KtxSetupMissingInput` reason; only a genuine interactive abort exits 0. Adds a non-TTY regression test and fixes the abandonment test to use a real TTY. |
||
|
Andrey Avtomonov
|
66517fc320
|
feat(cli): add Slack community CTA on errors, crashes, setup, and help (#277)
* feat(cli): show Slack CTA on help and unexpected errors * feat(cli): show Slack CTA after crashes * feat(setup): show Slack community note after setup * chore: refresh Python lockfile versions |
||
|
Andrey Avtomonov
|
6b2f7c3365
|
fix(cli): ensure git committer identity during ktx setup (#276)
* fix(cli): ensure git committer identity during ktx setup ktx setup threw "Failed to initialize git repository" when the project directory was already a git repo with no commits and the machine had no configured git identity (e.g. a fresh Mac with no ~/.gitconfig). GitService only set the identity on the path where it created the repo itself, so the bootstrap commit had no resolvable committer. Carry ktx's identity via GIT_AUTHOR/GIT_COMMITTER env on the shared git client so every commit succeeds regardless of whether ktx created the repo, without mutating the user's repo config. Also preserve the underlying git error when rethrowing so the failure is diagnosable in telemetry and actionable for the user. * chore: sync uv.lock ktx-daemon and ktx-sl versions to 0.10.0 |
||
|
semantic-release-bot
|
48676c74fa |
chore(release): 0.10.0 [skip ci]
## [0.10.0](https://github.com/Kaelio/ktx/compare/v0.9.0...v0.10.0) (2026-06-08) ### Features * add GitHub star nudges to CLI build view and docs sidebar ([#271](https://github.com/Kaelio/ktx/issues/271)) ([ |
||
|
Andrey Avtomonov
|
795a97485a
|
feat: add GitHub star nudges to CLI build view and docs sidebar (#271)
* feat: load star count during context builds * docs: document star prompt opt-out * fix: initialize demo context star count * feat(docs-site): add GitHub star count widget to docs sidebar * test: isolate star-prompt build-view tests from ambient CI env |
||
|
Andrey Avtomonov
|
5232578d44
|
fix(sl): stop baking drift-prone counts into overlay summaries (#270)
The auto-generated semantic-layer overlay description embedded measure/segment/column counts that were computed once and never recomputed, so the summary drifted and misreported its source after measures were later appended. Make the auto fallback count-free, since those counts are already rendered live from the body at `ktx sl list`/ `read` time; this removes the drift class without ever overwriting human-authored descriptions (the fill-only-when-empty guard is untouched). Adds a regression test that fails on main and passes after the fix, plus guards for description preservation and the no-measures fallback. |
||
|
Andrey Avtomonov
|
2c18a62de4
|
feat(setup): apply per-role LLM model presets, remove --llm-model (#268)
* feat(setup): write per-role llm model presets * feat(setup): remove llm model setup flag * chore(setup): update llm preset guidance * docs(setup): document llm model presets * chore(release): sync uv.lock to 0.9.0 * fix(cli): make sl query --execute work on secret-backed connections sl query --execute used a parallel SQL executor (createDefaultLocalQueryExecutor) that passed connection.url verbatim into pg, so file:/env: secret references failed with "SASL: SCRAM-SERVER-FIRST-MESSAGE: client password must be a string". Collapse onto the connector-based executor already used by MCP and ingest (createKtxCliIngestQueryExecutor), which resolves secret references and supports every driver. Delete the now-dead local/postgres/sqlite query executors, their tests, and the orphaned hasLocalQueryExecutor driver flag. * docs(agents): require one implementation per capability Add a design-reasoning default and a matching self-check question telling agents to route callers through a single shared implementation of a capability rather than forking a parallel one, and to fix the shared layer rather than patch one branch. Encodes the lesson from a divergent SQL-execution-path bug, stated generally. CLAUDE.md is a symlink to AGENTS.md, so both agent-instruction files are covered. |
||
|
Andrey Avtomonov
|
2896f9fb91
|
fix(ingest): drive work-unit progress from tool calls, not turn counts (#269)
The ingest HUD showed "step 70/40" because the Claude subscription runtime re-derived a per-turn counter that could not match the SDK's num_turns and overshot the maxTurns budget. Replace the turn-based work_unit_step heartbeat with a real, observed tool-call count (no denominator), report metrics.stepCount from the SDK's authoritative num_turns, and delete the brittle countsAsAssistantTurn denylist plus the now-unused onStepFinish callback across the runtime port and all three runtimes. Reconcile and curator progress move to the same tool-call heartbeat. |
||
|
Mayorkun Ayanshina
|
18245c2373
|
feat(mysql): implement columnStats using INFORMATION_SCHEMA.STATISTICS (#233)
* feat(mysql): implement columnStats using INFORMATION_SCHEMA.STATISTICS Enable column cardinality statistics for the MySQL connector by querying INFORMATION_SCHEMA.STATISTICS, which provides index-based cardinality estimates without requiring additional permissions. - Add generateColumnStatisticsQuery() to KtxMysqlDialect - Add getColumnStatistics() and columnStats() to KtxMysqlScanConnector - Flip columnStats capability from false to true - Add MysqlStatsRow and KtxMysqlColumnStatisticsResult interfaces - Add tests for dialect query generation and connector stats retrieval - Update dialect conformance fixture for mysql * fix(mysql): filter to leading index columns to avoid inflated cardinality Add AND SEQ_IN_INDEX = 1 to INFORMATION_SCHEMA.STATISTICS query to ensure only leading index columns are returned. For composite indexes, non-leading columns report the cardinality of the index prefix rather than the column's own distinct count, which inflates distinctCount. Add regression test asserting SEQ_IN_INDEX = 1 is present in the query. * fix: add trailing newline to dialect.test.ts --------- Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com> |
||
|
Luca Martial
|
bf1fe9748e
|
docs: minor README and docs-site touch-ups (#266)
- Link the Y Combinator badge and the docs "by Kaelio" label - Add a maintainer line to the README - Set the npm author field on @kaelio/ktx Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Andrey Avtomonov
|
698efdcef8
|
feat(cli): add channel-aware update notifier (#265)
* feat(cli): show cached update notices after commands * docs(cli): describe update notices * fix(cli): type update check environment * fix(cli): decouple update notice display from refresh and harden suppression Display a cached "update available" notice based solely on the lastNoticeAt 24h throttle, independent of checkedAt refresh freshness, matching the design's independent display/refresh decisions. Suppress the check unconditionally under --json, CI, and non-TTY before consulting output-mode preferences, so a KTX_OUTPUT=pretty override can no longer make CI/non-TTY contexts phone npm. |
||
|
Andrey Avtomonov
|
fb7b94b60e
|
feat(telemetry): collect PostHog $exception error reports in CLI and daemon (#262)
* feat(telemetry): add node exception reporter * feat(telemetry): report node cli exceptions * feat(telemetry): add daemon exception reporter * feat(telemetry): report daemon exceptions * docs(telemetry): document error reports * fix(telemetry): pass redaction snapshots from node call sites * test(telemetry): verify prepared node exception payload * fix(telemetry): close daemon exception lifecycle gaps * test(telemetry): verify prepared daemon exception payload * test(telemetry): close error collection acceptance gaps * test(telemetry): close posthog exception acceptance gaps |
||
|
Andrey Avtomonov
|
c3d8cedb0b
|
feat(cli): add ingest LLM rate-limit governor with paced retries (#261)
* feat(cli): add ingest rate limit governor * feat(cli): wire ingest rate-limit config * feat(cli): report provider rate-limit signals * feat(cli): show ingest rate-limit waits * fix(cli): complete rate-limit event coverage * fix(cli): abort ingest provider calls cleanly * fix(cli): propagate ingest cancellation * fix(cli): reject pre-aborted ingest rate-limit waits * fix(cli): honor Claude rate-limit reset waits * fix(cli): retry thrown Codex rate-limit failures * fix(cli): type Claude rate-limit result details * fix(cli): emit ingest rate-limit countdowns from rejected signals * fix(cli): report ai sdk rate-limit header utilization * fix(cli): gate LLM rate-limit retries on the governor budget The AI SDK and Codex runtimes retried 429 / opaque rate-limit failures up to 6-7 times with no backoff when constructed without a RateLimitGovernor (scan, memory, setup) or with pacing disabled, ignoring Retry-After and worsening the limit. The outer retry loop only cooperates with the governor's pause, so without active pacing there is no backoff to apply. Route the retry bound through a single source: RateLimitGovernor .maxRetryAttempts(), which returns retry.maxAttempts when enabled and 1 (no outer retry) when absent or disabled. All three runtimes (ai-sdk, codex, claude-code) now use it, so ingest.rateLimit.retry.maxAttempts genuinely controls attempts and the hard-coded 6 (plus Codex's off-by-one extra attempt) is gone. Backend-native retry (e.g. the AI SDK's maxRetries) still handles transient 429s. Also correct the ktx.yaml docs for maxWaitMs (caps each wait, not the whole run) and maxAttempts, and sync uv.lock ktx-sl/ktx-daemon to 0.9.0. |
||
|
Andrey Avtomonov
|
ec7edf8f50
|
fix(telemetry): preserve driver error class and code in connection_test (#260)
Native connector test failures were flattened to `new Error(message)`, collapsing every driver's error class to `Error` and dropping `.code` / `.number`. connection_test telemetry could therefore not tell a SQL Server login rejection (ELOGIN / 18456) apart from a network or TLS error, and the only field that varied was a raw message. Connectors now return `connectorTestFailure(error)`, which preserves the original driver error as `cause`, and `testNativeConnection` re-throws that cause. `scrubErrorClass` then records the real class (e.g. ConnectionError) and `formatErrorDetail` keeps the code prefix (e.g. "ELOGIN: ..."). The helper is the single source of truth for the failure shape across all seven native connectors. User-facing terminal output is unchanged. |
||
|
Andrey Avtomonov
|
c2beaf7d55
|
feat(setup): wizard prompt tweaks and quieter query-history filter output (#259)
Setup wizard flow tweaks: - Add a reveal-tail password prompt (reveal-password-prompt.ts) that unmasks the last few characters of a typed/pasted secret, and wire it into the setup prompt adapter in place of clack's password(); adds the @clack/core dep. - Reorder wizard select options: surface "Paste a key" before the environment-variable option across embeddings/models/sources, promote Metabase/Notion in the source list, put Git URL before Local path, reorder the Notion crawl-mode choices, and relabel the sources "Done" action. Query-history filter picker output: - Collapse the per-template parse-failure lines into a single count in the setup output and route the full template-id list to --debug stderr. - Model parse failures as a structured parseFailedTemplateIds field instead of warning strings. - Add a privacy-safe query_history_filter_completed telemetry event (counts/enums only), mirrored into the Python daemon schema. |
||
|
semantic-release-bot
|
7ba948a135 |
chore(release): 0.9.0 [skip ci]
## [0.9.0](https://github.com/Kaelio/ktx/compare/v0.8.0...v0.9.0) (2026-06-03) ### Features * add codex llm backend for ktx runtime work ([#253](https://github.com/Kaelio/ktx/issues/253)) ([ |
||
|
Andrey Avtomonov
|
e70ae1e63b
|
feat(query-history): scope mining to modeled schemas by default (#258)
* feat(query-history): structure SQL analysis table refs * feat(query-history): qualify SQL analysis table refs * feat(query-history): wire modeled scope floor through ingest * chore(query-history): verify scope floor * test(query-history): align daemon SQL batch endpoint contract * feat(query-history): build scope from same-run scan catalog * feat(query-history): fail open on scope-floor catalog failures * chore(query-history): verify scope-floor v1 closure * refactor(query-history): share scope membership * feat(setup): apply derived query history filters * docs: document derived query history filters * fix(query-history): redact filter picker LLM prompt SQL * fix(setup): run filter picker SQL analysis through managed daemon * chore(query-history): verify filter picker v1 closure * fix(query-history): fail open on partial service-account attribution * fix(query-history): aggregate BigQuery users by execution count * fix(query-history): aggregate Snowflake users by execution count * fix(query-history): use BigQuery query info hash |