* feat: add codex sdk runner foundation
* feat: parse codex runtime events
* feat: expose codex runtime mcp tools
* feat: add codex llm runtime
* feat: wire codex llm backend
* test: avoid Array.fromAsync in codex runner test
* docs: document codex llm backend
* fix: tighten codex runtime config ownership
* fix: use codex sdk env and thread options
* fix: parse codex sdk event shapes
* test: add codex backend live smoke
* docs: clarify codex backend isolation
* fix: drive codex loop metrics from mcp events
* fix: enforce codex local step budget
* docs: disclose codex isolation limits
* fix: count all codex agent steps and stream step callbacks live
The agent-loop step budget only counted completed mcp_tool_call items, so
built-in command_execution steps (which the public Codex SDK/CLI surface can
still expose) never decremented the budget, letting ingest/reconciliation run
past stepBudget until Codex stopped on its own. onStepFinish was also replayed
only after the whole stream drained, so live work_unit_step / reconciliation
progress appeared stuck until the Codex process exited.
collectEvents is now the single live step accumulator: it counts every
completed agent-action item via a shared isCompletedAgentStep predicate
(command_execution, mcp_tool_call, file_change, web_search), fires onStepFinish
as each step completes, and enforces the budget on that broader count. A
no-tool turn still counts as one step. toolFailures stays MCP-specific, since a
non-zero command exit is normal agent exploration, not a loop failure.
* test: align ingest llm-guard assertions with codex backend
The skip-llm ingest guard message now lists codex as a valid backend and
mentions a Claude Code/Codex session plus a codex setup hint, but this slow
suite test still asserted the pre-codex wording. Update it to match the
production message (already covered by the local-bundle-runtime unit test) and
add the codex setup-line assertion.
* fix: treat codex error:null tool calls as success
The Codex SDK serializes error: null on successful mcp_tool_call items, so
the failure check (item.error !== undefined) flagged every successful tool
call as failed with the empty-payload default "Codex turn failed". This
killed every ingest work unit under the codex backend before it could
produce a patch.
Key on status === 'failed' (authoritative, always set) and only treat a
populated error object as a failure. Add a regression test built from a
verbatim real-SDK event capture.
* fix: default codex backend to gpt-5.5 and report real probe errors
The previous default gpt-5.3-codex is an API-key-only model that the OpenAI
API rejects under ChatGPT-account (subscription) auth, so codex status/setup
failed with a misleading "authentication is not usable" message even though
auth was fine.
- Default codex model is now gpt-5.5 (works on both subscription and API-key
auth); the curated setup picker offers gpt-5.5 / gpt-5.4 / gpt-5.4-mini and
keeps free-form entry for account-specific ids (e.g. gpt-5.3-codex-spark).
- runCodexAuthProbe now distinguishes "model not available" from an auth
failure and surfaces the real API error: collectEvents retains stream
events when the SDK throws on a non-zero exit, and the API error JSON
envelope is unwrapped to its human-readable message.
- The Codex isolation warning now renders inside the clack setup frame.
- Docs updated to gpt-5.5 with a note that *-codex ids require API-key auth.
* fix: require llm.models.default in status and match codex probe remediation
Status reported a project ready when a non-none LLM backend was configured
without llm.models.default, but the runtime (resolveModelSlots) hard-requires
it, so ingest/scan/memory threw after `ktx status` said the project was usable.
buildLlmStatus now fails for any non-none backend missing models.default and no
longer invents a fallback model for claude-code/codex.
Codex probe failures now carry a category-matched fix: a model-access failure
steers the user at llm.models.default instead of the auth/install remediation.
runCodexAuthProbe returns the fix and status consumes it; the message stays
self-sufficient so setup output is unchanged.
Docs: README now lists the codex backend and local Codex auth; ktx-setup.mdx
states --llm-model only accepts codex/default or gpt-*/codex-* ids.
Repaired four doctor fixtures that configured a backend without models.default
(the now-correctly-blocked config) and added coverage for the new behavior.
The GitHub repo was renamed back from Kaelio/ktx-ai-data-agents-context to Kaelio/ktx, reverting the URL changes from #250 across package metadata, CI (codecov + star-history slugs), issue/security templates, the release runbook, and docs/install commands.
Also removes the rename-resilience machinery #250 added: semantic-release now reads the repository URL straight from package.json (Kaelio/ktx) again, so the repositoryUrl() derivation in scripts/semantic-release-config.cjs, its tests, and the rename note in docs/release.md are no longer needed.
* feat(cli): share public ingest progress adapter
* feat(cli): stream plain public ingest progress
* test(cli): update plain ingest progress assertions
* chore(cli): satisfy plain ingest progress checks
* fix(artifacts): expect plain ingest stderr progress in installed-CLI smoke
* ci(coverage): make Codecov upload non-fatal and fix repo slug
The Coverage job failed because the Codecov upload returned
'Repository not found' while fail_ci_if_error was true, turning a
Codecov-side issue into a hard CI failure even though all tests pass.
- Set fail_ci_if_error: false on both uploads so Codecov outages or an
unlinked repo no longer break CI (upload stays best-effort).
- Correct the stale slug Kaelio/ktx -> Kaelio/ktx-ai-data-agents-context
to match the actual GitHub repo (aligns with main).
* fix(cli): isolate query-history failure capture from scan output
The plain public-ingest progress path passes one captured IO as the
target-level `io`. With progress deps set, both the schema scan and the
query-history ingest resolved their capture to that same shared buffer,
so a non-actionable query-history failure surfaced leftover scan report
text (e.g. "Mode: enriched") as the skipped-facet detail instead of the
real query-history message.
Give the query-history ingest a phase-local capture while preserving the
flow-to-io branch the foreground context-build view relies on.
---------
Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
* fix(release): point repository URLs at renamed GitHub repo
The GitHub repo was renamed from Kaelio/ktx to
Kaelio/ktx-ai-data-agents-context. semantic-release reads repositoryUrl
from package.json's repository field and the @semantic-release/github
plugin failed verifyConditions with EMISMATCHGITHUBURL because it no
longer matched the live clone URL.
Update every Kaelio/ktx reference to the renamed repo: package metadata
(root + CLI repository/bugs/homepage), the codecov upload slugs and
star-history slug in CI, the issue-template and security-advisory links,
the release runbook, and all docs/install commands.
* fix(release): derive semantic-release repositoryUrl from the CI repo
@semantic-release/github exact-matches repositoryUrl against the live
GitHub clone_url (no redirect following), so any repo rename re-breaks the
release when repositoryUrl is the static package.json value.
Derive repositoryUrl from the runner's GITHUB_REPOSITORY/GITHUB_SERVER_URL
so it always tracks the current repo name. A future rename (including back
to Kaelio/ktx) now resolves with no code change. Outside CI the option is
omitted, so semantic-release falls back to package.json as documented.
The package.json repository field stays ktx-ai-data-agents-context as
npm-display metadata, decoupled from the release-time match.
* feat(cli): profile ingest runs to find where wall-clock time goes
Add opt-in profiling for `ktx ingest`. Each timed phase, work unit, and
agent loop now records durationMs / step count / token usage in the
trace, and a post-run aggregator rolls them up into a "where did the
time go" report printed to stderr.
Enable per run with KTX_PROFILE_INGEST (1/true -> human table, json ->
raw structured profile) or persistently via `ingest.profile` in
ktx.yaml. The json form emits raw milliseconds, token counts, and a
summary.headline one-line diagnosis so coding agents can parse it
directly; json wins when both env and config request profiling.
- runtime-port: RunLoopMetrics (totalMs, usage, stepCount,
stepBoundariesMs) plus onMetrics callbacks on text/object generation
- ai-sdk + claude-code runtimes: capture per-loop timing and token usage
- work-unit-executor and stages 3/4: thread metrics into trace events
- ingest-bundle.runner: time worktree / triage / clustering / index /
reconcile / squash phases and emit the profile in a finally block
(best-effort; never affects the run outcome)
- ingest-profile: new trace+transcript aggregator with table/json formatters
- config: ingest.profile flag; docs: profiling section in ktx-ingest.mdx
* fix(cli): flush tool-call logs before reading ingest profile
Tool transcripts are appended fire-and-forget so the agent hot path never
blocks on logging. The ingest profiler read them before the writes settled,
so per-work-unit toolMs (and the model-vs-tool split derived from it) could
be incomplete. Track in-flight appends and expose flushToolCallLogs() —
bounded by a timeout so it can never hang — and flush before the profiler
reads the transcript.
Add a clickable launch-video poster (linking to YouTube) directly after
the intro note and before the architecture diagrams. GitHub Markdown can
not embed a YouTube player, so the poster image links out instead.
Fold field-tested fixes into the ktx skill, verified against current CLI source:
- prefer file: secret refs over env: (env: re-resolves per-process and resolves
empty in later ingest/mcp shells)
- pass --skip-agents on data-only setup runs; explain the trailing agent step's
misleading exit 1 on otherwise-successful runs
- dbt ignores --source-warehouse-connection-id (maps by table name); required
only for Metabase/Looker/LookML
- never go silent during slow setup/ingest: poll .ktx mtimes and post progress
so a long run does not look stuck
- judge readiness from verdict, connections[].status, localStats.semanticLayer
and wikiPages; perConnection under-reports
- add troubleshooting entries for the 'Run in a TTY' exit 1 and secrets that
resolve empty only during ingest/mcp
Replace the tall portrait README ingestion SVG with two landscape
diagrams — "1 · Ingestion" (build the context layer) and "2 · Serving"
(agents query it through MCP) — wired in as transparent 2x PNGs that
read on GitHub light and dark.
Add docs-site/diagram-studio: a static React Flow page with custom
themed nodes and the inlined ktx mascot that renders both diagrams and
exports them to PNG via html-to-image (the diagrams' reproducible
source). Remove the superseded ingestion-flow SVGs.
* feat(completion): complete known argument values
* fix(completion): hide Commander-hidden subcommands from completions
Replace the `__`-prefix name heuristic with Commander's `_hidden` flag so
internal subcommands registered with { hidden: true } (e.g. `mcp serve-internal`)
are excluded from completions, mirroring `ktx --help`.
* test: cover wiki and sl read command routing
* test: cover raw wiki and sl reads
* feat: add wiki read command
* feat: add sl read command
* feat: complete read command entity names
* docs: document wiki and sl read commands
* test: include read commands in command tree
* feat(sl): read and validate unique sources by name
* feat(sl): make read and validate connection id optional
* fix(completion): dedupe semantic source names
* docs(sl): document connection-optional read and validate
* fix(sl): require connection id for query command
* docs(sl): clarify query connection requirement
* fix(completion): don't resolve option values as subcommands
resolveCommand skipped flag tokens but not the value consumed by a
value-taking option in the `--flag value` form, so a connection id like
`query` was matched as the `sl query` subcommand and yielded no `sl`
completions. Track value-taking options and skip their consumed value
before matching subcommands.
* test(telemetry): assert first-run notice via TELEMETRY_NOTICE constant
CI (which tests this branch merged with main) failed because #243 changed
the first-run notice wording in identity.ts (dropped "anonymous") but left
this test grepping for the old literal 'ktx collects anonymous usage data',
so indexOf returned -1. Assert against the exported TELEMETRY_NOTICE
constant instead so the test tracks the source of truth and cannot drift
when the notice text changes again.
Set disableGeoip: false on the CLI telemetry client so events are enriched with approximate, IP-based location at ingest. Update the first-run notice, public telemetry docs, and the AGENTS telemetry policy to drop the prior "anonymous" wording to match.
The star-history refresh workflow committed the API's SVG verbatim, but the
response has no trailing newline. Because the refresh commit uses [skip ci],
the file never ran end-of-file-fixer at commit time, so pre-commit's
`--all-files` run failed end-of-file-fixer on every open PR (e.g. #240), even
PRs that never touched the file.
Normalize the downloaded SVG to exactly one trailing newline in the workflow
(idempotent, so the "unchanged" guard still works), and fix the currently
committed file so open PRs go green now.
The scheduled star-history workflow checked out with the default
GITHUB_TOKEN, so its git push to main was rejected by the branch
protection hook (GH006). Check out with RELEASE_PAT instead, matching
release.yml, whose semantic-release step already pushes to the protected
main branch with the same token.
Point the README chart at a committed assets/star-history.svg instead of
the star-history API URL so GitHub serves it directly and bypasses the Camo
proxy cache. A scheduled workflow regenerates the SVG at 06:00/18:00 UTC,
busting star-history's server-side cache, and commits it when it changes.
* fix(cli): derive ingest outcomes from saved artifacts
* fix(cli): treat artifact-producing ingests with failures as partial
* fix(cli): route memory-flow run status through shared ingest outcome
* fix(cli): treat partial ingest as saved context in setup status
* test(cli): align memory-flow replay expectations with partial ingests
Fast mode (the ktx ingest --fast/--deep database-ingest depth toggle) is removed.
ktx ingest now always builds the full enriched ("deep") context. There is no
structural fallback: a database connection without a configured model and
embeddings fails the enrichment-readiness preflight before any work runs, with
a 'Run ktx setup to configure a model and embeddings' hint.
- Remove --fast/--deep flags, the per-connection context.depth field, and the
ktx setup depth prompt (delete setup-database-context-depth.ts).
- Rename ingest-depth.ts -> connection-drivers.ts; ingest always requests scan
mode 'enriched'; readiness gate (enrichmentReadinessGaps) runs for every
database target.
- Drop the database-context-depth telemetry step (Node + Python schema mirrors
regenerated).
- Update CLI, setup, context-build view, docs, the public ktx skill, and the
release-smoke / artifacts scripts (now assert the no-LLM guard failure).
ktx status --fast (a separate network-probe flag) is unchanged.
Follow-ups: KLO-726 (live progress for ktx ingest --all), KLO-727 (restore
credentialed successful-ingest release smoke coverage).
Notion's setup path read --source-api-key-ref while writing the auth_token_ref
config field, so --source-auth-token-ref was silently dropped. Align Notion to
the flag=field convention every other connector follows: it now reads
--source-auth-token-ref, and --source-api-key-ref becomes Metabase-only.
Also add validation rejecting any credential-ref flag not applicable to the
chosen --source, with a pointer to the correct flag, closing the silent-drop
class for all connectors.
Update CLI-reference docs, the ktx skill Notion example, and tests.
Fixes KLO-724.
The pre-commit job failed because tombi-format reformats uv.lock to a
layout uv does not produce, so once CI's uv sync re-resolved the stale
lock (workspace members still at 0.6.0) and rewrote it, tombi rewrote it
back and the hook reported a modified file.
Exclude uv.lock from tombi-format so uv stays authoritative for its
generated lockfile, and bump the workspace members to 0.7.0 so the lock
is current and uv stops re-resolving it (uv lock --check now passes).
- skills/ktx/SKILL.md: add an "Add context sources" section with the generic
`ktx setup --source ...` flags per connector (dbt, Metabase, Notion, ...),
warehouse mapping, the --metabase-database-id discovery note, and the
`ktx ingest` follow-up. The skill previously only documented database setup
with --skip-sources, so agents couldn't wire up dbt/Metabase/Notion (KLO-723).
- docs-site quickstart: the kaelio.com/start callout now points at the
"copy agent setup" one-shot prompt that installs the full four-source demo.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An external agent ran the skill end-to-end against `ktx setup` and reported
seven concrete failures, all verified against the CLI source:
- All useful setup flags are `.hideHelp()`, so the skill's "verify with
--help" rule led the agent to conclude its own examples were wrong
(setup-commands.ts:208-332).
- The non-interactive LLM default is `anthropic` (and requires a key), not
`claude-code` as the skill claimed (setup-models.ts:505-507).
- `ktx status` exits 1 whenever the LLM is `none`, even with healthy
embeddings and connections (status-project.ts:204-211, doctor.ts:647).
- `ktx ingest` rejects `--yes`+`--no-input` while `ktx setup` accepts both
(managed-python-command.ts:23-24).
- `--database-url <raw>` auto-externalizes to `.ktx/secrets/<id>-url` —
worth telling the agent (setup-databases.ts:671-683).
- Resuming setup with only `--llm-backend` fails on missing DB flags even
when `ktx.yaml` already has one (setup-databases.ts:1778-1782).
- The `--agents` step prints `Required before using agents: ktx mcp start`
but the skill never told agents to run it (setup-agents.ts:989,1227).
Rewrite SKILL.md to: lead with the scripted (non-interactive) path; add a
single "gather inputs once" checklist; correct the LLM default; document
`--skip-*` flags and resumability; warn that `status` exit code ≠
readiness; fix the `ktx ingest` example to use `--no-input` only; require
`ktx mcp start` after `--agents`; add a ktx-monorepo branch that avoids
`npm install -g`.
Add skills/ktx/troubleshooting.md (one level deep, per Anthropic's
progressive-disclosure guidance) covering the five real failure signatures
the agent hit: invalid ELF header, missing native CLI binary, missing
Anthropic key, claude-code probe failure, and the resume-without-DB error.
Description rewritten to combine what + when per the official skill
authoring guidelines.
ktx setup wiped ktx.yaml, .ktx/setup/state.json, wiki/, semantic-layer/,
raw-sources/, and .git/ — or removed the entire project dir — whenever any
single source in the context-build step failed, destroying hours of ingest
work and the persisted resume state. The cleanup hint was designed for an
"early abort, leave no trace" semantic but was applied indiscriminately to
every later step failure, in direct conflict with the .ktx/setup/state.json
resume mechanism.
Drop the cleanup mechanism entirely (KtxSetupCreatedProjectCleanup,
cleanupForFolderState, createProjectWithCleanup, cleanupCreatedProjectScaffold,
and the createdProjectCleanup plumbing through KtxSetupProjectResult). Step
failures now return non-zero without touching the filesystem, so re-running
ktx setup continues from completed steps and only re-attempts failed sources.
Rewrites the two tests that documented the wipe behavior to assert
preservation, and adds a regression test that simulates partial context-build
artifacts (state.json, wiki/, semantic-layer/) and verifies all survive a
failed context step.
Refs KLO-719
Geist Mono fuses `--` into an em-dash glyph that visually swallows the
adjacent space, so prompts like `npx skills add Kaelio/ktx --skill ktx`
rendered as `Kaelio/ktx--skill ktx` on the quickstart page. The existing
ligature-off rule only covered <code>/<pre> and the .ktx-code wrapper —
quickstart.mdx puts the prompt in a plain <div className="font-mono">,
so the rule didn't apply. Extend the selector to also match the
.font-mono Tailwind utility and any inline-style opt-in via the mono
font CSS variable.
Document the convention in AGENTS.md so future docs additions keep
ligatures off on any new monospace container.
The Claude Code runtime counted every SDKAssistantMessage with
parent_tool_use_id === null as a step, but the SDK emits extra messages
within a single num_turns round-trip — `stop_reason: 'pause_turn'`
continuations and errored partials it retries internally. The local
counter then outran maxTurns and the ingest HUD rendered confusing
ratios like `step 69/40`.
Filter both cases in collectResult so stepIndex tracks num_turns and
stays bounded by the work-unit stepBudget.
Add a host-scoped redirect for /slack on ktx.sh before the existing
catch-all so the path resolves to the community invite link instead of
docs.kaelio.com/ktx/slack.
* docs(readme): restructure for clarity and add FAQ + comparison table
Restructure the README: trim Common Commands to the 6 essentials and link
to the CLI Reference, add a "How ktx compares" table and "Who is ktx for"
qualifier, introduce a small FAQ, wrap key prompts in GitHub callouts,
merge the duplicate workspace-layout section into Development, move
Telemetry next to License, and add a Star History chart.
* docs(readme): tighten Skip-ktx list and convert FAQ to bullets
Add a Clack note pointing to https://docs.kaelio.com/ktx right after the
setup intro, and a second note pointing to https://kaelio.com/start
above the database driver multiselect — mirroring the docs-site CTA
wording. Closes KLO-715 and KLO-716.
The emit_historic_sql_evidence tool took rawPath as LLM-supplied input,
so projection actions frequently lacked defensible raw paths and every
row in bundle_ingest_reports fell through as actionType: 'skipped' with
null artifact metadata, hiding the wiki pages and SL merges the run had
actually produced (KLO-698).
The tool now reads the work unit's rawFiles from session.allowedRawPaths
and stores them on the evidence envelope; the projection emits actions
with those paths, and stale/archive actions are anchored to manifest.json
so they also surface as non-skipped provenance rows.
* feat(cli): define full warehouse dialect contract
* test(cli): keep dialect edge tests focused
* fix(cli): stabilize dialect contract foundation
* refactor(connectors): own read-only query preparation
* refactor(connectors): resolve dialects through registry
* refactor(connectors): keep concrete dialect classes internal
* chore(workspace): enforce dialect import boundary
* refactor(cli): resolve relationship dialect at scan boundary
* refactor(cli): use dialect display parsing for entity details
* refactor(cli): use dialect display parsing for warehouse catalog
* refactor(cli): use dialect SQL in relationship workflows
* test(cli): verify solid dialect scan workflow closure
* test: split cli tests from source tree
* refactor(cli): standardize BigQuery scope listing
* feat(sqlite): implement connector scope listing
* test(connectors): cover required table listing
* feat(cli): add warehouse driver registry
* refactor(setup): route scope discovery through driver registry
* refactor(cli): route local query execution through driver registry
* refactor(historic-sql): route dialect support through driver registry
* refactor(cli): test warehouse connections through driver registry
* fix(cli): close driver registry type export gaps
* Improve setup daemon diagnostics
* refactor(setup): centralize rail-prefixed diagnostics + query-history fallback
Extract errorMessage, writePrefixedLines, and flushPrefixedBufferedCommandOutput
into clack.ts so the setup wizard, managed daemons, and embedding/agent steps
share one rail-formatted writer. setup-databases.ts also adds a
"disable query history and retry" option when the schema-context build fails
and query history is the likely culprit, surfaced via a new
failed-query-history-unavailable status.
* fix(cli): carry catalog through the picker so BigQuery/Snowflake/SQL Server scope filters match
The setup picker's KtxTableListEntry was a 2-level { schema, name }, so
qualifiedTableId always wrote db.name into enabled_tables. When BigQuery,
Snowflake, or SQL Server later ran fast ingest, their introspect step filtered
the scope set with scopedTableNames(scope, { catalog: projectId|database, db })
— catalog was non-null on the introspect side but null in the scope refs, so
every entry was rejected, the live-database adapter staged zero table files,
and detect() failed with 'Adapter "live-database" did not recognize fetched
source output'.
Align the picker boundary with the canonical 3-level KtxTableRef:
- Add catalog: string | null to KtxTableListEntry.
- BigQuery/Snowflake/SQL Server listTables populate catalog from the
resolved projectId / database; Postgres/MySQL/ClickHouse/SQLite set null.
- qualifiedTableId emits catalog.schema.name when catalog is non-null
(resolveEnabledTables already accepts the 3-part shape) and
schemasFromEnabledTables now goes through parseDottedTableEntry so it
recovers the schema correctly from both 2-part and 3-part entries.
- Export parseDottedTableEntry from enabled-tables.ts (@internal) for picker
reuse.
Update listTables expectations in all seven connector tests and the setup /
picker test fixtures. Add a picker regression test that covers the
catalog-bearing round-trip (save + refine).
* fix(cli): allow debug telemetry under opt-out env
