Compare commits

..

No commits in common. "main" and "v0.8.0" have entirely different histories.
main ... v0.8.0

277 changed files with 4154 additions and 20476 deletions

View file

@ -10,5 +10,5 @@ contact_links:
url: https://docs.kaelio.com/ktx/docs/community/support
about: Full guide on where to ask what — Slack vs. GitHub Issues vs. docs.
- name: Security issues
url: https://github.com/Kaelio/ktx/security/advisories/new
url: https://github.com/Kaelio/ktx-ai-data-agents-context/security/advisories/new
about: Report security vulnerabilities privately via GitHub Security Advisories. Please do not file security issues publicly.

View file

@ -212,12 +212,12 @@ jobs:
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe # v5.5.4
with:
token: ${{ secrets.CODECOV_TOKEN }}
slug: Kaelio/ktx
slug: Kaelio/ktx-ai-data-agents-context
files: ./packages/cli/coverage/lcov.info
flags: typescript
name: typescript
disable_search: true
fail_ci_if_error: false
fail_ci_if_error: true
- name: Warn when Codecov token is missing for TypeScript
if: env.CODECOV_TOKEN_CONFIGURED != 'true'
@ -231,12 +231,12 @@ jobs:
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe # v5.5.4
with:
token: ${{ secrets.CODECOV_TOKEN }}
slug: Kaelio/ktx
slug: Kaelio/ktx-ai-data-agents-context
files: ./coverage/python.xml
flags: python
name: python
disable_search: true
fail_ci_if_error: false
fail_ci_if_error: true
- name: Warn when Codecov token is missing for Python
if: env.CODECOV_TOKEN_CONFIGURED != 'true'

View file

@ -35,7 +35,7 @@ jobs:
set -euo pipefail
# cachebust forces star-history to regenerate instead of serving its
# own server-side cache; --location follows the slug-normalizing 301.
url="https://api.star-history.com/svg?repos=Kaelio/ktx&type=Date&cachebust=${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
url="https://api.star-history.com/svg?repos=Kaelio/ktx-ai-data-agents-context&type=Date&cachebust=${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
curl --fail --location --silent --show-error \
--retry 3 --retry-delay 5 --max-time 60 \
-o assets/star-history.svg.new "$url"

129
AGENTS.md
View file

@ -64,25 +64,6 @@ When rules conflict, follow this order:
4. Code quality: types, readable boundaries, focused modules
5. Performance where it matters
## Opinionated Product Defaults
- **MUST**: Prefer one canonical behavior over configurable alternatives. A new
flag, config field, environment variable, mode, strategy option, adapter hook,
or fallback path is a product feature and must be justified by an explicit
user request or a real correctness requirement.
- **MUST NOT**: Add speculative flexibility for imagined users, migrations,
review preferences, local workflows, or "just in case" scenarios. If the
requested behavior can work with one solid default, implement that default.
- **MUST NOT**: Add boolean switches that create two runtime paths unless both
paths are essential and the user explicitly asked for the choice. Boolean
policy knobs are especially suspect because they double the state space and
test surface.
- **MUST**: When a design seems to need a new option, first try to remove the
need by choosing the stronger default, tightening the invariant, or failing
clearly. Ask the user before adding the option if it still seems necessary.
- **MUST**: Delete obsolete branches, tests, docs, and config when removing a
behavior. Do not preserve dormant compatibility paths.
## Repository Shape
**ktx** is a pnpm + uv workspace.
@ -178,91 +159,6 @@ and naming asymmetries are bugs in waiting — see
[`docs/code-design.md`](docs/code-design.md). Treat the `MUST` / `MUST NOT`
rules there with the same weight as the ones in this file.
## Design Reasoning Defaults
When proposing a design, an approach, or any non-trivial change, apply these
defaults and run the self-check before presenting it. They encode the
corrections users most often have to make; reaching these conclusions
autonomously — without being asked the leading question — is the bar.
- **MUST**: Optimize for the best outcome, not for an unstated constraint. Do not
silently adopt "smallest change", "least effort", "cheapest", or "least user
intervention" as the goal unless the user said so. Default to the most correct,
durable solution, and present cost / effort / scope as information for the user
to weigh — not as a ceiling you impose on their behalf.
- **MUST**: Separate one-time cost from recurring cost before discarding an
option. A fixed cost paid once (a setup-time computation, an extra LLM call
during setup, a contract change) to make every later run cheaper or more
correct is usually worth it. Do not reject it with recurring-cost reasoning;
quantify both sides. (Example smell: "don't add an LLM call to a cost-cutting
feature" — wrong when the call is one-time and the savings recur.)
- **MUST**: Treat a user's example as a representative of a class, not as the
spec. Design for the general population the example stands for, then stress-test
against deliberately different instances — another warehouse, dialect, stack
layout, or input shape — before committing. If a design only works because of an
incidental property of the example (e.g. "the noise happened to be in a separate
schema *on this demo*"), it is curve-fitting; generalize it or state the
assumption explicitly.
- **MUST**: Prefer deriving from the system's own state over enumerating cases.
Favor an allowlist computed from declared/observed state (config, scanned
catalog, query log, the user's own inputs) over a denylist of known-bad
specifics (particular tables, schemas, tools, or vendors). A hardcoded or
hand-maintained list of external specifics is a smell: it rots and fails on the
next stack. The only acceptable static patterns are genuinely universal
invariants (e.g. DB-engine system catalogs) and ktx's own self-emitted
signatures.
- **MUST**: Give each capability one implementation and route every caller
through it. When some behavior — running a query, resolving a credential or
config reference, authenticating, selecting a dialect, loading config —
already has a working implementation that some call sites use, make new or
divergent call sites depend on that path instead of standing up a second one.
Parallel implementations of one capability drift apart silently: a fix, a
newly supported input, or an added case lands on one path and not the other,
so one entry point (a CLI command, an MCP tool, an ingest stage) succeeds
while another fails on the same input. When two paths already do the same
job, collapse onto the shared one and delete the duplicate instead of
keeping both. When fixing a defect that lives on one path, fix the shared
implementation; do not patch the symptom on a forked branch, which preserves
the divergence you set out to remove.
- **SHOULD**: Before inventing an abstraction or hand-rolling structural logic,
search for what already exists and reuse it — the codebase's canonical
representation (a structured ref/key type) instead of a parallel string scheme,
and a mandated/available tool (e.g. `sqlglot` for SQL structure; see
[SQL and Structured Parsing](#sql-and-structured-parsing)) instead of
hand-parsing. Normalize ambiguous input to the canonical form at the boundary;
do not carry the ambiguity downstream. This is the single-source-of-truth / DRY
item from the Priority Hierarchy applied at design time.
Before presenting a design, answer these explicitly:
1. Am I optimizing for a goal the user actually stated, or one I assumed?
2. Does this generalize beyond the example in front of me? Name a real case where
it would break.
3. Am I enumerating known-bad cases when I could derive scope from the system's
own declared/observed state?
4. Is there an existing canonical representation or mandated tool I should reuse
instead of building or parsing my own?
5. Am I discarding the better option on a weak or misapplied constraint
(one-time vs recurring cost, "more surface area", "more work now")?
6. Does another entry point already perform this operation through a shared
implementation? If so, am I routing through that path instead of forking a
parallel one — and if I'm fixing a bug, am I fixing the shared layer rather
than one branch?
7. Am I adding a user-visible option or alternate runtime path that the user did
not ask for? If yes, can one opinionated default solve the problem instead?
8. Does this option multiply behavior by caller path, config value, or local
state? If yes, remove it unless it is explicitly required.
A user question that nudges toward an alternative ("would X help?", "should I
always do Y?", "will you hardcode Z?") is a signal that a better option exists.
Investigate the implied direction and reason it through *before* defending the
original proposal — and prefer to have asked yourself the question first.
Example: If generated context changes should be saved, choose one save policy
and route ingest, setup, memory, indexing, and docs through it. Do not add an
`auto_commit`-style switch unless the user explicitly asks for staged-only runs
and accepts the extra runtime path.
## TypeScript Standards
- Use Node 22+ and pnpm workspace commands.
@ -382,8 +278,7 @@ use `PascalCase` without the suffix.
## Telemetry
**ktx** ships PostHog usage telemetry. Catalog telemetry events use strict
schemas. When adding commands or events:
**ktx** ships PostHog usage telemetry. When adding commands or events:
- **MUST NOT**: Add fields that carry user data — file paths, hostnames,
environment values, SQL text, schema/table/column names, error messages,
@ -400,24 +295,6 @@ schemas. When adding commands or events:
of collected data changes. Adding another event with no new field types
needs no docs change.
### Error reports
**ktx** also sends PostHog Error Tracking `$exception` events when telemetry is
enabled. This channel is separate from the strict catalog event schema and is
used only for exception diagnostics.
`$exception` events may include stack frames, error class names, raw error
messages, cause chains, `source`, `handled`, `fatal`, runtime version fields,
OS/runtime fields, and the hashed `projectId` when known. Stack frames may
include local file paths and the local username when those appear in paths.
`$exception` events must never intentionally include secrets, credentials,
database URLs, auth headers, raw argv, raw environment values, SQL text,
schema/table/column names as explicit properties, customer row data, user prompt
text, or raw MCP arguments. Reporters must redact call-site-provided secret
snapshots and common static credential patterns before the SDK serializes the
exception.
## Documentation and Specs
- Keep public documentation in `README.md`, package READMEs, example READMEs,
@ -504,8 +381,8 @@ rather than silently skipping it.
- **MUST**: Disable monospace ligatures on every surface that uses the
`var(--font-mono)` family (Geist Mono). Geist Mono fuses `--` into an
em-dash glyph that visually eats the adjacent space, so prompts like
`npx skills add Kaelio/ktx --skill ktx` render as
`Kaelio/ktx--skill ktx`.
`npx skills add Kaelio/ktx-ai-data-agents-context --skill ktx` render as
`Kaelio/ktx-ai-data-agents-context--skill ktx`.
- **MUST**: When adding a new container that renders user-visible monospace
text outside `<code>` / `<pre>` (e.g. a styled `<div className="font-mono">`
for a copyable prompt), verify the global ligature-off rule in

View file

@ -8,8 +8,8 @@ layout, and verification commands, see the
## How to contribute
1. Browse open issues labeled
[`good first issue`](https://github.com/Kaelio/ktx/labels/good%20first%20issue)
or [`help wanted`](https://github.com/Kaelio/ktx/labels/help%20wanted).
[`good first issue`](https://github.com/Kaelio/ktx-ai-data-agents-context/labels/good%20first%20issue)
or [`help wanted`](https://github.com/Kaelio/ktx-ai-data-agents-context/labels/help%20wanted).
2. Comment on the issue to claim it. A maintainer will confirm scope and
assign it to you.
3. For changes not covered by an existing issue, open one first so we can
@ -82,7 +82,7 @@ page for the full guide. The short version:
- **Feature requests**: use the
[Feature request](.github/ISSUE_TEMPLATE/feature_request.yml) template.
- **Security**: report privately via
[GitHub Security Advisories](https://github.com/Kaelio/ktx/security/advisories/new),
[GitHub Security Advisories](https://github.com/Kaelio/ktx-ai-data-agents-context/security/advisories/new),
not as a public issue.
## Code of conduct

View file

@ -8,25 +8,21 @@
<p align="center">
<a href="https://www.npmjs.com/package/@kaelio/ktx"><img src="https://img.shields.io/npm/v/@kaelio/ktx?style=flat-square&color=f97316" alt="npm version" /></a>
<a href="https://codecov.io/gh/Kaelio/ktx"><img src="https://codecov.io/gh/Kaelio/ktx/graph/badge.svg?branch=main" alt="Codecov" /></a>
<a href="https://github.com/Kaelio/ktx/actions/workflows/ci.yml?query=branch%3Amain"><img src="https://img.shields.io/github/actions/workflow/status/Kaelio/ktx/ci.yml?branch=main&label=tests&style=flat-square" alt="Tests" /></a>
<a href="https://codecov.io/gh/Kaelio/ktx-ai-data-agents-context"><img src="https://codecov.io/gh/Kaelio/ktx-ai-data-agents-context/graph/badge.svg?branch=main" alt="Codecov" /></a>
<a href="https://github.com/Kaelio/ktx-ai-data-agents-context/actions/workflows/ci.yml?query=branch%3Amain"><img src="https://img.shields.io/github/actions/workflow/status/Kaelio/ktx-ai-data-agents-context/ci.yml?branch=main&label=tests&style=flat-square" alt="Tests" /></a>
<a href="https://docs.kaelio.com/ktx/docs/"><img src="https://img.shields.io/badge/docs-ktx-22c55e?style=flat-square" alt="Documentation" /></a>
<a href="https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ"><img src="https://img.shields.io/badge/slack-join%20community-4A154B?style=flat-square&logo=slack&logoColor=white" alt="Join the ktx Slack community" /></a>
<a href="https://github.com/Kaelio/ktx/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
<a href="https://www.ycombinator.com/companies/kaelio"><img src="https://img.shields.io/badge/Y%20Combinator-P25-orange?style=flat-square" alt="Y Combinator P25" /></a>
<a href="https://github.com/Kaelio/ktx-ai-data-agents-context/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
<a href="https://www.ycombinator.com/companies?batch=P25"><img src="https://img.shields.io/badge/Y%20Combinator-P25-orange?style=flat-square" alt="Y Combinator P25" /></a>
</p>
<p align="center">
<a href="https://docs.kaelio.com/ktx/docs/getting-started/quickstart"><b>Quickstart</b></a> ·
<a href="https://docs.kaelio.com/ktx/docs/cli-reference/ktx"><b>CLI Reference</b></a> ·
<a href="https://docs.kaelio.com/ktx/docs/community/ai-resources"><b>Agent Setup</b></a> ·
<a href="https://docs.kaelio.com/ktx/docs/ai-resources/agent-quickstart"><b>Agent Setup</b></a> ·
<a href="https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ"><b>Slack</b></a>
</p>
<p align="center">
<sub>Built and maintained by <a href="https://www.kaelio.com"><b>Kaelio</b></a></sub>
</p>
---
**ktx** is a self-improving context layer that teaches agents how to query your
@ -34,9 +30,8 @@ warehouse accurately - from approved metric definitions, joinable columns, and
business knowledge it builds and maintains for you.
> [!NOTE]
> Run **ktx** with your own LLM API keys or a local agent sign-in — a
> **Claude Pro/Max** subscription through Claude Code, or your local Codex
> authentication. No extra usage billing from **ktx**.
> Run **ktx** with your own LLM API keys or a **Claude Pro/Max** subscription.
> No extra usage billing from **ktx**.
<p align="center">
<a href="https://youtu.be/5V4TuzYVlrA">
@ -135,7 +130,7 @@ Agent integration ready: yes (codex:project)
> your project directory:
>
> ```text
> Run npx skills add Kaelio/ktx --skill ktx and use the ktx skill to install
> Run npx skills add Kaelio/ktx-ai-data-agents-context --skill ktx and use the ktx skill to install
> and configure ktx in this project.
> ```
@ -143,14 +138,6 @@ Agent integration ready: yes (codex:project)
> If `ktx status` prints `ktx mcp start --project-dir ...`, run it before
> opening your agent client.
## Upgrading
Re-run the global install with the `@latest` tag:
```bash
npm install -g @kaelio/ktx@latest
```
## First commands
| Command | Purpose |
@ -188,9 +175,8 @@ then the current directory. Pass `--project-dir <path>` when scripting.
No. **ktx** runs locally. The only data leaving your machine is what you
send to the LLM provider you configured.
- **Which LLM backends are supported?**
Anthropic API, Google Vertex AI, AI Gateway, the local Claude Code session
through the Claude Agent SDK, and your local Codex authentication through the
Codex SDK. See
Anthropic API, Google Vertex AI, AI Gateway, and the local Claude Code
session through the Claude Agent SDK. See
[LLM configuration](https://docs.kaelio.com/ktx/docs/guides/llm-configuration).
- **How is ktx different from a dbt or MetricFlow semantic layer?**
**ktx** *ingests* those layers and combines them with raw-table
@ -209,13 +195,13 @@ then the current directory. Pass `--project-dir <path>` when scripting.
- [The Context Layer](https://docs.kaelio.com/ktx/docs/concepts/the-context-layer)
- [Building Context](https://docs.kaelio.com/ktx/docs/guides/building-context)
- [CLI Reference](https://docs.kaelio.com/ktx/docs/cli-reference/ktx)
- [AI Resources](https://docs.kaelio.com/ktx/docs/community/ai-resources)
- [Agent Quickstart](https://docs.kaelio.com/ktx/docs/ai-resources/agent-quickstart)
- [Community & Support](https://docs.kaelio.com/ktx/docs/community/support)
## Community
- **[Slack](https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ)** — ask questions, share what you're building, and chat with maintainers.
- **[GitHub Issues](https://github.com/Kaelio/ktx/issues)** — report bugs and request features.
- **[GitHub Issues](https://github.com/Kaelio/ktx-ai-data-agents-context/issues)** — report bugs and request features.
- **[Contributing](https://docs.kaelio.com/ktx/docs/community/contributing)** — set up the repo, run tests, and open a PR.
## Development
@ -259,17 +245,11 @@ uv run pytest -q
## Telemetry
**ktx** collects privacy-conscious usage telemetry to understand installs and
improve setup, command reliability, and data-agent workflows. Catalog telemetry
events do not record file paths, hostnames, SQL, schema names, table names,
column names, error messages, raw environment values, or argv. Error reports use
PostHog Error Tracking and can include stack frames and raw error messages,
which may contain local file paths or the local username in those paths.
**ktx** redacts secrets, credentials, database URLs, auth headers, argv, raw
environment values, SQL text, row data, and user-typed prompt or MCP argument
text from the explicit `$exception` payload. See
[Telemetry](https://docs.kaelio.com/ktx/docs/community/telemetry) for the event
catalog and opt-out options.
**ktx** collects anonymous usage telemetry from interactive CLI runs to
improve setup, command reliability, and data-agent workflows. No file paths,
hostnames, SQL, schema names, error messages, or argv are recorded. See
[Telemetry](https://docs.kaelio.com/ktx/docs/community/telemetry) for the
event catalog and opt-out options.
## License
@ -278,7 +258,7 @@ catalog and opt-out options.
## Star History
<p align="center">
<a href="https://star-history.com/#Kaelio/ktx&Date">
<a href="https://star-history.com/#Kaelio/ktx-ai-data-agents-context&Date">
<img src="assets/star-history.svg" alt="ktx Star History Chart" width="700" />
</a>
</p>

View file

@ -5,7 +5,7 @@
If you believe you've found a security vulnerability in KTX, please report it
**privately** through GitHub Security Advisories:
[Report a vulnerability](https://github.com/Kaelio/ktx/security/advisories/new)
[Report a vulnerability](https://github.com/Kaelio/ktx-ai-data-agents-context/security/advisories/new)
If you cannot use GitHub Security Advisories, email `support@kaelio.com`
instead. Please do **not** open a public issue, post in the KTX Slack, or

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 63 KiB

After

Width:  |  Height:  |  Size: 63 KiB

Before After
Before After

View file

@ -2,21 +2,10 @@ import { source } from "@/lib/source";
import { DocsLayout } from "fumadocs-ui/layouts/docs";
import type { ReactNode } from "react";
import { baseOptions } from "@/app/layout.config";
import { GitHubStars } from "@/components/github-stars";
export default function Layout({ children }: { children: ReactNode }) {
return (
<DocsLayout
tree={source.pageTree}
{...baseOptions}
sidebar={{
banner: (
<div className="flex">
<GitHubStars />
</div>
),
}}
>
<DocsLayout tree={source.pageTree} {...baseOptions}>
{children}
</DocsLayout>
);

View file

@ -869,147 +869,6 @@ body::after {
50% { opacity: 0.65; transform: scale(0.9); }
}
/*
GitHub star widget (navbar)
Split pill: GitHub mark + "Star" gold star + count.
*/
.ktx-stars {
display: inline-flex;
align-items: stretch;
height: 32px;
border-radius: 999px;
border: 1px solid var(--color-fd-border);
background: color-mix(in oklch, var(--color-fd-card) 72%, transparent);
backdrop-filter: blur(8px);
-webkit-backdrop-filter: blur(8px);
font-family: var(--font-display), var(--font-sans), sans-serif;
font-size: 13px;
line-height: 1;
color: var(--color-fd-foreground);
text-decoration: none;
overflow: hidden;
box-shadow: 0 1px 2px rgba(27, 27, 24, 0.04);
transition:
transform 0.3s var(--ktx-ease),
box-shadow 0.3s var(--ktx-ease),
border-color 0.3s ease;
animation: ktx-stars-in 0.5s var(--ktx-ease) both;
}
@keyframes ktx-stars-in {
from { opacity: 0; transform: translateY(-4px); }
to { opacity: 1; transform: translateY(0); }
}
.ktx-stars:hover {
transform: translateY(-1px);
border-color: color-mix(in oklch, var(--color-fd-primary) 45%, var(--color-fd-border));
box-shadow:
0 6px 18px -8px rgba(14, 116, 144, 0.28),
0 1px 2px rgba(27, 27, 24, 0.05);
}
.ktx-stars:focus-visible {
outline: 2px solid var(--color-fd-ring);
outline-offset: 2px;
}
.dark .ktx-stars {
background: color-mix(in oklch, var(--color-fd-card) 60%, transparent);
box-shadow: 0 1px 2px rgba(0, 0, 0, 0.25);
}
.dark .ktx-stars:hover {
border-color: rgba(34, 211, 238, 0.4);
box-shadow:
0 6px 18px -8px rgba(34, 211, 238, 0.3),
0 1px 2px rgba(0, 0, 0, 0.3);
}
.ktx-stars-seg {
display: inline-flex;
align-items: center;
gap: 6px;
padding: 0 11px;
}
.ktx-stars-seg--count {
border-left: 1px solid var(--color-fd-border);
background: color-mix(in oklch, var(--color-fd-primary) 6%, transparent);
transition: background 0.3s var(--ktx-ease);
}
.ktx-stars:hover .ktx-stars-seg--count {
background: color-mix(in oklch, var(--color-fd-primary) 12%, transparent);
}
.ktx-stars-gh {
width: 15px;
height: 15px;
opacity: 0.85;
}
.ktx-stars-text {
font-weight: 500;
letter-spacing: -0.01em;
}
.ktx-stars-star {
width: 14px;
height: 14px;
fill: #f5b301;
transition: transform 0.3s var(--ktx-ease), filter 0.3s var(--ktx-ease);
}
.ktx-stars:hover .ktx-stars-star {
transform: scale(1.18) rotate(-8deg);
filter: drop-shadow(0 1px 4px rgba(245, 179, 1, 0.55));
}
.ktx-stars-count {
font-weight: 600;
font-variant-numeric: tabular-nums;
color: var(--color-fd-foreground);
}
/* Skeleton shown only on the rare cold (uncached) fetch */
.ktx-stars--skeleton {
animation: none;
}
.ktx-stars-skeleton-bar {
display: inline-block;
width: 26px;
height: 11px;
border-radius: 4px;
background: linear-gradient(
90deg,
var(--color-fd-muted) 25%,
color-mix(in oklch, var(--color-fd-muted-foreground) 28%, var(--color-fd-muted)) 50%,
var(--color-fd-muted) 75%
);
background-size: 200% 100%;
animation: ktx-stars-shimmer 1.4s ease-in-out infinite;
}
@keyframes ktx-stars-shimmer {
from { background-position: 200% 0; }
to { background-position: -200% 0; }
}
/* Compact on phones: drop the "Star" word, keep mark + count */
@media (max-width: 640px) {
.ktx-stars-text { display: none; }
.ktx-stars-seg { padding: 0 9px; }
}
@media (prefers-reduced-motion: reduce) {
.ktx-stars { animation: none; transition: none; }
.ktx-stars:hover { transform: none; }
.ktx-stars:hover .ktx-stars-star { transform: none; }
.ktx-stars-skeleton-bar { animation: none; }
}
/* Dot grid */
.dot-grid {
background-image: radial-gradient(

View file

@ -1,13 +1,22 @@
import type { BaseLayoutProps } from "fumadocs-ui/layouts/shared";
import { GitHubIcon } from "@/components/github-icon";
import { Logo } from "@/components/logo";
import { SlackIcon } from "@/components/slack-icon";
export const baseOptions: BaseLayoutProps = {
nav: {
title: Logo,
title: <Logo />,
transparentMode: "top",
},
links: [
{
type: "icon",
label: "GitHub",
icon: <GitHubIcon />,
text: "GitHub",
url: "https://github.com/kaelio/ktx",
external: true,
},
{
type: "icon",
label: "Join the ktx Slack community",

View file

@ -305,8 +305,8 @@ export const runtimeEdges: Edge[] = [
sourceHandle: "to-context",
target: "context",
targetHandle: "in",
type: "smoothstep",
label: "search + read",
type: "default",
label: "search",
...labelBg,
style: edgeStyle,
markerStart: marker,
@ -318,7 +318,7 @@ export const runtimeEdges: Edge[] = [
sourceHandle: "to-warehouse",
target: "warehouse",
targetHandle: "in",
type: "smoothstep",
type: "default",
label: "read-only",
...labelBg,
style: edgeStyle,

View file

@ -1,93 +0,0 @@
import { Suspense } from "react";
import { GitHubIcon } from "@/components/github-icon";
const REPO = "kaelio/ktx";
const REPO_URL = `https://github.com/${REPO}`;
const API_URL = `https://api.github.com/repos/${REPO}`;
async function fetchStarCount(): Promise<number | null> {
try {
const res = await fetch(API_URL, {
headers: { Accept: "application/vnd.github+json" },
// Revalidate hourly. GitHub's unauthenticated REST limit is 60 req/h per
// IP, so a single cached server-side fetch keeps the count fresh while
// never exposing visitors to rate limits or layout shift.
next: { revalidate: 3600 },
});
if (!res.ok) return null;
const data = (await res.json()) as { stargazers_count?: unknown };
return typeof data.stargazers_count === "number"
? data.stargazers_count
: null;
} catch {
return null;
}
}
/** Compact, GitHub-style count: 847 → "847", 1234 → "1.2k", 12345 → "12.3k". */
function formatStars(count: number): string {
if (count < 1000) return count.toLocaleString("en-US");
const thousands = count / 1000;
const rounded =
thousands >= 100 ? Math.round(thousands) : Math.round(thousands * 10) / 10;
return `${rounded}k`;
}
function StarGlyph() {
return (
<svg className="ktx-stars-star" viewBox="0 0 24 24" aria-hidden="true">
<path d="M12 2.6l2.9 5.88 6.49.95-4.7 4.57 1.11 6.46L12 17.4l-5.8 3.06 1.11-6.46-4.7-4.57 6.49-.95z" />
</svg>
);
}
async function StarsContent() {
const count = await fetchStarCount();
const label =
count === null
? "Star ktx on GitHub"
: `Star ktx on GitHub — ${count.toLocaleString("en-US")} stars`;
return (
<a
href={REPO_URL}
target="_blank"
rel="noopener noreferrer"
aria-label={label}
className="ktx-stars"
>
<span className="ktx-stars-seg ktx-stars-seg--label">
<GitHubIcon className="ktx-stars-gh" />
<span className="ktx-stars-text">Star</span>
</span>
{count !== null && (
<span className="ktx-stars-seg ktx-stars-seg--count">
<StarGlyph />
<span className="ktx-stars-count">{formatStars(count)}</span>
</span>
)}
</a>
);
}
function StarsSkeleton() {
return (
<span className="ktx-stars ktx-stars--skeleton" aria-hidden="true">
<span className="ktx-stars-seg ktx-stars-seg--label">
<GitHubIcon className="ktx-stars-gh" />
<span className="ktx-stars-text">Star</span>
</span>
<span className="ktx-stars-seg ktx-stars-seg--count">
<span className="ktx-stars-skeleton-bar" />
</span>
</span>
);
}
export function GitHubStars() {
return (
<Suspense fallback={<StarsSkeleton />}>
<StarsContent />
</Suspense>
);
}

View file

@ -1,56 +1,40 @@
"use client";
import Link from "next/link";
const brandFont = {
fontFamily: "var(--font-display), var(--font-sans), sans-serif",
} as const;
export function Logo({ href = "/", className }: { href?: string; className?: string }) {
export function Logo() {
return (
<div className={className}>
<div className="flex items-center gap-3.5 group">
<Link href={href} aria-label="ktx documentation home" className="flex items-center no-underline">
<span className="relative flex items-center justify-center transition-transform duration-300 ease-out group-hover:rotate-[-4deg]">
<img
src="/ktx/brand/ktx-mascot.svg"
alt=""
aria-hidden="true"
className="h-20 w-20 object-contain block dark:hidden"
/>
<img
src="/ktx/brand/ktx-mascot-dark.svg"
alt=""
aria-hidden="true"
className="h-20 w-20 object-contain hidden dark:block"
/>
</span>
</Link>
<div className="flex flex-col items-start leading-none">
<Link
href={href}
className="text-[42px] font-semibold text-fd-foreground tracking-tight no-underline"
style={brandFont}
>
ktx
</Link>
<a
href="https://www.kaelio.com"
target="_blank"
rel="noreferrer"
className="mt-1 whitespace-nowrap text-[13px] font-medium text-fd-muted-foreground/80 tracking-tight no-underline transition-colors hover:text-fd-foreground"
style={brandFont}
>
by Kaelio
</a>
</div>
<div className="flex items-center gap-3.5 group">
<div className="relative flex items-center justify-center transition-transform duration-300 ease-out group-hover:rotate-[-4deg]">
<img
src="/ktx/brand/ktx-mascot.svg"
alt=""
aria-hidden="true"
className="h-20 w-20 object-contain block dark:hidden"
/>
<img
src="/ktx/brand/ktx-mascot-dark.svg"
alt=""
aria-hidden="true"
className="h-20 w-20 object-contain hidden dark:block"
/>
</div>
<div className="flex flex-col items-start leading-none">
<span
className="text-[19px] font-medium text-fd-muted-foreground/80 tracking-tight border-l border-fd-border pl-3 ml-1"
style={brandFont}
className="text-[42px] font-semibold text-fd-foreground tracking-tight"
style={{ fontFamily: "var(--font-display), var(--font-sans), sans-serif" }}
>
Docs
ktx
</span>
<span
className="mt-1 whitespace-nowrap text-[13px] font-medium text-fd-muted-foreground/80 tracking-tight"
style={{ fontFamily: "var(--font-display), var(--font-sans), sans-serif" }}
>
by Kaelio
</span>
</div>
<span
className="text-[19px] font-medium text-fd-muted-foreground/80 tracking-tight border-l border-fd-border pl-3 ml-1"
style={{ fontFamily: "var(--font-display), var(--font-sans), sans-serif" }}
>
Docs
</span>
</div>
);
}

View file

@ -1,576 +0,0 @@
"use client";
import {
type Edge,
type EdgeProps,
getSmoothStepPath,
Handle,
MarkerType,
type Node,
type NodeProps,
Position,
} from "@xyflow/react";
import { FlowCanvas } from "./flow-canvas";
type AgentNodeData = {
title: string;
items: string[];
};
type HubNodeData = {
title: string;
badge: string;
rows: string[];
};
type TargetNodeData = {
accent: string;
title: string;
body: string;
rows: { text: string; color?: string; mono?: boolean }[];
badge?: string;
};
type AgentNode = Node<AgentNodeData, "agent">;
type HubNode = Node<HubNodeData, "hub">;
type TargetNode = Node<TargetNodeData, "target">;
type FlowNode = AgentNode | HubNode | TargetNode;
const AGENT_W = 252;
const AGENT_H = 96;
const HUB_W = 306;
const HUB_H = 190;
const TARGET_W = 268;
const TARGET_H = 148;
const CENTER_X = 470;
const ROW_AGENT_Y = 0;
const ROW_HUB_Y = 196;
const ROW_TARGET_Y = 488;
const AGENT_X = CENTER_X - AGENT_W / 2;
const HUB_X = CENTER_X - HUB_W / 2;
const TARGET_GAP_X = 38;
const TARGETS_TOTAL = TARGET_W * 2 + TARGET_GAP_X;
const TARGETS_START_X = CENTER_X - TARGETS_TOTAL / 2;
const CONTEXT_X = TARGETS_START_X;
const WAREHOUSE_X = TARGETS_START_X + TARGET_W + TARGET_GAP_X;
const EDGE_STROKE = "#94a3b8";
const CYCLE_STROKE = "#0e7490";
const EMERALD = "#059669";
const TEAL = "#0e7490";
const nodes: FlowNode[] = [
{
id: "agent",
type: "agent",
position: { x: AGENT_X, y: ROW_AGENT_Y },
data: {
title: "Your agent",
items: ["Claude Code", "Cursor", "Codex"],
},
draggable: false,
selectable: false,
},
{
id: "hub",
type: "hub",
position: { x: HUB_X, y: ROW_HUB_Y },
data: {
title: "ktx",
badge: "MCP + CLI",
rows: [
"Search wiki + semantic layer",
"Return approved metrics",
"Compile metrics → SQL",
],
},
draggable: false,
selectable: false,
},
{
id: "context",
type: "target",
position: { x: CONTEXT_X, y: ROW_TARGET_Y },
data: {
accent: TEAL,
title: "Context layer",
body: "Approved definitions agents search before they answer.",
rows: [
{ text: "wiki/*.md", color: EMERALD, mono: true },
{ text: "semantic-layer/*.yaml", color: TEAL, mono: true },
],
},
draggable: false,
selectable: false,
},
{
id: "warehouse",
type: "target",
position: { x: WAREHOUSE_X, y: ROW_TARGET_Y },
data: {
accent: "#334155",
title: "Database",
badge: "read-only",
body: "Runs the compiled SQL. ktx never writes to it.",
rows: [],
},
draggable: false,
selectable: false,
},
];
const labelBg = {
labelBgPadding: [6, 3] as [number, number],
labelBgBorderRadius: 4,
labelStyle: {
fontSize: 13,
fontWeight: 600,
fill: "var(--color-fd-muted-foreground)",
},
labelBgStyle: {
fill: "var(--color-fd-background)",
stroke: "var(--color-fd-border)",
strokeWidth: 1,
},
};
const requestMarker = {
type: MarkerType.ArrowClosed,
color: EDGE_STROKE,
width: 16,
height: 16,
};
const flowEdges: Edge[] = [
{
id: "e-ask",
source: "agent",
sourceHandle: "ask",
target: "hub",
targetHandle: "ask",
type: "straight",
label: "ask",
...labelBg,
style: { stroke: EDGE_STROKE, strokeWidth: 1.5 },
markerEnd: requestMarker,
},
{
id: "e-answer",
source: "hub",
sourceHandle: "answer",
target: "agent",
targetHandle: "answer",
type: "straight",
label: "answer",
...labelBg,
style: { stroke: EDGE_STROKE, strokeWidth: 1.5 },
markerEnd: requestMarker,
},
{
id: "e-search",
source: "hub",
sourceHandle: "to-context",
target: "context",
targetHandle: "in",
type: "smoothstep",
label: "search + read",
...labelBg,
style: { stroke: CYCLE_STROKE, strokeWidth: 1.5 },
markerStart: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
markerEnd: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
},
{
id: "e-readonly",
source: "hub",
sourceHandle: "to-warehouse",
target: "warehouse",
targetHandle: "in",
type: "smoothstep",
label: "read-only",
...labelBg,
style: { stroke: CYCLE_STROKE, strokeWidth: 1.5 },
markerStart: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
markerEnd: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
},
];
function AgentNodeView({ data }: NodeProps<AgentNode>) {
return (
<div
style={{ width: AGENT_W, height: AGENT_H }}
className="flex flex-col justify-center rounded-md border border-fd-border bg-fd-card px-3.5 py-2.5 shadow-sm"
>
<Handle
id="ask"
type="source"
position={Position.Bottom}
className="!opacity-0"
style={{ left: "35%" }}
/>
<Handle
id="answer"
type="target"
position={Position.Bottom}
className="!opacity-0"
style={{ left: "65%" }}
/>
<div className="flex items-center gap-2.5">
<span className="flex h-8 w-8 flex-none items-center justify-center rounded-full bg-fd-primary/15 text-fd-primary">
<svg
xmlns="http://www.w3.org/2000/svg"
width="18"
height="18"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
strokeWidth="1.75"
strokeLinecap="round"
strokeLinejoin="round"
aria-hidden="true"
>
<rect x="3" y="6" width="18" height="12" rx="3" />
<circle cx="9" cy="12" r="1.25" fill="currentColor" stroke="none" />
<circle cx="15" cy="12" r="1.25" fill="currentColor" stroke="none" />
<path d="M12 3v3" />
</svg>
</span>
<p className="text-[17px] font-semibold leading-6 text-fd-foreground">
{data.title}
</p>
</div>
<div className="mt-2 flex flex-wrap gap-1.5">
{data.items.map((item) => (
<span
key={item}
className="rounded border border-fd-border bg-fd-background px-1.5 py-0.5 text-[12px] leading-5 text-fd-muted-foreground"
>
{item}
</span>
))}
</div>
</div>
);
}
function HubNodeView({ data }: NodeProps<HubNode>) {
return (
<div
style={{ width: HUB_W, height: HUB_H }}
className="relative flex flex-col rounded-md border border-cyan-200/20 bg-[#0f1f23] px-4 py-3.5 text-white shadow-sm dark:bg-[#0b181b]"
>
<Handle
id="ask"
type="target"
position={Position.Top}
className="!opacity-0"
style={{ left: "37.5%" }}
/>
<Handle
id="answer"
type="source"
position={Position.Top}
className="!opacity-0"
style={{ left: "62.5%" }}
/>
<Handle
id="to-context"
type="source"
position={Position.Bottom}
className="!opacity-0"
style={{ left: "44%" }}
/>
<Handle
id="to-warehouse"
type="source"
position={Position.Bottom}
className="!opacity-0"
style={{ left: "56%" }}
/>
<div className="flex items-center gap-2.5">
<span className="flex h-7 w-7 flex-none items-center justify-center rounded-md bg-cyan-300/95 font-mono text-sm font-bold text-[#0b1c20]">
k
</span>
<span className="text-[19px] font-bold leading-6 text-white">
{data.title}
</span>
<span className="ml-1 rounded border border-cyan-200/30 bg-white/5 px-1.5 py-0.5 font-mono text-[11px] leading-5 text-cyan-100/85">
{data.badge}
</span>
</div>
<div className="mt-3 flex flex-1 flex-col justify-center gap-2">
{data.rows.map((row) => (
<div key={row} className="flex items-center gap-2.5">
<span className="h-1.5 w-1.5 flex-none rounded-full bg-cyan-300/95" />
<span className="text-[14px] font-medium leading-5 text-cyan-50/90">
{row}
</span>
</div>
))}
</div>
</div>
);
}
function TargetNodeView({ data }: NodeProps<TargetNode>) {
return (
<div
style={{
width: TARGET_W,
height: TARGET_H,
borderTop: `3px solid ${data.accent}`,
}}
className="overflow-hidden rounded-md border border-fd-border bg-fd-card px-3.5 py-3 shadow-sm"
>
<Handle id="in" type="target" position={Position.Top} className="!opacity-0" />
<div className="flex items-center gap-2">
<p className="text-[17px] font-semibold leading-6 text-fd-foreground">
{data.title}
</p>
{data.badge ? (
<span
className="rounded-full px-1.5 py-0.5 text-[11px] font-semibold leading-5"
style={{
color: data.accent,
background: "color-mix(in oklch, var(--color-fd-card) 86%, #64748b)",
}}
>
{data.badge}
</span>
) : null}
</div>
{data.rows.length > 0 ? (
<div className="mt-1 flex flex-col gap-0.5">
{data.rows.map((row) => (
<span
key={row.text}
className={
row.mono
? "font-mono text-[13px] font-semibold tracking-tight"
: "text-[12px] leading-4 text-fd-muted-foreground"
}
style={row.color ? { color: row.color } : undefined}
>
{row.text}
</span>
))}
</div>
) : null}
<p className="mt-1.5 line-clamp-2 text-[13px] leading-[18px] text-fd-muted-foreground">
{data.body}
</p>
</div>
);
}
/* ------------------------------- Particles ------------------------------- */
const PARTICLE_SPEED_PX_PER_SEC = 150;
const PARTICLE_MIN_DURATION_SEC = 5;
type Leg = {
sx: number;
sy: number;
sPos: Position;
tx: number;
ty: number;
tPos: Position;
};
const AGENT_ASK_X = AGENT_X + AGENT_W * 0.35;
const AGENT_ANSWER_X = AGENT_X + AGENT_W * 0.65;
const AGENT_BOTTOM_Y = ROW_AGENT_Y + AGENT_H;
const HUB_ASK_X = HUB_X + HUB_W * 0.375;
const HUB_ANSWER_X = HUB_X + HUB_W * 0.625;
const HUB_TO_CONTEXT_X = HUB_X + HUB_W * 0.44;
const HUB_TO_WAREHOUSE_X = HUB_X + HUB_W * 0.56;
const HUB_BOTTOM_Y = ROW_HUB_Y + HUB_H;
const CONTEXT_TOP_X = CONTEXT_X + TARGET_W / 2;
const WAREHOUSE_TOP_X = WAREHOUSE_X + TARGET_W / 2;
function buildCyclePath(spokeX: number, targetX: number): {
d: string;
length: number;
} {
const legs: Leg[] = [
// agent → hub (ask, down)
{ sx: AGENT_ASK_X, sy: AGENT_BOTTOM_Y, sPos: Position.Bottom, tx: HUB_ASK_X, ty: ROW_HUB_Y, tPos: Position.Top },
// through the hub to its spoke handle (down, drawn behind the hub)
{ sx: HUB_ASK_X, sy: ROW_HUB_Y, sPos: Position.Bottom, tx: spokeX, ty: HUB_BOTTOM_Y, tPos: Position.Top },
// hub → target (down)
{ sx: spokeX, sy: HUB_BOTTOM_Y, sPos: Position.Bottom, tx: targetX, ty: ROW_TARGET_Y, tPos: Position.Top },
// target → hub (up)
{ sx: targetX, sy: ROW_TARGET_Y, sPos: Position.Top, tx: spokeX, ty: HUB_BOTTOM_Y, tPos: Position.Bottom },
// through the hub to its answer handle (up, drawn behind the hub)
{ sx: spokeX, sy: HUB_BOTTOM_Y, sPos: Position.Top, tx: HUB_ANSWER_X, ty: ROW_HUB_Y, tPos: Position.Bottom },
// hub → agent (answer, up)
{ sx: HUB_ANSWER_X, sy: ROW_HUB_Y, sPos: Position.Top, tx: AGENT_ANSWER_X, ty: AGENT_BOTTOM_Y, tPos: Position.Bottom },
];
const segments = legs.map((leg) => {
const [segment] = getSmoothStepPath({
sourceX: leg.sx,
sourceY: leg.sy,
sourcePosition: leg.sPos,
targetX: leg.tx,
targetY: leg.ty,
targetPosition: leg.tPos,
});
return segment;
});
let d = segments[0];
for (let i = 1; i < segments.length; i += 1) {
d += ` ${segments[i].replace(/^M/, "L")}`;
}
const length = legs.reduce(
(sum, leg) => sum + Math.abs(leg.tx - leg.sx) + Math.abs(leg.ty - leg.sy),
0,
);
return { d, length };
}
type ParticleEdgeData = {
d: string;
duration: number;
beginOffset: number;
color: string;
};
type ParticleEdge = Edge<ParticleEdgeData, "particle">;
function ParticleEdgeView({ id, data }: EdgeProps<ParticleEdge>) {
if (!data) return null;
const pathId = `runtime-particle-path-${id}`;
return (
<>
<path id={pathId} d={data.d} fill="none" stroke="none" pointerEvents="none" />
<g className="runtime-particle" style={{ color: data.color }}>
<circle r={7.5} fill="currentColor" opacity={0.16} />
<circle r={3.75} fill="currentColor" opacity={0.32} />
<circle r={2.1} fill="currentColor" />
<animateMotion
dur={`${data.duration.toFixed(2)}s`}
begin={`-${data.beginOffset.toFixed(2)}s`}
repeatCount="indefinite"
>
<mpath href={`#${pathId}`} />
</animateMotion>
</g>
</>
);
}
function makeCycleEdge(
id: string,
source: string,
spokeX: number,
targetX: number,
beginFraction: number,
): ParticleEdge {
const { d, length } = buildCyclePath(spokeX, targetX);
const duration = Math.max(
PARTICLE_MIN_DURATION_SEC,
length / PARTICLE_SPEED_PX_PER_SEC,
);
return {
id,
source,
target: source,
type: "particle",
data: { d, duration, beginOffset: duration * beginFraction, color: CYCLE_STROKE },
};
}
const particleEdges: ParticleEdge[] = [
makeCycleEdge("p-context", "context", HUB_TO_CONTEXT_X, CONTEXT_TOP_X, 0),
makeCycleEdge("p-warehouse", "warehouse", HUB_TO_WAREHOUSE_X, WAREHOUSE_TOP_X, 0.5),
];
const nodeTypes = {
agent: AgentNodeView,
hub: HubNodeView,
target: TargetNodeView,
};
const edgeTypes = {
particle: ParticleEdgeView,
};
const edges = [...flowEdges, ...particleEdges];
export function ProductRuntime() {
return (
<section
className="not-prose my-12 w-full max-w-full min-w-0 space-y-5"
aria-labelledby="runtime-title"
>
<div className="max-w-3xl">
<h2
id="runtime-title"
className="text-xl font-semibold tracking-normal text-fd-foreground sm:text-2xl"
style={{ fontFamily: "var(--font-display)" }}
>
How serving works
</h2>
<p className="mt-3 text-sm leading-6 text-fd-muted-foreground">
At runtime, agents reach ktx through MCP. ktx searches the context
layer, returns approved metrics, and compiles them into read-only SQL
the warehouse runs.
</p>
</div>
<article
className="max-w-full min-w-0 overflow-hidden rounded-lg border border-fd-border bg-fd-card shadow-sm"
aria-label="ktx serving flow from an agent request to a governed answer"
>
<div className="border-b border-fd-border bg-fd-muted/35 px-5 py-4">
<p className="text-xs font-semibold uppercase tracking-wide text-fd-primary">
Serving flow
</p>
<h3
className="mt-1 text-base font-semibold tracking-normal text-fd-foreground sm:text-lg"
style={{ fontFamily: "var(--font-display)" }}
>
From an agent request to a governed answer
</h3>
<p className="mt-2 max-w-3xl text-xs leading-5 text-fd-muted-foreground">
The agent asks in plain language. ktx is the only thing that touches
the context layer and the warehouse, and every database connection
is read-only.
</p>
</div>
<FlowCanvas
nodes={nodes}
edges={edges}
nodeTypes={nodeTypes}
edgeTypes={edgeTypes}
canvasStyle={{
height: "min(620px, 98vw)",
minHeight: 430,
}}
className="runtime-canvas"
fitViewOptions={{ padding: 0.06 }}
ariaLabel="ktx serving flow diagram"
/>
</article>
<style>{`
.runtime-canvas .runtime-particle {
pointer-events: none;
filter: drop-shadow(0 0 6px currentColor);
}
@media (prefers-reduced-motion: reduce) {
.runtime-canvas .runtime-particle {
display: none;
}
}
`}</style>
</section>
);
}

View file

@ -0,0 +1,40 @@
---
title: Agent Instructions
description: Suggested instructions for coding assistants that need to read and cite ktx docs.
---
Use these instructions when a coding assistant needs to answer questions from the **ktx** documentation.
```text
When answering ktx docs questions:
1. Start with https://docs.kaelio.com/ktx/llms.txt.
2. Fetch the smallest relevant Markdown page from the index.
3. Prefer /docs/<path>.md over rendered HTML.
4. Use https://docs.kaelio.com/ktx/llms-full.txt only when the task needs broad docs context.
5. Quote commands exactly from docs pages.
6. If docs and local repository behavior disagree, say what differs and prefer local verified output for code changes.
```
## What this is for
This page is for documentation consumption only:
- answering questions about **ktx**
- finding the right docs page
- citing setup or CLI guidance
- helping an assistant avoid stale or invented commands
It does not describe local tool configuration.
## Minimal project prompt
```text
You are helping with ktx. Read https://docs.kaelio.com/ktx/llms.txt first, then fetch only the Markdown pages needed for the task. Do not scrape the rendered docs site when a .md route exists.
```
## Repository prompt
```text
Before editing ktx docs, read /llms.txt and the affected .md docs pages. Keep AI Resources focused on docs consumption. After editing, verify /llms.txt, /llms-full.txt, and any changed .md routes.
```

View file

@ -0,0 +1,54 @@
---
title: Agent Quickstart
description: A task-first route for coding agents that need to understand ktx docs.
---
This page is for coding assistants reading or citing the **ktx** docs. It is intentionally limited to documentation lookup, docs navigation, and safe command discovery.
For Markdown endpoints, use [Markdown Access](/docs/ai-resources/markdown-access).
For reusable task prompts, use [Prompt Recipes](/docs/ai-resources/prompt-recipes).
To install **ktx** into an agent client, use [Agent Clients](/docs/integrations/agent-clients).
## First read
Agents should start with the smallest source that answers the task:
1. [`/llms.txt`](/llms.txt) - discover the docs and preferred entry points.
2. The relevant per-page Markdown URL, for example `/docs/getting-started/quickstart.md`.
3. [`/llms-full.txt`](/llms-full.txt) - use only when the task needs broad context across many pages.
## Task router
| User asks the agent to explain... | Read first | Then read |
|------------------------------------|------------|-----------|
| What **ktx** does | [Introduction](/docs/getting-started/introduction) | [The Context Layer](/docs/concepts/the-context-layer) |
| How to start from a checkout | [Quickstart](/docs/getting-started/quickstart) | [ktx setup](/docs/cli-reference/ktx-setup) |
| How to check project readiness | [ktx status](/docs/cli-reference/ktx-status) | [Quickstart](/docs/getting-started/quickstart) |
| How context gets built | [Building Context](/docs/guides/building-context) | [ktx ingest](/docs/cli-reference/ktx-ingest) |
| How semantic YAML works | [Writing Context](/docs/guides/writing-context) | [ktx sl](/docs/cli-reference/ktx-sl) |
| How machine-readable CLI output is shaped | [ktx sl](/docs/cli-reference/ktx-sl) | [ktx wiki](/docs/cli-reference/ktx-wiki) |
## Operating workflow
Use this workflow when the user asks an assistant to answer a **ktx** docs question:
1. Read [`/llms.txt`](/llms.txt).
2. Pick the smallest relevant `.md` page.
3. Use [`/llms-full.txt`](/llms-full.txt) only if the answer needs multiple sections of the docs.
4. Quote commands exactly from the docs page.
5. If a command affects a local project, ask the user before assuming credentials or live services are available.
## Docs lookup from a shell
```bash
curl https://docs.kaelio.com/ktx/llms.txt
curl https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
```
## Guardrails
- Do not invent CLI flags. Fetch the relevant CLI reference page.
- Do not scrape rendered HTML when a `.md` route exists.
- Do not assume docs lookup requires agent-client configuration.
- Do not include credentials or secrets in prompts, URLs, or copied docs snippets.
- When docs and local CLI behavior disagree, prefer the local CLI output and mention the mismatch.

View file

@ -0,0 +1,76 @@
---
title: Markdown Access
description: Fetch ktx docs as llms.txt, llms-full.txt, or per-page Markdown.
---
**ktx** docs are available as plain Markdown so assistants do not need to parse the rendered HTML site.
## Index
Fetch the curated index:
```text
https://docs.kaelio.com/ktx/llms.txt
```
Use this file to discover high-value pages, task-specific entry points, and Markdown URLs.
## Full corpus
Fetch the complete docs corpus:
```text
https://docs.kaelio.com/ktx/llms-full.txt
```
Use this when an assistant needs broad context across setup, concepts, CLI reference, integrations, and troubleshooting. Prefer the smaller per-page Markdown route for narrow tasks.
## Per-page Markdown
Every docs page has a Markdown route:
```text
https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
https://docs.kaelio.com/ktx/docs/cli-reference/ktx-sl.md
https://docs.kaelio.com/ktx/docs/cli-reference/ktx-wiki.md
https://docs.kaelio.com/ktx/docs/guides/building-context.md
```
Requests that ask for Markdown can also use the normal docs URL with `Accept: text/markdown`:
```bash
curl -H "Accept: text/markdown" https://docs.kaelio.com/ktx/docs/getting-started/quickstart
```
## Recommended retrieval order
1. Fetch `/llms.txt`.
2. Select one or two relevant page Markdown URLs.
3. Fetch `/llms-full.txt` only when page-level docs are not enough.
## Output contract
Markdown responses are designed for agent consumption:
- Frontmatter is removed.
- Each page includes a title, description, canonical URL, and Markdown URL.
- Code blocks stay as code blocks.
- Tables stay as Markdown tables.
- Missing docs pages return a plain-text `404` instead of silently falling back to HTML.
## Page actions
Rendered docs pages include page-level actions near the title:
- **Copy MD** copies the generated Markdown for the current page.
- **View MD** opens the generated Markdown route.
- **Copy MDX** copies the source MDX for the current page.
## Common mistakes
| Mistake | Better path |
|---------|-------------|
| Scraping the HTML page for a docs answer | Fetch the `.md` route instead |
| Loading `/llms-full.txt` for a single CLI flag lookup | Fetch the relevant CLI reference page |
| Treating `/llms.txt` as complete documentation | Use it as an index, then fetch linked pages |
| Copying rendered text by hand | Use **Copy MD** or **Copy MDX** from the page actions |

View file

@ -0,0 +1,10 @@
{
"title": "AI Resources",
"defaultOpen": true,
"pages": [
"agent-quickstart",
"markdown-access",
"agent-instructions",
"prompt-recipes"
]
}

View file

@ -0,0 +1,55 @@
---
title: Prompt Recipes
description: Copyable prompts for common ktx agent workflows.
---
Use these prompts when asking a coding assistant to work with **ktx**. Replace project names, connection ids, and business terms with your own values.
## Learn the docs
```text
Read https://docs.kaelio.com/ktx/llms.txt first. Then fetch only the ktx Markdown pages needed for this task. Do not scrape rendered HTML unless no Markdown route exists.
```
## Set up a project
```text
Run npx skills add Kaelio/ktx-ai-data-agents-context --skill ktx and use the ktx skill to install
and configure ktx in this project.
```
## Find a command
```text
Find the correct ktx command for this task: <task>. Start with /llms.txt, then fetch the smallest relevant CLI reference .md page. Quote the exact command and flags from the docs.
```
## Explain setup
```text
Explain how to set up ktx for this repo. Read /docs/getting-started/quickstart.md and the relevant CLI reference pages. Summarize prerequisites, commands, generated files, and any credentials the user must provide manually.
```
## Compare concepts
```text
Explain the difference between these ktx concepts: <concepts>. Start from /llms.txt, fetch the relevant concept and guide pages as Markdown, and answer with links to the source pages.
```
## Review semantic changes
```text
Review the ktx semantic-layer and knowledge changes in this branch. Check that measures have clear definitions, joins use valid keys, hidden/internal columns are not exposed to agents, and validation passes. List concrete file and line issues first.
```
## Copy exact docs source
```text
Open the relevant ktx docs page and use the page action to copy the generated Markdown or source MDX. Preserve code fences and tables exactly.
```
## Update docs
```text
Update the ktx docs for agent readability. Keep AI Resources focused on docs consumption. After editing, verify /llms.txt, /llms-full.txt, and the affected .md routes.
```

View file

@ -104,6 +104,6 @@ configured connection and exit non-zero if any probe fails.
| Error | Cause | Recovery |
|-------|-------|----------|
| No connections configured | The project has no entries under `connections` | Run `ktx setup` and add a database or context-source connection |
| Connection test fails | Credentials, network access, database, warehouse, or schema is invalid | Use the setup recovery menu to retry or re-enter details; if it still fails, verify the same URL with the database's native client |
| Mapping validation fails during setup | BI database mappings do not point at valid warehouse connections | Use the setup recovery menu to retry validation or re-enter mapping selections; rerun `ktx setup` if you already exited |
| Connection test fails | Credentials, network access, database, warehouse, or schema is invalid | Verify the same URL with the database's native client, then rerun `ktx setup` and reconfigure the connection |
| Mapping validation fails during setup | BI database mappings do not point at valid warehouse connections | Rerun `ktx setup` and update the context-source mapping selections |
| Notion page picker cannot run | The terminal is non-interactive or Notion discovery failed | Rerun interactive `ktx setup`, or use non-interactive setup flags with explicit root page ids |

View file

@ -177,9 +177,7 @@ Slowest phase: reconciliation (2m 05s, 48% of wall time). 2 work units (1 failed
Work units run serially by default (`ingest.workUnits.maxConcurrency` is `1`);
raise it in `ktx.yaml` if the profile shows the run is bound by serialized
work-unit agent loops. If the provider reports an LLM rate limit, **ktx** shows
a transient wait message and temporarily reduces effective work-unit concurrency
according to `ingest.rateLimit`.
work-unit agent loops.
## Common errors

View file

@ -51,9 +51,9 @@ prompts.
| Flag | Description |
|------|-------------|
| `--llm-backend <backend>` | LLM backend: `anthropic`, `vertex`, `claude-code`, or `codex` |
| `--llm-backend <backend>` | LLM backend: `anthropic`, `vertex`, or `claude-code` |
| `--llm-backend claude-code` | Use the local Claude Code session for **ktx** LLM calls |
| `--llm-backend codex` | Use local Codex authentication for **ktx** LLM calls |
| `--llm-model <model>` | LLM model ID or backend model alias to validate and save |
| `--anthropic-api-key-env <name>` | Environment variable containing the Anthropic API key |
| `--anthropic-api-key-file <path>` | File containing the Anthropic API key |
| `--vertex-project <project>` | Vertex AI project ID, `env:NAME`, or `file:/path` reference |
@ -62,17 +62,9 @@ prompts.
Choose only one Anthropic credential source. Anthropic credential flags are only
valid with the Anthropic backend; Vertex flags are only valid with the Vertex
backend. The `claude-code` and `codex` backends use local authentication instead
of Anthropic API key or Vertex flags. After you choose a backend, `ktx setup`
writes that backend's per-role model preset to `ktx.yaml`. To change a model,
edit the matching `llm.models.<role>` value in `ktx.yaml`.
With `--no-input`, `ktx setup` does not assume a default LLM provider, because
every backend needs credentials only you can supply. Pass `--llm-backend`
explicitly. Note that `--target` selects the agent integration, not the LLM
provider: `ktx setup --target claude-code --no-input` still needs
`--llm-backend claude-code` to use your Claude subscription for **ktx** LLM
calls.
backend. The `claude-code` backend uses local Claude Code authentication instead
of Anthropic API key or Vertex flags. For Claude Code, `--llm-model` accepts
`sonnet`, `opus`, `haiku`, or a full Claude model ID.
### Embeddings
@ -150,13 +142,6 @@ fix the prerequisite. If the later schema-context build also fails, interactive
setup offers **Disable query history and retry** so you can finish database
setup with `connections.<id>.context.queryHistory.enabled: false`.
After the schema scan completes, setup can derive query-history service-account
filters from in-scope history. If **ktx** finds clear operational roles, it
prints each proposed exclusion with a reason and writes
`connections.<id>.context.queryHistory.filters.serviceAccounts` only when you
apply the proposal. In non-interactive setup with `--yes`, the proposal is
applied automatically. Existing `serviceAccounts` blocks are never overwritten.
For BigQuery, the remediation tells you to grant `roles/bigquery.resourceViewer`
on the BigQuery project, or grant a custom role that contains
`bigquery.jobs.listAll`.
@ -200,22 +185,12 @@ ktx setup
# Run setup for a specific project directory
ktx setup --project-dir ./analytics
# Use Claude Code for ktx LLM calls
# Use Claude Code with Opus for ktx LLM calls
ktx setup \
--project-dir ./analytics \
--llm-backend claude-code
--llm-backend claude-code \
--llm-model opus
# Configure **ktx** to use local Codex authentication for LLM work
ktx setup --llm-backend codex --no-input
```
When you choose `--llm-backend codex`, setup prints a warning if the public
Codex SDK and CLI surface cannot prove full Claude-Code-style isolation. The
backend restricts **ktx** runtime MCP tools to each run, but Codex may still
load user Codex config and built-in command execution or read-only file
capabilities.
```bash
# Script a Postgres connection that reads its URL from the environment
ktx setup \
--project-dir ./analytics \
@ -283,7 +258,6 @@ Use `ktx status` for repeatable readiness checks after setup exits.
|-------|-------|----------|
| Setup resumes an unexpected project | `KTX_PROJECT_DIR` or nearest `ktx.yaml` points to another directory | Pass `--project-dir <path>` explicitly |
| Setup cannot run in CI | Required values are missing and `--no-input` disables prompts | Provide the relevant automation flags or create a fixture `ktx.yaml` |
| `Missing LLM backend: pass --llm-backend …` | `--no-input` setup ran without an LLM backend; `--target` does not select one | Pass `--llm-backend claude-code`, `codex`, `anthropic`, or `vertex` (with that backend's credential flags) |
| Provider health check fails | Provider key, model id, Vertex project, or Vertex location is invalid | Fix the `env:` or `file:` reference and rerun setup |
| Python runtime is missing | The selected setup needs runtime-backed agent, query-history, Looker, or local embedding features | Accept the interactive prompt, rerun with `--yes`, or run the suggested `ktx admin runtime install` command |
| `--enable-query-history` is rejected | The selected database driver does not support query history | Use Postgres, BigQuery, or Snowflake, or rerun without query-history flags |

View file

@ -21,7 +21,7 @@ ktx status [options]
| `--json` | Print JSON output | `false` |
| `-v`, `--verbose` | Show every check, including passing ones | `false` |
| `--validate` | Only validate the `ktx.yaml` schema; skip readiness checks | `false` |
| `--fast` | Skip checks that require external communication (query-history readiness probes, Claude Code auth probe, and Codex auth probe) | `false` |
| `--fast` | Skip checks that require external communication (query-history readiness probes and Claude Code auth probe) | `false` |
| `--no-input` | Disable interactive terminal input | - |
## Examples
@ -39,7 +39,7 @@ ktx status --verbose
# Validate ktx.yaml without running readiness checks
ktx status --validate
# Skip slow probes (query-history readiness, Claude Code auth, Codex auth)
# Skip slow probes (query-history readiness, Claude Code auth)
ktx status --fast
# Check a project from another directory
@ -57,16 +57,6 @@ flow, then rerun `ktx status`. Use `--fast` to skip this probe (useful in CI
or offline contexts); skipped checks render as `-` and carry
`"status": "skipped"` in JSON output.
For `llm.provider.backend: codex`, `ktx status` runs a minimal non-interactive
Codex request. If the probe fails, authenticate Codex locally with the Codex CLI
and verify the Codex CLI installation.
When `llm.provider.backend: codex` is configured, `ktx status` also prints a
warning when the installed public Codex SDK and CLI surface cannot prove full
Claude-Code-style isolation. The warning does not block authenticated Codex
usage, but it marks the project status as partial so you can make an explicit
runtime-isolation decision.
A `Local data` section summarises what the project has accumulated locally:
ingest run counts, last completed timestamp per connection, knowledge page
counts by scope, semantic-layer source and dictionary value counts, and the

View file

@ -74,56 +74,6 @@ The public context-build entrypoint is `ktx ingest [connectionId]` or
| `-v`, `--version` | Show the CLI package name and version. |
| `-h`, `--help` | Show help for the current command. |
## Update notices
> **Note:** The update notifier writes only to stderr and keeps command stdout
> unchanged.
When a newer package is available on your installed release channel, `ktx`
prints a short notice after the command finishes:
```text
↑ Update available: ktx 0.9.0 → 0.10.0
npm i -g @kaelio/ktx
```
Stable installs compare against the npm `latest` dist-tag.
Release-candidate installs compare against the `next` dist-tag and show:
```text
npm i -g @kaelio/ktx@next
```
The check is skipped for JSON output, CI, non-TTY stdout, and hidden completion
commands. To opt out explicitly, set any of these environment variables:
```bash
KTX_NO_UPDATE_CHECK=1
NO_UPDATE_NOTIFIER=1
DO_NOT_TRACK=1
```
The `ktx` CLI prints one npm command because globally installed binaries don't
expose a reliable runtime package-manager signal. If you prefer another global
package manager, use the equivalent command:
```bash
pnpm add -g @kaelio/ktx
yarn global add @kaelio/ktx
```
## Build-view star prompt
During an interactive context build, `ktx setup` and `ktx ingest` can show a dim
GitHub star reminder above the `Ctrl+C to stop` hint. **ktx** skips this prompt
for CI, non-TTY output, and `DO_NOT_TRACK=1`.
To suppress only this prompt while keeping other notices enabled, set:
```bash
KTX_NO_STAR=1
```
## Project resolution
Most commands are project-aware. Pass `--project-dir <path>` when scripting or

View file

@ -1,111 +0,0 @@
---
title: AI Resources
description: How coding agents read, cite, and act on the ktx docs - Markdown endpoints, a task router, and copy-paste prompts.
---
This page is for coding assistants that read or cite the **ktx** docs. It covers
the machine-readable endpoints, a task router for common questions, and
copy-paste prompts. It is scoped to documentation lookup and safe command
discovery - to wire **ktx** into an agent client, see
[Agent Clients](/docs/integrations/agent-clients).
## Markdown endpoints
**ktx** docs are available as plain Markdown so assistants never have to parse
the rendered HTML site.
- [`/llms.txt`](/llms.txt) - a curated index of high-value pages and agent entry
points. **Start here.**
- [`/llms-full.txt`](/llms-full.txt) - the entire docs corpus in one response.
Use only when a task needs broad context across many pages.
- **Per-page Markdown** - append `.md` to any docs URL:
```text
https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
https://docs.kaelio.com/ktx/docs/cli-reference/ktx-sl.md
https://docs.kaelio.com/ktx/docs/guides/building-context.md
```
A request for any docs URL with an `Accept: text/markdown` header returns the
same Markdown without the `.md` suffix:
```bash
curl -H "Accept: text/markdown" https://docs.kaelio.com/ktx/docs/getting-started/quickstart
```
Each Markdown response leads with the page title, description, canonical URL, and
Markdown URL; frontmatter is stripped; code blocks and tables are preserved; and
missing pages return a plain-text `404` instead of falling back to HTML. Rendered
pages also expose a **Copy as Markdown** action near the title.
### Retrieval order
1. Fetch [`/llms.txt`](/llms.txt).
2. Pick one or two relevant per-page `.md` URLs.
3. Fetch [`/llms-full.txt`](/llms-full.txt) only when page-level docs are not
enough.
```bash
curl https://docs.kaelio.com/ktx/llms.txt
curl https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
```
## Task router
| User asks the agent to explain... | Read first | Then read |
|------------------------------------|------------|-----------|
| What **ktx** does | [Introduction](/docs/getting-started/introduction) | [The Context Layer](/docs/concepts/the-context-layer) |
| How to start from a checkout | [Quickstart](/docs/getting-started/quickstart) | [ktx setup](/docs/cli-reference/ktx-setup) |
| How to check project readiness | [ktx status](/docs/cli-reference/ktx-status) | [Quickstart](/docs/getting-started/quickstart) |
| How context gets built | [Building Context](/docs/guides/building-context) | [ktx ingest](/docs/cli-reference/ktx-ingest) |
| How semantic YAML works | [Writing Context](/docs/guides/writing-context) | [ktx sl](/docs/cli-reference/ktx-sl) |
| How machine-readable CLI output is shaped | [ktx sl](/docs/cli-reference/ktx-sl) | [ktx wiki](/docs/cli-reference/ktx-wiki) |
## Agent instructions
Paste this into a project or system prompt when an assistant needs to answer
from the **ktx** docs:
```text
When answering ktx docs questions:
1. Start with https://docs.kaelio.com/ktx/llms.txt.
2. Fetch the smallest relevant Markdown page (append .md to its docs URL).
3. Prefer the .md route over rendered HTML.
4. Use https://docs.kaelio.com/ktx/llms-full.txt only when the task needs broad docs context.
5. Quote commands exactly from docs pages.
6. If docs and local CLI behavior disagree, say what differs and prefer local verified output.
```
## Prompts
Replace project names, connection ids, and business terms with your own values.
**Install and configure ktx in a project**
```text
Run npx skills add Kaelio/ktx --skill ktx and use the ktx skill to install and configure ktx
```
**Find the right command**
```text
Find the correct ktx command for this task: <task>. Start with /llms.txt, then fetch the smallest relevant CLI reference .md page. Quote the exact command and flags from the docs.
```
**Review semantic changes**
```text
Review the ktx semantic-layer and wiki changes in this branch. Check that measures have clear definitions, joins use valid keys, hidden or internal columns are not exposed to agents, and validation passes. List concrete file and line issues first.
```
## Guardrails
- Do not invent CLI flags - fetch the relevant CLI reference page.
- Do not scrape rendered HTML when a `.md` route exists.
- Do not treat `/llms.txt` as complete documentation - use it as an index, then
fetch the linked pages.
- Do not include credentials or secrets in prompts, URLs, or copied docs
snippets.
- When docs and local CLI behavior disagree, prefer the local CLI output and
mention the mismatch.

View file

@ -1,5 +1,5 @@
{
"title": "Community & Resources",
"title": "Community",
"defaultOpen": true,
"pages": ["support", "contributing", "telemetry", "ai-resources"]
"pages": ["support", "contributing", "telemetry"]
}

View file

@ -11,7 +11,7 @@ the core team trade questions, share patterns, and shape the roadmap.
| You want to... | Go here |
|----------------|---------|
| Ask a question or chat with the community | [**ktx** Slack](https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ) |
| Report a bug or request a feature | [GitHub Issues](https://github.com/Kaelio/ktx/issues) |
| Report a bug or request a feature | [GitHub Issues](https://github.com/Kaelio/ktx-ai-data-agents-context/issues) |
| Read or contribute to the docs | [docs.kaelio.com/ktx](https://docs.kaelio.com/ktx/docs/) |
| Contribute code | [Contributing guide](/docs/community/contributing) |
@ -30,14 +30,14 @@ Slack is the right place for:
- **Feedback** on the roadmap and early features
For anything reproducible - a crash, a wrong result, an unexpected CLI error -
open a [GitHub issue](https://github.com/Kaelio/ktx/issues) instead. Issues are
open a [GitHub issue](https://github.com/Kaelio/ktx-ai-data-agents-context/issues) instead. Issues are
searchable, get triaged, and stay attached to the eventual fix.
## GitHub
- **[Issues](https://github.com/Kaelio/ktx/issues)** - bugs and feature requests
- **[Pull requests](https://github.com/Kaelio/ktx/pulls)** - code, docs, and connector contributions
- **[Releases](https://github.com/Kaelio/ktx/releases)** - changelog and published versions
- **[Issues](https://github.com/Kaelio/ktx-ai-data-agents-context/issues)** - bugs and feature requests
- **[Pull requests](https://github.com/Kaelio/ktx-ai-data-agents-context/pulls)** - code, docs, and connector contributions
- **[Releases](https://github.com/Kaelio/ktx-ai-data-agents-context/releases)** - changelog and published versions
## Code of conduct

View file

@ -6,10 +6,11 @@ description: Understand what usage telemetry ktx collects and how to opt out.
**ktx** collects aggregated usage telemetry so maintainers can see
which commands work, where setup fails, and which parts of the data-agent
workflow need improvement. Telemetry is opt-out: it turns on the first time you
run **ktx** in any way — an interactive command, a script, or an
agent-launched MCP server — and prints a one-time notice (to the terminal when
there is one, otherwise to standard error). It stays disabled in CI and whenever
an opt-out is set.
run **ktx** in an interactive terminal, which prints a one-time notice. From
then on the same install also reports background activity that has no terminal
of its own, such as the local MCP server your agent calls. It stays disabled in
CI, whenever an opt-out is set, and until that first interactive run has shown
the notice.
## Opt out
@ -24,11 +25,10 @@ Use any of these mechanisms to disable telemetry:
## What we collect
High-level signals: which commands run, how long they take, whether they
High-level signals only: which commands run, how long they take, whether they
succeed or fail, and basic environment metadata (CLI version, Node version, OS
platform). When an operation fails, we also include diagnostic detail about the
error so we can debug it. For project-level analysis, **ktx** sends a salted
hash of the project directory to group events.
platform). For project-level analysis, **ktx** sends a salted hash of the
project directory — never the raw path.
When an agent reaches **ktx** through MCP, we also record the connecting client
tool's self-reported name and version (for example Claude Desktop, Cursor, or
@ -37,41 +37,11 @@ tool, never you or your data.
## What we never collect
We build telemetry around counts and coarse signals, not the contents of your
data or configuration. We don't deliberately collect your `ktx.yaml`, query
results, passwords, API keys, or access tokens.
The one place environment-specific text can appear is failure diagnostics: when
an operation errors, the detail we record is the error as your tools reported
it, which can include identifiers from your setup. If you'd rather send nothing
at all, turn telemetry off using any of the options above.
## Error reports
When telemetry is enabled, **ktx** sends PostHog Error Tracking `$exception`
events for CLI and daemon exceptions. Error reports help group crashes and
handled failures into PostHog issues.
Error reports can include:
- Stack frames, including function names, local file paths, line numbers, and
SDK-provided source context.
- Error class names and raw error messages.
- Cause chains when the runtime exposes them.
- `source`, `handled`, and `fatal` diagnostic fields.
- Runtime version, OS, architecture, and CI fields.
- The hashed `projectId` when **ktx** knows the project.
Error reports never intentionally include:
- Secrets, credentials, API keys, tokens, cookies, signed URLs, or auth headers.
- Database URLs, connection strings, DSNs, raw argv, or raw environment values.
- SQL text, schema names, table names, or column names as explicit payload
properties.
- Customer row data.
- User prompt text or raw MCP arguments.
The same opt-out controls listed above disable error reports.
- File paths, hostnames, environment variable values, or command arguments
- `ktx.yaml` contents, connection passwords, API keys, or tokens
- Schema names, table names, column names, SQL text, or query results
- Error messages or stack traces
- Git remote URLs, Git user email, OS user, or hostname
## Storage and retention

View file

@ -179,22 +179,9 @@ connections:
context:
queryHistory:
enabled: true
enabledSchemas:
- orbit_raw
- orbit_analytics
minExecutions: 5
```
- `enabledSchemas`: Optional list of schema or dataset names that query-history
ingest may mine. Omit it to let **ktx** derive the modeled schema floor from
the connection and semantic-layer sources. Use `["*"]` to disable the floor
for discovery runs.
- `filters.serviceAccounts`: Optional service-account filter block. During
setup, when query history is enabled and no service-account block already
exists, **ktx** can propose exact role patterns such as `^svc_loader$` from
observed in-scope query history. The block uses `mode: exclude` and remains
hand-editable.
### Metabase
```yaml
@ -377,10 +364,6 @@ llm:
models:
default: claude-sonnet-4-6
triage: claude-haiku-4-5
candidateExtraction: claude-sonnet-4-6
curator: claude-opus-4-7
reconcile: claude-opus-4-7
repair: claude-haiku-4-5
promptCaching:
enabled: true
systemTtl: 1h
@ -393,28 +376,13 @@ llm:
| Field | Type | Default | Purpose |
|-------|------|---------|---------|
| `provider.backend` | `none` \| `anthropic` \| `vertex` \| `gateway` \| `claude-code` \| `codex` | `none` | Selected backend. `none` disables LLM features. `claude-code` uses the local Claude Code session and needs no API key. `codex` uses local Codex authentication and needs no API key. |
| `provider.backend` | `none` \| `anthropic` \| `vertex` \| `gateway` \| `claude-code` | `none` | Selected backend. `none` disables LLM features. `claude-code` uses the local Claude Code session and needs no API key. |
| `provider.anthropic.api_key` | `string` | - | Anthropic API key. Required when `backend: anthropic`. Accepts `env:` or `file:` references. |
| `provider.anthropic.base_url` | `string` | - | Override the Anthropic API base URL (proxy, self-hosted gateway). |
| `provider.gateway.api_key` / `base_url` | `string` | - | Credentials for an AI Gateway provider. Required when `backend: gateway`. |
| `provider.vertex.project` | `string` | - | Google Cloud project ID hosting the Vertex AI endpoint. |
| `provider.vertex.location` | `string` | - | Vertex AI region (for example `us-east5`). Required when the `vertex` block is present. |
Use `codex` when local Codex authentication should power **ktx** LLM work:
```yaml
llm:
provider:
backend: codex
models:
default: gpt-5.5
triage: gpt-5.5
candidateExtraction: gpt-5.5
curator: gpt-5.5
reconcile: gpt-5.5
repair: gpt-5.5
```
### Model roles
`models` overrides the per-role model. Keys are fixed; values are
@ -461,16 +429,6 @@ ingest:
stepBudget: 40
maxConcurrency: 2
failureMode: continue
rateLimit:
enabled: true
throttleThreshold: 0.8
minConcurrencyUnderPressure: 1
maxWaitMs: 600000
retry:
maxAttempts: 6
baseDelayMs: 1000
maxDelayMs: 60000
jitter: true
```
### Adapters
@ -517,24 +475,6 @@ handles failures.
| `workUnits.maxConcurrency` | `int > 0` | `1` | How many work units run in parallel. |
| `workUnits.failureMode` | `abort` \| `continue` | `continue` | `abort` stops the whole ingest run on the first failure; `continue` records it and keeps going. |
### Rate limits
`rateLimit` controls provider-neutral pacing for LLM calls during ingest. When a
provider reports a subscription window, retry-after delay, or HTTP 429,
**ktx** pauses new work-unit model calls, shows a transient wait in the CLI,
and reduces work-unit concurrency while the provider is under pressure.
| Field | Type | Default | Purpose |
|-------|------|---------|---------|
| `rateLimit.enabled` | `boolean` | `true` | Master switch for ingest LLM rate-limit pacing and visible waits. |
| `rateLimit.throttleThreshold` | `number between 0 and 1` | `0.8` | Fraction of a known provider window at which **ktx** starts reducing concurrency. |
| `rateLimit.minConcurrencyUnderPressure` | `int > 0` | `1` | Effective work-unit concurrency while a provider is under rate-limit pressure. |
| `rateLimit.maxWaitMs` | `int > 0` | unset | Caps how long a single provider-reset wait can last. This bounds each wait, not the whole run: after a capped wait elapses **ktx** retries and may pause again. Omit to wait until the provider's reset time. |
| `rateLimit.retry.maxAttempts` | `int > 0` | `6` | Maximum attempts for a single rate-limited LLM call before the failure surfaces (counts the first try). Also bounds how far opaque backoff grows for responses without a reset time or retry-after value. |
| `rateLimit.retry.baseDelayMs` | `int > 0` | `1000` | Initial opaque retry delay in milliseconds. |
| `rateLimit.retry.maxDelayMs` | `int > 0` | `60000` | Maximum opaque retry delay in milliseconds. |
| `rateLimit.retry.jitter` | `boolean` | `true` | Add jitter to opaque retry delays. |
## `scan`
`scan` configures how schema-level inputs become structured context:
@ -652,11 +592,6 @@ llm:
backend: claude-code
models:
default: sonnet
triage: haiku
candidateExtraction: sonnet
curator: opus
reconcile: opus
repair: haiku
ingest:
adapters:
- live-database

View file

@ -4,7 +4,6 @@ description: ktx is an open-source, self-improving context layer for data agents
---
import { ProductMechanics } from "@/components/product-mechanics";
import { ProductRuntime } from "@/components/product-runtime";
<div className="not-prose mb-10">
<div>
@ -24,7 +23,7 @@ import { ProductRuntime } from "@/components/product-runtime";
>
Make analytics context usable by agents
</h1>
<p className="mt-4 max-w-full text-lg text-fd-muted-foreground" style={{ lineHeight: '1.7' }}>
<p className="mt-4 max-w-2xl text-lg text-fd-muted-foreground" style={{ lineHeight: '1.7' }}>
{'ktx is an open-source context layer for data agents. It turns warehouse metadata, BI tool definitions, query history, docs, and approved metric definitions into reviewable files agents can search and execute.'}
</p>
</div>
@ -60,8 +59,6 @@ serves that context to agents at runtime.
<ProductMechanics />
<ProductRuntime />
## Use it for
Use **ktx** when agents need more than raw database access. Agents can search wiki
@ -95,8 +92,8 @@ best first step for users; contributor setup lives in the community docs.
<Card title="CLI Reference" href="/docs/cli-reference/ktx">
Complete flag and subcommand reference for every **ktx** command.
</Card>
<Card title="AI Resources" href="/docs/community/ai-resources">
Machine-readable docs, a task router, and copy-paste agent prompts.
<Card title="Agent Quickstart" href="/docs/ai-resources/agent-quickstart">
Machine-readable docs and agent-facing setup notes.
</Card>
</Cards>
@ -105,6 +102,6 @@ best first step for users; contributor setup lives in the community docs.
Have questions, want to share what you're building, or chat with maintainers?
Join the [**ktx** Slack community](https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ).
For bug reports and feature requests, open a
[GitHub issue](https://github.com/Kaelio/ktx/issues). See
[GitHub issue](https://github.com/Kaelio/ktx-ai-data-agents-context/issues). See
[Community & Support](/docs/community/support) for the full guide on where to
ask what.

View file

@ -9,8 +9,8 @@ This guide takes a local analytics project from empty to agent-ready. You'll
install the CLI, run one guided setup command, and hand the context to a
coding assistant.
If you're a coding assistant choosing a docs route, start with
[AI Resources](/docs/community/ai-resources) instead.
If you're a coding assistant choosing a docs route, start with the
[Agent Quickstart](/docs/ai-resources/agent-quickstart) instead.
<div
className="not-prose my-8 overflow-hidden rounded-2xl border"
@ -171,14 +171,14 @@ If you're a coding assistant choosing a docs route, start with
</span>
<CopyButton
text={[
'Run npx skills add Kaelio/ktx --skill ktx and use the ktx skill',
'Run npx skills add Kaelio/ktx-ai-data-agents-context --skill ktx and use the ktx skill',
'to install and configure ktx',
].join(' ')}
className="-my-1"
/>
</div>
<div className="p-3 font-mono text-[13.5px] leading-6 text-fd-foreground">
Run {'`npx skills add Kaelio/ktx --skill ktx`'} and use the ktx skill to install and configure ktx
Run {'`npx skills add Kaelio/ktx-ai-data-agents-context --skill ktx`'} and use the ktx skill to install and configure ktx
</div>
</div>
</div>
@ -191,12 +191,6 @@ Install the published package globally:
npm install -g @kaelio/ktx
```
To upgrade an existing install later, re-run with the `@latest` tag:
```bash
npm install -g @kaelio/ktx@latest
```
**ktx** is open source. If you'd like to hack on it or run from a local checkout,
the source lives at [github.com/kaelio/ktx](https://github.com/kaelio/ktx) -
see [Contributing](/docs/community/contributing) to get set up.
@ -221,8 +215,8 @@ The wizard walks you through everything **ktx** needs in one pass:
SQLite, PostgreSQL, MySQL, SQL Server, BigQuery, and Snowflake.
5. **Context sources** - optionally adds dbt, MetricFlow, LookML, Looker,
Metabase, or Notion. You can skip and add them later.
6. **Build** - offers to run the first ingest so semantic sources and wiki
pages are ready for agents. If you skip it, build later with `ktx ingest`.
6. **Build** - runs the first ingest so semantic sources and wiki pages
are ready for agents.
7. **Agent integration** - installs project-local rules for Claude Code,
Codex, Cursor, OpenCode, or universal `.agents`.
@ -253,18 +247,6 @@ progress under `.ktx/setup/` and resumes from the remaining work.
> resuming setup, connecting an agent, checking status, or exploring a
> pre-built demo project.
When the wizard finishes, it states where you stand and the single next action:
- **Context built** - **ktx** confirms it is ready for agents and points you to
open your coding agent and ask a data question.
- **Build skipped** - **ktx** tells you setup is complete and that the only step
left is to build context with `ktx ingest`.
Re-running `ktx setup` on an already-configured project goes straight to the
remaining step - building context or connecting an agent - instead of
re-asking every question. Once everything is ready, it confirms you are set
rather than reopening the configuration menu.
## Verify
When setup finishes, check readiness:
@ -286,9 +268,6 @@ Agent integration ready: yes (codex:project)
For a structured check inside scripts, use `ktx status --json`.
If you skipped the build, `ktx context built` shows `no`. Build it with
`ktx ingest` - there is no need to re-run `ktx setup`.
When setup finishes building context, its final context check looks like:
```text
@ -301,26 +280,6 @@ Context sources:
dbt_main: memory update complete
```
Before the build starts, **ktx** runs a live test for every connection the
build depends on. A context build can take several minutes, so if any required
connection is unreachable or misconfigured the build is blocked up front and
**ktx** names the failing connection by id and connector type:
```text
KTX cannot build context: a required connection failed its live test.
Failed connections:
warehouse (postgres)
Each connection must be reachable before KTX builds context.
Run `ktx connection test <id>` to see the error, fix the connection, then retry.
```
Run `ktx connection test <connection-id>` to see the underlying error, fix the
connection, then continue. In interactive setup you can retry without
restarting; with `--no-input` the build exits non-zero and names the failing
connection so scripts can stop early.
## Connect a coding agent
The setup wizard installs project-local agent rules in the last step. To
@ -380,8 +339,7 @@ surface.
| `ktx: command not found` | Reinstall `@kaelio/ktx` and open a new shell |
| Setup resumes the wrong project | Pass `--project-dir <path>` |
| LLM or embeddings health check fails | Rerun setup and pick a different credential, model, or backend |
| Database test fails | Use the setup recovery menu to retry or re-enter details; if it still fails, verify the same connection with the database's native client |
| Context build blocked: a connection failed its live test | Run `ktx connection test <connection-id>` to see the error, fix the connection, then retry the build |
| Database test fails | Verify the same connection with the database's native client, then rerun setup |
| Agent integration is incomplete | Run `ktx setup --agents --target <target>` |
## Next steps

View file

@ -39,28 +39,13 @@ ktx ingest --all
Enriched ingest needs a configured model and embeddings. Run `ktx setup` first;
connections without that configuration fail before any work starts.
Local-auth backends keep provider credentials out of `ktx.yaml`:
```bash
ktx setup --llm-backend claude-code --no-input
ktx setup --llm-backend codex --no-input
```
With `claude-code`, **ktx** agent loops can invoke only the **ktx** MCP tools
for the current run. With `codex`, **ktx** restricts the temporary runtime MCP
server to the current run's tool set, disables Codex web search, requests a
read-only sandbox, and sets `approval_policy=never`. The public Codex SDK and
CLI surface may still load user Codex config and built-in command execution or
read-only file capabilities, so use `claude-code` for stricter runtime tool
isolation.
With `claude-code`, **ktx** agent loops can invoke only the **ktx** MCP tools for the
current run.
## Query history
PostgreSQL, BigQuery, and Snowflake can add query-history context: common joins,
filters, redaction rules, high-usage templates, and service-account exclusions.
When query history is enabled during setup, **ktx** reviews observed in-scope
roles and can write exact `filters.serviceAccounts` patterns for operational
traffic such as loader or refresh roles.
filters, service-account patterns, redaction rules, and high-usage templates.
Enable it during setup, store it under `connections.<id>.context.queryHistory`,
or request it for one run:

View file

@ -16,7 +16,6 @@ Set `llm.provider.backend` to one of these values:
- `gateway`: Use AI Gateway-compatible Anthropic model ids.
- `claude-code`: Use your local Claude Code session through the Claude Agent
SDK. **ktx** strips provider-routing environment variables from child processes.
- `codex`: Use your local Codex authentication through the Codex SDK.
## Claude Code
@ -30,65 +29,24 @@ llm:
default: sonnet
triage: haiku
candidateExtraction: sonnet
curator: opus
reconcile: opus
repair: haiku
curator: sonnet
reconcile: sonnet
repair: sonnet
```
During setup, choose the backend interactively or pass it in automation:
During setup, choose the backend interactively or pass the model in automation:
```bash
ktx setup --llm-backend claude-code --no-input
ktx setup --llm-backend claude-code --llm-model opus --no-input
```
Setup writes `sonnet`, `haiku`, and `opus` aliases into `llm.models`. You can
edit any role to another alias or a full Claude model ID after setup.
For Claude Code, `sonnet`, `opus`, and `haiku` map to **ktx** defaults. Full Claude
model IDs are also accepted.
`claude-code` exposes only **ktx** MCP tools for the current agent loop. SDK init
metadata may still list host slash commands, skills, and subagents; **ktx** does not
grant execution access to them.
## Codex backend
Use `codex` when you want **ktx** to run LLM-backed workflows through your
local Codex authentication instead of a direct provider API key.
```yaml
llm:
provider:
backend: codex
models:
default: gpt-5.5
triage: gpt-5.5
candidateExtraction: gpt-5.5
curator: gpt-5.5
reconcile: gpt-5.5
repair: gpt-5.5
```
Configure it non-interactively:
```bash
ktx setup --llm-backend codex --no-input
```
This is separate from Codex agent-client setup. `ktx setup --agents --target
codex` installs instructions and MCP access for an end-user Codex session.
`ktx setup --llm-backend codex` makes **ktx** itself execute ingest, scan
enrichment, memory, and other LLM-backed work through Codex.
During runtime loops, **ktx** starts a temporary loopback MCP server for the
current run, exposes only the tools passed to that run, asks Codex to use a
read-only sandbox, sets `approval_policy=never`, auto-approves only those
run-scoped MCP tools, and disables Codex web search.
Codex backend isolation is currently limited by the public Codex SDK and CLI
surface. Codex may still load user Codex config and built-in command execution
or read-only file capabilities. Use `llm.provider.backend: claude-code` when
you need stricter Claude-Code-style runtime tool isolation, or remove host
Codex MCP and tool config before running untrusted prompts through the `codex`
backend.
## Prompt caching
`llm.promptCaching` has partial parity on `claude-code`. Status and doctor warn

View file

@ -8,6 +8,7 @@
"integrations",
"configuration",
"cli-reference",
"ai-resources",
"community"
]
}

View file

@ -52,9 +52,11 @@ ktx provides semantic-layer files, warehouse scans, wiki pages, provenance, and
## Agent Entry Points
- Installable setup skill: run \`npx skills add Kaelio/ktx --skill ktx\` from
- Installable setup skill: run \`npx skills add Kaelio/ktx-ai-data-agents-context --skill ktx\` from
the project you want to configure.
${link("/docs/community/ai-resources", "AI Resources", "How coding agents read, cite, and act on the ktx docs")}
${link("/docs/ai-resources/agent-quickstart", "Agent Quickstart", "Task-first route for coding assistants using ktx")}
${link("/docs/ai-resources/markdown-access", "Markdown Access", "Fetch ktx docs as llms.txt, llms-full.txt, or per-page Markdown")}
${link("/docs/ai-resources/agent-instructions", "Agent Instructions", "Suggested instructions for coding assistants that need to read and cite ktx docs")}
## Start Here
@ -65,7 +67,7 @@ ${link("/docs/guides/writing-context", "Writing Context", "Write semantic source
## Machine-Readable Documentation
- [Full documentation](${absoluteUrl("/llms-full.txt")}): All docs pages in one plain-text markdown response
- [AI Resources guide](${absoluteUrl("/docs/community/ai-resources.md")}): How agents fetch llms.txt, llms-full.txt, and per-page Markdown
- [Markdown access guide](${absoluteUrl("/docs/ai-resources/markdown-access.md")}): How to fetch llms.txt, llms-full.txt, and per-page Markdown
- [Quickstart markdown](${absoluteUrl("/docs/getting-started/quickstart.md")}): Human setup walkthrough
- [Semantic-layer CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-sl.md")}): Semantic-layer commands and JSON output
- [Wiki CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-wiki.md")}): Wiki page commands and JSON output
@ -145,8 +147,8 @@ function absoluteUrl(path: string) {
function formatCategoryName(category: string) {
const labels: Record<string, string> = {
"ai-resources": "AI Resources",
"cli-reference": "CLI Reference",
community: "Community & Resources",
};
if (labels[category]) {

View file

@ -30,36 +30,7 @@ const config = {
};
},
async redirects() {
// Alias-host canonicalization MUST come before the generic root/docs
// redirects below. Those generic rules have no host guard, so if they ran
// first they would inject a "/ktx" basePath into the path on the alias
// hosts, which the alias catch-alls would then prepend a second time —
// producing https://docs.kaelio.com/ktx/ktx/docs/... Redirects also run
// before beforeFiles rewrites, so the ktx.sh catch-all must exclude
// /stars* to let the stars dashboard rewrite proxy through.
return [
{
source: "/slack",
has: [{ type: "host", value: "ktx.sh" }],
destination:
"https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ",
permanent: false,
basePath: false,
},
{
source: "/:path*",
has: [{ type: "host", value: "docs.ktx.sh" }],
destination: "https://docs.kaelio.com/ktx/:path*",
permanent: true,
basePath: false,
},
{
source: "/:path((?!stars(?:/|$)).*)",
has: [{ type: "host", value: "ktx.sh" }],
destination: "https://docs.kaelio.com/ktx/:path",
permanent: true,
basePath: false,
},
{
source: "/",
destination: "/ktx/docs/getting-started/introduction",
@ -73,30 +44,26 @@ const config = {
basePath: false,
},
{
// AI Resources collapsed from four pages to one and now lives under the
// Community & Resources section. Redirect the old top-level URL and the
// retired per-page slugs to the new home. Redirects run before the .md
// rewrite, so the Markdown variants must be matched first and keep their
// .md suffix; otherwise a cached Markdown URL would 308 to the HTML page
// and break the agent Markdown contract.
source: "/docs/ai-resources.md",
destination: "/docs/community/ai-resources.md",
source: "/:path*",
has: [{ type: "host", value: "docs.ktx.sh" }],
destination: "https://docs.kaelio.com/ktx/:path*",
permanent: true,
basePath: false,
},
{
source: "/docs/ai-resources/:slug([^/]+\\.md)",
destination: "/docs/community/ai-resources.md",
permanent: true,
source: "/slack",
has: [{ type: "host", value: "ktx.sh" }],
destination:
"https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ",
permanent: false,
basePath: false,
},
{
source: "/docs/ai-resources",
destination: "/docs/community/ai-resources",
permanent: true,
},
{
source: "/docs/ai-resources/:slug",
destination: "/docs/community/ai-resources",
source: "/:path((?!stars(?:/|$)).*)",
has: [{ type: "host", value: "ktx.sh" }],
destination: "https://docs.kaelio.com/ktx/:path",
permanent: true,
basePath: false,
},
];
},

View file

@ -2,8 +2,6 @@ import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { once } from "node:events";
import { readFile, writeFile } from "node:fs/promises";
import http from "node:http";
import https from "node:https";
import { dirname, join } from "node:path";
import { createServer } from "node:net";
import { after, before, test } from "node:test";
@ -102,37 +100,6 @@ after(async () => {
}
});
// Node's fetch (undici) overwrites the Host header with the connection host,
// so the alias-host redirect rules never match. The low-level http(s) client
// sends Host verbatim, which is what the alias canonicalization keys off of.
function requestWithHost(hostHeader, path) {
const target = new URL(docsSiteUrl);
const client = target.protocol === "https:" ? https : http;
const port =
target.port || (target.protocol === "https:" ? "443" : "80");
return new Promise((resolve, reject) => {
const request = client.request(
{
hostname: target.hostname,
port,
path,
method: "GET",
headers: { Host: hostHeader },
},
(response) => {
response.resume();
resolve({
status: response.statusCode,
location: response.headers.location,
});
},
);
request.on("error", reject);
request.end();
});
}
test("/ktx/docs redirects to the docs introduction", async () => {
const response = await fetch(`${docsSiteUrl}${docsBasePath}/docs`, {
redirect: "manual",
@ -145,53 +112,6 @@ test("/ktx/docs redirects to the docs introduction", async () => {
);
});
test("retired AI Resources URLs redirect to the page under Community", async () => {
// The former top-level URL.
const bare = await fetch(
`${docsSiteUrl}${docsBasePath}/docs/ai-resources`,
{ redirect: "manual" },
);
assert.equal(bare.status, 308);
assert.equal(
bare.headers.get("location"),
`${docsBasePath}/docs/community/ai-resources`,
);
// A retired per-page slug.
const slug = await fetch(
`${docsSiteUrl}${docsBasePath}/docs/ai-resources/agent-quickstart`,
{ redirect: "manual" },
);
assert.equal(slug.status, 308);
assert.equal(
slug.headers.get("location"),
`${docsBasePath}/docs/community/ai-resources`,
);
// A retired per-page Markdown URL must stay Markdown: it has to redirect to
// the new .md route, not fall through to the HTML page.
const markdown = await fetch(
`${docsSiteUrl}${docsBasePath}/docs/ai-resources/agent-quickstart.md`,
{ redirect: "manual" },
);
assert.equal(markdown.status, 308);
assert.equal(
markdown.headers.get("location"),
`${docsBasePath}/docs/community/ai-resources.md`,
);
// Following that redirect end to end must land on Markdown, not HTML.
const followed = await fetch(
`${docsSiteUrl}${docsBasePath}/docs/ai-resources/agent-quickstart.md`,
);
assert.equal(followed.status, 200);
assert.match(followed.headers.get("content-type") ?? "", /text\/markdown/);
});
test("/ redirects into the /ktx docs site", async () => {
const response = await fetch(`${docsSiteUrl}/`, {
redirect: "manual",
@ -221,51 +141,3 @@ test("/ktx/api/search returns docs search results", async () => {
"search should return at least one docs result",
);
});
test("ktx.sh canonicalizes to a single /ktx basePath on the docs host", async () => {
const root = await requestWithHost("ktx.sh", "/");
assert.equal(root.status, 308);
assert.equal(root.location, "https://docs.kaelio.com/ktx/");
assert.ok(
!root.location.includes("/ktx/ktx"),
"the basePath must not be doubled",
);
const page = await requestWithHost(
"ktx.sh",
"/docs/getting-started/quickstart",
);
assert.equal(page.status, 308);
assert.equal(
page.location,
"https://docs.kaelio.com/ktx/docs/getting-started/quickstart",
);
});
test("docs.ktx.sh canonicalizes to a single /ktx basePath on the docs host", async () => {
const root = await requestWithHost("docs.ktx.sh", "/");
assert.equal(root.status, 308);
assert.equal(root.location, "https://docs.kaelio.com/ktx");
assert.ok(
!root.location.includes("/ktx/ktx"),
"the basePath must not be doubled",
);
const page = await requestWithHost("docs.ktx.sh", "/llms.txt");
assert.equal(page.status, 308);
assert.equal(page.location, "https://docs.kaelio.com/ktx/llms.txt");
});
test("ktx.sh keeps the /slack and /stars exceptions", async () => {
const slack = await requestWithHost("ktx.sh", "/slack");
assert.equal(slack.status, 307);
assert.match(slack.location, /^https:\/\/join\.slack\.com\//);
// /stars is proxied by a beforeFiles rewrite, so the apex catch-all must not
// canonicalize it to the docs host.
const stars = await requestWithHost("ktx.sh", "/stars");
assert.ok(
!(stars.location ?? "").startsWith("https://docs.kaelio.com"),
"the stars dashboard must not be redirected to the docs host",
);
});

View file

@ -85,7 +85,7 @@ test("product mechanics component explains ingestion outputs", async () => {
"compile into SQL",
'"use client"',
"@xyflow/react",
"<FlowCanvas",
"<ReactFlow",
"getSmoothStepPath",
"animateMotion",
"mechanics-particle",
@ -97,21 +97,21 @@ test("product mechanics component explains ingestion outputs", async () => {
);
}
// The ReactFlow canvas config lives in the shared FlowCanvas wrapper, which
// product-mechanics renders. Assert the static read-only behavior there.
const flowCanvas = await readDocsFile("components/flow-canvas.tsx");
for (const guard of [
assert.match(
component,
/nodesDraggable=\{false\}/,
/nodesConnectable=\{false\}/,
"ReactFlow canvas should disable node dragging",
);
assert.match(
component,
/panOnDrag=\{false\}/,
"ReactFlow canvas should disable panning",
);
assert.match(
component,
/zoomOnScroll=\{false\}/,
/elementsSelectable=\{false\}/,
]) {
assert.match(
flowCanvas,
guard,
`shared FlowCanvas should enforce static read-only behavior: ${guard}`,
);
}
"ReactFlow canvas should disable scroll zoom",
);
assert.doesNotMatch(component, /raw-sources/);
assert.doesNotMatch(component, /\.ktx/);

View file

@ -1,74 +0,0 @@
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { test } from "node:test";
import { fileURLToPath } from "node:url";
const docsSiteDir = join(dirname(fileURLToPath(import.meta.url)), "..");
async function readDocsFile(path) {
return readFile(join(docsSiteDir, path), "utf8");
}
test("docs introduction renders the serving phase after ingestion", async () => {
const introduction = await readDocsFile(
"content/docs/getting-started/introduction.mdx",
);
assert.match(
introduction,
/import\s+\{\s*ProductRuntime\s*\}\s+from\s+"@\/components\/product-runtime";/,
);
assert.match(introduction, /<ProductRuntime\s*\/>/);
const mechanicsIndex = introduction.indexOf("<ProductMechanics />");
const runtimeIndex = introduction.indexOf("<ProductRuntime />");
const useCaseIndex = introduction.indexOf("## Use it for");
assert.ok(
runtimeIndex > mechanicsIndex,
"serving diagram should appear after the ingestion diagram",
);
assert.ok(
runtimeIndex < useCaseIndex,
"serving diagram should appear before use-case sections",
);
});
test("product runtime component explains the serving cycle", async () => {
const component = await readDocsFile("components/product-runtime.tsx");
for (const expectedText of [
"How serving works",
"Serving flow",
"From an agent request to a governed answer",
"Your agent",
"Claude Code",
"Cursor",
"Codex",
"Search wiki + semantic layer",
"Return approved metrics",
"Compile metrics → SQL",
"Context layer",
"Database",
"search + read",
"read-only",
"wiki/*.md",
"semantic-layer/*.yaml",
'"use client"',
"@xyflow/react",
"FlowCanvas",
"getSmoothStepPath",
"animateMotion",
"runtime-particle",
"buildCyclePath",
]) {
assert.ok(
component.includes(expectedText),
`component should include: ${expectedText}`,
);
}
assert.doesNotMatch(component, /raw-sources/);
assert.doesNotMatch(component, /<img/);
});

View file

@ -26,7 +26,7 @@ The workflow rejects releases from any branch other than `main`.
Before you publish, confirm these requirements:
- npm Trusted Publishing is configured for `@kaelio/ktx`.
- The trusted publisher points at the `Kaelio/ktx` repository and the
- The trusted publisher points at the `Kaelio/ktx-ai-data-agents-context` repository and the
`.github/workflows/release.yml` workflow.
- The workflow keeps `id-token: write` permission so npm can verify the
GitHub Actions run through OpenID Connect.
@ -35,6 +35,15 @@ Before you publish, confirm these requirements:
- The repository has a stable baseline tag when you need semantic-release to
publish the first stable version as `0.1.0`.
If you rename the GitHub repository, the semantic-release run adapts on its
own: `scripts/semantic-release-config.cjs` derives `repositoryUrl` from the
runner's `GITHUB_REPOSITORY`, so `@semantic-release/github` always matches the
current clone URL. The one thing that does **not** auto-update is the npm
Trusted Publishing config — re-point it at the new repository name (plus
`release.yml`) on npm, or `npm publish --provenance` will fail OIDC
verification. The `repository` field in `package.json` is npm-display metadata
only and can stay whatever public name you prefer.
semantic-release doesn't support choosing an arbitrary first `0.x` stable
release. If KTX has no stable tag yet and you need the first stable release to
be `0.1.0`, create and push the baseline tag once before running the live

View file

@ -37,9 +37,6 @@
"@semantic-release/release-notes-generator",
"conventional-changelog-conventionalcommits"
],
"ignore": [
".context/**"
],
"ignoreBinaries": [
"uv",
"lsof"

View file

@ -1,6 +1,6 @@
{
"name": "ktx-workspace",
"version": "0.11.0",
"version": "0.8.0",
"description": "Workspace root for ktx packages",
"private": true,
"type": "module",
@ -32,7 +32,6 @@
"setup:dev": "node scripts/setup-dev.mjs",
"release:published-smoke": "node scripts/published-package-smoke.mjs --require-config",
"release:local-embeddings-smoke": "node scripts/local-embeddings-runtime-smoke.mjs --require-opt-in",
"release:codex-backend-smoke": "node scripts/codex-backend-live-smoke.mjs",
"release:readiness": "node scripts/release-readiness.mjs",
"release:update-version": "node scripts/update-public-release-version.mjs",
"relationships:acquire-public-fixtures": "node scripts/acquire-public-benchmark-fixtures.mjs",
@ -69,13 +68,18 @@
"typescript": "^6.0.3",
"yaml": "^2.9.0"
},
"pnpm": {
"onlyBuiltDependencies": [
"better-sqlite3"
]
},
"license": "Apache-2.0",
"repository": {
"type": "git",
"url": "git+https://github.com/Kaelio/ktx.git"
"url": "git+https://github.com/Kaelio/ktx-ai-data-agents-context.git"
},
"bugs": {
"url": "https://github.com/Kaelio/ktx/issues"
"url": "https://github.com/Kaelio/ktx-ai-data-agents-context/issues"
},
"homepage": "https://github.com/Kaelio/ktx#readme"
"homepage": "https://github.com/Kaelio/ktx-ai-data-agents-context#readme"
}

View file

@ -1,11 +1,7 @@
{
"name": "@kaelio/ktx",
"version": "0.11.0",
"version": "0.8.0",
"description": "Standalone ktx context layer for data agents",
"author": {
"name": "Kaelio",
"url": "https://www.kaelio.com"
},
"type": "module",
"engines": {
"node": ">=22.0.0"
@ -51,7 +47,6 @@
"@ai-sdk/devtools": "0.0.18",
"@ai-sdk/google-vertex": "^4.0.134",
"@anthropic-ai/claude-agent-sdk": "0.3.146",
"@clack/core": "1.3.1",
"@clack/prompts": "1.4.0",
"@clickhouse/client": "^1.18.5",
"@commander-js/extra-typings": "14.0.0",
@ -61,7 +56,6 @@
"@looker/sdk-rtl": "^21.6.5",
"@modelcontextprotocol/sdk": "^1.29.0",
"@notionhq/client": "^5.22.0",
"@openai/codex-sdk": "^0.133.0",
"ai": "^6.0.188",
"better-sqlite3": "^12.10.0",
"commander": "14.0.3",
@ -77,7 +71,6 @@
"pg": "^8.21.0",
"posthog-node": "^5.34.9",
"react": "^19.2.6",
"semver": "^7.8.1",
"simple-git": "3.36.0",
"snowflake-sdk": "^2.4.2",
"yaml": "^2.9.0",
@ -91,7 +84,6 @@
"@types/node": "^25.9.1",
"@types/pg": "^8.20.0",
"@types/react": "^19.2.15",
"@types/semver": "^7.7.1",
"@vitest/coverage-v8": "^4.1.7",
"ajv": "8.20.0",
"ink-testing-library": "^4.0.0",
@ -101,11 +93,11 @@
"license": "Apache-2.0",
"repository": {
"type": "git",
"url": "https://github.com/Kaelio/ktx",
"url": "https://github.com/Kaelio/ktx-ai-data-agents-context",
"directory": "packages/cli"
},
"bugs": {
"url": "https://github.com/Kaelio/ktx/issues"
"url": "https://github.com/Kaelio/ktx-ai-data-agents-context/issues"
},
"homepage": "https://github.com/Kaelio/ktx#readme"
"homepage": "https://github.com/Kaelio/ktx-ai-data-agents-context#readme"
}

View file

@ -3,30 +3,6 @@ import type { KtxCliIo } from './cli-runtime.js';
const ESC = String.fromCharCode(0x1b);
export interface CliStyleEnv {
NO_COLOR?: string;
TERM?: string;
}
function ansiEnabled(env: CliStyleEnv = process.env): boolean {
return !env.NO_COLOR && env.TERM !== 'dumb';
}
function ansiColor(text: string, open: number, close: number, env?: CliStyleEnv): string {
if (!ansiEnabled(env)) {
return text;
}
return `${ESC}[${open}m${text}${ESC}[${close}m`;
}
export function dim(text: string, env?: CliStyleEnv): string {
return ansiColor(text, 2, 22, env);
}
export function cyan(text: string, env?: CliStyleEnv): string {
return ansiColor(text, 36, 39, env);
}
export interface RailBufferedSource {
stdoutText(): string;
stderrText(): string;
@ -85,11 +61,11 @@ export function createClackSpinner(): KtxCliSpinner {
}
function magenta(text: string): string {
return ansiColor(text, 35, 39);
return `${ESC}[35m${text}${ESC}[39m`;
}
function red(text: string): string {
return ansiColor(text, 31, 39);
return `${ESC}[31m${text}${ESC}[39m`;
}
export function createStaticCliSpinner(io: KtxCliSpinnerIo): KtxCliSpinner {

View file

@ -2,7 +2,6 @@ import { existsSync } from 'node:fs';
import { join } from 'node:path';
import { Command, type CommandUnknownOpts, InvalidArgumentError } from '@commander-js/extra-typings';
import type { KtxCliDeps, KtxCliIo, KtxCliPackageInfo } from './cli-runtime.js';
import { SLACK_HELP_FOOTER, writeErrorCommunityHint } from './community-cta.js';
import { registerCompletionCommands } from './commands/completion-commands.js';
import { registerConnectionCommands } from './commands/connection-commands.js';
import { registerIngestCommands } from './commands/ingest-commands.js';
@ -17,7 +16,6 @@ import { renderMissingProjectMessage } from './doctor.js';
import { findNearestKtxProjectDir, resolveKtxProjectDir } from './project-resolver.js';
import { profileMark, profileSpan } from './startup-profile.js';
import type { CommandOutcome } from './telemetry/index.js';
import { prepareUpdateCheckNotice, type PrepareUpdateCheckNoticeOptions } from './update-check/update-check.js';
profileMark('module:cli-program');
@ -41,8 +39,6 @@ interface KtxCommanderProgramOptions {
runInit: (args: { projectDir: string; force: boolean }, io: KtxCliIo) => Promise<number>;
}
type KtxCliUpdateCheckOptions = Pick<PrepareUpdateCheckNoticeOptions, 'env' | 'fetchDistTags' | 'homeDir' | 'now'>;
export interface BuildKtxProgramOptions {
io: KtxCliIo;
deps: KtxCliDeps;
@ -51,7 +47,6 @@ export interface BuildKtxProgramOptions {
setExitCode?: (code: number) => void;
argv?: string[];
setTelemetryModule?: (telemetry: typeof import('./telemetry/index.js')) => void;
updateCheck?: KtxCliUpdateCheckOptions;
}
type CommanderExitLike = { exitCode: number; code: string; message: string };
@ -259,7 +254,6 @@ function createBaseProgram(info: KtxCliPackageInfo, io: KtxCliIo): Command {
.helpOption('-h, --help', 'Show this help text')
.configureHelp({ showGlobalOptions: true })
.showHelpAfterError()
.addHelpText('after', `\n${SLACK_HELP_FOOTER}`)
.exitOverride()
.configureOutput({
writeOut: (chunk) => io.stdout.write(chunk),
@ -437,29 +431,16 @@ export function collectCommandFlagsPresent(command: CommandUnknownOpts): Record<
export function buildKtxProgram(options: BuildKtxProgramOptions): Command {
const program = createBaseProgram(options.packageInfo, options.io);
let pendingUpdateNotice: string | null = null;
program.hook('preAction', async (_thisCommand, actionCommand) => {
// The hidden completion command must stay silent and side-effect free: skip
// the telemetry notice, command span, project checks, and update checks entirely.
// the telemetry notice, command span, and project checks entirely.
if (commandPath(actionCommand as CommandPathNode).includes('__complete')) {
return;
}
const commandNode = actionCommand as CommandPathNode;
const updateCheck = await prepareUpdateCheckNotice({
io: options.io,
env: options.updateCheck?.env,
fetchDistTags: options.updateCheck?.fetchDistTags,
homeDir: options.updateCheck?.homeDir,
installedVersion: options.packageInfo.version,
now: options.updateCheck?.now,
commandOptions: commandOptions(commandNode),
});
pendingUpdateNotice = updateCheck.notice;
const telemetry = await import('./telemetry/index.js');
options.setTelemetryModule?.(telemetry);
await telemetry.showTelemetryNoticeIfNeeded(options.io, options.packageInfo);
const commandNode = actionCommand as CommandPathNode;
const path = commandPath(commandNode);
const projectDir = resolveCommandProjectDir(commandNode);
const hasProject = ktxYamlExists(projectDir);
@ -476,13 +457,6 @@ export function buildKtxProgram(options: BuildKtxProgramOptions): Command {
ensureProjectAvailable(options.io, commandNode);
});
program.hook('postAction', () => {
if (pendingUpdateNotice) {
options.io.stderr.write(pendingUpdateNotice);
pendingUpdateNotice = null;
}
});
const context: KtxCliCommandContext = {
io: options.io,
deps: options.deps,
@ -555,15 +529,7 @@ export async function runCommanderKtxCli(
try {
return await runBareInteractiveCommand(program, io, context);
} catch (error) {
const telemetry = await import('./telemetry/index.js');
await telemetry.reportException({
error,
context: { source: 'bare-interactive', handled: true, fatal: false },
packageInfo: info,
io,
});
io.stderr.write(`${formatCliError(error)}\n`);
writeErrorCommunityHint(io, 'error');
return 1;
}
}
@ -588,7 +554,6 @@ export async function runCommanderKtxCli(
exitCode = error.exitCode === 0 ? 0 : 1;
} else {
io.stderr.write(`${formatCliError(error)}\n`);
writeErrorCommunityHint(io, 'error');
exitCode = 1;
}
} finally {
@ -598,23 +563,6 @@ export async function runCommanderKtxCli(
outcome: commandOutcomeForParseResult(parseError, exitCode),
error: parseError,
});
if (
parseError &&
!isCommanderExit(parseError) &&
!isKtxProjectMissingAbortError(parseError)
) {
await telemetryModule.reportException({
error: parseError,
context: {
source: completed?.commandPath.join(' ') ?? 'commander parseAsync',
handled: true,
fatal: false,
},
projectDir: completed?.projectGroupAttached ? completed.projectDir : undefined,
packageInfo: info,
io,
});
}
await telemetryModule.emitCompletedCommand({ completed, packageInfo: info, io });
await telemetryModule.shutdownTelemetryEmitter();
}

View file

@ -12,7 +12,6 @@ import type { KtxSqlArgs } from './sql.js';
import { profileMark, profileSpan } from './startup-profile.js';
import type { KtxTextIngestArgs } from './text-ingest.js';
import { assertCliVersion } from './release-version.js';
import { writeErrorCommunityHint } from './community-cta.js';
profileMark('module:cli-runtime');
@ -90,94 +89,6 @@ export async function runInitForCommander(
return await runInit(args, io);
}
function signalExitCode(signal: NodeJS.Signals): number {
// 128 + signal number: SIGINT (2) -> 130, SIGTERM (15) -> 143.
return signal === 'SIGTERM' ? 143 : 130;
}
/**
* Flush telemetry on interrupt for the real CLI process. `capture()` is
* fire-and-forget and the only flush guarantee lives in a `finally` a signal
* skips, so Ctrl-C / `kill` of a long-running command (ingest, `mcp stdio`)
* would otherwise drop its `command` event and queued events. Installed only
* when driving the actual process; programmatic/test callers pass their own
* `io` and never reach here. Returns a disposer that removes the listeners.
*/
function installTelemetrySignalFlush(io: KtxCliIo, info: KtxCliPackageInfo): () => void {
let handling = false;
const handle = (signal: NodeJS.Signals): void => {
if (handling) {
process.exit(signalExitCode(signal));
}
handling = true;
void (async () => {
try {
const { emitAbortedCommandAndShutdown } = await import('./telemetry/index.js');
await emitAbortedCommandAndShutdown({ packageInfo: info, io });
} catch {
// Best-effort: never let a telemetry hiccup block the interrupt exit.
}
process.exit(signalExitCode(signal));
})();
};
const onSigint = (): void => handle('SIGINT');
const onSigterm = (): void => handle('SIGTERM');
process.on('SIGINT', onSigint);
process.on('SIGTERM', onSigterm);
return () => {
process.off('SIGINT', onSigint);
process.off('SIGTERM', onSigterm);
};
}
/** @internal */
export function createGlobalExceptionReporter(io: KtxCliIo, info: KtxCliPackageInfo) {
return async (source: 'uncaughtException' | 'unhandledRejection', error: unknown): Promise<void> => {
const { reportException, shutdownTelemetryEmitter } = await import('./telemetry/index.js');
await reportException({
error,
context: { source, handled: false, fatal: true },
io,
packageInfo: info,
immediate: true,
});
await shutdownTelemetryEmitter();
};
}
/** @internal */
export function writeGlobalExceptionToStderr(io: KtxCliIo, error: unknown): void {
if (error instanceof Error && error.stack) {
io.stderr.write(`${error.stack}\n`);
} else {
io.stderr.write(`${String(error)}\n`);
}
writeErrorCommunityHint(io, 'crash');
}
export function installGlobalExceptionHandlers(io: KtxCliIo, info: KtxCliPackageInfo): () => void {
const report = createGlobalExceptionReporter(io, info);
const handle = (source: 'uncaughtException' | 'unhandledRejection', error: unknown): void => {
void (async () => {
try {
await report(source, error);
} catch {
// Best-effort: preserve Node's process termination behavior.
}
writeGlobalExceptionToStderr(io, error);
process.exit(1);
})();
};
const onUncaught = (error: Error): void => handle('uncaughtException', error);
const onUnhandled = (reason: unknown): void => handle('unhandledRejection', reason);
process.on('uncaughtException', onUncaught);
process.on('unhandledRejection', onUnhandled);
return () => {
process.off('uncaughtException', onUncaught);
process.off('unhandledRejection', onUnhandled);
};
}
export async function runKtxCli(
argv = process.argv.slice(2),
io: KtxCliIo = process,
@ -187,17 +98,7 @@ export async function runKtxCli(
profileMark('runtime:runKtxCli');
const { runCommanderKtxCli } = await profileSpan('import ./cli-program.js', () => import('./cli-program.js'));
// Real-process entry only: flush telemetry if interrupted. Test/programmatic
// callers pass their own `io`, so they never install process-level handlers.
const removeSignalFlush = (io as unknown) === process ? installTelemetrySignalFlush(io, info) : undefined;
const removeGlobalExceptionHandlers =
(io as unknown) === process ? installGlobalExceptionHandlers(io, info) : undefined;
try {
return await runCommanderKtxCli(argv, io, deps, info, {
runInit: runInitForCommander,
});
} finally {
removeGlobalExceptionHandlers?.();
removeSignalFlush?.();
}
return await runCommanderKtxCli(argv, io, deps, info, {
runInit: runInitForCommander,
});
}

View file

@ -2,7 +2,7 @@ import { type Command, InvalidArgumentError, Option } from '@commander-js/extra-
import type { KtxCliCommandContext } from '../cli-program.js';
import { resolveCommandProjectDir } from '../cli-program.js';
import type { KtxSetupDatabaseDriver } from '../setup-databases.js';
import { isKtxSetupLlmBackend, type KtxSetupLlmBackend } from '../setup-models.js';
import type { KtxSetupLlmBackend } from '../setup-models.js';
import type { KtxSetupSourceType } from '../setup-sources.js';
async function runSetupArgs(
@ -29,7 +29,7 @@ function embeddingBackend(value: string): 'openai' | 'sentence-transformers' {
}
function llmBackend(value: string): KtxSetupLlmBackend {
if (isKtxSetupLlmBackend(value)) {
if (value === 'anthropic' || value === 'vertex' || value === 'claude-code') {
return value;
}
throw new InvalidArgumentError(`invalid choice '${value}'`);
@ -95,6 +95,7 @@ function shouldShowSetupEntryMenu(
llmBackend?: KtxSetupLlmBackend;
anthropicApiKeyEnv?: string;
anthropicApiKeyFile?: string;
llmModel?: string;
vertexProject?: string;
vertexLocation?: string;
skipLlm?: boolean;
@ -165,6 +166,7 @@ function shouldShowSetupEntryMenu(
'llmBackend',
'anthropicApiKeyEnv',
'anthropicApiKeyFile',
'llmModel',
'vertexProject',
'vertexLocation',
'skipLlm',
@ -227,6 +229,7 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
.addOption(
new Option('--anthropic-api-key-file <path>', 'File containing the Anthropic API key').hideHelp(),
)
.addOption(new Option('--llm-model <model>', 'LLM model ID or backend model alias').hideHelp())
.addOption(new Option('--vertex-project <project>', 'Google Vertex AI project ID, env:NAME, or file:/path').hideHelp())
.addOption(new Option('--vertex-location <location>', 'Google Vertex AI location, env:NAME, or file:/path').hideHelp())
.addOption(new Option('--skip-llm', 'Leave LLM setup incomplete for now').hideHelp().default(false))
@ -403,8 +406,6 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
}
const resolvedAgentScope = options.local ? 'local' : options.global ? 'global' : 'project';
const debugEnabled =
((command.optsWithGlobals ? command.optsWithGlobals() : command.opts()) as { debug?: unknown }).debug === true;
await runSetupArgs(context, {
command: 'run',
projectDir: resolveCommandProjectDir(command),
@ -414,12 +415,12 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
agentScope: resolvedAgentScope,
skipAgents: options.skipAgents === true,
inputMode: options.input === false ? 'disabled' : 'auto',
...(debugEnabled ? { debug: true } : {}),
yes: options.yes === true,
cliVersion: context.packageInfo.version,
...(options.llmBackend ? { llmBackend: options.llmBackend } : {}),
...(options.anthropicApiKeyEnv ? { anthropicApiKeyEnv: options.anthropicApiKeyEnv } : {}),
...(options.anthropicApiKeyFile ? { anthropicApiKeyFile: options.anthropicApiKeyFile } : {}),
...(options.llmModel ? { llmModel: options.llmModel } : {}),
...(options.vertexProject ? { vertexProject: options.vertexProject } : {}),
...(options.vertexLocation ? { vertexLocation: options.vertexLocation } : {}),
skipLlm: options.skipLlm === true,

View file

@ -1,28 +0,0 @@
import type { KtxCliIo } from './cli-runtime.js';
import { isWritableTtyOutput } from './io/tty.js';
import { dim } from './io/symbols.js';
import { SLACK_URL } from './links.js';
type ErrorCtaVariant = 'error' | 'crash';
/** @internal */
export const SLACK_HELP_FOOTER = `Community & support: ${SLACK_URL}`;
/** @internal */
export const SLACK_SETUP_NOTE = {
title: 'Community',
body: `Questions or feedback? Join the ktx Slack: ${SLACK_URL}`,
} as const;
export function writeErrorCommunityHint(io: KtxCliIo, variant: ErrorCtaVariant): void {
if (!isWritableTtyOutput(io.stderr)) {
return;
}
const line =
variant === 'crash'
? `This may be a bug - report it or ask in the ktx community: ${SLACK_URL}`
: `Stuck? The ktx community can help: ${SLACK_URL}`;
io.stderr.write(`${dim(line)}\n`);
}

View file

@ -1,132 +0,0 @@
import type { KtxCliIo } from './cli-runtime.js';
import type { KtxSetupPromptOption } from './setup-prompts.js';
export type RecoveryOutcome = 'ready' | 'skip' | 'back' | 'failed';
/** @internal */
export interface RecoveryAction {
value: string;
label: string;
run: () => Promise<void>;
}
export type ConfigureResult = 'configured' | 'back' | 'cancelled';
export type ValidateResult =
| { status: 'ok' }
| { status: 'back' }
| { status: 'failed'; extraActions?: RecoveryAction[] };
export interface ConnectionRecoveryInput {
label: string;
interactive: boolean;
allowSkip: boolean;
io: KtxCliIo;
prompts: {
select(options: { message: string; options: KtxSetupPromptOption[] }): Promise<string>;
};
snapshot: () => Promise<() => Promise<void>>;
configure: () => Promise<ConfigureResult>;
validate: () => Promise<ValidateResult>;
}
async function runRollbackOnce(input: {
rollback: () => Promise<void>;
state: { rolledBack: boolean };
}): Promise<void> {
if (input.state.rolledBack) {
return;
}
input.state.rolledBack = true;
await input.rollback();
}
function recoveryOptions(input: {
allowSkip: boolean;
extraActions?: RecoveryAction[];
}): KtxSetupPromptOption[] {
return [
{ value: 'retry', label: 'Retry connection test' },
{ value: 're-enter', label: 'Re-enter connection details' },
...(input.extraActions ?? []).map((action) => ({
value: action.value,
label: action.label,
})),
...(input.allowSkip ? [{ value: 'skip', label: 'Skip this connection' }] : []),
{ value: 'back', label: 'Back' },
];
}
export async function runConnectionSetupWithRecovery(
input: ConnectionRecoveryInput,
): Promise<RecoveryOutcome> {
const rollback = await input.snapshot();
const rollbackState = { rolledBack: false };
const firstConfig = await input.configure();
if (firstConfig === 'back') {
await runRollbackOnce({ rollback, state: rollbackState });
return 'back';
}
if (firstConfig === 'cancelled') {
await runRollbackOnce({ rollback, state: rollbackState });
return 'failed';
}
let validation = await input.validate();
while (validation.status !== 'ok') {
if (validation.status === 'back') {
await runRollbackOnce({ rollback, state: rollbackState });
return 'back';
}
if (!input.interactive) {
return 'failed';
}
const action = await input.prompts.select({
message: `Connection setup failed for ${input.label}`,
options: recoveryOptions({
allowSkip: input.allowSkip,
extraActions: validation.extraActions,
}),
});
if (action === 'back') {
await runRollbackOnce({ rollback, state: rollbackState });
return 'back';
}
if (action === 'skip' && input.allowSkip) {
await runRollbackOnce({ rollback, state: rollbackState });
return 'skip';
}
if (action === 're-enter') {
const nextConfig = await input.configure();
if (nextConfig === 'back') {
await runRollbackOnce({ rollback, state: rollbackState });
return 'back';
}
if (nextConfig === 'cancelled') {
await runRollbackOnce({ rollback, state: rollbackState });
return 'failed';
}
validation = await input.validate();
continue;
}
if (action === 'retry') {
validation = await input.validate();
continue;
}
const extraAction = validation.extraActions?.find((candidate) => candidate.value === action);
if (extraAction) {
await extraAction.run();
validation = await input.validate();
continue;
}
validation = await input.validate();
}
return 'ready';
}

View file

@ -16,9 +16,8 @@ import { bold, dim, green, red, SYMBOLS } from './io/symbols.js';
import { createKtxCliScanConnector } from './local-scan-connectors.js';
import { profileMark } from './startup-profile.js';
import { isDemoConnection } from './telemetry/demo-detect.js';
import { emitTelemetryEvent, reportException } from './telemetry/index.js';
import { collectTelemetryRedactionSecrets } from './telemetry/redaction-secrets.js';
import { formatErrorDetail, scrubErrorClass } from './telemetry/scrubber.js';
import { emitTelemetryEvent } from './telemetry/index.js';
import { scrubErrorClass } from './telemetry/scrubber.js';
profileMark('module:connection');
@ -75,12 +74,6 @@ async function testNativeConnection(
}
const result = await connector.testConnection();
if (!result.success) {
// Re-throw the driver's original error so connection_test telemetry records
// its real class (e.g. ConnectionError) and code (e.g. ELOGIN) instead of
// collapsing every native failure to a generic Error with no code.
if (result.cause instanceof Error) {
throw result.cause;
}
throw new Error(result.error ?? 'connection test failed');
}
return { driver: connector.driver };
@ -311,7 +304,6 @@ async function emitConnectionTest(input: {
io: KtxCliIo;
}): Promise<void> {
const errorClass = input.error ? scrubErrorClass(input.error) : undefined;
const errorDetail = input.error ? formatErrorDetail(input.error) : undefined;
await emitTelemetryEvent({
name: 'connection_test',
projectDir: input.project.projectDir,
@ -322,24 +314,8 @@ async function emitConnectionTest(input: {
outcome: input.outcome,
durationMs: input.durationMs,
...(errorClass ? { errorClass } : {}),
...(errorDetail ? { errorDetail } : {}),
},
});
if (input.error) {
await reportException({
error: input.error,
context: { source: 'connection test', handled: true, fatal: false },
projectDir: input.project.projectDir,
io: input.io,
redactionSecrets: await collectTelemetryRedactionSecrets({
project: input.project,
connectionId: input.connectionId,
includeLlm: false,
includeEmbeddings: false,
env: process.env,
}),
});
}
}
function visualWidth(text: string): number {

View file

@ -5,9 +5,7 @@ import { assertReadOnlySql, limitSqlForExecution } from '../../context/connectio
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
import {
connectorTestFailure,
createKtxConnectorCapabilities,
type KtxConnectorTestResult,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
@ -322,7 +320,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
this.id = `bigquery:${options.connectionId}`;
}
async testConnection(): Promise<KtxConnectorTestResult> {
async testConnection(): Promise<{ success: boolean; error?: string }> {
try {
const client = this.getClient();
await client.getDatasets({ maxResults: 1 });
@ -331,7 +329,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
}
return { success: true };
} catch (error) {
return connectorTestFailure(error);
return { success: false, error: error instanceof Error ? error.message : String(error) };
}
}

View file

@ -1,7 +1,7 @@
import { createClient } from '@clickhouse/client';
import { getDialectForDriver } from '../../context/connections/dialects.js';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { connectorTestFailure, createKtxConnectorCapabilities, type KtxConnectorTestResult, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
import { readFileSync } from 'node:fs';
import { Agent as HttpsAgent } from 'node:https';
@ -317,12 +317,12 @@ export class KtxClickHouseScanConnector implements KtxScanConnector {
this.id = `clickhouse:${options.connectionId}`;
}
async testConnection(): Promise<KtxConnectorTestResult> {
async testConnection(): Promise<{ success: boolean; error?: string }> {
try {
await this.query('SELECT 1');
return { success: true };
} catch (error) {
return connectorTestFailure(error);
return { success: false, error: error instanceof Error ? error.message : String(error) };
}
}

View file

@ -11,9 +11,7 @@ import {
} from '../../context/scan/constraint-discovery.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
import {
connectorTestFailure,
createKtxConnectorCapabilities,
type KtxConnectorTestResult,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
@ -159,15 +157,6 @@ interface MysqlDistinctValueRow extends RowDataPacket {
val: unknown;
}
interface MysqlStatsRow extends RowDataPacket {
column_name: string;
estimated_cardinality: number | null;
}
export interface KtxMysqlColumnStatisticsResult {
cardinalityByColumn: Map<string, number>;
}
class DefaultMysqlPoolFactory implements KtxMysqlPoolFactory {
createPool(config: KtxMysqlPoolConfig): KtxMysqlPool {
return mysql.createPool(config) as Pool;
@ -393,7 +382,7 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
readonly capabilities = createKtxConnectorCapabilities({
tableSampling: true,
columnSampling: true,
columnStats: true,
columnStats: false,
readOnlySql: true,
nestedAnalysis: true,
formalForeignKeys: true,
@ -424,12 +413,12 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
this.id = `mysql:${options.connectionId}`;
}
async testConnection(): Promise<KtxConnectorTestResult> {
async testConnection(): Promise<{ success: boolean; error?: string }> {
try {
await this.query('SELECT 1');
return { success: true };
} catch (error) {
return connectorTestFailure(error);
return { success: false, error: error instanceof Error ? error.message : String(error) };
}
}
@ -571,29 +560,8 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
return { values, nullCount: null, distinctCount: null };
}
async columnStats(input: KtxColumnStatsInput, _ctx: KtxScanContext): Promise<KtxColumnStatsResult | null> {
const stats = await this.getColumnStatistics(input.table);
const value = stats?.cardinalityByColumn.get(input.column);
return value === undefined
? null
: { min: null, max: null, average: null, nullCount: null, distinctCount: value };
}
async getColumnStatistics(table: KtxTableRef): Promise<KtxMysqlColumnStatisticsResult | null> {
const schema = table.db ?? this.poolConfig.database;
const sql = this.dialect.generateColumnStatisticsQuery(schema, table.name);
if (!sql) {
return null;
}
const rows = await this.queryRaw<MysqlStatsRow>(sql);
const cardinalityByColumn = new Map<string, number>();
for (const row of rows) {
const cardinality = Number(row.estimated_cardinality);
if (Number.isFinite(cardinality) && cardinality >= 0) {
cardinalityByColumn.set(row.column_name, cardinality);
}
}
return cardinalityByColumn.size > 0 ? { cardinalityByColumn } : null;
async columnStats(_input: KtxColumnStatsInput, _ctx: KtxScanContext): Promise<KtxColumnStatsResult | null> {
return null;
}
async executeReadOnly(input: KtxMysqlReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {

View file

@ -171,18 +171,8 @@ export class KtxMysqlDialect implements KtxDialect {
`;
}
generateColumnStatisticsQuery(schemaName: string, tableName: string): string | null {
return `
SELECT
COLUMN_NAME AS column_name,
MAX(CARDINALITY) AS estimated_cardinality
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = '${schemaName.replace(/'/g, "''")}'
AND TABLE_NAME = '${tableName.replace(/'/g, "''")}'
AND CARDINALITY IS NOT NULL
AND SEQ_IN_INDEX = 1
GROUP BY COLUMN_NAME
`;
generateColumnStatisticsQuery(_schemaName: string, _tableName: string): string | null {
return null;
}
generateRandomizedCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {

View file

@ -6,9 +6,7 @@ import { assertReadOnlySql, limitSqlForExecution } from '../../context/connectio
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
import {
connectorTestFailure,
createKtxConnectorCapabilities,
type KtxConnectorTestResult,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
@ -444,12 +442,12 @@ export class KtxPostgresScanConnector implements KtxScanConnector {
this.id = `postgres:${options.connectionId}`;
}
async testConnection(): Promise<KtxConnectorTestResult> {
async testConnection(): Promise<{ success: boolean; error?: string }> {
try {
await this.query('SELECT 1');
return { success: true };
} catch (error) {
return connectorTestFailure(error);
return { success: false, error: error instanceof Error ? error.message : String(error) };
}
}

View file

@ -7,9 +7,7 @@ import { assertReadOnlySql, limitSqlForExecution } from '../../context/connectio
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
import {
connectorTestFailure,
createKtxConnectorCapabilities,
type KtxConnectorTestResult,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
@ -466,7 +464,7 @@ class SnowflakeSdkDriver implements KtxSnowflakeDriver {
await this.query('SELECT 1');
return { success: true };
} catch (error) {
return connectorTestFailure(error);
return { success: false, error: error instanceof Error ? error.message : String(error) };
}
}
@ -575,7 +573,7 @@ export class KtxSnowflakeScanConnector implements KtxScanConnector {
}
}
async testConnection(): Promise<KtxConnectorTestResult> {
async testConnection(): Promise<{ success: boolean; error?: string }> {
return this.getDriver().test();
}

View file

@ -6,7 +6,7 @@ import { fileURLToPath } from 'node:url';
import { getDialectForDriver } from '../../context/connections/dialects.js';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { normalizeQueryRows } from '../../context/connections/query-executor.js';
import { connectorTestFailure, createKtxConnectorCapabilities, type KtxConnectorTestResult, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
export interface KtxSqliteConnectionConfig {
@ -167,7 +167,7 @@ export class KtxSqliteScanConnector implements KtxScanConnector {
this.id = `sqlite:${options.connectionId}`;
}
async testConnection(): Promise<KtxConnectorTestResult> {
async testConnection(): Promise<{ success: boolean; error?: string }> {
try {
if (!existsSync(this.dbPath) || !statSync(this.dbPath).isFile()) {
return { success: false, error: `File not found: ${this.dbPath}` };
@ -175,7 +175,7 @@ export class KtxSqliteScanConnector implements KtxScanConnector {
this.database().prepare('SELECT 1').get();
return { success: true };
} catch (error) {
return connectorTestFailure(error);
return { success: false, error: error instanceof Error ? error.message : String(error) };
}
}

View file

@ -3,9 +3,7 @@ import { getDialectForDriver } from '../../context/connections/dialects.js';
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
import {
connectorTestFailure,
createKtxConnectorCapabilities,
type KtxConnectorTestResult,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
@ -386,12 +384,12 @@ export class KtxSqlServerScanConnector implements KtxScanConnector {
this.id = `sqlserver:${options.connectionId}`;
}
async testConnection(): Promise<KtxConnectorTestResult> {
async testConnection(): Promise<{ success: boolean; error?: string }> {
try {
await this.query('SELECT 1');
return { success: true };
} catch (error) {
return connectorTestFailure(error);
return { success: false, error: error instanceof Error ? error.message : String(error) };
}
}

View file

@ -1,6 +1,8 @@
import type { KtxProgressPort, KtxProgressUpdateOptions } from './context/scan/types.js';
import type { KtxCliIo } from './index.js';
import type { KtxIngestProgressUpdate } from './ingest.js';
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
import { publicDatabaseIngestMessage, publicQueryHistoryMessage } from './public-ingest-copy.js';
import type {
KtxPublicIngestArgs,
KtxPublicIngestDeps,
@ -8,17 +10,9 @@ import type {
KtxPublicIngestProject,
KtxPublicIngestTargetResult,
} from './public-ingest.js';
import { buildPublicIngestPlan, executePublicIngestTarget, publicProgressMessage } from './public-ingest.js';
import { createAggregateProgressPort } from './progress-port-adapter.js';
import { buildPublicIngestPlan, executePublicIngestTarget } from './public-ingest.js';
import { formatDuration } from './demo-metrics.js';
import { profileMark } from './startup-profile.js';
import {
isFreshStarCountCache,
readStarCountCache,
writeStarCountCache,
} from './star-prompt/cache.js';
import { fetchGitHubStarCount as defaultFetchGitHubStarCount } from './star-prompt/star-count.js';
import { renderStarPromptLine } from './star-prompt/star-line.js';
profileMark('module:context-build-view');
@ -86,7 +80,6 @@ export interface ContextBuildViewState {
frame: number;
startedAt: number | null;
totalElapsedMs: number;
starCount: number | null;
}
export interface ContextBuildArgs {
@ -129,8 +122,6 @@ interface CompletedItemName {
interface ContextBuildRenderOptions {
styled?: boolean;
showHint?: boolean;
showStarPrompt?: boolean;
columns?: number;
hintText?: string;
projectDir?: string;
title?: string;
@ -148,15 +139,6 @@ export interface ContextBuildDeps {
now?: () => number;
onSourceProgress?: (sources: ContextBuildSourceProgressUpdate[]) => void;
sourceProgressThrottleMs?: number;
fetchStarCount?: typeof defaultFetchGitHubStarCount;
starPromptEnv?: StarPromptEnv;
starPromptHomeDir?: string;
}
interface StarPromptEnv extends NodeJS.ProcessEnv {
CI?: string;
DO_NOT_TRACK?: string;
KTX_NO_STAR?: string;
}
// --- Rendering ---
@ -446,14 +428,6 @@ export function renderContextBuildView(
lines.push('');
}
if (options.showStarPrompt && hasActive) {
const starPrompt = renderStarPromptLine({
count: state.starCount,
columns: options.columns ?? 80,
});
lines.push(styled ? dim(starPrompt) : starPrompt);
}
if (options.showHint && hasActive) {
const hintContent = options.hintText ?? 'Ctrl+C to stop';
const hint = ` ${hintContent}`;
@ -611,7 +585,6 @@ export function viewStateFromSourceProgress(
frame: 0,
startedAt: startedAtMs ?? null,
totalElapsedMs: startedAtMs ? now - startedAtMs : 0,
starCount: null,
};
}
@ -659,9 +632,6 @@ export function createRepainter(io: KtxCliIo) {
hasPainted = true;
lastCursorUpRows = cursorUpRowsAfterWrite(content);
},
columns() {
return terminalColumns();
},
};
}
@ -837,10 +807,20 @@ export function initViewState(targets: KtxPublicIngestPlanTarget[]): ContextBuil
frame: 0,
startedAt: null,
totalElapsedMs: 0,
starCount: null,
};
}
function publicProgressMessage(message: string, target: KtxPublicIngestPlanTarget): string {
let current = message;
if (target.operation === 'database-ingest') {
current = publicDatabaseIngestMessage(current);
}
if (target.steps.includes('query-history')) {
current = publicQueryHistoryMessage(current, target.connectionId);
}
return current;
}
function formatProgressDetail(
update: Pick<KtxIngestProgressUpdate, 'percent' | 'message'>,
target: KtxPublicIngestPlanTarget,
@ -849,48 +829,27 @@ function formatProgressDetail(
return `[${percent}%] ${publicProgressMessage(update.message, target)}`;
}
const STAR_COUNT_CACHE_TTL_MS = 24 * 60 * 60 * 1000;
function envFlag(value: string | undefined): boolean {
return value !== undefined && value !== '' && value !== '0' && value !== 'false';
}
function shouldSuppressStarPrompt(env: StarPromptEnv): boolean {
return envFlag(env.CI) || envFlag(env.DO_NOT_TRACK) || envFlag(env.KTX_NO_STAR);
}
function startStarPromptCountRefresh(input: {
fetchStarCount: typeof defaultFetchGitHubStarCount;
homeDir?: string;
now: () => number;
paint: () => void;
state: ContextBuildViewState;
}): void {
const cached = readStarCountCache({ homeDir: input.homeDir });
if (cached) {
input.state.starCount = cached.count;
}
if (isFreshStarCountCache(cached, new Date(input.now()), STAR_COUNT_CACHE_TTL_MS)) {
return;
}
void input.fetchStarCount()
.then((count) => {
if (typeof count !== 'number' || !Number.isFinite(count)) {
return;
}
input.state.starCount = count;
input.paint();
void writeStarCountCache(
{
count,
fetchedAt: new Date(input.now()).toISOString(),
},
{ homeDir: input.homeDir },
);
})
.catch(() => undefined);
function createContextBuildProgressPort(
onProgress: (update: KtxIngestProgressUpdate) => void,
state: { progress: number } = { progress: 0 },
start = 0,
weight = 1,
): KtxProgressPort {
return {
async update(value: number, message?: string, options?: KtxProgressUpdateOptions): Promise<void> {
const absoluteValue = start + Math.max(0, Math.min(1, value)) * weight;
state.progress = Math.max(state.progress, Math.min(1, absoluteValue));
if (!message) return;
onProgress({
percent: Math.max(0, Math.min(100, Math.round(state.progress * 100))),
message,
...(options?.transient !== undefined ? { transient: options.transient } : {}),
});
},
startPhase(phaseWeight: number): KtxProgressPort {
return createContextBuildProgressPort(onProgress, state, state.progress, weight * phaseWeight);
},
};
}
export async function runContextBuild(
@ -914,31 +873,13 @@ export async function runContextBuild(
state.startedAt = nowFn();
const repainter = isTTY ? createRepainter(io) : null;
const starPromptEnabled = repainter !== null && !shouldSuppressStarPrompt(deps.starPromptEnv ?? process.env);
const viewOpts = {
styled: true,
projectDir: args.projectDir,
notices: plan.notices ?? [],
warnings: plan.warnings,
};
const paint = (hint: boolean) =>
repainter?.paint(
renderContextBuildView(state, {
...viewOpts,
showHint: hint,
showStarPrompt: starPromptEnabled && hint,
columns: repainter.columns(),
}),
);
if (starPromptEnabled) {
startStarPromptCountRefresh({
fetchStarCount: deps.fetchStarCount ?? defaultFetchGitHubStarCount,
homeDir: deps.starPromptHomeDir,
now: nowFn,
paint: () => paint(true),
state,
});
}
const paint = (hint: boolean) => repainter?.paint(renderContextBuildView(state, { ...viewOpts, showHint: hint }));
paint(true);
let spinnerInterval: ReturnType<typeof setInterval> | null = null;
@ -1081,7 +1022,7 @@ export async function runContextBuild(
};
const progressDeps: KtxPublicIngestDeps = {
scanProgress: createAggregateProgressPort(updateSchemaPhase),
scanProgress: createContextBuildProgressPort(updateSchemaPhase),
ingestProgress: updateIngestPhase,
runtimeIo: io,
onPhaseStart,
@ -1091,7 +1032,7 @@ export async function runContextBuild(
let result: KtxPublicIngestTargetResult | null = null;
let thrownError: unknown = null;
try {
result = await execTarget(targetState.target, runArgs, capture.io, progressDeps, project);
result = await execTarget(targetState.target, runArgs, capture.io, progressDeps);
} catch (error) {
thrownError = error;
}

View file

@ -17,6 +17,7 @@ export interface KtxDriverRegistration {
readonly driver: KtxConnectionDriver;
readonly scopeConfigKey: KtxScopeConfigKey | null;
readonly hasHistoricSqlReader: boolean;
readonly hasLocalQueryExecutor: boolean;
load(): Promise<KtxDriverConnectorModule>;
}
@ -30,6 +31,7 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
driver: 'bigquery',
scopeConfigKey: 'dataset_ids',
hasHistoricSqlReader: true,
hasLocalQueryExecutor: false,
load: async () => {
const m = await import('../../connectors/bigquery/connector.js');
return {
@ -51,6 +53,7 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
driver: 'clickhouse',
scopeConfigKey: 'databases',
hasHistoricSqlReader: false,
hasLocalQueryExecutor: false,
load: async () => {
const m = await import('../../connectors/clickhouse/connector.js');
return {
@ -72,6 +75,7 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
driver: 'mysql',
scopeConfigKey: 'schemas',
hasHistoricSqlReader: false,
hasLocalQueryExecutor: false,
load: async () => {
const m = await import('../../connectors/mysql/connector.js');
return {
@ -93,6 +97,7 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
driver: 'postgres',
scopeConfigKey: 'schemas',
hasHistoricSqlReader: true,
hasLocalQueryExecutor: true,
load: async () => {
const m = await import('../../connectors/postgres/connector.js');
return {
@ -114,6 +119,7 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
driver: 'sqlite',
scopeConfigKey: null,
hasHistoricSqlReader: false,
hasLocalQueryExecutor: true,
load: async () => {
const m = await import('../../connectors/sqlite/connector.js');
return {
@ -135,6 +141,7 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
driver: 'snowflake',
scopeConfigKey: 'schema_names',
hasHistoricSqlReader: true,
hasLocalQueryExecutor: false,
load: async () => {
const m = await import('../../connectors/snowflake/connector.js');
return {
@ -156,6 +163,7 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
driver: 'sqlserver',
scopeConfigKey: 'schemas',
hasHistoricSqlReader: false,
hasLocalQueryExecutor: false,
load: async () => {
const m = await import('../../connectors/sqlserver/connector.js');
return {

View file

@ -0,0 +1,59 @@
import { driverRegistrations, getDriverRegistration } from './drivers.js';
import { createPostgresQueryExecutor } from './postgres-query-executor.js';
import type {
KtxSqlQueryExecutionInput,
KtxSqlQueryExecutionResult,
KtxSqlQueryExecutorPort,
} from './query-executor.js';
import { createSqliteQueryExecutor } from './sqlite-query-executor.js';
import type { KtxConnectionDriver } from '../scan/types.js';
export interface DefaultLocalQueryExecutorOptions {
postgres?: KtxSqlQueryExecutorPort;
sqlite?: KtxSqlQueryExecutorPort;
}
function driverFor(input: KtxSqlQueryExecutionInput): string {
return String(input.connection?.driver ?? '').toLowerCase();
}
function localExecutorMap(
options: DefaultLocalQueryExecutorOptions,
): Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> {
const wiredExecutors: Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> = {
postgres: options.postgres ?? createPostgresQueryExecutor(),
sqlite: options.sqlite ?? createSqliteQueryExecutor(),
};
const executors: Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> = {};
for (const registration of Object.values(driverRegistrations)) {
if (!registration.hasLocalQueryExecutor) continue;
const executor = wiredExecutors[registration.driver];
if (executor) {
executors[registration.driver] = executor;
}
}
return executors;
}
export function createDefaultLocalQueryExecutor(options: DefaultLocalQueryExecutorOptions = {}): KtxSqlQueryExecutorPort {
const executors = localExecutorMap(options);
return {
async execute(input: KtxSqlQueryExecutionInput): Promise<KtxSqlQueryExecutionResult> {
const driver = driverFor(input);
const registration = getDriverRegistration(driver);
if (!registration?.hasLocalQueryExecutor) {
throw new Error(`No local query executor is configured for driver "${input.connection?.driver ?? 'unknown'}".`);
}
const executor = executors[registration.driver];
if (!executor) {
throw new Error(
`Local query executor flag is enabled for driver "${registration.driver}", but no executor factory is wired.`,
);
}
return executor.execute(input);
},
};
}

View file

@ -0,0 +1,78 @@
import { Client, type ClientConfig } from 'pg';
import type {
KtxSqlQueryExecutionInput,
KtxSqlQueryExecutionResult,
KtxSqlQueryExecutorPort,
} from './query-executor.js';
import { limitSqlForExecution } from './read-only-sql.js';
interface PgClientLike {
connect(): Promise<unknown>;
query(input: string | { text: string; rowMode: 'array' }): Promise<{
fields: Array<{ name: string }>;
rows: unknown[][];
command: string;
rowCount: number | null;
}>;
end(): Promise<void>;
}
interface PostgresQueryExecutorOptions {
statementTimeoutMs?: number;
queryTimeoutMs?: number;
connectionTimeoutMs?: number;
clientFactory?: (config: ClientConfig) => PgClientLike;
}
function connectionDriver(input: KtxSqlQueryExecutionInput): string {
return String(input.connection?.driver ?? '').toLowerCase();
}
function createDefaultClient(config: ClientConfig): PgClientLike {
return new Client(config);
}
export function createPostgresQueryExecutor(options: PostgresQueryExecutorOptions = {}): KtxSqlQueryExecutorPort {
const clientFactory = options.clientFactory ?? createDefaultClient;
return {
async execute(input: KtxSqlQueryExecutionInput): Promise<KtxSqlQueryExecutionResult> {
const driver = connectionDriver(input);
const connection = input.connection;
if (driver !== 'postgres') {
throw new Error(`Local Postgres execution cannot run driver "${connection?.driver ?? 'unknown'}".`);
}
if (typeof connection?.url !== 'string' || connection.url.trim().length === 0) {
throw new Error(`Local Postgres execution requires connections.${input.connectionId}.url.`);
}
const client = clientFactory({
connectionString: connection.url,
statement_timeout: options.statementTimeoutMs ?? 30_000,
query_timeout: options.queryTimeoutMs ?? 35_000,
connectionTimeoutMillis: options.connectionTimeoutMs ?? 5_000,
application_name: 'ktx-local-query',
});
await client.connect();
try {
await client.query('BEGIN READ ONLY');
const result = await client.query({
text: limitSqlForExecution(input.sql, input.maxRows),
rowMode: 'array',
});
await client.query('COMMIT');
return {
headers: result.fields.map((field) => field.name),
rows: result.rows,
totalRows: result.rows.length,
command: result.command,
rowCount: result.rowCount,
};
} catch (error) {
await client.query('ROLLBACK').catch(() => undefined);
throw error;
} finally {
await client.end();
}
},
};
}

View file

@ -8,7 +8,7 @@ export interface KtxSqlQueryExecutionInput {
maxRows?: number;
}
interface KtxSqlQueryExecutionResult {
export interface KtxSqlQueryExecutionResult {
headers: string[];
rows: unknown[][];
totalRows: number;

View file

@ -0,0 +1,92 @@
import { isAbsolute, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import Database from 'better-sqlite3';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import type {
KtxSqlQueryExecutionInput,
KtxSqlQueryExecutionResult,
KtxSqlQueryExecutorPort,
} from './query-executor.js';
import { normalizeQueryRows } from './query-executor.js';
import { limitSqlForExecution } from './read-only-sql.js';
type SqliteConnectionConfig = Record<string, unknown> | undefined;
function connectionDriver(input: KtxSqlQueryExecutionInput): string {
return String(input.connection?.driver ?? '').toLowerCase();
}
function stringConfigValue(connection: SqliteConnectionConfig, key: string): string | undefined {
const value = connection?.[key];
return typeof value === 'string' && value.trim().length > 0 ? resolveStringReference(key, value.trim()) : undefined;
}
function resolveStringReference(key: string, value: string): string {
if (value.startsWith('env:')) {
return process.env[value.slice('env:'.length)] ?? '';
}
if (key !== 'url' && value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}
function sqlitePathFromUrl(url: string): string {
if (url.startsWith('file:')) {
return fileURLToPath(url);
}
if (url.startsWith('sqlite:')) {
const parsed = new URL(url);
if (parsed.pathname.length > 0) {
return decodeURIComponent(parsed.pathname);
}
}
return url;
}
/** @internal */
export function sqliteDatabasePathFromConnection(input: KtxSqlQueryExecutionInput): string {
const driver = connectionDriver(input);
if (driver !== 'sqlite') {
throw new Error(`Local SQLite execution cannot run driver "${input.connection?.driver ?? 'unknown'}".`);
}
const pathValue = stringConfigValue(input.connection, 'path');
const urlValue = stringConfigValue(input.connection, 'url');
if (!pathValue && !urlValue) {
throw new Error(
`Local SQLite execution requires connections.${input.connectionId}.path or connections.${input.connectionId}.url.`,
);
}
const candidate = pathValue ?? sqlitePathFromUrl(urlValue as string);
return isAbsolute(candidate) ? candidate : resolve(input.projectDir ?? process.cwd(), candidate);
}
export function createSqliteQueryExecutor(): KtxSqlQueryExecutorPort {
return {
async execute(input: KtxSqlQueryExecutionInput): Promise<KtxSqlQueryExecutionResult> {
const sql = limitSqlForExecution(input.sql, input.maxRows);
const dbPath = sqliteDatabasePathFromConnection(input);
const db = new Database(dbPath, { readonly: true, fileMustExist: true });
try {
const statement = db.prepare(sql);
const rows = statement.all() as unknown[];
return {
headers: statement.columns().map((column) => column.name),
rows: normalizeQueryRows(rows),
totalRows: rows.length,
command: 'SELECT',
rowCount: rows.length,
};
} finally {
db.close();
}
},
};
}

View file

@ -1,39 +0,0 @@
/** @internal */
export function createAbortError(message = 'Aborted'): DOMException {
return new DOMException(message, 'AbortError');
}
export function isAbortError(error: unknown): boolean {
if (error instanceof DOMException && error.name === 'AbortError') {
return true;
}
if (!error || typeof error !== 'object') {
return false;
}
const record = error as { name?: unknown; code?: unknown };
return record.name === 'AbortError' || record.code === 'ABORT_ERR';
}
/** @internal */
export function throwIfAborted(signal?: AbortSignal): void {
if (signal?.aborted) {
throw createAbortError();
}
}
export function linkAbortSignal(parent?: AbortSignal): { controller: AbortController; dispose: () => void } {
const controller = new AbortController();
if (!parent) {
return { controller, dispose: () => undefined };
}
if (parent.aborted) {
controller.abort(createAbortError());
return { controller, dispose: () => undefined };
}
const onAbort = () => controller.abort(createAbortError());
parent.addEventListener('abort', onAbort, { once: true });
return {
controller,
dispose: () => parent.removeEventListener('abort', onAbort),
};
}

View file

@ -24,21 +24,6 @@ function sanitizedGitEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEn
return sanitized;
}
/**
* Create a simple-git client scoped to `baseDir`. When an identity is provided, ktx's own
* commits carry it through the GIT_AUTHOR and GIT_COMMITTER environment variables instead of
* relying on repo-local or global git config. This keeps commits working when the project
* directory is an existing repo ktx did not create and the machine has no configured git
* identity (e.g. a fresh Mac with no ~/.gitconfig), without mutating the user's repo config.
* Explicit `--author` flags on individual commits still take precedence over GIT_AUTHOR_NAME.
*/
export function createSimpleGit(baseDir: string, identity?: { name: string; email: string }): SimpleGit {
const env = sanitizedGitEnv();
if (identity?.name && identity.email) {
env.GIT_AUTHOR_NAME = identity.name;
env.GIT_AUTHOR_EMAIL = identity.email;
env.GIT_COMMITTER_NAME = identity.name;
env.GIT_COMMITTER_EMAIL = identity.email;
}
return simpleGit({ baseDir, unsafe: { allowUnsafeAskPass: true } }).env(env);
export function createSimpleGit(baseDir: string): SimpleGit {
return simpleGit({ baseDir, unsafe: { allowUnsafeAskPass: true } }).env(sanitizedGitEnv());
}

View file

@ -1,6 +1,6 @@
import { promises as fs } from 'node:fs';
import { dirname, join } from 'node:path';
import { CheckRepoActions, type SimpleGit } from 'simple-git';
import type { SimpleGit } from 'simple-git';
import { noopLogger, resolveConfigDir, type KtxCoreConfig, type KtxLogger } from './config.js';
import { createSimpleGit } from './git-env.js';
@ -85,12 +85,8 @@ export class GitService {
await fs.mkdir(this.configDir, { recursive: true });
this.logger.log(`Config directory ensured at: ${this.configDir}`);
// Initialize simple-git. Carry ktx's identity in the environment so commits succeed even
// when this repo already exists and the machine has no configured git identity.
this.git = createSimpleGit(this.configDir, {
name: this.config.git.userName,
email: this.config.git.userEmail,
});
// Initialize simple-git
this.git = createSimpleGit(this.configDir);
// Initialize git repository
await this.initialize();
@ -98,16 +94,14 @@ export class GitService {
private async initialize(): Promise<void> {
try {
// Adopt an existing repo ONLY when this directory is itself that repo's root.
// When it sits below an enclosing repo, a plain checkIsRepo() is true and ktx
// would silently piggyback on the enclosing tree — but every ktx relative path
// (file-store writes, session worktrees, squash-merges, reindex scans) assumes
// this directory IS the working-tree root. So treat "inside an enclosing repo"
// the same as "no repo" and initialize a dedicated repo rooted here.
const isRepoRoot = await this.git.checkIsRepo(CheckRepoActions.IS_REPO_ROOT);
// Check if already initialized
const isRepo = await this.git.checkIsRepo();
if (!isRepoRoot) {
if (!isRepo) {
await this.git.init();
const gitConfig = this.config.git;
await this.git.addConfig('user.name', gitConfig.userName);
await this.git.addConfig('user.email', gitConfig.userEmail);
this.logger.log('Initialized git repository');
}
@ -131,11 +125,7 @@ export class GitService {
}
} catch (error) {
this.logger.error('Failed to initialize git repository', error);
// Preserve the underlying git error: the generic message alone is undiagnosable in
// telemetry and unactionable for the user. The exception reporter walks `cause` and
// redacts secrets before send.
const detail = error instanceof Error ? error.message : String(error);
throw new Error(`Failed to initialize git repository: ${detail}`, { cause: error });
throw new Error('Failed to initialize git repository');
}
}
@ -909,10 +899,7 @@ export class GitService {
*/
forWorktree(workdir: string): GitService {
const scoped = new GitService(this.config, this.logger);
scoped.git = createSimpleGit(workdir, {
name: this.config.git.userName,
email: this.config.git.userEmail,
});
scoped.git = createSimpleGit(workdir);
scoped.configDir = workdir;
return scoped;
}

View file

@ -200,78 +200,27 @@ export class BigQueryHistoricSqlQueryHistoryReader {
config: HistoricSqlUnifiedPullConfig,
): AsyncIterable<AggregatedTemplate> {
const sql = `
WITH filtered_jobs AS (
SELECT
COALESCE(query_info.query_hashes.normalized_literals, TO_HEX(SHA256(query))) AS template_id,
query,
user_email,
creation_time,
end_time,
error_result
FROM ${this.viewPath}
WHERE job_type = 'QUERY'
AND statement_type IN ('SELECT', 'MERGE')
AND creation_time >= ${timestampExpression(window.start)}
AND creation_time < ${timestampExpression(window.end)}
AND query IS NOT NULL
),
template_stats AS (
SELECT
template_id,
MIN(query) AS canonical_sql,
COUNT(*) AS executions,
COUNT(DISTINCT user_email) AS distinct_users,
MIN(creation_time) AS first_seen,
MAX(creation_time) AS last_seen,
APPROX_QUANTILES(TIMESTAMP_DIFF(end_time, creation_time, MILLISECOND), 100)[OFFSET(50)] AS p50_ms,
APPROX_QUANTILES(TIMESTAMP_DIFF(end_time, creation_time, MILLISECOND), 100)[OFFSET(95)] AS p95_ms,
SAFE_DIVIDE(COUNTIF(error_result IS NOT NULL), COUNT(*)) AS error_rate,
CAST(NULL AS INT64) AS rows_produced
FROM filtered_jobs
GROUP BY template_id
HAVING COUNT(*) >= ${config.minExecutions}
),
template_users AS (
SELECT
template_id,
user_email AS user,
COUNT(*) AS executions,
MAX(creation_time) AS last_seen
FROM filtered_jobs
GROUP BY template_id, user_email
)
SELECT
stats.template_id,
stats.canonical_sql,
stats.executions,
stats.distinct_users,
stats.first_seen,
stats.last_seen,
stats.p50_ms,
stats.p95_ms,
stats.error_rate,
stats.rows_produced,
TO_JSON_STRING(
ARRAY_AGG(
STRUCT(users.user AS user, users.executions AS executions)
ORDER BY users.executions DESC, users.last_seen DESC
)
) AS top_users
FROM template_stats AS stats
JOIN template_users AS users
ON users.template_id = stats.template_id
GROUP BY
stats.template_id,
stats.canonical_sql,
stats.executions,
stats.distinct_users,
stats.first_seen,
stats.last_seen,
stats.p50_ms,
stats.p95_ms,
stats.error_rate,
stats.rows_produced
ORDER BY stats.executions DESC`.trim();
query_hash AS template_id,
MIN(query) AS canonical_sql,
COUNT(*) AS executions,
COUNT(DISTINCT user_email) AS distinct_users,
MIN(creation_time) AS first_seen,
MAX(creation_time) AS last_seen,
APPROX_QUANTILES(TIMESTAMP_DIFF(end_time, creation_time, MILLISECOND), 100)[OFFSET(50)] AS p50_ms,
APPROX_QUANTILES(TIMESTAMP_DIFF(end_time, creation_time, MILLISECOND), 100)[OFFSET(95)] AS p95_ms,
SAFE_DIVIDE(COUNTIF(error_result IS NOT NULL), COUNT(*)) AS error_rate,
CAST(NULL AS INT64) AS rows_produced,
TO_JSON_STRING(ARRAY_AGG(STRUCT(user_email AS user, 1 AS executions) ORDER BY creation_time DESC LIMIT 5)) AS top_users
FROM ${this.viewPath}
WHERE job_type = 'QUERY'
AND statement_type IN ('SELECT', 'MERGE')
AND creation_time >= ${timestampExpression(window.start)}
AND creation_time < ${timestampExpression(window.end)}
AND query IS NOT NULL
GROUP BY query_hash
HAVING COUNT(*) >= ${config.minExecutions}
ORDER BY executions DESC`.trim();
const result = await queryClient(client).executeQuery(sql);
if (result.error) {
throw grantsError(result.error);

View file

@ -1,7 +1,6 @@
import { createHash } from 'node:crypto';
import { readFile, readdir } from 'node:fs/promises';
import { join, relative } from 'node:path';
import { tableRefKey } from '../../../scan/table-ref.js';
import type { ChunkResult, DiffSet, ScopeDescriptor, WorkUnit } from '../../types.js';
import { isHistoricSqlPatternInputShardPath } from './pattern-inputs.js';
import { stagedManifestSchema, stagedPatternsInputSchema, stagedTableInputSchema } from './types.js';
@ -38,7 +37,7 @@ export async function chunkHistoricSqlUnifiedStagedDir(stagedDir: string, diffSe
}
const table = stagedTableInputSchema.parse(await readJson(stagedDir, path));
workUnits.push({
unitKey: `historic-sql-table-${safeUnitKey(tableRefKey(table.tableRef))}`,
unitKey: `historic-sql-table-${safeUnitKey(table.table)}`,
displayLabel: `Historic SQL usage: ${table.table}`,
rawFiles: [path],
dependencyPaths: ['manifest.json'],

View file

@ -26,21 +26,6 @@ export function isQueryHistoryEnabled(connection: unknown): boolean {
return queryHistoryRecord(connection)?.enabled === true;
}
/**
* Resolves the query-history dialect from the connection's driver capability
* alone, ignoring whether query history is enabled in ktx.yaml. Use this on the
* adapter-registration path when query history has been explicitly requested
* for the run (e.g. via `--query-history`, which is itself the opt-in): the
* persisted `context.queryHistory.enabled` flag must not gate registration.
* Returns null when the connection's driver has no query-history reader.
*/
export function historicSqlDialectForConnectionDriver(connection: unknown): HistoricSqlDialect | null {
const conn = recordOrNull(connection);
const driver = String(conn?.driver ?? '').toLowerCase();
const registration = getDriverRegistration(driver);
return registration?.hasHistoricSqlReader ? historicSqlDialectForDriver(registration.driver) : null;
}
/**
* Resolves the query-history dialect for a connection. Returns null when
* query history is disabled, or when the connection's driver has no
@ -50,5 +35,8 @@ export function queryHistoryDialectForConnection(connection: unknown): HistoricS
if (!isQueryHistoryEnabled(connection)) {
return null;
}
return historicSqlDialectForConnectionDriver(connection);
const conn = recordOrNull(connection);
const driver = String(conn?.driver ?? '').toLowerCase();
const registration = getDriverRegistration(driver);
return registration?.hasHistoricSqlReader ? historicSqlDialectForDriver(registration.driver) : null;
}

View file

@ -1,5 +1,4 @@
import { Buffer } from 'node:buffer';
import { tableRefKey } from '../../../scan/table-ref.js';
import type { StagedPatternsInput } from './types.js';
const HISTORIC_SQL_PATTERN_WORKUNIT_DIR = 'patterns-input';
@ -45,16 +44,11 @@ function sortedAuditTemplates(templates: readonly PatternTemplate[]): PatternTem
function sortedPatternCandidates(templates: readonly PatternTemplate[]): PatternTemplate[] {
return [...templates]
.filter((template) => template.tablesTouched.length >= 2)
.map((template) => ({
...template,
tablesTouched: [...template.tablesTouched].sort((left, right) => tableRefKey(left).localeCompare(tableRefKey(right))),
}))
.map((template) => ({ ...template, tablesTouched: [...template.tablesTouched].sort() }))
.sort((left, right) => {
const cardinality = right.tablesTouched.length - left.tablesTouched.length;
if (cardinality !== 0) return cardinality;
const leftSignature = left.tablesTouched.map(tableRefKey).join('\0');
const rightSignature = right.tablesTouched.map(tableRefKey).join('\0');
const tableSignature = leftSignature.localeCompare(rightSignature);
const tableSignature = left.tablesTouched.join('\0').localeCompare(right.tablesTouched.join('\0'));
if (tableSignature !== 0) return tableSignature;
return left.id.localeCompare(right.id);
});

View file

@ -1,283 +0,0 @@
import { z } from 'zod';
import type { KtxLlmRuntimePort } from '../../../../context/llm/runtime-port.js';
import type { SqlAnalysisPort } from '../../../../context/sql-analysis/ports.js';
import { tableRefKey } from '../../../scan/table-ref.js';
import type { KtxTableRef } from '../../../scan/types.js';
import { bucketDistinctUsers, bucketExecutions, bucketRecency } from './buckets.js';
import {
compileHistoricSqlRedactionPatterns,
redactHistoricSqlText,
type HistoricSqlRedactionPattern,
} from './redaction.js';
import { includedQueryHistoryTableRefs } from './scope-membership.js';
import {
aggregatedTemplateSchema,
historicSqlUnifiedPullConfigSchema,
type AggregatedTemplate,
type HistoricSqlDialect,
type HistoricSqlReader,
} from './types.js';
export interface QueryHistoryFilterProposal {
excludedRoles: Array<{ role: string; reason: string; pattern: string }>;
consideredRoleCount: number;
skipped: { reason: 'no-llm' | 'no-daemon' | 'no-in-scope-history' | 'user-block-present' } | null;
warnings: string[];
parseFailedTemplateIds: string[];
}
export interface ProposeQueryHistoryServiceAccountFiltersInput {
connectionId: string;
dialect: HistoricSqlDialect;
queryClient: unknown;
reader: HistoricSqlReader;
sqlAnalysis: SqlAnalysisPort;
llmRuntime: KtxLlmRuntimePort | null;
pullConfig: unknown;
now?: Date;
userServiceAccountsPresent?: boolean;
}
interface ParsedTemplateForPicker {
template: AggregatedTemplate;
tablesTouched: KtxTableRef[];
includedTables: KtxTableRef[];
}
interface RoleAccumulator {
role: string;
executions: number;
distinctUsers: number;
lastSeen: string;
tables: Map<string, KtxTableRef>;
templates: AggregatedTemplate[];
}
interface QueryHistoryRoleRecord {
role: string;
inScopeTables: string[];
executionsBucket: string;
distinctUsersBucket: string;
recencyBucket: string;
representativeTemplates: Array<{ id: string; canonicalSql: string; dialect: HistoricSqlDialect }>;
}
const queryHistoryFilterAdjudicationSchema = z.object({
roles: z.array(
z.object({
role: z.string().min(1),
exclude: z.boolean(),
reason: z.string().min(1),
}).strict(),
),
}).strict();
type QueryHistoryFilterAdjudication = z.infer<typeof queryHistoryFilterAdjudicationSchema>;
function emptyProposal(skipped: QueryHistoryFilterProposal['skipped'], warnings: string[] = []): QueryHistoryFilterProposal {
return { excludedRoles: [], consideredRoleCount: 0, skipped, warnings, parseFailedTemplateIds: [] };
}
function displayTableRef(ref: KtxTableRef): string {
return [ref.catalog, ref.db, ref.name].filter((part): part is string => !!part && part.length > 0).join('.');
}
function redactTemplateSqlForPicker(
template: AggregatedTemplate,
redactors: readonly HistoricSqlRedactionPattern[],
): AggregatedTemplate {
if (redactors.length === 0) {
return template;
}
return {
...template,
canonicalSql: redactHistoricSqlText(template.canonicalSql, redactors),
};
}
/** @internal */
export function regexEscapeForExactRolePattern(role: string): string {
return `^${role.replace(/[\\^$.*+?()[\]{}|]/g, '\\$&')}$`;
}
function recordRole(
acc: RoleAccumulator,
template: AggregatedTemplate,
tables: readonly KtxTableRef[],
executions: number,
): void {
acc.executions += executions;
acc.distinctUsers = Math.max(acc.distinctUsers, template.stats.distinctUsers);
acc.lastSeen = template.stats.lastSeen > acc.lastSeen ? template.stats.lastSeen : acc.lastSeen;
for (const table of tables) {
acc.tables.set(tableRefKey(table), table);
}
acc.templates.push(template);
}
function roleRecords(parsedTemplates: readonly ParsedTemplateForPicker[], now: Date): QueryHistoryRoleRecord[] {
const byRole = new Map<string, RoleAccumulator>();
for (const parsed of parsedTemplates) {
for (const entry of parsed.template.topUsers) {
if (!entry.user || entry.user.trim().length === 0 || entry.executions <= 0) {
continue;
}
const role = entry.user.trim();
const acc =
byRole.get(role) ??
{
role,
executions: 0,
distinctUsers: 0,
lastSeen: '1970-01-01T00:00:00.000Z',
tables: new Map<string, KtxTableRef>(),
templates: [],
};
recordRole(acc, parsed.template, parsed.includedTables, entry.executions);
byRole.set(role, acc);
}
}
return [...byRole.values()]
.sort((left, right) => right.executions - left.executions || left.role.localeCompare(right.role))
.map((acc) => ({
role: acc.role,
inScopeTables: [...acc.tables.entries()]
.sort(([left], [right]) => left.localeCompare(right))
.slice(0, 25)
.map(([, ref]) => displayTableRef(ref)),
executionsBucket: bucketExecutions(acc.executions),
distinctUsersBucket: bucketDistinctUsers(acc.distinctUsers),
recencyBucket: bucketRecency(acc.lastSeen, now),
representativeTemplates: [...acc.templates]
.sort((left, right) => right.stats.executions - left.stats.executions || left.templateId.localeCompare(right.templateId))
.slice(0, 3)
.map((template) => ({
id: template.templateId,
canonicalSql: template.canonicalSql,
dialect: template.dialect,
})),
}));
}
function adjudicationSystemPrompt(): string {
return [
'You are helping ktx decide whether observed query-history roles are operational service accounts.',
'Default every role to keep. Mark exclude true only when the aggregate evidence clearly shows loader, ELT, reverse-ETL, export, refresh, or maintenance traffic rather than analyst or BI-dashboard usage.',
'Use only the observed role records. Do not rely on a hardcoded denylist. Return structured output only.',
].join('\n');
}
export async function proposeQueryHistoryServiceAccountFilters(
input: ProposeQueryHistoryServiceAccountFiltersInput,
): Promise<QueryHistoryFilterProposal> {
if (!input.llmRuntime) {
return emptyProposal({ reason: 'no-llm' });
}
const config = historicSqlUnifiedPullConfigSchema.parse(input.pullConfig);
const redactors = compileHistoricSqlRedactionPatterns(config.redactionPatterns);
const now = input.now ?? new Date();
const windowDays = 'windowDays' in config ? config.windowDays : 90;
const windowStart = new Date(now.getTime() - windowDays * 24 * 60 * 60 * 1000);
const warnings: string[] = [];
const parseFailedTemplateIds: string[] = [];
const snapshot: AggregatedTemplate[] = [];
try {
for await (const row of input.reader.fetchAggregated(input.queryClient, { start: windowStart, end: now }, config)) {
snapshot.push(aggregatedTemplateSchema.parse(row));
}
} catch (error) {
return emptyProposal(null, [
`query_history_filter_picker_read_failed:${error instanceof Error ? error.message : String(error)}`,
]);
}
if (snapshot.length === 0) {
return emptyProposal({ reason: 'no-in-scope-history' });
}
const analysisItems = snapshot.map((template) => ({ id: template.templateId, sql: template.canonicalSql }));
const analysisOptions =
config.modeledTableCatalog.length > 0 ? { catalog: { tables: config.modeledTableCatalog } } : undefined;
let analysis: Awaited<ReturnType<SqlAnalysisPort['analyzeBatch']>>;
try {
analysis = await input.sqlAnalysis.analyzeBatch(analysisItems, input.dialect, analysisOptions);
} catch (error) {
return emptyProposal({ reason: 'no-daemon' }, [
`query_history_filter_picker_analysis_failed:${error instanceof Error ? error.message : String(error)}`,
]);
}
const parsedTemplates: ParsedTemplateForPicker[] = [];
for (const template of snapshot) {
const parsed = analysis.get(template.templateId);
if (!parsed || parsed.error) {
parseFailedTemplateIds.push(template.templateId);
continue;
}
const tablesTouched = [...new Map(parsed.tablesTouched.map((ref) => [tableRefKey(ref), ref])).values()]
.filter((ref) => ref.name.length > 0)
.sort((left, right) => tableRefKey(left).localeCompare(tableRefKey(right)));
const includedTables = includedQueryHistoryTableRefs(tablesTouched, config);
if (includedTables.length === 0) {
continue;
}
parsedTemplates.push({
template: redactTemplateSqlForPicker(template, redactors),
tablesTouched,
includedTables,
});
}
const records = roleRecords(parsedTemplates, now);
if (records.length <= 1) {
return {
excludedRoles: [],
consideredRoleCount: records.length,
skipped: { reason: 'no-in-scope-history' },
warnings,
parseFailedTemplateIds,
};
}
let generated: QueryHistoryFilterAdjudication;
try {
generated = await input.llmRuntime.generateObject<QueryHistoryFilterAdjudication, typeof queryHistoryFilterAdjudicationSchema>({
role: 'candidateExtraction',
system: adjudicationSystemPrompt(),
prompt: JSON.stringify({ connectionId: input.connectionId, dialect: input.dialect, roles: records }),
schema: queryHistoryFilterAdjudicationSchema,
});
} catch (error) {
return {
excludedRoles: [],
consideredRoleCount: records.length,
skipped: { reason: 'no-llm' },
warnings: [
...warnings,
`query_history_filter_picker_llm_failed:${error instanceof Error ? error.message : String(error)}`,
],
parseFailedTemplateIds,
};
}
const knownRoles = new Set(records.map((record) => record.role));
const excludedRoles = generated.roles
.filter((role) => role.exclude && knownRoles.has(role.role))
.sort((left, right) => left.role.localeCompare(right.role))
.map((role) => ({
role: role.role,
reason: role.reason,
pattern: regexEscapeForExactRolePattern(role.role),
}));
return {
excludedRoles,
consideredRoleCount: records.length,
skipped: input.userServiceAccountsPresent ? { reason: 'user-block-present' } : null,
warnings,
parseFailedTemplateIds,
};
}

View file

@ -1,260 +0,0 @@
import type { Dirent } from 'node:fs';
import { access, readdir, readFile } from 'node:fs/promises';
import { join, relative } from 'node:path';
import YAML from 'yaml';
import { getDriverRegistration } from '../../../connections/drivers.js';
import { parseDottedTableEntry } from '../../../scan/enabled-tables.js';
import { tableRefKey, tableRefSet, type KtxTableRefKey } from '../../../scan/table-ref.js';
import type { KtxTableRef } from '../../../scan/types.js';
import { readLiveDatabaseTableFiles } from '../live-database/stage.js';
export interface QueryHistoryScopeFloorInput {
projectDir: string;
connectionId: string;
driver: string;
connection: Record<string, unknown>;
storedQueryHistory: Record<string, unknown>;
}
export interface QueryHistoryScopeFloor {
enabledTables: KtxTableRef[];
enabledTableKeys: ReadonlySet<KtxTableRefKey> | null;
enabledSchemas: string[];
modeledTableCatalog: KtxTableRef[];
floorDisabled: boolean;
warnings: string[];
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value);
}
function stringArray(value: unknown): string[] {
return Array.isArray(value)
? value
.filter((item): item is string => typeof item === 'string' && item.trim().length > 0)
.map((item) => item.trim())
: [];
}
function tableRefsFromValues(values: unknown): KtxTableRef[] {
if (!Array.isArray(values)) return [];
return values.flatMap((value) => {
if (typeof value === 'string') {
const ref = parseDottedTableEntry(value);
return ref ? [ref] : [];
}
if (isRecord(value) && typeof value.name === 'string' && value.name.length > 0) {
return [
{
catalog: typeof value.catalog === 'string' ? value.catalog : null,
db: typeof value.db === 'string' ? value.db : null,
name: value.name,
},
];
}
return [];
});
}
function declaredSchemas(driver: string, connection: Record<string, unknown>): string[] {
const key = getDriverRegistration(driver)?.scopeConfigKey;
if (!key) return [];
return [...new Set(stringArray(connection[key]))].sort();
}
function uniqueSortedTableRefs(refs: readonly KtxTableRef[]): KtxTableRef[] {
const byKey = new Map<KtxTableRefKey, KtxTableRef>();
for (const ref of refs) {
byKey.set(tableRefKey(ref), ref);
}
return [...byKey.entries()]
.sort(([left], [right]) => left.localeCompare(right))
.map(([, ref]) => ref);
}
async function latestLiveDatabaseScanDir(projectDir: string, connectionId: string): Promise<string | null> {
const root = join(projectDir, 'raw-sources', connectionId, 'live-database');
let entries: Dirent[];
try {
entries = await readdir(root, { withFileTypes: true });
} catch (error) {
if (error && typeof error === 'object' && 'code' in error && error.code === 'ENOENT') return null;
throw error;
}
const syncDirs = entries
.filter((entry) => entry.isDirectory())
.map((entry) => entry.name)
.sort()
.reverse();
for (const syncDir of syncDirs) {
const absolute = join(root, syncDir);
try {
await access(join(absolute, 'connection.json'));
return absolute;
} catch {
continue;
}
}
return null;
}
async function scannedTableRefs(
projectDir: string,
connectionId: string,
): Promise<{ refs: KtxTableRef[]; catalogAvailable: boolean; warnings: string[] }> {
const scanDir = await latestLiveDatabaseScanDir(projectDir, connectionId);
if (!scanDir) {
return { refs: [], catalogAvailable: false, warnings: [] };
}
try {
const tableFiles = await readLiveDatabaseTableFiles(scanDir);
return {
refs: uniqueSortedTableRefs(
tableFiles.map(({ table }) => ({ catalog: table.catalog, db: table.db, name: table.name })),
),
catalogAvailable: true,
warnings: [],
};
} catch (error) {
return {
refs: [],
catalogAvailable: false,
warnings: [
`query_history_scope_floor_catalog_read_failed:live_database_scan:${error instanceof Error ? error.message : String(error)}`,
],
};
}
}
async function listYamlFiles(root: string): Promise<string[]> {
try {
const entries = await readdir(root, { withFileTypes: true, recursive: true });
return entries
.filter((entry) => entry.isFile() && /\.ya?ml$/i.test(entry.name))
.map((entry) => relative(root, join(entry.parentPath, entry.name)).replace(/\\/g, '/'))
.sort();
} catch (error) {
if (error && typeof error === 'object' && 'code' in error && error.code === 'ENOENT') return [];
throw error;
}
}
function refsFromManifest(content: string): KtxTableRef[] {
const parsed = YAML.parse(content) as unknown;
if (!isRecord(parsed) || !isRecord(parsed.tables)) return [];
return Object.values(parsed.tables).flatMap((entry) => {
if (!isRecord(entry) || typeof entry.table !== 'string') return [];
const ref = parseDottedTableEntry(entry.table);
return ref ? [ref] : [];
});
}
function refsFromStandaloneSource(content: string): KtxTableRef[] {
const parsed = YAML.parse(content) as unknown;
if (!isRecord(parsed) || typeof parsed.table !== 'string') return [];
const ref = parseDottedTableEntry(parsed.table);
return ref ? [ref] : [];
}
async function semanticTableRefs(
projectDir: string,
connectionId: string,
): Promise<{ refs: KtxTableRef[]; warnings: string[] }> {
const root = join(projectDir, 'semantic-layer', connectionId);
const files = await listYamlFiles(root);
const refs: KtxTableRef[] = [];
const warnings: string[] = [];
for (const file of files) {
try {
const content = await readFile(join(root, file), 'utf-8');
refs.push(...(file.startsWith('_schema/') ? refsFromManifest(content) : refsFromStandaloneSource(content)));
} catch (error) {
warnings.push(
`query_history_scope_floor_catalog_read_failed:${file}:${error instanceof Error ? error.message : String(error)}`,
);
}
}
return { refs: uniqueSortedTableRefs(refs), warnings };
}
export async function resolveQueryHistoryScopeFloor(input: QueryHistoryScopeFloorInput): Promise<QueryHistoryScopeFloor> {
const explicitEnabledTables = [
...tableRefsFromValues(input.storedQueryHistory.enabledTables),
...tableRefsFromValues(input.connection.enabled_tables),
];
const semanticTables = await semanticTableRefs(input.projectDir, input.connectionId);
const scannedTables = await scannedTableRefs(input.projectDir, input.connectionId);
const modeledTables = uniqueSortedTableRefs([
...semanticTables.refs,
...scannedTables.refs,
...explicitEnabledTables,
]);
const warnings = [...semanticTables.warnings, ...scannedTables.warnings];
if (explicitEnabledTables.length > 0) {
return {
enabledTables: explicitEnabledTables,
enabledTableKeys: tableRefSet(explicitEnabledTables),
enabledSchemas: [],
modeledTableCatalog: modeledTables,
floorDisabled: false,
warnings,
};
}
const explicitSchemas = stringArray(input.storedQueryHistory.enabledSchemas);
if (explicitSchemas.includes('*')) {
return {
enabledTables: [],
enabledTableKeys: null,
enabledSchemas: ['*'],
modeledTableCatalog: modeledTables,
floorDisabled: true,
warnings,
};
}
if (explicitSchemas.length > 0) {
if (!scannedTables.catalogAvailable || modeledTables.length === 0) {
return {
enabledTables: [],
enabledTableKeys: null,
enabledSchemas: ['*'],
modeledTableCatalog: modeledTables,
floorDisabled: true,
warnings: [...warnings, 'query_history_scope_floor_disabled:catalog_unavailable'],
};
}
return {
enabledTables: [],
enabledTableKeys: null,
enabledSchemas: [...new Set(explicitSchemas)].sort(),
modeledTableCatalog: modeledTables,
floorDisabled: false,
warnings,
};
}
const schemas = new Set(declaredSchemas(input.driver, input.connection));
for (const ref of semanticTables.refs) {
if (ref.db) schemas.add(ref.db);
}
if (schemas.size > 0 && (!scannedTables.catalogAvailable || modeledTables.length === 0)) {
return {
enabledTables: [],
enabledTableKeys: null,
enabledSchemas: ['*'],
modeledTableCatalog: modeledTables,
floorDisabled: true,
warnings: [...warnings, 'query_history_scope_floor_disabled:catalog_unavailable'],
};
}
return {
enabledTables: [],
enabledTableKeys: null,
enabledSchemas: [...schemas].sort(),
modeledTableCatalog: modeledTables,
floorDisabled: false,
warnings,
};
}

View file

@ -1,45 +0,0 @@
import { tableRefKey, tableRefSet } from '../../../scan/table-ref.js';
import type { KtxTableRef } from '../../../scan/types.js';
export interface QueryHistoryScopeMembershipConfig {
enabledTables: readonly KtxTableRef[];
enabledSchemas: readonly string[];
}
function schemaNameForRef(ref: KtxTableRef): string | null {
return ref.db && ref.db.length > 0 ? ref.db : null;
}
function schemaNamesFromConfig(enabledSchemas: readonly string[]): Set<string> {
return new Set(enabledSchemas.filter((schema) => schema !== '*'));
}
export function isQueryHistoryScopeFloorDisabled(config: QueryHistoryScopeMembershipConfig): boolean {
return config.enabledSchemas.includes('*');
}
export function shouldFailOpenQueryHistoryScope(config: QueryHistoryScopeMembershipConfig): boolean {
return (
config.enabledTables.length === 0 &&
!isQueryHistoryScopeFloorDisabled(config) &&
config.enabledSchemas.length === 0
);
}
export function includedQueryHistoryTableRefs(
tablesTouched: readonly KtxTableRef[],
config: QueryHistoryScopeMembershipConfig,
): KtxTableRef[] {
if (config.enabledTables.length > 0) {
const enabled = tableRefSet(config.enabledTables);
return tablesTouched.filter((ref) => enabled.has(tableRefKey(ref)));
}
if (isQueryHistoryScopeFloorDisabled(config) || shouldFailOpenQueryHistoryScope(config)) {
return [...tablesTouched];
}
const schemas = schemaNamesFromConfig(config.enabledSchemas);
return tablesTouched.filter((ref) => {
const schema = schemaNameForRef(ref);
return schema !== null && schemas.has(schema);
});
}

View file

@ -188,75 +188,26 @@ export class SnowflakeHistoricSqlQueryHistoryReader {
config: HistoricSqlUnifiedPullConfig,
): AsyncIterable<AggregatedTemplate> {
const sql = `
WITH filtered_queries AS (
SELECT
query_hash,
query_text,
user_name,
start_time,
total_elapsed_time,
execution_status,
rows_produced
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE query_text IS NOT NULL
AND query_type IN ('SELECT', 'MERGE')
AND start_time >= ${timestampLiteral(window.start)}
AND start_time < ${timestampLiteral(window.end)}
),
template_stats AS (
SELECT
query_hash AS template_id,
MIN(query_text) AS canonical_sql,
COUNT(*) AS executions,
COUNT(DISTINCT user_name) AS distinct_users,
MIN(start_time) AS first_seen,
MAX(start_time) AS last_seen,
APPROX_PERCENTILE(total_elapsed_time, 0.50) AS p50_ms,
APPROX_PERCENTILE(total_elapsed_time, 0.95) AS p95_ms,
DIV0(COUNT_IF(execution_status != 'SUCCESS'), COUNT(*)) AS error_rate,
SUM(rows_produced) AS rows_produced
FROM filtered_queries
GROUP BY query_hash
HAVING COUNT(*) >= ${config.minExecutions}
),
template_users AS (
SELECT
query_hash AS template_id,
user_name AS user,
COUNT(*) AS executions,
MAX(start_time) AS last_seen
FROM filtered_queries
GROUP BY query_hash, user_name
)
SELECT
stats.template_id,
stats.canonical_sql,
stats.executions,
stats.distinct_users,
stats.first_seen,
stats.last_seen,
stats.p50_ms,
stats.p95_ms,
stats.error_rate,
stats.rows_produced,
ARRAY_AGG(
OBJECT_CONSTRUCT('user', users.user, 'executions', users.executions)
) WITHIN GROUP (ORDER BY users.executions DESC, users.last_seen DESC)::string AS top_users
FROM template_stats AS stats
JOIN template_users AS users
ON users.template_id = stats.template_id
GROUP BY
stats.template_id,
stats.canonical_sql,
stats.executions,
stats.distinct_users,
stats.first_seen,
stats.last_seen,
stats.p50_ms,
stats.p95_ms,
stats.error_rate,
stats.rows_produced
ORDER BY stats.executions DESC`.trim();
query_hash AS template_id,
MIN(query_text) AS canonical_sql,
COUNT(*) AS executions,
COUNT(DISTINCT user_name) AS distinct_users,
MIN(start_time) AS first_seen,
MAX(start_time) AS last_seen,
APPROX_PERCENTILE(total_elapsed_time, 0.50) AS p50_ms,
APPROX_PERCENTILE(total_elapsed_time, 0.95) AS p95_ms,
DIV0(COUNT_IF(execution_status != 'SUCCESS'), COUNT(*)) AS error_rate,
SUM(rows_produced) AS rows_produced,
ARRAY_AGG(OBJECT_CONSTRUCT('user', user_name, 'executions', 1)) WITHIN GROUP (ORDER BY start_time DESC)::string AS top_users
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE query_text IS NOT NULL
AND query_type IN ('SELECT', 'MERGE')
AND start_time >= ${timestampLiteral(window.start)}
AND start_time < ${timestampLiteral(window.end)}
GROUP BY query_hash
HAVING COUNT(*) >= ${config.minExecutions}
ORDER BY executions DESC`.trim();
const result = await queryClient(client).executeQuery(sql);
if (result.error) {
throw grantsError(result.error);

View file

@ -1,8 +1,6 @@
import { mkdir, writeFile } from 'node:fs/promises';
import { dirname, join } from 'node:path';
import type { SqlAnalysisPort } from '../../../../context/sql-analysis/ports.js';
import { tableRefKey, type KtxTableRefKey } from '../../../scan/table-ref.js';
import type { KtxTableRef } from '../../../scan/types.js';
import {
bucketDistinctUsers,
bucketErrorRate,
@ -17,11 +15,6 @@ import {
redactHistoricSqlText,
type HistoricSqlRedactionPattern,
} from './redaction.js';
import {
includedQueryHistoryTableRefs,
isQueryHistoryScopeFloorDisabled,
shouldFailOpenQueryHistoryScope,
} from './scope-membership.js';
import {
HISTORIC_SQL_SOURCE_KEY,
aggregatedTemplateSchema,
@ -45,13 +38,17 @@ interface StageHistoricSqlAggregatedSnapshotInput {
interface ParsedTemplate {
template: AggregatedTemplate;
tablesTouched: KtxTableRef[];
includedTables: KtxTableRef[];
tablesTouched: string[];
includedTables: string[];
columnsByClause: Record<string, string[]>;
}
interface EnabledTableFilter {
exact: Set<string>;
uniqueUnqualified: Set<string>;
}
interface TableAccumulator {
tableRef: KtxTableRef;
table: string;
executions: number;
distinctUsers: number;
@ -82,21 +79,8 @@ function matchesAny(value: string | null, patterns: RegExp[]): boolean {
return !!value && patterns.some((pattern) => pattern.test(value));
}
// ktx's own warehouse scan emits relationship- and column-profiling probes that land in
// pg_stat_statements (relationship-validation, relationship-composite-candidates, and each
// dialect's relationship value aggregation). They are ktx introspection, not genuine query
// usage, so they must not be mined back as query history. The markers are ktx-owned
// identifiers, stable across dialects.
function isKtxScanProbe(sql: string): boolean {
if (/\brelationship_profile_values\b/i.test(sql)) {
return true;
}
return /\bchild_values\b/i.test(sql) && /\bparent_values\b/i.test(sql);
}
function shouldDropBySql(sql: string, config: HistoricSqlUnifiedPullConfig): boolean {
if (NOISE_PREFIX_RE.test(sql) || SYSTEM_TABLE_RE.test(sql)) return true;
if (isKtxScanProbe(sql)) return true;
if (config.filters.dropTrivialProbes !== false && TRIVIAL_SQL_RE.test(sql)) return true;
return false;
}
@ -108,7 +92,8 @@ function shouldDropByUsers(template: AggregatedTemplate, config: HistoricSqlUnif
const matchingExecutions = template.topUsers
.filter((entry) => matchesAny(entry.user, patterns))
.reduce((sum, entry) => sum + entry.executions, 0);
const serviceOnly = template.stats.executions > 0 && matchingExecutions >= template.stats.executions;
const allExecutions = template.topUsers.reduce((sum, entry) => sum + entry.executions, 0);
const serviceOnly = allExecutions > 0 && matchingExecutions >= allExecutions;
return service.mode === 'exclude' ? serviceOnly : !serviceOnly;
}
@ -124,8 +109,43 @@ function shouldDropTemplate(template: AggregatedTemplate, config: HistoricSqlUni
return false;
}
function displayTableRef(ref: KtxTableRef): string {
return [ref.catalog, ref.db, ref.name].filter((part): part is string => !!part && part.length > 0).join('.');
function normalizeTableIdentifier(value: string): string {
return value.trim().toLowerCase();
}
function unqualifiedTableIdentifier(value: string): string {
const parts = normalizeTableIdentifier(value).split('.').filter(Boolean);
return parts.at(-1) ?? '';
}
function buildEnabledTableFilter(enabledTables: string[]): EnabledTableFilter | null {
if (enabledTables.length === 0) {
return null;
}
const exact = new Set(enabledTables.map(normalizeTableIdentifier).filter((value) => value.length > 0));
const unqualifiedCounts = new Map<string, number>();
for (const table of exact) {
const unqualified = unqualifiedTableIdentifier(table);
if (unqualified.length > 0) {
unqualifiedCounts.set(unqualified, (unqualifiedCounts.get(unqualified) ?? 0) + 1);
}
}
return {
exact,
uniqueUnqualified: new Set(
[...unqualifiedCounts.entries()]
.filter(([, count]) => count === 1)
.map(([table]) => table),
),
};
}
function isEnabledTable(table: string, filter: EnabledTableFilter | null): boolean {
if (!filter) {
return true;
}
const normalized = normalizeTableIdentifier(table);
return filter.exact.has(normalized) || filter.uniqueUnqualified.has(unqualifiedTableIdentifier(normalized));
}
function historicSqlWindowDays(config: HistoricSqlUnifiedPullConfig): number {
@ -160,10 +180,9 @@ function recordJoin(acc: TableAccumulator, otherTable: string, columns: string[]
}
}
function accumulatorFor(tableRef: KtxTableRef): TableAccumulator {
function accumulatorFor(table: string): TableAccumulator {
return {
tableRef,
table: displayTableRef(tableRef),
table,
executions: 0,
distinctUsers: 0,
errorRateNumerator: 0,
@ -193,8 +212,8 @@ function addTemplate(acc: TableAccumulator, parsed: ParsedTemplate): void {
}
}
const joinColumns = parsed.columnsByClause.join ?? [];
for (const otherTable of parsed.tablesTouched.filter((table) => tableRefKey(table) !== tableRefKey(acc.tableRef))) {
recordJoin(acc, displayTableRef(otherTable), joinColumns, executions);
for (const otherTable of parsed.tablesTouched.filter((table) => table !== acc.table)) {
recordJoin(acc, otherTable, joinColumns, executions);
}
acc.topTemplates.push(parsed.template);
}
@ -231,7 +250,6 @@ function toStagedTable(acc: TableAccumulator, now: Date): StagedTableInput {
return {
table: acc.table,
tableRef: acc.tableRef,
stats: {
executionsBucket: bucketExecutions(acc.executions),
distinctUsersBucket: bucketDistinctUsers(acc.distinctUsers),
@ -251,7 +269,7 @@ function toPatternsInput(parsedTemplates: ParsedTemplate[]): StagedPatternsInput
.map(({ template, tablesTouched }) => ({
id: template.templateId,
canonicalSql: template.canonicalSql,
tablesTouched: [...tablesTouched].sort((left, right) => tableRefKey(left).localeCompare(tableRefKey(right))),
tablesTouched: [...tablesTouched].sort(),
executionsBucket: bucketExecutions(template.stats.executions),
distinctUsersBucket: bucketDistinctUsers(template.stats.distinctUsers),
dialect: template.dialect,
@ -262,6 +280,7 @@ function toPatternsInput(parsedTemplates: ParsedTemplate[]): StagedPatternsInput
export async function stageHistoricSqlAggregatedSnapshot(input: StageHistoricSqlAggregatedSnapshotInput): Promise<void> {
const config = historicSqlUnifiedPullConfigSchema.parse(input.pullConfig);
const enabledTableFilter = buildEnabledTableFilter(config.enabledTables);
const redactors = compileHistoricSqlRedactionPatterns(config.redactionPatterns);
const now = input.now ?? new Date();
const windowStart = new Date(now.getTime() - historicSqlWindowDays(config) * 24 * 60 * 60 * 1000);
@ -277,25 +296,11 @@ export async function stageHistoricSqlAggregatedSnapshot(input: StageHistoricSql
}
}
const analysisItems = snapshot.map((template) => ({ id: template.templateId, sql: template.canonicalSql }));
const analysisOptions =
config.modeledTableCatalog.length > 0 ? { catalog: { tables: config.modeledTableCatalog } } : undefined;
const warnings: string[] = [
...config.scopeFloorWarnings,
...(shouldFailOpenQueryHistoryScope(config) ? ['query_history_scope_floor_disabled:empty_modeled_scope'] : []),
];
let scopeDisabledByQualificationFailure = false;
let analysis: Awaited<ReturnType<SqlAnalysisPort['analyzeBatch']>>;
try {
analysis = await input.sqlAnalysis.analyzeBatch(analysisItems, config.dialect, analysisOptions);
} catch (error) {
if (!analysisOptions || config.enabledTables.length > 0 || isQueryHistoryScopeFloorDisabled(config)) {
throw error;
}
warnings.push('query_history_scope_floor_disabled:catalog_qualification_failed');
scopeDisabledByQualificationFailure = true;
analysis = await input.sqlAnalysis.analyzeBatch(analysisItems, config.dialect, undefined);
}
const analysis = await input.sqlAnalysis.analyzeBatch(
snapshot.map((template) => ({ id: template.templateId, sql: template.canonicalSql })),
config.dialect,
);
const warnings: string[] = [];
const parsedTemplates: ParsedTemplate[] = [];
for (const template of snapshot) {
const parsed = analysis.get(template.templateId);
@ -303,12 +308,8 @@ export async function stageHistoricSqlAggregatedSnapshot(input: StageHistoricSql
warnings.push(`parse_failed:${template.templateId}`);
continue;
}
const tablesTouched = [...new Map(parsed.tablesTouched.map((ref) => [tableRefKey(ref), ref])).values()]
.filter((ref) => ref.name.length > 0)
.sort((left, right) => tableRefKey(left).localeCompare(tableRefKey(right)));
const includedTables = scopeDisabledByQualificationFailure
? [...tablesTouched]
: includedQueryHistoryTableRefs(tablesTouched, config);
const tablesTouched = [...new Set(parsed.tablesTouched)].filter((table) => table.length > 0).sort();
const includedTables = tablesTouched.filter((table) => isEnabledTable(table, enabledTableFilter));
if (includedTables.length === 0) {
continue;
}
@ -322,23 +323,22 @@ export async function stageHistoricSqlAggregatedSnapshot(input: StageHistoricSql
});
}
const byTable = new Map<KtxTableRefKey, TableAccumulator>();
const byTable = new Map<string, TableAccumulator>();
for (const parsed of parsedTemplates) {
for (const tableRef of parsed.includedTables) {
const key = tableRefKey(tableRef);
const acc = byTable.get(key) ?? accumulatorFor(tableRef);
for (const table of parsed.includedTables) {
const acc = byTable.get(table) ?? accumulatorFor(table);
addTemplate(acc, parsed);
byTable.set(key, acc);
byTable.set(table, acc);
}
}
await mkdir(input.stagedDir, { recursive: true });
for (const [, acc] of [...byTable.entries()].sort((left, right) => left[0].localeCompare(right[0]))) {
await writeJson(input.stagedDir, `tables/${acc.table}.json`, toStagedTable(acc, now));
for (const [table, acc] of [...byTable.entries()].sort(([left], [right]) => left.localeCompare(right))) {
await writeJson(input.stagedDir, `tables/${table}.json`, toStagedTable(acc, now));
}
const patternsInput = toPatternsInput(parsedTemplates);
const patternInputSplit = splitHistoricSqlPatternInputs(patternsInput);
const allWarnings = [...new Set([...warnings, ...patternInputSplit.warnings])];
const allWarnings = [...warnings, ...patternInputSplit.warnings];
await writeJson(input.stagedDir, 'patterns-input.json', patternInputSplit.auditInput);
for (const shard of patternInputSplit.shards) {
await writeJson(input.stagedDir, shard.path, shard.input);

View file

@ -8,22 +8,9 @@ export type HistoricSqlDialect = z.infer<typeof historicSqlDialectSchema>;
const filterModeSchema = z.enum(['exclude', 'include', 'mark-only']);
const ktxTableRefSchema = z.object({
catalog: z.string().nullable(),
db: z.string().nullable(),
name: z.string().min(1),
}).strict();
const ktxTableRefWithColumnsSchema = ktxTableRefSchema.extend({
columns: z.array(z.string().min(1)).optional(),
}).strict();
const historicSqlCommonPullConfigSchema = z.object({
minExecutions: z.number().int().nonnegative().default(5),
enabledTables: z.array(ktxTableRefSchema).default([]),
enabledSchemas: z.array(z.string().min(1)).default([]),
modeledTableCatalog: z.array(ktxTableRefWithColumnsSchema).default([]),
scopeFloorWarnings: z.array(z.string()).default([]),
enabledTables: z.array(z.string().min(1)).default([]),
filters: z.object({
serviceAccounts: z.object({
patterns: z.array(z.string()).default([]),
@ -81,7 +68,6 @@ export type AggregatedTemplate = z.infer<typeof aggregatedTemplateSchema>;
export const stagedTableInputSchema = z.object({
table: z.string().min(1),
tableRef: ktxTableRefSchema,
stats: z.object({
executionsBucket: z.string(),
distinctUsersBucket: z.string(),
@ -107,7 +93,7 @@ export const stagedPatternsInputSchema = z.object({
templates: z.array(z.object({
id: z.string(),
canonicalSql: z.string(),
tablesTouched: z.array(ktxTableRefSchema),
tablesTouched: z.array(z.string()),
executionsBucket: z.string(),
distinctUsersBucket: z.string(),
dialect: historicSqlDialectSchema,

View file

@ -39,7 +39,7 @@ export interface CuratorPaginationInput {
buildUserPrompt: (input: CuratorPaginationPromptInput) => string;
buildToolSet: (passNumber: number) => KtxRuntimeToolSet;
getReconciliationActions: () => MemoryAction[];
abortSignal?: AbortSignal;
onStepFinish?: (info: { passNumber: number; stepIndex: number; stepBudget: number }) => void;
}
interface CuratorPaginationResult extends ReconciliationOutcome {
@ -243,7 +243,10 @@ export class CuratorPaginationService implements CuratorPaginationPort {
sourceKey: params.input.sourceKey,
jobId: params.input.jobId,
forceRun: params.forceRun,
abortSignal: params.input.abortSignal,
onStepFinish: params.input.onStepFinish
? ({ stepIndex, stepBudget }) =>
params.input.onStepFinish?.({ passNumber: params.passNumber, stepIndex, stepBudget })
: undefined,
});
}

View file

@ -21,7 +21,6 @@ export interface RepairFinalGateFailureInput {
repairKind: FinalGateRepairKind;
maxAttempts?: number;
stepBudget?: number;
abortSignal?: AbortSignal;
}
const readRepairFileSchema = z.object({
@ -201,7 +200,6 @@ export async function repairFinalGateFailure(
jobId: input.trace.context.jobId,
repairKind: input.repairKind,
},
abortSignal: input.abortSignal,
}),
);

View file

@ -3,7 +3,6 @@ import { dirname, join } from 'node:path';
import pLimit from 'p-limit';
import { z } from 'zod';
import { type KtxLogger, noopLogger } from '../../context/core/config.js';
import type { RateLimitWaitState } from '../../context/llm/rate-limit-governor.js';
import { createRuntimeToolDescriptorFromAiTool } from '../../context/llm/runtime-tools.js';
import type { KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
import type { CaptureSession, MemoryAction } from '../../context/memory/types.js';
@ -220,10 +219,6 @@ export class IngestBundleRunner {
}
async run(job: IngestBundleJob, ctx?: IngestJobContext): Promise<IngestBundleResult> {
const unsubscribeRateLimitGovernor = this.subscribeRateLimitGovernor({
trace: this.createTrace(job),
memoryFlow: ctx?.memoryFlow,
});
const key = job.connectionId;
const previous = this.chainByConnection.get(key);
if (previous) {
@ -246,72 +241,10 @@ export class IngestBundleRunner {
ctx?.memoryFlow?.finish('error', [sanitizeMemoryFlowError(error)]);
throw error;
} finally {
unsubscribeRateLimitGovernor();
await this.maybeEmitIngestProfile(job.jobId);
}
}
private formatRateLimitWait(
state: Extract<RateLimitWaitState, { kind: 'wait_tick' | 'wait_started' | 'wait_finished' }>,
): string {
const seconds = Math.ceil(state.remainingMs / 1_000);
const minutes = Math.floor(seconds / 60);
const remainder = seconds % 60;
const duration = minutes > 0 ? `${minutes}m${String(remainder).padStart(2, '0')}s` : `${seconds}s`;
const type = state.rateLimitType ? ` ${state.rateLimitType}` : '';
return `Rate-limited (${state.provider}${type}); resuming in ${duration}; Ctrl+C to stop`;
}
private subscribeRateLimitGovernor(input: {
trace: IngestTraceWriter;
memoryFlow?: MemoryFlowEventSink;
}): () => void {
const governor = this.deps.settings.rateLimitGovernor;
if (!governor) {
return () => undefined;
}
return governor.subscribe((state: RateLimitWaitState) => {
if (state.kind === 'rate_limit_observed') {
void input.trace.event('info', 'rate_limit', 'rate_limit_observed', { ...state });
return;
}
if (state.kind === 'concurrency_adjusted') {
void input.trace.event('info', 'rate_limit', 'concurrency_adjusted', { ...state });
return;
}
void input.trace.event('info', 'rate_limit', state.kind, { ...state });
if (state.kind === 'wait_tick' || state.kind === 'wait_started') {
input.memoryFlow?.emit({
type: 'rate_limit_wait',
provider: state.provider,
...(state.rateLimitType ? { rateLimitType: state.rateLimitType } : {}),
resumeAtMs: state.resumeAtMs,
remainingMs: state.remainingMs,
});
input.memoryFlow?.emit({
type: 'stage_progress',
stage: 'integration',
percent: 50,
message: this.formatRateLimitWait(state),
transient: true,
});
}
});
}
private async withRateLimitWorkSlot<T>(abortSignal: AbortSignal | undefined, fn: () => Promise<T>): Promise<T> {
const governor = this.deps.settings.rateLimitGovernor;
if (!governor) {
return fn();
}
const release = await governor.acquireWorkSlot(abortSignal);
try {
return await fn();
} finally {
release();
}
}
/**
* When profiling is enabled via the `KTX_PROFILE_INGEST` env var or the
* `ingest.profile` config setting read the job's trace + tool transcripts
@ -939,13 +872,13 @@ export class IngestBundleRunner {
workUnitSettings: { maxConcurrency: number; stepBudget: number; failureMode: 'abort' | 'continue' };
transcriptDir: string;
transcriptSummaries: Map<string, MutableToolTranscriptSummary>;
recordTranscriptEntry(path: string): (entry: ToolCallLogEntry) => MutableToolTranscriptSummary;
recordTranscriptEntry(path: string): (entry: ToolCallLogEntry) => void;
stageIndex: StageIndex;
includeContextEvidenceTools: boolean;
currentTableExists(tableRef: string): Promise<boolean>;
memoryFlow?: MemoryFlowEventSink;
abortSignal?: AbortSignal;
wuSkillNames: string[];
onStepFinish?: (info: { stepIndex: number; stepBudget: number }) => void;
}): Promise<WorkUnitOutcome> {
const session: CaptureSession = {
userId: 'system',
@ -1049,6 +982,7 @@ export class IngestBundleRunner {
type: 'work_unit_started',
unitKey: input.wu.unitKey,
skills: input.wuSkillNames,
stepBudget: input.workUnitSettings.stepBudget,
});
return executeWorkUnit(
{
@ -1072,10 +1006,8 @@ export class IngestBundleRunner {
slIndex: input.slIndex,
priorProvenance: input.priorProvenance,
}),
buildToolSet: (wuInner) => {
const transcriptPath = join(input.transcriptDir, `${wuInner.unitKey}.jsonl`);
const record = input.recordTranscriptEntry(transcriptPath);
return wrapToolsWithLogger(
buildToolSet: (wuInner) =>
wrapToolsWithLogger(
buildWuToolSet({
sourceKey: input.job.sourceKey,
stagedDir: input.stagedDir,
@ -1084,23 +1016,10 @@ export class IngestBundleRunner {
emitUnmappedFallbackTool: wuEmitUnmappedFallbackTool,
toolsetTools: wuToolset.toRuntimeTools(wuToolContext),
}),
transcriptPath,
join(input.transcriptDir, `${wuInner.unitKey}.jsonl`),
wuInner.unitKey,
{
// Drive the live HUD heartbeat from real tool calls: each invocation
// ticks the running per-unit count. This is an observed signal, not a
// re-derived turn count, so it can never overshoot a budget.
onEntry: (entry) => {
const summary = record(entry);
input.memoryFlow?.emit({
type: 'work_unit_step',
unitKey: wuInner.unitKey,
toolCalls: summary.toolCallCount,
});
},
},
);
},
{ onEntry: input.recordTranscriptEntry(join(input.transcriptDir, `${wuInner.unitKey}.jsonl`)) },
),
captureSession: session,
sessionActions,
modelRole: 'candidateExtraction',
@ -1109,7 +1028,7 @@ export class IngestBundleRunner {
connectionId: input.job.connectionId,
jobId: input.job.jobId,
toolFailureCount: (unitKey) => input.transcriptSummaries.get(unitKey)?.fatalErrorCount ?? 0,
abortSignal: input.abortSignal,
onStepFinish: input.onStepFinish,
},
input.wu,
);
@ -1178,12 +1097,11 @@ export class IngestBundleRunner {
const transcriptDir = this.deps.storage.resolveTranscriptDir(job.jobId);
const recordTranscriptEntry =
(path: string) =>
(entry: ToolCallLogEntry): MutableToolTranscriptSummary => {
(entry: ToolCallLogEntry): void => {
const current =
transcriptSummaries.get(entry.wuKey) ?? createMutableToolTranscriptSummary(entry.wuKey, path);
recordToolTranscriptEntry(current, entry);
transcriptSummaries.set(entry.wuKey, current);
return current;
};
const overrideReport = await this.loadOverrideReport(job);
@ -1606,8 +1524,7 @@ export class IngestBundleRunner {
try {
await Promise.all(
workUnits.map((wu, index) =>
limitWorkUnit(() =>
this.withRateLimitWorkSlot(ctx?.abortSignal, async () => {
limitWorkUnit(async () => {
const outcome = await runIsolatedWorkUnit({
unitIndex: index,
ingestionBaseSha,
@ -1615,7 +1532,6 @@ export class IngestBundleRunner {
patchDir,
trace: runTrace,
workUnit: wu,
abortSignal: ctx?.abortSignal,
afterSuccess: (child) => copyTransientIngestEvidence(child.workdir, sessionWorktree.workdir),
run: async (child) => {
const scopedWikiService = this.deps.wikiService.forWorktree(child.workdir);
@ -1649,9 +1565,11 @@ export class IngestBundleRunner {
includeContextEvidenceTools: adapter.evidenceIndexing === 'documents' && !!contextReport,
currentTableExists: (tableRef) =>
this.tableRefExistsInSemanticLayer(scopedSemanticLayerService, slConnectionIds, tableRef),
abortSignal: ctx?.abortSignal,
memoryFlow,
wuSkillNames,
onStepFinish: ({ stepIndex, stepBudget }) => {
memoryFlow?.emit({ type: 'work_unit_step', unitKey: wu.unitKey, stepIndex, stepBudget });
},
});
},
});
@ -1676,8 +1594,7 @@ export class IngestBundleRunner {
completedWorkUnits / workUnits.length,
`${completedWorkUnits} of ${workUnits.length} work units complete`,
);
}),
),
}),
),
);
} catch (error) {
@ -1776,7 +1693,6 @@ export class IngestBundleRunner {
reason: context.reason,
maxAttempts: 1,
stepBudget: 12,
abortSignal: ctx?.abortSignal,
});
emitStageProgress(
'integration',
@ -1798,7 +1714,6 @@ export class IngestBundleRunner {
repairKind: 'patch_semantic_gate',
maxAttempts: 1,
stepBudget: 16,
abortSignal: ctx?.abortSignal,
});
emitStageProgress(
'integration',
@ -2023,45 +1938,6 @@ export class IngestBundleRunner {
let curatorWarnings: string[] = [];
let reconcileOutcome: Awaited<ReturnType<typeof runReconciliationStage4>>;
// Reconcile shares the work-unit liveness model: the HUD heartbeat is driven
// by real tool calls (a monotonic, observed count), not a re-derived turn
// counter. The soft cap only paces the phase progress bar; it is never shown
// to the user, so it cannot read as a misleading "X/Y" fraction.
const reconcileTranscriptPath = join(transcriptDir, 'reconcile.jsonl');
const reconcileProgressSoftCap = 40;
const buildReconcileToolSetWithHeartbeat = (): KtxRuntimeToolSet => {
const record = recordTranscriptEntry(reconcileTranscriptPath);
return wrapToolsWithLogger(
buildReconcileToolSet({
loadSkillTool: rcLoadSkill,
stageListTool: rcStageListTool,
stageDiffTool: rcStageDiffTool,
evictionListTool: rcEvictionListTool,
emitConflictResolutionTool: rcEmitConflictResolutionTool,
emitEvictionDecisionTool: rcEmitEvictionDecisionTool,
emitArtifactResolutionTool: rcEmitArtifactResolutionTool,
emitUnmappedFallbackTool: rcEmitUnmappedFallbackTool,
readRawSpanTool: rcRawSpanTool,
toolsetTools: rcToolset.toRuntimeTools(rcToolContext),
}),
reconcileTranscriptPath,
'reconcile',
{
onEntry: (entry) => {
const summary = record(entry);
if (!stage4) {
return;
}
const label = `Reconciling results · ${summary.toolCallCount} action${
summary.toolCallCount === 1 ? '' : 's'
}`;
emitStageProgress('reconciliation', 85, label, { transient: true });
void stage4.updateProgress(Math.min(0.95, summary.toolCallCount / reconcileProgressSoftCap), label);
},
},
);
};
const reconcileStartedAt = Date.now();
const reconcileMode = contextReport && this.deps.curatorPagination ? 'curator' : 'single';
if (contextReport && this.deps.curatorPagination) {
@ -2084,9 +1960,39 @@ export class IngestBundleRunner {
}),
buildUserPrompt: ({ summary, items, runState }) =>
buildReconcileUserPrompt(stageIndex, eviction, { summary, items }, reconcileNotes, runState),
buildToolSet: (_passNumber) => buildReconcileToolSetWithHeartbeat(),
buildToolSet: (_passNumber) =>
wrapToolsWithLogger(
buildReconcileToolSet({
loadSkillTool: rcLoadSkill,
stageListTool: rcStageListTool,
stageDiffTool: rcStageDiffTool,
evictionListTool: rcEvictionListTool,
emitConflictResolutionTool: rcEmitConflictResolutionTool,
emitEvictionDecisionTool: rcEmitEvictionDecisionTool,
emitArtifactResolutionTool: rcEmitArtifactResolutionTool,
emitUnmappedFallbackTool: rcEmitUnmappedFallbackTool,
readRawSpanTool: rcRawSpanTool,
toolsetTools: rcToolset.toRuntimeTools(rcToolContext),
}),
join(transcriptDir, 'reconcile.jsonl'),
'reconcile',
{ onEntry: recordTranscriptEntry(join(transcriptDir, 'reconcile.jsonl')) },
),
getReconciliationActions: () => reconcileActions,
abortSignal: ctx?.abortSignal,
onStepFinish: stage4
? ({ passNumber, stepIndex, stepBudget }) => {
emitStageProgress(
'reconciliation',
85,
`Reconciling results: pass ${passNumber} step ${stepIndex}/${stepBudget}`,
{ transient: true },
);
void stage4.updateProgress(
stepIndex / stepBudget,
`Reconciling results · pass ${passNumber} step ${stepIndex}`,
);
}
: undefined,
});
curatorReport = curatorOutcome.report;
curatorWarnings = curatorOutcome.warnings;
@ -2109,13 +2015,37 @@ export class IngestBundleRunner {
canonicalPins: relevantCanonicalPins,
}),
buildUserPrompt: (idx, ev) => buildReconcileUserPrompt(idx, ev, undefined, reconcileNotes),
buildToolSet: () => buildReconcileToolSetWithHeartbeat(),
buildToolSet: () =>
wrapToolsWithLogger(
buildReconcileToolSet({
loadSkillTool: rcLoadSkill,
stageListTool: rcStageListTool,
stageDiffTool: rcStageDiffTool,
evictionListTool: rcEvictionListTool,
emitConflictResolutionTool: rcEmitConflictResolutionTool,
emitEvictionDecisionTool: rcEmitEvictionDecisionTool,
emitArtifactResolutionTool: rcEmitArtifactResolutionTool,
emitUnmappedFallbackTool: rcEmitUnmappedFallbackTool,
readRawSpanTool: rcRawSpanTool,
toolsetTools: rcToolset.toRuntimeTools(rcToolContext),
}),
join(transcriptDir, 'reconcile.jsonl'),
'reconcile',
{ onEntry: recordTranscriptEntry(join(transcriptDir, 'reconcile.jsonl')) },
),
modelRole: 'reconcile',
stepBudget: 60,
sourceKey: job.sourceKey,
jobId: job.jobId,
force: !!overrideReport,
abortSignal: ctx?.abortSignal,
onStepFinish: stage4
? ({ stepIndex, stepBudget }) => {
emitStageProgress('reconciliation', 85, `Reconciling results: step ${stepIndex}/${stepBudget}`, {
transient: true,
});
void stage4.updateProgress(stepIndex / stepBudget, `Reconciling results · step ${stepIndex}`);
}
: undefined,
});
}
await runTrace.event(
@ -2540,7 +2470,6 @@ export class IngestBundleRunner {
repairKind: 'final_artifact_gate',
maxAttempts: 1,
stepBudget: 16,
abortSignal: ctx?.abortSignal,
});
isolatedDiffSummary.gateRepairAttempts += gateRepair.attempts;

View file

@ -155,103 +155,18 @@ export async function integrateWorkUnitPatch(input: IntegrateWorkUnitPatchInput)
},
);
} catch (semanticError) {
const reason = errorMessage(semanticError);
if (preApplyHead) {
await input.integrationGit.resetHardTo(preApplyHead);
}
await input.trace.event('error', 'integration', 'patch_semantic_conflict_after_textual_resolution', {
unitKey: input.unitKey,
patchPath: input.patchPath,
touchedPaths: textualResolution.changedPaths,
reason,
reason: errorMessage(semanticError),
});
// A textual conflict and a semantic-gate failure can co-occur: the resolver
// reconciles the text but can leave wiki sl_refs pointing at measures the
// merged source no longer defines. Recover via the same gate repair the
// clean-apply branch uses, instead of hard-failing the whole job.
if (input.repairGateFailure) {
const gateRepair = await input.repairGateFailure({
unitKey: input.unitKey,
patchPath: input.patchPath,
touchedPaths: textualResolution.changedPaths,
reason,
});
if (gateRepair.status !== 'failed') {
// The resolver wrote its merge to the worktree (unstaged); the repair
// edited a subset on top. Commit the union so neither is dropped.
const resolvedAndRepairedPaths = [
...new Set([...textualResolution.changedPaths, ...gateRepair.changedPaths]),
].sort();
try {
await traceTimed(
input.trace,
'integration',
'semantic_gate_after_gate_repair',
{ unitKey: input.unitKey, touchedPaths: gateRepair.changedPaths },
async () => {
await input.validateAppliedTree(gateRepair.changedPaths);
},
);
const commit = await input.integrationGit.commitFiles(
resolvedAndRepairedPaths,
`ingest: resolve WorkUnit ${input.unitKey} conflict`,
input.author.name,
input.author.email,
);
if (commit.created) {
await input.trace.event('debug', 'integration', 'patch_accepted_after_textual_resolution', {
unitKey: input.unitKey,
commitSha: commit.commitHash,
touchedPaths: resolvedAndRepairedPaths,
attempts: textualResolution.attempts,
gateRepairAttempts: gateRepair.attempts,
});
return {
status: 'accepted',
commitSha: commit.commitHash,
touchedPaths: resolvedAndRepairedPaths,
textualResolution,
gateRepair,
};
}
} catch (repairValidationError) {
if (preApplyHead) {
await input.integrationGit.resetHardTo(preApplyHead);
}
await input.trace.event('error', 'integration', 'patch_semantic_conflict_after_textual_resolution', {
unitKey: input.unitKey,
patchPath: input.patchPath,
touchedPaths: gateRepair.changedPaths,
reason: errorMessage(repairValidationError),
});
return {
status: 'semantic_conflict',
reason: errorMessage(repairValidationError),
touchedPaths: gateRepair.changedPaths,
textualResolution,
gateRepair,
};
}
}
if (preApplyHead) {
await input.integrationGit.resetHardTo(preApplyHead);
}
return {
status: 'semantic_conflict',
reason: gateRepair.status === 'failed' ? gateRepair.reason : reason,
touchedPaths: textualResolution.changedPaths,
textualResolution,
gateRepair,
};
}
if (preApplyHead) {
await input.integrationGit.resetHardTo(preApplyHead);
}
return {
status: 'semantic_conflict',
reason,
reason: errorMessage(semanticError),
touchedPaths: textualResolution.changedPaths,
textualResolution,
};

View file

@ -19,7 +19,6 @@ export interface ResolveTextualConflictInput {
reason: string;
maxAttempts?: number;
stepBudget?: number;
abortSignal?: AbortSignal;
}
const readIntegrationFileSchema = z.object({
@ -209,7 +208,6 @@ export async function resolveTextualConflict(
jobId: input.trace.context.jobId,
unitKey: input.unitKey,
},
abortSignal: input.abortSignal,
}),
);

View file

@ -14,7 +14,6 @@ export interface RunIsolatedWorkUnitInput {
patchDir: string;
trace: IngestTraceWriter;
workUnit: WorkUnit;
abortSignal?: AbortSignal;
run(child: IngestSessionWorktree): Promise<WorkUnitOutcome>;
afterSuccess?(child: IngestSessionWorktree): Promise<void>;
}

View file

@ -9,7 +9,6 @@ import { DbtSourceAdapter } from './adapters/dbt/dbt.adapter.js';
import { FakeSourceAdapter } from './adapters/fake/fake.adapter.js';
import { HistoricSqlSourceAdapter } from './adapters/historic-sql/historic-sql.adapter.js';
import { PostgresPgssReader } from './adapters/historic-sql/postgres-pgss-reader.js';
import { resolveQueryHistoryScopeFloor } from './adapters/historic-sql/scope-floor.js';
import {
HISTORIC_SQL_SOURCE_KEY,
historicSqlUnifiedPullConfigSchema,
@ -180,39 +179,12 @@ function queryHistoryRecord(connection: unknown): Record<string, unknown> | null
return queryHistory;
}
async function queryHistoryPullConfig(
project: KtxLocalProject,
connectionId: string,
connection: unknown,
): Promise<Record<string, unknown> | null> {
function queryHistoryPullConfig(connection: unknown): Record<string, unknown> | null {
const queryHistory = queryHistoryRecord(connection);
if (queryHistory?.enabled !== true || !isRecord(connection)) return null;
const driver = String(connection.driver ?? '').toLowerCase();
const dialect = historicSqlDialectByDriver.get(driver);
const dialect = historicSqlDialectByDriver.get(String(connection.driver ?? '').toLowerCase());
if (!dialect) return null;
const scopeFloor = await resolveQueryHistoryScopeFloor({
projectDir: project.projectDir,
connectionId,
driver,
connection,
storedQueryHistory: queryHistory,
});
const {
enabled: _enabled,
dialect: _dialect,
enabledTables: _enabledTables,
enabledSchemas: _enabledSchemas,
scopeFloorWarnings: _scopeFloorWarnings,
...stored
} = queryHistory;
return {
...stored,
dialect,
...(scopeFloor.enabledTables.length > 0 ? { enabledTables: scopeFloor.enabledTables } : {}),
...(scopeFloor.enabledSchemas.length > 0 ? { enabledSchemas: scopeFloor.enabledSchemas } : {}),
...(scopeFloor.modeledTableCatalog.length > 0 ? { modeledTableCatalog: scopeFloor.modeledTableCatalog } : {}),
...(scopeFloor.warnings.length > 0 ? { scopeFloorWarnings: scopeFloor.warnings } : {}),
};
return { ...queryHistory, dialect };
}
function stringField(value: unknown): string | null {
@ -273,7 +245,7 @@ export async function localPullConfigForAdapter(
if (options.historicSqlPullConfigOverride) {
return historicSqlUnifiedPullConfigSchema.parse(options.historicSqlPullConfigOverride);
}
const queryHistory = await queryHistoryPullConfig(project, connectionId, connection);
const queryHistory = queryHistoryPullConfig(connection);
if (!queryHistory) {
throw new Error(`Connection "${connectionId}" does not have context.queryHistory.enabled: true`);
}

View file

@ -12,7 +12,6 @@ import type { KtxSemanticLayerComputePort } from '../../context/daemon/semantic-
import { createRuntimeToolDescriptorFromAiTool } from '../../context/llm/runtime-tools.js';
import { createLocalKtxLlmRuntimeFromConfig } from '../../context/llm/local-config.js';
import { KtxIngestEmbeddingPortAdapter } from '../../context/llm/embedding-port.js';
import { createRateLimitGovernorConfig, RateLimitGovernor } from '../../context/llm/rate-limit-governor.js';
import { RuntimeAgentRunner, type AgentRunnerPort, type KtxLlmRuntimePort, type KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
import type { KtxEmbeddingProvider } from '../../llm/types.js';
import type { KtxLocalProject } from '../../context/project/project.js';
@ -612,15 +611,14 @@ function nextLocalJobId(): string {
function localIngestLlmProviderGuardMessage(projectDir: string): string {
return [
'ktx ingest requires llm.provider.backend: anthropic, vertex, gateway, claude-code, or codex, or an injected agentRunner.',
'Configure a local Claude Code/Codex session or API-backed LLM, then rerun ingest:',
'ktx ingest requires llm.provider.backend: anthropic, vertex, gateway, or claude-code, or an injected agentRunner.',
'Configure a local Claude Code session or API-backed LLM, then rerun ingest:',
` ktx setup --project-dir ${projectDir} --llm-backend claude-code --no-input`,
` ktx setup --project-dir ${projectDir} --llm-backend codex --no-input`,
` ktx setup --project-dir ${projectDir} --llm-backend anthropic --anthropic-api-key-env ANTHROPIC_API_KEY --no-input`,
` ktx setup --project-dir ${projectDir} --llm-backend anthropic --anthropic-api-key-env ANTHROPIC_API_KEY --llm-model claude-sonnet-4-6 --no-input`,
].join('\n');
}
function resolveAgentRunner(options: CreateLocalBundleIngestRuntimeOptions, rateLimitGovernor: RateLimitGovernor): {
function resolveAgentRunner(options: CreateLocalBundleIngestRuntimeOptions): {
agentRunner: AgentRunnerPort;
llmRuntime?: KtxLlmRuntimePort;
} {
@ -629,7 +627,6 @@ function resolveAgentRunner(options: CreateLocalBundleIngestRuntimeOptions, rate
(options.createLlmRuntime ?? createLocalKtxLlmRuntimeFromConfig)(options.project.config.llm, {
projectDir: options.project.projectDir,
env: process.env,
rateLimitGovernor,
}) ??
undefined;
@ -679,13 +676,7 @@ export function createLocalBundleIngestRuntime(
const knowledgeIndex = new LocalKnowledgeIndex(options.project, embedding);
const knowledgeEvents = new NoopKnowledgeEventPort();
const wikiService = new KnowledgeWikiService(rootFileStore, embedding, knowledgeIndex, options.project.git, logger);
const rateLimitGovernor = new RateLimitGovernor(
createRateLimitGovernorConfig({
...options.project.config.ingest.rateLimit,
maxConcurrency: options.project.config.ingest.workUnits.maxConcurrency,
}),
);
const { agentRunner, llmRuntime } = resolveAgentRunner(options, rateLimitGovernor);
const { agentRunner, llmRuntime } = resolveAgentRunner(options);
const promptService = new PromptService({ promptsDir, partials: [], logger });
const storage = new LocalIngestStorage(options.project);
const registry = registerAdapters(options.adapters);
@ -725,7 +716,6 @@ export function createLocalBundleIngestRuntime(
workUnitMaxConcurrency: options.project.config.ingest.workUnits.maxConcurrency,
workUnitStepBudget: options.project.config.ingest.workUnits.stepBudget,
workUnitFailureMode: options.project.config.ingest.workUnits.failureMode,
rateLimitGovernor,
profileIngest: options.project.config.ingest.profile,
ingestTraceLevel: ingestTraceLevelFromEnv(),
},

View file

@ -3,7 +3,6 @@ import { cp, mkdir, rm } from 'node:fs/promises';
import { isAbsolute, resolve } from 'node:path';
import type { KtxSqlQueryExecutorPort } from '../../context/connections/query-executor.js';
import type { KtxLogger } from '../../context/core/config.js';
import { createAbortError, isAbortError } from '../../context/core/abort.js';
import type { KtxSemanticLayerComputePort } from '../../context/daemon/semantic-layer-compute.js';
import type { AgentRunnerPort, KtxLlmRuntimePort } from '../../context/llm/runtime-port.js';
import type { KtxLocalProject } from '../../context/project/project.js';
@ -37,7 +36,6 @@ export interface RunLocalIngestOptions {
queryExecutor?: KtxSqlQueryExecutorPort;
logger?: KtxLogger;
embeddingProvider?: import('../../llm/types.js').KtxEmbeddingProvider | null;
abortSignal?: AbortSignal;
}
export interface LocalIngestResult {
@ -125,11 +123,10 @@ function findAdapter(adapters: SourceAdapter[], source: string): SourceAdapter {
return adapter;
}
function localJobContext(jobId: string, memoryFlow?: MemoryFlowEventSink, abortSignal?: AbortSignal): IngestJobContext {
function localJobContext(jobId: string, memoryFlow?: MemoryFlowEventSink): IngestJobContext {
return {
jobId,
...(memoryFlow ? { memoryFlow } : {}),
...(abortSignal ? { abortSignal } : {}),
startPhase() {
return new LocalIngestPhase();
},
@ -161,7 +158,6 @@ async function runScheduledPullJob(options: {
queryExecutor?: KtxSqlQueryExecutorPort;
logger?: KtxLogger;
embeddingProvider?: import('../../llm/types.js').KtxEmbeddingProvider | null;
abortSignal?: AbortSignal;
}): Promise<LocalIngestResult> {
const runtime = createLocalBundleIngestRuntime(options);
const jobId = options.jobId ?? runtime.nextJobId();
@ -173,7 +169,7 @@ async function runScheduledPullJob(options: {
trigger: options.trigger ?? 'manual_resync',
bundleRef: { kind: 'scheduled_pull', config: options.pullConfig },
},
localJobContext(jobId, options.memoryFlow, options.abortSignal),
localJobContext(jobId, options.memoryFlow),
);
const report = await runtime.store.findByJobId(jobId);
if (!report) {
@ -216,7 +212,6 @@ export async function runLocalIngest(options: RunLocalIngestOptions): Promise<Lo
queryExecutor: options.queryExecutor,
logger: options.logger,
embeddingProvider: options.embeddingProvider,
abortSignal: options.abortSignal,
});
}
@ -228,7 +223,7 @@ export async function runLocalIngest(options: RunLocalIngestOptions): Promise<Lo
trigger: options.trigger ?? (options.sourceDir ? 'upload' : 'manual_resync'),
bundleRef,
},
localJobContext(jobId, options.memoryFlow, options.abortSignal),
localJobContext(jobId, options.memoryFlow),
);
const report = await runtime.store.findByJobId(jobId);
if (!report) {
@ -367,9 +362,6 @@ export async function runLocalMetabaseIngest(
const children: LocalMetabaseFanoutChild[] = [];
for (const childPlan of childPlans) {
if (options.abortSignal?.aborted) {
throw createAbortError();
}
const targetConnectionId = safeSegment('target connection id', childPlan.targetConnectionId);
if (!options.project.config.connections[targetConnectionId]) {
throw new Error(`Target connection "${targetConnectionId}" is not configured in ktx.yaml`);
@ -399,12 +391,8 @@ export async function runLocalMetabaseIngest(
queryExecutor: options.queryExecutor,
logger: options.logger,
embeddingProvider: options.embeddingProvider,
abortSignal: options.abortSignal,
});
} catch (error) {
if (isAbortError(error)) {
throw error;
}
child = await recordLocalMetabaseChildFailure({
project: options.project,
jobId: childJobId,

View file

@ -174,7 +174,7 @@ export function ingestReportToMemoryFlowReplay(
const actions = allReportActions(report);
const workUnitEvents: MemoryFlowEvent[] = report.body.workUnits.flatMap((workUnit) => [
{ type: 'work_unit_started', unitKey: workUnit.unitKey, skills: [] } satisfies MemoryFlowEvent,
{ type: 'work_unit_started', unitKey: workUnit.unitKey, skills: [], stepBudget: 0 } satisfies MemoryFlowEvent,
...workUnit.actions.map(
(action): MemoryFlowEvent => ({
type: 'candidate_action',

View file

@ -70,22 +70,17 @@ const memoryFlowEventSchema = z.discriminatedUnion('type', [
message: z.string().min(1),
transient: z.boolean().optional(),
}),
eventSchema({
type: z.literal('rate_limit_wait'),
provider: z.string(),
rateLimitType: z.string().optional(),
resumeAtMs: z.number().int().nonnegative(),
remainingMs: z.number().int().nonnegative(),
}),
eventSchema({
type: z.literal('work_unit_started'),
unitKey: z.string().min(1),
skills: z.array(z.string().min(1)),
stepBudget: z.number().int().min(0),
}),
eventSchema({
type: z.literal('work_unit_step'),
unitKey: z.string().min(1),
toolCalls: z.number().int().min(0),
stepIndex: z.number().int().min(0),
stepBudget: z.number().int().min(0),
}),
eventSchema({
type: z.literal('candidate_action'),

View file

@ -60,22 +60,17 @@ type MemoryFlowEventPayload =
message: string;
transient?: boolean;
}
| {
type: 'rate_limit_wait';
provider: string;
rateLimitType?: string;
resumeAtMs: number;
remainingMs: number;
}
| {
type: 'work_unit_started';
unitKey: string;
skills: string[];
stepBudget: number;
}
| {
type: 'work_unit_step';
unitKey: string;
toolCalls: number;
stepIndex: number;
stepBudget: number;
}
| {
type: 'candidate_action';

View file

@ -5,7 +5,6 @@ import type { KtxFileStorePort } from '../../context/core/file-store.js';
import type { KtxLogger } from '../../context/core/config.js';
import type { SessionOutcome } from '../../context/core/session-worktree.service.js';
import type { AgentRunnerPort, KtxLlmRuntimePort, KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
import type { RateLimitGovernor } from '../llm/rate-limit-governor.js';
import type { MemoryAction, MemoryKnowledgeSlRefsPort } from '../../context/memory/types.js';
import type { PromptService } from '../../context/prompts/prompt.service.js';
import type { SkillsRegistryService } from '../../context/skills/skills-registry.service.js';
@ -145,7 +144,6 @@ interface IngestSettingsPort {
workUnitMaxConcurrency?: number;
workUnitStepBudget?: number;
workUnitFailureMode?: 'abort' | 'continue';
rateLimitGovernor?: RateLimitGovernor;
/** Print a timing breakdown to stderr at the end of each run (config-driven; see also KTX_PROFILE_INGEST). `'json'` emits the raw structured profile. */
profileIngest?: boolean | 'json';
ingestTraceLevel?: IngestTraceLevel;
@ -324,7 +322,7 @@ export interface CuratorPaginationPort {
}) => string;
buildToolSet: (passNumber: number) => KtxRuntimeToolSet;
getReconciliationActions: () => MemoryAction[];
abortSignal?: AbortSignal;
onStepFinish?: (info: { passNumber: number; stepIndex: number; stepBudget: number }) => void;
}): Promise<ReconciliationOutcome & { report: CuratorPaginationReport; warnings: string[] }>;
}

View file

@ -1,5 +1,4 @@
import type { KtxModelRole } from '../../../llm/types.js';
import { isAbortError } from '../../core/abort.js';
import type { AgentRunnerPort, KtxRuntimeToolSet, RunLoopMetrics } from '../../../context/llm/runtime-port.js';
import type { CaptureSession, MemoryAction } from '../../../context/memory/types.js';
import { listTouchedSlSources, type TouchedSlSource } from '../../../context/tools/touched-sl-sources.js';
@ -28,7 +27,7 @@ export interface WorkUnitExecutionDeps {
sourceKey: string;
connectionId: string;
jobId: string;
abortSignal?: AbortSignal;
onStepFinish?: (info: { stepIndex: number; stepBudget: number }) => void;
toolFailureCount?: (unitKey: string) => number;
}
@ -106,12 +105,9 @@ export async function executeWorkUnit(deps: WorkUnitExecutionDeps, wu: WorkUnit)
unitKey: wu.unitKey,
jobId: deps.jobId,
},
abortSignal: deps.abortSignal,
onStepFinish: deps.onStepFinish,
});
} catch (error) {
if (isAbortError(error)) {
throw error;
}
return failWithResetFromCurrentHead(error instanceof Error ? error.message : String(error));
}

View file

@ -15,7 +15,7 @@ export interface ReconciliationContext {
sourceKey: string;
jobId: string;
force?: boolean;
abortSignal?: AbortSignal;
onStepFinish?: (info: { stepIndex: number; stepBudget: number }) => void;
forceRun?: boolean;
}
@ -39,7 +39,7 @@ export async function runReconciliationStage4(ctx: ReconciliationContext): Promi
toolSet: ctx.buildToolSet(),
stepBudget: ctx.stepBudget,
telemetryTags: { operationName: 'ingest-bundle-reconcile', source: ctx.sourceKey, jobId: ctx.jobId },
abortSignal: ctx.abortSignal,
onStepFinish: ctx.onStepFinish,
});
return { skipped: false, stopReason: run.stopReason, error: run.error, ...(run.metrics ? { metrics: run.metrics } : {}) };
}

View file

@ -220,6 +220,5 @@ export interface IngestJobPhase {
export interface IngestJobContext {
jobId: string;
memoryFlow?: MemoryFlowEventSink;
abortSignal?: AbortSignal;
startPhase(weight: number): IngestJobPhase;
}

View file

@ -3,9 +3,7 @@ import type { KtxLlmProvider } from '../../llm/types.js';
import { generateText, Output, stepCountIs, type FlexibleSchema, type TelemetrySettings, type ToolSet } from 'ai';
import type { z } from 'zod';
import { noopLogger, type KtxLogger } from '../../context/core/config.js';
import { isAbortError } from '../core/abort.js';
import { summarizeKtxLlmDebugRequest, type KtxLlmDebugRequestRecorder } from './debug-request-recorder.js';
import type { RateLimitGovernor, RateLimitProvider, RateLimitSignal } from './rate-limit-governor.js';
import { createAiSdkToolSet } from './runtime-tools.js';
import type {
KtxGenerateObjectInput,
@ -42,129 +40,12 @@ export interface AiSdkKtxLlmRuntimeDeps {
telemetry?: AgentTelemetryPort;
logger?: KtxLogger;
debugRequestRecorder?: KtxLlmDebugRequestRecorder;
rateLimitGovernor?: Pick<RateLimitGovernor, 'waitForReady' | 'report' | 'maxRetryAttempts'>;
}
function hasTools(tools: Record<string, unknown>): boolean {
return Object.keys(tools).length > 0;
}
function modelProviderName(model: unknown): RateLimitProvider {
const provider = (model as { provider?: string }).provider ?? '';
return provider.includes('vertex') || provider.includes('google') ? 'vertex' : 'anthropic-api';
}
interface HeaderLimitPair {
limit: string;
remaining: string;
rateLimitType: string;
}
const RATE_LIMIT_HEADER_PAIRS: HeaderLimitPair[] = [
{
limit: 'anthropic-ratelimit-requests-limit',
remaining: 'anthropic-ratelimit-requests-remaining',
rateLimitType: 'rpm',
},
{
limit: 'anthropic-ratelimit-tokens-limit',
remaining: 'anthropic-ratelimit-tokens-remaining',
rateLimitType: 'tpm',
},
{
limit: 'anthropic-ratelimit-input-tokens-limit',
remaining: 'anthropic-ratelimit-input-tokens-remaining',
rateLimitType: 'itpm',
},
{
limit: 'anthropic-ratelimit-output-tokens-limit',
remaining: 'anthropic-ratelimit-output-tokens-remaining',
rateLimitType: 'otpm',
},
{
limit: 'x-ratelimit-limit-requests',
remaining: 'x-ratelimit-remaining-requests',
rateLimitType: 'rpm',
},
{
limit: 'x-ratelimit-limit-tokens',
remaining: 'x-ratelimit-remaining-tokens',
rateLimitType: 'tpm',
},
];
function normalizeHeaders(headers: unknown): Record<string, string> {
if (!headers || typeof headers !== 'object') {
return {};
}
const get = (headers as { get?: unknown }).get;
if (typeof get === 'function') {
const out: Record<string, string> = {};
for (const pair of RATE_LIMIT_HEADER_PAIRS) {
const limit = get.call(headers, pair.limit);
const remaining = get.call(headers, pair.remaining);
if (typeof limit === 'string') out[pair.limit] = limit;
if (typeof remaining === 'string') out[pair.remaining] = remaining;
}
return out;
}
return Object.fromEntries(
Object.entries(headers as Record<string, unknown>)
.filter((entry): entry is [string, string | number] => typeof entry[1] === 'string' || typeof entry[1] === 'number')
.map(([key, value]) => [key.toLowerCase(), String(value)]),
);
}
function numericHeader(headers: Record<string, string>, key: string): number | undefined {
const value = Number(headers[key]);
return Number.isFinite(value) && value >= 0 ? value : undefined;
}
function utilizationForPair(headers: Record<string, string>, pair: HeaderLimitPair): number | undefined {
const limit = numericHeader(headers, pair.limit);
const remaining = numericHeader(headers, pair.remaining);
if (limit === undefined || remaining === undefined || limit <= 0) {
return undefined;
}
return 1 - Math.min(limit, remaining) / limit;
}
function aiSdkHeaderRateLimitSignal(provider: RateLimitProvider, result: unknown): RateLimitSignal | undefined {
const headers = normalizeHeaders((result as { response?: { headers?: unknown } }).response?.headers);
let best: { utilization: number; rateLimitType: string } | undefined;
for (const pair of RATE_LIMIT_HEADER_PAIRS) {
const utilization = utilizationForPair(headers, pair);
if (utilization === undefined) {
continue;
}
if (!best || utilization > best.utilization) {
best = { utilization, rateLimitType: pair.rateLimitType };
}
}
if (!best) {
return undefined;
}
return {
provider,
status: 'allowed',
rateLimitType: best.rateLimitType,
utilization: Number(best.utilization.toFixed(4)),
};
}
function retryAfterMs(error: unknown): number | undefined {
const value = (error as { retryAfter?: unknown }).retryAfter;
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
return value < 1_000 ? value * 1_000 : value;
}
return undefined;
}
function isAiSdkRateLimitError(error: unknown): boolean {
const record = error as { name?: string; statusCode?: number; status?: number };
return record.name === 'TooManyRequestsError' || record.statusCode === 429 || record.status === 429;
}
export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
private readonly logger: KtxLogger;
@ -172,41 +53,6 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
this.logger = deps.logger ?? noopLogger;
}
private async generateTextWithRateLimitRetry<T>(
provider: RateLimitProvider,
abortSignal: AbortSignal | undefined,
run: () => Promise<T>,
): Promise<T> {
// maxRetryAttempts() returns 1 when no governor is present or pacing is
// disabled, so a 429 throws immediately instead of hammering the provider
// with no backoff; the AI SDK's own maxRetries still handles transient 429s.
const maxAttempts = this.deps.rateLimitGovernor?.maxRetryAttempts() ?? 1;
let attempt = 0;
while (true) {
await this.deps.rateLimitGovernor?.waitForReady(abortSignal);
try {
const result = await run();
const signal = aiSdkHeaderRateLimitSignal(provider, result);
if (signal) {
this.deps.rateLimitGovernor?.report(signal);
}
return result;
} catch (error) {
if (isAbortError(error) || !isAiSdkRateLimitError(error) || attempt >= maxAttempts - 1) {
throw error;
}
attempt += 1;
const retryAfter = retryAfterMs(error);
this.deps.rateLimitGovernor?.report({
provider,
status: 'rejected',
rateLimitType: 'http_429',
...(retryAfter !== undefined ? { retryAfterMs: retryAfter } : {}),
});
}
}
}
async generateText(input: KtxGenerateTextInput): Promise<string> {
const model = this.deps.llmProvider.getModel(input.role);
if ((model as { provider?: string }).provider === 'deterministic') {
@ -221,13 +67,12 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
});
const split = splitKtxSystemMessages(built.messages);
const startedAt = Date.now();
const request = {
const result = await generateText({
model,
temperature: input.temperature ?? 0,
...(split.system ? { system: split.system } : {}),
messages: split.messages,
tools: built.tools as ToolSet,
...(input.abortSignal ? { abortSignal: input.abortSignal } : {}),
...(hasTools(tools)
? {
experimental_repairToolCall: this.deps.llmProvider.repairToolCallHandler({
@ -235,8 +80,7 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
}),
}
: {}),
};
const result = await this.generateTextWithRateLimitRetry(modelProviderName(model), input.abortSignal, () => generateText(request));
});
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: toLlmTokenUsage(result.totalUsage ?? result.usage) });
if (typeof result.text !== 'string') {
throw new Error('KTX LLM text generation returned no text');
@ -257,13 +101,12 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
});
const split = splitKtxSystemMessages(built.messages);
const startedAt = Date.now();
const request = {
const result = await generateText({
model,
temperature: input.temperature ?? 0,
...(split.system ? { system: split.system } : {}),
messages: split.messages,
tools: built.tools as ToolSet,
...(input.abortSignal ? { abortSignal: input.abortSignal } : {}),
...(hasTools(tools)
? {
experimental_repairToolCall: this.deps.llmProvider.repairToolCallHandler({
@ -272,8 +115,7 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
}
: {}),
output: Output.object({ schema: input.schema as unknown as FlexibleSchema<TOutput> }),
};
const result = await this.generateTextWithRateLimitRetry(modelProviderName(model), input.abortSignal, () => generateText(request));
});
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: toLlmTokenUsage(result.totalUsage ?? result.usage) });
if (result.output == null) {
throw new Error('KTX LLM object generation returned no output');
@ -310,7 +152,7 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
}),
);
const request = {
const result = await generateText({
model,
temperature: 0,
stopWhen: stepCountIs(params.stepBudget),
@ -321,15 +163,23 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
...(promptMessages.system ? { system: promptMessages.system } : {}),
messages: promptMessages.messages,
tools: built.tools as ToolSet,
...(params.abortSignal ? { abortSignal: params.abortSignal } : {}),
// Count model round-trips locally for metrics. `stepCountIs(stepBudget)`
// caps the loop, so this counter never exceeds the budget.
onStepFinish: () => {
onStepFinish: async () => {
stepIndex += 1;
stepBoundariesMs.push(Date.now() - startedAt);
if (!params.onStepFinish) {
return;
}
try {
await params.onStepFinish({ stepIndex, stepBudget: params.stepBudget });
} catch (err) {
this.logger.warn(
`[agent-runner] onStepFinish callback threw; ignoring: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
},
};
const result = await this.generateTextWithRateLimitRetry(modelProviderName(model), params.abortSignal, () => generateText(request));
});
return {
stopReason: 'natural',
metrics: {
@ -340,9 +190,6 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
},
};
} catch (error) {
if (isAbortError(error)) {
throw error;
}
const err = error instanceof Error ? error : new Error(String(error));
this.logger.warn(`[agent-runner] loop failed: ${err.message}`);
return {

View file

@ -6,10 +6,9 @@ import {
type SDKResultMessage,
} from '@anthropic-ai/claude-agent-sdk';
import { z } from 'zod';
import { createAbortError, isAbortError, throwIfAborted } from '../core/abort.js';
import { noopLogger, type KtxLogger } from '../../context/core/config.js';
import { createKtxClaudeCodeEnv } from './claude-code-env.js';
import { resolveClaudeCodeModel } from './claude-code-models.js';
import type { RateLimitGovernor, RateLimitSignal } from './rate-limit-governor.js';
import { createClaudeSdkTools, mcpToolIds } from './runtime-tools.js';
import type {
KtxGenerateObjectInput,
@ -22,16 +21,7 @@ import type {
RunLoopStopReason,
} from './runtime-port.js';
type QueryResult = AsyncIterable<SDKMessage> & {
interrupt?: () => void | Promise<void>;
};
type QueryFn = (params: Parameters<typeof defaultQuery>[0]) => QueryResult;
interface ClaudeQueryOutcome {
result: SDKResultMessage;
rejectedRateLimitSignal?: RateLimitSignal;
}
type QueryFn = (params: Parameters<typeof defaultQuery>[0]) => AsyncIterable<SDKMessage>;
function claudeTokenUsage(result: SDKResultMessage): LlmTokenUsage {
const usage = (result as { usage?: { input_tokens?: number; output_tokens?: number } }).usage;
@ -52,7 +42,7 @@ export interface ClaudeCodeKtxLlmRuntimeDeps {
modelSlots: { default: string } & Partial<Record<string, string>>;
query?: QueryFn;
env?: NodeJS.ProcessEnv;
rateLimitGovernor?: Pick<RateLimitGovernor, 'waitForReady' | 'report' | 'maxRetryAttempts'>;
logger?: KtxLogger;
}
const BUILTIN_TOOLS = [
@ -83,6 +73,22 @@ function isResult(message: SDKMessage): message is SDKResultMessage {
return message.type === 'result';
}
// Skip emissions the SDK does not count toward `num_turns`: `pause_turn` continuations and
// errored partials (e.g. `max_output_tokens`) it retries internally. Without this, the
// runtime's step counter outruns `maxTurns` and the HUD renders e.g. `step 69/40`.
function countsAsAssistantTurn(message: SDKMessage): boolean {
if (message.type !== 'assistant' || message.parent_tool_use_id !== null) {
return false;
}
if (message.error !== undefined) {
return false;
}
if (message.message.stop_reason === 'pause_turn') {
return false;
}
return true;
}
function resultError(result: SDKResultMessage): Error | undefined {
if (result.subtype === 'success') {
return undefined;
@ -151,74 +157,6 @@ function expectedMcpServerNames(tools: KtxRuntimeToolSet | undefined): Set<strin
return tools && Object.keys(tools).length > 0 ? new Set([KTX_MCP_SERVER_NAME]) : new Set();
}
const CLAUDE_RATE_LIMIT_ERROR_MARKERS = /\b429\b|rate limit|too many requests|quota exceeded|overloaded|max_retries/i;
function normalizeClaudeResetAtMs(value: unknown): number | undefined {
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
return Math.round(value < 10_000_000_000 ? value * 1_000 : value);
}
if (typeof value === 'string') {
const numeric = Number(value);
if (Number.isFinite(numeric) && numeric > 0) {
return normalizeClaudeResetAtMs(numeric);
}
const parsed = Date.parse(value);
return Number.isFinite(parsed) ? parsed : undefined;
}
return undefined;
}
function isClaudeRateLimitResult(result: SDKResultMessage, rejectedSignal: RateLimitSignal | undefined): boolean {
const error = resultError(result);
if (!error) {
return false;
}
if (rejectedSignal?.status === 'rejected') {
return true;
}
const resultDetails = result as {
stop_reason?: unknown;
terminal_reason?: unknown;
errors?: unknown[];
};
const details = [
error.message,
resultDetails.stop_reason,
resultDetails.terminal_reason,
...(resultDetails.errors ?? []),
]
.filter((value): value is string => typeof value === 'string' && value.length > 0)
.join('\n');
return CLAUDE_RATE_LIMIT_ERROR_MARKERS.test(details);
}
function claudeRateLimitSignal(message: SDKMessage): RateLimitSignal | null {
const record = message as unknown as Record<string, unknown>;
if (record.type === 'rate_limit_event') {
const info = record.rate_limit_info as Record<string, unknown> | undefined;
if (!info) return null;
const rawStatus = typeof info.status === 'string' ? info.status : 'allowed';
const resetAtMs = normalizeClaudeResetAtMs(info.resetsAt);
return {
provider: 'claude-subscription',
status: rawStatus === 'rejected' ? 'rejected' : rawStatus === 'allowed_warning' ? 'warning' : 'allowed',
...(resetAtMs !== undefined ? { resetAtMs } : {}),
...(typeof info.rateLimitType === 'string' ? { rateLimitType: info.rateLimitType } : {}),
...(typeof info.utilization === 'number' ? { utilization: info.utilization } : {}),
};
}
if (record.subtype === 'api_retry' || record.type === 'api_retry') {
const retryDelayMs = typeof record.retry_delay_ms === 'number' ? record.retry_delay_ms : undefined;
return {
provider: 'claude-subscription',
status: 'warning',
...(retryDelayMs !== undefined ? { retryAfterMs: retryDelayMs } : {}),
rateLimitType: 'api_retry',
};
}
return null;
}
function managedMcpSettings(serverNames: string[]): NonNullable<Options['managedSettings']> {
return {
allowManagedMcpServersOnly: true,
@ -278,67 +216,31 @@ async function collectResult(params: {
options: Options;
allowedToolIds: Set<string>;
expectedMcpServerNames: Set<string>;
rateLimitGovernor?: Pick<RateLimitGovernor, 'waitForReady' | 'report' | 'maxRetryAttempts'>;
abortSignal?: AbortSignal;
}): Promise<ClaudeQueryOutcome> {
onAssistantTurn?: () => Promise<void>;
}): Promise<SDKResultMessage> {
let result: SDKResultMessage | undefined;
let rejectedRateLimitSignal: RateLimitSignal | undefined;
throwIfAborted(params.abortSignal);
await params.rateLimitGovernor?.waitForReady(params.abortSignal);
throwIfAborted(params.abortSignal);
const queryResult = params.query({ prompt: params.prompt, options: params.options });
const onAbort = () => {
void Promise.resolve(queryResult.interrupt?.()).catch(() => undefined);
};
params.abortSignal?.addEventListener('abort', onAbort, { once: true });
try {
for await (const message of queryResult) {
throwIfAborted(params.abortSignal);
const rateLimitSignal = claudeRateLimitSignal(message);
if (rateLimitSignal) {
if (rateLimitSignal.status === 'rejected') {
rejectedRateLimitSignal = rateLimitSignal;
}
params.rateLimitGovernor?.report(rateLimitSignal);
}
assertInitIsolation(message, params.allowedToolIds, params.expectedMcpServerNames);
if (isResult(message)) {
result = message;
}
for await (const message of params.query({ prompt: params.prompt, options: params.options })) {
assertInitIsolation(message, params.allowedToolIds, params.expectedMcpServerNames);
if (countsAsAssistantTurn(message)) {
await params.onAssistantTurn?.();
}
if (isResult(message)) {
result = message;
}
} finally {
params.abortSignal?.removeEventListener('abort', onAbort);
}
if (params.abortSignal?.aborted) {
throw createAbortError();
}
if (!result) {
throw new Error('Claude Code query returned no result message');
}
return {
result,
...(rejectedRateLimitSignal ? { rejectedRateLimitSignal } : {}),
};
}
async function collectResultWithRateLimitRetry(params: Parameters<typeof collectResult>[0]): Promise<SDKResultMessage> {
// maxRetryAttempts() returns 1 when no governor is present or pacing is
// disabled, so a rate-limited result surfaces without an extra query; the
// Claude Code SDK applies its own backoff for transient rejections.
const maxAttempts = params.rateLimitGovernor?.maxRetryAttempts() ?? 1;
for (let attempt = 0; ; attempt += 1) {
const outcome = await collectResult(params);
if (!isClaudeRateLimitResult(outcome.result, outcome.rejectedRateLimitSignal) || attempt >= maxAttempts - 1) {
return outcome.result;
}
}
return result;
}
export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
private readonly runQuery: QueryFn;
private readonly logger: KtxLogger;
constructor(private readonly deps: ClaudeCodeKtxLlmRuntimeDeps) {
this.runQuery = deps.query ?? defaultQuery;
this.logger = deps.logger ?? noopLogger;
}
async generateText(input: KtxGenerateTextInput): Promise<string> {
@ -350,14 +252,12 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
tools: input.tools,
});
const startedAt = Date.now();
const result = await collectResultWithRateLimitRetry({
const result = await collectResult({
query: this.runQuery,
prompt: [input.system, input.prompt].filter(Boolean).join('\n\n'),
options,
allowedToolIds: new Set(mcpToolIds(input.tools ?? {})),
expectedMcpServerNames: expectedMcpServerNames(input.tools),
rateLimitGovernor: this.deps.rateLimitGovernor,
abortSignal: input.abortSignal,
});
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: claudeTokenUsage(result) });
const error = resultError(result);
@ -389,14 +289,12 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
outputFormat: { type: 'json_schema' as const, schema: jsonSchema(input.schema as z.ZodType) },
};
const startedAt = Date.now();
const result = await collectResultWithRateLimitRetry({
const result = await collectResult({
query: this.runQuery,
prompt: [input.system, input.prompt].filter(Boolean).join('\n\n'),
options,
allowedToolIds: new Set([...mcpToolIds(input.tools ?? {}), STRUCTURED_OUTPUT_TOOL_NAME]),
expectedMcpServerNames: expectedMcpServerNames(input.tools),
rateLimitGovernor: this.deps.rateLimitGovernor,
abortSignal: input.abortSignal,
});
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: claudeTokenUsage(result) });
const error = resultError(result);
@ -410,7 +308,9 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
}
async runAgentLoop(params: RunLoopParams): Promise<RunLoopResult> {
let stepIndex = 0;
const startedAt = Date.now();
const stepBoundariesMs: number[] = [];
try {
const options = baseOptions({
projectDir: this.deps.projectDir,
@ -419,14 +319,28 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
maxTurns: params.stepBudget,
tools: params.toolSet,
});
const result = await collectResultWithRateLimitRetry({
const result = await collectResult({
query: this.runQuery,
prompt: params.userPrompt,
options: { ...options, systemPrompt: params.systemPrompt },
allowedToolIds: new Set(mcpToolIds(params.toolSet)),
expectedMcpServerNames: expectedMcpServerNames(params.toolSet),
rateLimitGovernor: this.deps.rateLimitGovernor,
abortSignal: params.abortSignal,
onAssistantTurn: async () => {
stepIndex += 1;
stepBoundariesMs.push(Date.now() - startedAt);
if (!params.onStepFinish) {
return;
}
try {
await params.onStepFinish({ stepIndex, stepBudget: params.stepBudget });
} catch (error) {
this.logger.warn(
`[claude-code-runner] onStepFinish callback threw; ignoring: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
},
});
const stopReason = mapClaudeCodeStopReason(result);
const error = resultError(result);
@ -435,24 +349,17 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
...(stopReason === 'error' && error ? { error } : {}),
metrics: {
totalMs: Date.now() - startedAt,
// Authoritative turn count from the SDK result. The runtime no longer
// re-derives a per-turn counter: it could not match the SDK's `num_turns`
// and overshot `maxTurns` (the source of the misleading `step 70/40`).
// Per-step boundaries require that counter and are not consumed anywhere.
stepCount: result.num_turns,
stepBoundariesMs: [],
stepCount: stepIndex,
stepBoundariesMs,
usage: claudeTokenUsage(result),
},
};
} catch (error) {
if (isAbortError(error)) {
throw error;
}
const err = error instanceof Error ? error : new Error(String(error));
return {
stopReason: 'error',
error: err,
metrics: { totalMs: Date.now() - startedAt, stepCount: 0, stepBoundariesMs: [], usage: {} },
metrics: { totalMs: Date.now() - startedAt, stepCount: stepIndex, stepBoundariesMs, usage: {} },
};
}
}
@ -481,7 +388,7 @@ export async function runClaudeCodeAuthProbe(input: {
env: input.env,
maxTurns: 1,
});
const result = await collectResultWithRateLimitRetry({
const result = await collectResult({
query: input.query ?? defaultQuery,
prompt: 'Reply with exactly: ok',
options,

View file

@ -1,194 +0,0 @@
import type { LlmTokenUsage, RunLoopStopReason } from './runtime-port.js';
export interface CodexExecEventSummary {
finalText: string;
stopReason: RunLoopStopReason;
usage: LlmTokenUsage;
stepCount: number;
stepBoundariesMs: number[];
toolCallCount: number;
toolFailures: string[];
error?: Error;
}
interface CodexEventParseOptions {
startedAt?: number;
now?: () => number;
}
function record(value: unknown): Record<string, unknown> | undefined {
return value && typeof value === 'object' ? (value as Record<string, unknown>) : undefined;
}
/**
* Codex thread items that represent a discrete agent action consuming one loop
* step. The step budget caps the total number of these regardless of which
* capability the agent reaches for, so built-in `command_execution` (and any
* file/web action the public Codex surface still exposes) count alongside our
* own `mcp_tool_call` items rather than only the MCP ones.
*/
const AGENT_STEP_ITEM_TYPES = new Set(['command_execution', 'mcp_tool_call', 'file_change', 'web_search']);
export function isCompletedAgentStep(event: unknown): boolean {
const eventRecord = record(event);
if (eventRecord?.type !== 'item.completed') {
return false;
}
const itemType = record(eventRecord.item)?.type;
return typeof itemType === 'string' && AGENT_STEP_ITEM_TYPES.has(itemType);
}
function text(value: unknown): string | undefined {
return typeof value === 'string' && value.trim().length > 0 ? value : undefined;
}
function numberValue(value: unknown): number | undefined {
return typeof value === 'number' && Number.isFinite(value) ? value : undefined;
}
function usageFrom(value: unknown): LlmTokenUsage {
const usage = record(value);
if (!usage) {
return {};
}
const inputTokens = numberValue(usage.input_tokens ?? usage.inputTokens);
const outputTokens = numberValue(usage.output_tokens ?? usage.outputTokens);
const explicitTotalTokens = numberValue(usage.total_tokens ?? usage.totalTokens);
const totalTokens =
explicitTotalTokens ??
(inputTokens !== undefined && outputTokens !== undefined ? inputTokens + outputTokens : undefined);
return {
...(inputTokens !== undefined ? { inputTokens } : {}),
...(outputTokens !== undefined ? { outputTokens } : {}),
...(totalTokens !== undefined ? { totalTokens } : {}),
};
}
function stopReasonFrom(value: unknown): RunLoopStopReason {
const reason = text(value)?.toLowerCase();
if (reason && /(budget|max_turn|max-turn|limit)/.test(reason)) {
return 'budget';
}
return 'natural';
}
function errorMessageFrom(value: unknown): string {
if (value instanceof Error) {
return value.message;
}
const asRecord = record(value);
const message = text(asRecord?.message);
return message ?? text(value) ?? 'Codex turn failed';
}
/**
* Codex serializes API failures as a JSON envelope inside the event message
* (e.g. `{"type":"error","status":400,"error":{"message":"…"}}`). Surface the
* human-readable inner message so callers don't leak raw JSON; pass plain
* strings through unchanged.
*/
function unwrapCodexApiErrorMessage(raw: string): string {
const trimmed = raw.trim();
if (!trimmed.startsWith('{')) {
return raw;
}
try {
const parsed = record(JSON.parse(trimmed));
return text(record(parsed?.error)?.message) ?? text(parsed?.message) ?? raw;
} catch {
return raw;
}
}
/** @internal */
export function parseCodexExecEventLine(line: string): unknown {
try {
return JSON.parse(line) as unknown;
} catch (error) {
throw new Error(`Codex JSONL event stream was malformed: ${error instanceof Error ? error.message : String(error)}`);
}
}
export function summarizeCodexExecEvents(
events: Iterable<unknown>,
options: CodexEventParseOptions = {},
): CodexExecEventSummary {
const startedAt = options.startedAt ?? Date.now();
const now = options.now ?? Date.now;
let finalText = '';
let stopReason: RunLoopStopReason = 'natural';
let usage: LlmTokenUsage = {};
let turnCount = 0;
let completedStepCount = 0;
const stepBoundariesMs: number[] = [];
let toolCallCount = 0;
const toolFailures: string[] = [];
let error: Error | undefined;
for (const event of events) {
const eventRecord = record(event);
const eventType = text(eventRecord?.type);
if (!eventRecord || !eventType) {
continue;
}
if (eventType === 'turn.started') {
turnCount += 1;
continue;
}
const item = record(eventRecord.item);
const itemType = text(item?.type);
if (eventType === 'item.started' && itemType === 'mcp_tool_call') {
toolCallCount += 1;
continue;
}
if (isCompletedAgentStep(event)) {
completedStepCount += 1;
stepBoundariesMs.push(now() - startedAt);
// Only MCP tool calls fail the loop: a non-zero `command_execution` exit
// is normal agent exploration, not a runtime error. `status` is the
// authoritative signal (the SDK always sets it); the SDK also serializes
// `error: null` on successful calls, so an explicit-null `error` must NOT
// be read as a failure — only a populated error object counts.
if (itemType === 'mcp_tool_call' && (item?.status === 'failed' || (item?.error !== undefined && item?.error !== null))) {
const name = text(item?.name) ?? text(item?.tool) ?? text(item?.tool_name) ?? 'unknown';
toolFailures.push(`${name}: ${errorMessageFrom(item?.error)}`);
}
continue;
}
if (eventType === 'item.completed' && itemType === 'agent_message') {
finalText = text(item?.text) ?? finalText;
continue;
}
if (eventType === 'turn.completed') {
usage = usageFrom(eventRecord.usage);
if (completedStepCount === 0) {
stepBoundariesMs.push(now() - startedAt);
}
stopReason = stopReasonFrom(eventRecord.reason ?? eventRecord.stop_reason ?? eventRecord.terminal_reason);
continue;
}
if (eventType === 'turn.failed' || eventType === 'error') {
stopReason = 'error';
error = new Error(unwrapCodexApiErrorMessage(errorMessageFrom(eventRecord.error ?? eventRecord.message)));
continue;
}
}
return {
finalText,
stopReason,
usage,
stepCount: completedStepCount > 0 ? completedStepCount : turnCount,
stepBoundariesMs,
toolCallCount,
toolFailures,
...(error ? { error } : {}),
};
}

View file

@ -1,9 +0,0 @@
export const CODEX_ISOLATION_WARNING =
'Codex backend isolation is limited by the public Codex SDK/CLI surface: ktx restricts the runtime MCP server to the current ktx tool set, disables Codex web search, asks for a read-only sandbox, and sets approval_policy=never, but Codex may still load user Codex config and built-in command execution or read-only file capabilities.';
export const CODEX_ISOLATION_WARNING_FIX =
'Use llm.provider.backend: claude-code when you need stricter Claude-Code-style runtime tool isolation, or remove host Codex MCP/tool config before running untrusted prompts through the codex backend.';
export function formatCodexIsolationWarning(): string {
return `${CODEX_ISOLATION_WARNING} ${CODEX_ISOLATION_WARNING_FIX}`;
}

View file

@ -1,87 +0,0 @@
import { randomBytes } from 'node:crypto';
import type { Server } from 'node:http';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import type { KtxMcpServerLike } from '../mcp/types.js';
import { runKtxMcpHttpServer, type KtxMcpHttpServerHandle } from '../../mcp-http-server.js';
import type { KtxRuntimeToolSet } from './runtime-port.js';
import { normalizeKtxRuntimeToolOutput } from './runtime-tools.js';
/** @internal */
export interface CreateCodexRuntimeMcpServerInput {
server?: KtxMcpServerLike;
toolSet: KtxRuntimeToolSet;
}
export interface CodexRuntimeMcpServerHandle {
url: string;
bearerTokenEnvVar: 'KTX_CODEX_RUNTIME_MCP_TOKEN';
bearerToken: string;
close(): Promise<void>;
}
type RunServer = typeof runKtxMcpHttpServer;
export interface StartCodexRuntimeMcpServerInput {
projectDir: string;
toolSet: KtxRuntimeToolSet;
runServer?: RunServer;
}
/** @internal */
export function createCodexRuntimeMcpServer(input: CreateCodexRuntimeMcpServerInput): KtxMcpServerLike {
const server =
input.server ??
(new McpServer({
name: 'ktx-runtime',
version: '0.0.0',
}) as KtxMcpServerLike);
for (const descriptor of Object.values(input.toolSet)) {
server.registerTool(
descriptor.name,
{
description: descriptor.description,
inputSchema: descriptor.inputSchema.shape,
},
async (toolInput) => {
const normalized = normalizeKtxRuntimeToolOutput(await descriptor.execute(toolInput));
return {
content: [{ type: 'text', text: normalized.markdown }],
...(normalized.structured !== undefined && normalized.structured !== null && typeof normalized.structured === 'object'
? { structuredContent: normalized.structured as object }
: {}),
};
},
);
}
return server;
}
function serverPort(server: Server, fallback: number): number {
const address = server.address();
return typeof address === 'object' && address ? address.port : fallback;
}
export async function startCodexRuntimeMcpServer(
input: StartCodexRuntimeMcpServerInput,
): Promise<CodexRuntimeMcpServerHandle> {
const bearerToken = randomBytes(32).toString('hex');
const runServer = input.runServer ?? runKtxMcpHttpServer;
const handle = (await runServer({
projectDir: input.projectDir,
host: '127.0.0.1',
port: 0,
token: bearerToken,
allowedHosts: ['127.0.0.1', 'localhost'],
allowedOrigins: [],
createMcpServer: () => createCodexRuntimeMcpServer({ toolSet: input.toolSet }) as McpServer,
})) as KtxMcpHttpServerHandle;
const port = serverPort(handle.server, 0);
return {
url: `http://127.0.0.1:${port}/mcp`,
bearerTokenEnvVar: 'KTX_CODEX_RUNTIME_MCP_TOKEN',
bearerToken,
close: () => handle.close(),
};
}

View file

@ -1,20 +0,0 @@
export const DEFAULT_CODEX_MODEL = 'gpt-5.5';
const CODEX_MODEL_ALIASES: Record<string, string> = {
codex: DEFAULT_CODEX_MODEL,
default: DEFAULT_CODEX_MODEL,
};
const EXPLICIT_CODEX_MODEL_ID = /^(?:gpt|codex)-[a-z0-9][a-z0-9._-]*$/i;
export function resolveCodexModel(model: string): string {
const normalized = model.trim();
const alias = CODEX_MODEL_ALIASES[normalized];
if (alias) {
return alias;
}
if (EXPLICIT_CODEX_MODEL_ID.test(normalized)) {
return normalized;
}
throw new Error(`Unsupported Codex model "${model}". Use codex, default, or a gpt-* / codex-* model id.`);
}

Some files were not shown because too many files have changed in this diff Show more