mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-10 08:05:14 +02:00
Compare commits
35 commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fd18caa26a | ||
|
|
65de75ebd7 | ||
|
|
0425160857 | ||
|
|
9ff0e86bb8 | ||
|
|
07ab275662 | ||
|
|
7b0023471e | ||
|
|
470802e58e | ||
|
|
66517fc320 | ||
|
|
6b2f7c3365 | ||
|
|
bd3a375081 | ||
|
|
e5425b51a3 | ||
|
|
8050b59f6e | ||
|
|
7ece0b63d3 | ||
|
|
07bbdefa14 | ||
|
|
50dec7bf64 | ||
|
|
48676c74fa | ||
|
|
795a97485a | ||
|
|
5232578d44 | ||
|
|
2c18a62de4 | ||
|
|
2896f9fb91 | ||
|
|
18245c2373 | ||
|
|
0d0ea55184 | ||
|
|
dc39eb7ef9 | ||
|
|
2914407f09 | ||
|
|
bf1fe9748e | ||
|
|
698efdcef8 | ||
|
|
377f21acd7 | ||
|
|
d3e20df1d5 | ||
|
|
d14227468b | ||
|
|
fb7b94b60e | ||
|
|
c3d8cedb0b | ||
|
|
5a8821073b | ||
|
|
ec7edf8f50 | ||
|
|
c2beaf7d55 | ||
|
|
8eb1cd3e79 |
196 changed files with 10815 additions and 2553 deletions
66
AGENTS.md
66
AGENTS.md
|
|
@ -64,6 +64,25 @@ When rules conflict, follow this order:
|
||||||
4. Code quality: types, readable boundaries, focused modules
|
4. Code quality: types, readable boundaries, focused modules
|
||||||
5. Performance where it matters
|
5. Performance where it matters
|
||||||
|
|
||||||
|
## Opinionated Product Defaults
|
||||||
|
|
||||||
|
- **MUST**: Prefer one canonical behavior over configurable alternatives. A new
|
||||||
|
flag, config field, environment variable, mode, strategy option, adapter hook,
|
||||||
|
or fallback path is a product feature and must be justified by an explicit
|
||||||
|
user request or a real correctness requirement.
|
||||||
|
- **MUST NOT**: Add speculative flexibility for imagined users, migrations,
|
||||||
|
review preferences, local workflows, or "just in case" scenarios. If the
|
||||||
|
requested behavior can work with one solid default, implement that default.
|
||||||
|
- **MUST NOT**: Add boolean switches that create two runtime paths unless both
|
||||||
|
paths are essential and the user explicitly asked for the choice. Boolean
|
||||||
|
policy knobs are especially suspect because they double the state space and
|
||||||
|
test surface.
|
||||||
|
- **MUST**: When a design seems to need a new option, first try to remove the
|
||||||
|
need by choosing the stronger default, tightening the invariant, or failing
|
||||||
|
clearly. Ask the user before adding the option if it still seems necessary.
|
||||||
|
- **MUST**: Delete obsolete branches, tests, docs, and config when removing a
|
||||||
|
behavior. Do not preserve dormant compatibility paths.
|
||||||
|
|
||||||
## Repository Shape
|
## Repository Shape
|
||||||
|
|
||||||
**ktx** is a pnpm + uv workspace.
|
**ktx** is a pnpm + uv workspace.
|
||||||
|
|
@ -192,6 +211,19 @@ autonomously — without being asked the leading question — is the bar.
|
||||||
next stack. The only acceptable static patterns are genuinely universal
|
next stack. The only acceptable static patterns are genuinely universal
|
||||||
invariants (e.g. DB-engine system catalogs) and ktx's own self-emitted
|
invariants (e.g. DB-engine system catalogs) and ktx's own self-emitted
|
||||||
signatures.
|
signatures.
|
||||||
|
- **MUST**: Give each capability one implementation and route every caller
|
||||||
|
through it. When some behavior — running a query, resolving a credential or
|
||||||
|
config reference, authenticating, selecting a dialect, loading config —
|
||||||
|
already has a working implementation that some call sites use, make new or
|
||||||
|
divergent call sites depend on that path instead of standing up a second one.
|
||||||
|
Parallel implementations of one capability drift apart silently: a fix, a
|
||||||
|
newly supported input, or an added case lands on one path and not the other,
|
||||||
|
so one entry point (a CLI command, an MCP tool, an ingest stage) succeeds
|
||||||
|
while another fails on the same input. When two paths already do the same
|
||||||
|
job, collapse onto the shared one and delete the duplicate instead of
|
||||||
|
keeping both. When fixing a defect that lives on one path, fix the shared
|
||||||
|
implementation; do not patch the symptom on a forked branch, which preserves
|
||||||
|
the divergence you set out to remove.
|
||||||
- **SHOULD**: Before inventing an abstraction or hand-rolling structural logic,
|
- **SHOULD**: Before inventing an abstraction or hand-rolling structural logic,
|
||||||
search for what already exists and reuse it — the codebase's canonical
|
search for what already exists and reuse it — the codebase's canonical
|
||||||
representation (a structured ref/key type) instead of a parallel string scheme,
|
representation (a structured ref/key type) instead of a parallel string scheme,
|
||||||
|
|
@ -212,12 +244,25 @@ Before presenting a design, answer these explicitly:
|
||||||
instead of building or parsing my own?
|
instead of building or parsing my own?
|
||||||
5. Am I discarding the better option on a weak or misapplied constraint
|
5. Am I discarding the better option on a weak or misapplied constraint
|
||||||
(one-time vs recurring cost, "more surface area", "more work now")?
|
(one-time vs recurring cost, "more surface area", "more work now")?
|
||||||
|
6. Does another entry point already perform this operation through a shared
|
||||||
|
implementation? If so, am I routing through that path instead of forking a
|
||||||
|
parallel one — and if I'm fixing a bug, am I fixing the shared layer rather
|
||||||
|
than one branch?
|
||||||
|
7. Am I adding a user-visible option or alternate runtime path that the user did
|
||||||
|
not ask for? If yes, can one opinionated default solve the problem instead?
|
||||||
|
8. Does this option multiply behavior by caller path, config value, or local
|
||||||
|
state? If yes, remove it unless it is explicitly required.
|
||||||
|
|
||||||
A user question that nudges toward an alternative ("would X help?", "should I
|
A user question that nudges toward an alternative ("would X help?", "should I
|
||||||
always do Y?", "will you hardcode Z?") is a signal that a better option exists.
|
always do Y?", "will you hardcode Z?") is a signal that a better option exists.
|
||||||
Investigate the implied direction and reason it through *before* defending the
|
Investigate the implied direction and reason it through *before* defending the
|
||||||
original proposal — and prefer to have asked yourself the question first.
|
original proposal — and prefer to have asked yourself the question first.
|
||||||
|
|
||||||
|
Example: If generated context changes should be saved, choose one save policy
|
||||||
|
and route ingest, setup, memory, indexing, and docs through it. Do not add an
|
||||||
|
`auto_commit`-style switch unless the user explicitly asks for staged-only runs
|
||||||
|
and accepts the extra runtime path.
|
||||||
|
|
||||||
## TypeScript Standards
|
## TypeScript Standards
|
||||||
|
|
||||||
- Use Node 22+ and pnpm workspace commands.
|
- Use Node 22+ and pnpm workspace commands.
|
||||||
|
|
@ -337,7 +382,8 @@ use `PascalCase` without the suffix.
|
||||||
|
|
||||||
## Telemetry
|
## Telemetry
|
||||||
|
|
||||||
**ktx** ships PostHog usage telemetry. When adding commands or events:
|
**ktx** ships PostHog usage telemetry. Catalog telemetry events use strict
|
||||||
|
schemas. When adding commands or events:
|
||||||
|
|
||||||
- **MUST NOT**: Add fields that carry user data — file paths, hostnames,
|
- **MUST NOT**: Add fields that carry user data — file paths, hostnames,
|
||||||
environment values, SQL text, schema/table/column names, error messages,
|
environment values, SQL text, schema/table/column names, error messages,
|
||||||
|
|
@ -354,6 +400,24 @@ use `PascalCase` without the suffix.
|
||||||
of collected data changes. Adding another event with no new field types
|
of collected data changes. Adding another event with no new field types
|
||||||
needs no docs change.
|
needs no docs change.
|
||||||
|
|
||||||
|
### Error reports
|
||||||
|
|
||||||
|
**ktx** also sends PostHog Error Tracking `$exception` events when telemetry is
|
||||||
|
enabled. This channel is separate from the strict catalog event schema and is
|
||||||
|
used only for exception diagnostics.
|
||||||
|
|
||||||
|
`$exception` events may include stack frames, error class names, raw error
|
||||||
|
messages, cause chains, `source`, `handled`, `fatal`, runtime version fields,
|
||||||
|
OS/runtime fields, and the hashed `projectId` when known. Stack frames may
|
||||||
|
include local file paths and the local username when those appear in paths.
|
||||||
|
|
||||||
|
`$exception` events must never intentionally include secrets, credentials,
|
||||||
|
database URLs, auth headers, raw argv, raw environment values, SQL text,
|
||||||
|
schema/table/column names as explicit properties, customer row data, user prompt
|
||||||
|
text, or raw MCP arguments. Reporters must redact call-site-provided secret
|
||||||
|
snapshots and common static credential patterns before the SDK serializes the
|
||||||
|
exception.
|
||||||
|
|
||||||
## Documentation and Specs
|
## Documentation and Specs
|
||||||
|
|
||||||
- Keep public documentation in `README.md`, package READMEs, example READMEs,
|
- Keep public documentation in `README.md`, package READMEs, example READMEs,
|
||||||
|
|
|
||||||
34
README.md
34
README.md
|
|
@ -13,16 +13,20 @@
|
||||||
<a href="https://docs.kaelio.com/ktx/docs/"><img src="https://img.shields.io/badge/docs-ktx-22c55e?style=flat-square" alt="Documentation" /></a>
|
<a href="https://docs.kaelio.com/ktx/docs/"><img src="https://img.shields.io/badge/docs-ktx-22c55e?style=flat-square" alt="Documentation" /></a>
|
||||||
<a href="https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ"><img src="https://img.shields.io/badge/slack-join%20community-4A154B?style=flat-square&logo=slack&logoColor=white" alt="Join the ktx Slack community" /></a>
|
<a href="https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ"><img src="https://img.shields.io/badge/slack-join%20community-4A154B?style=flat-square&logo=slack&logoColor=white" alt="Join the ktx Slack community" /></a>
|
||||||
<a href="https://github.com/Kaelio/ktx/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
|
<a href="https://github.com/Kaelio/ktx/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
|
||||||
<a href="https://www.ycombinator.com/companies?batch=P25"><img src="https://img.shields.io/badge/Y%20Combinator-P25-orange?style=flat-square" alt="Y Combinator P25" /></a>
|
<a href="https://www.ycombinator.com/companies/kaelio"><img src="https://img.shields.io/badge/Y%20Combinator-P25-orange?style=flat-square" alt="Y Combinator P25" /></a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="https://docs.kaelio.com/ktx/docs/getting-started/quickstart"><b>Quickstart</b></a> ·
|
<a href="https://docs.kaelio.com/ktx/docs/getting-started/quickstart"><b>Quickstart</b></a> ·
|
||||||
<a href="https://docs.kaelio.com/ktx/docs/cli-reference/ktx"><b>CLI Reference</b></a> ·
|
<a href="https://docs.kaelio.com/ktx/docs/cli-reference/ktx"><b>CLI Reference</b></a> ·
|
||||||
<a href="https://docs.kaelio.com/ktx/docs/ai-resources/agent-quickstart"><b>Agent Setup</b></a> ·
|
<a href="https://docs.kaelio.com/ktx/docs/community/ai-resources"><b>Agent Setup</b></a> ·
|
||||||
<a href="https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ"><b>Slack</b></a>
|
<a href="https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ"><b>Slack</b></a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<sub>Built and maintained by <a href="https://www.kaelio.com"><b>Kaelio</b></a></sub>
|
||||||
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
**ktx** is a self-improving context layer that teaches agents how to query your
|
**ktx** is a self-improving context layer that teaches agents how to query your
|
||||||
|
|
@ -139,6 +143,14 @@ Agent integration ready: yes (codex:project)
|
||||||
> If `ktx status` prints `ktx mcp start --project-dir ...`, run it before
|
> If `ktx status` prints `ktx mcp start --project-dir ...`, run it before
|
||||||
> opening your agent client.
|
> opening your agent client.
|
||||||
|
|
||||||
|
## Upgrading
|
||||||
|
|
||||||
|
Re-run the global install with the `@latest` tag:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install -g @kaelio/ktx@latest
|
||||||
|
```
|
||||||
|
|
||||||
## First commands
|
## First commands
|
||||||
|
|
||||||
| Command | Purpose |
|
| Command | Purpose |
|
||||||
|
|
@ -197,7 +209,7 @@ then the current directory. Pass `--project-dir <path>` when scripting.
|
||||||
- [The Context Layer](https://docs.kaelio.com/ktx/docs/concepts/the-context-layer)
|
- [The Context Layer](https://docs.kaelio.com/ktx/docs/concepts/the-context-layer)
|
||||||
- [Building Context](https://docs.kaelio.com/ktx/docs/guides/building-context)
|
- [Building Context](https://docs.kaelio.com/ktx/docs/guides/building-context)
|
||||||
- [CLI Reference](https://docs.kaelio.com/ktx/docs/cli-reference/ktx)
|
- [CLI Reference](https://docs.kaelio.com/ktx/docs/cli-reference/ktx)
|
||||||
- [Agent Quickstart](https://docs.kaelio.com/ktx/docs/ai-resources/agent-quickstart)
|
- [AI Resources](https://docs.kaelio.com/ktx/docs/community/ai-resources)
|
||||||
- [Community & Support](https://docs.kaelio.com/ktx/docs/community/support)
|
- [Community & Support](https://docs.kaelio.com/ktx/docs/community/support)
|
||||||
|
|
||||||
## Community
|
## Community
|
||||||
|
|
@ -247,11 +259,17 @@ uv run pytest -q
|
||||||
|
|
||||||
## Telemetry
|
## Telemetry
|
||||||
|
|
||||||
**ktx** collects anonymous usage telemetry from interactive CLI runs to
|
**ktx** collects privacy-conscious usage telemetry to understand installs and
|
||||||
improve setup, command reliability, and data-agent workflows. No file paths,
|
improve setup, command reliability, and data-agent workflows. Catalog telemetry
|
||||||
hostnames, SQL, schema names, error messages, or argv are recorded. See
|
events do not record file paths, hostnames, SQL, schema names, table names,
|
||||||
[Telemetry](https://docs.kaelio.com/ktx/docs/community/telemetry) for the
|
column names, error messages, raw environment values, or argv. Error reports use
|
||||||
event catalog and opt-out options.
|
PostHog Error Tracking and can include stack frames and raw error messages,
|
||||||
|
which may contain local file paths or the local username in those paths.
|
||||||
|
**ktx** redacts secrets, credentials, database URLs, auth headers, argv, raw
|
||||||
|
environment values, SQL text, row data, and user-typed prompt or MCP argument
|
||||||
|
text from the explicit `$exception` payload. See
|
||||||
|
[Telemetry](https://docs.kaelio.com/ktx/docs/community/telemetry) for the event
|
||||||
|
catalog and opt-out options.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|
|
||||||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 63 KiB After Width: | Height: | Size: 63 KiB |
|
|
@ -2,10 +2,21 @@ import { source } from "@/lib/source";
|
||||||
import { DocsLayout } from "fumadocs-ui/layouts/docs";
|
import { DocsLayout } from "fumadocs-ui/layouts/docs";
|
||||||
import type { ReactNode } from "react";
|
import type { ReactNode } from "react";
|
||||||
import { baseOptions } from "@/app/layout.config";
|
import { baseOptions } from "@/app/layout.config";
|
||||||
|
import { GitHubStars } from "@/components/github-stars";
|
||||||
|
|
||||||
export default function Layout({ children }: { children: ReactNode }) {
|
export default function Layout({ children }: { children: ReactNode }) {
|
||||||
return (
|
return (
|
||||||
<DocsLayout tree={source.pageTree} {...baseOptions}>
|
<DocsLayout
|
||||||
|
tree={source.pageTree}
|
||||||
|
{...baseOptions}
|
||||||
|
sidebar={{
|
||||||
|
banner: (
|
||||||
|
<div className="flex">
|
||||||
|
<GitHubStars />
|
||||||
|
</div>
|
||||||
|
),
|
||||||
|
}}
|
||||||
|
>
|
||||||
{children}
|
{children}
|
||||||
</DocsLayout>
|
</DocsLayout>
|
||||||
);
|
);
|
||||||
|
|
|
||||||
|
|
@ -869,6 +869,147 @@ body::after {
|
||||||
50% { opacity: 0.65; transform: scale(0.9); }
|
50% { opacity: 0.65; transform: scale(0.9); }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ═══════════════════════════════════════════
|
||||||
|
GitHub star widget (navbar)
|
||||||
|
Split pill: GitHub mark + "Star" │ gold star + count.
|
||||||
|
═══════════════════════════════════════════ */
|
||||||
|
.ktx-stars {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: stretch;
|
||||||
|
height: 32px;
|
||||||
|
border-radius: 999px;
|
||||||
|
border: 1px solid var(--color-fd-border);
|
||||||
|
background: color-mix(in oklch, var(--color-fd-card) 72%, transparent);
|
||||||
|
backdrop-filter: blur(8px);
|
||||||
|
-webkit-backdrop-filter: blur(8px);
|
||||||
|
font-family: var(--font-display), var(--font-sans), sans-serif;
|
||||||
|
font-size: 13px;
|
||||||
|
line-height: 1;
|
||||||
|
color: var(--color-fd-foreground);
|
||||||
|
text-decoration: none;
|
||||||
|
overflow: hidden;
|
||||||
|
box-shadow: 0 1px 2px rgba(27, 27, 24, 0.04);
|
||||||
|
transition:
|
||||||
|
transform 0.3s var(--ktx-ease),
|
||||||
|
box-shadow 0.3s var(--ktx-ease),
|
||||||
|
border-color 0.3s ease;
|
||||||
|
animation: ktx-stars-in 0.5s var(--ktx-ease) both;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes ktx-stars-in {
|
||||||
|
from { opacity: 0; transform: translateY(-4px); }
|
||||||
|
to { opacity: 1; transform: translateY(0); }
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars:hover {
|
||||||
|
transform: translateY(-1px);
|
||||||
|
border-color: color-mix(in oklch, var(--color-fd-primary) 45%, var(--color-fd-border));
|
||||||
|
box-shadow:
|
||||||
|
0 6px 18px -8px rgba(14, 116, 144, 0.28),
|
||||||
|
0 1px 2px rgba(27, 27, 24, 0.05);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars:focus-visible {
|
||||||
|
outline: 2px solid var(--color-fd-ring);
|
||||||
|
outline-offset: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.dark .ktx-stars {
|
||||||
|
background: color-mix(in oklch, var(--color-fd-card) 60%, transparent);
|
||||||
|
box-shadow: 0 1px 2px rgba(0, 0, 0, 0.25);
|
||||||
|
}
|
||||||
|
|
||||||
|
.dark .ktx-stars:hover {
|
||||||
|
border-color: rgba(34, 211, 238, 0.4);
|
||||||
|
box-shadow:
|
||||||
|
0 6px 18px -8px rgba(34, 211, 238, 0.3),
|
||||||
|
0 1px 2px rgba(0, 0, 0, 0.3);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars-seg {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 6px;
|
||||||
|
padding: 0 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars-seg--count {
|
||||||
|
border-left: 1px solid var(--color-fd-border);
|
||||||
|
background: color-mix(in oklch, var(--color-fd-primary) 6%, transparent);
|
||||||
|
transition: background 0.3s var(--ktx-ease);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars:hover .ktx-stars-seg--count {
|
||||||
|
background: color-mix(in oklch, var(--color-fd-primary) 12%, transparent);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars-gh {
|
||||||
|
width: 15px;
|
||||||
|
height: 15px;
|
||||||
|
opacity: 0.85;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars-text {
|
||||||
|
font-weight: 500;
|
||||||
|
letter-spacing: -0.01em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars-star {
|
||||||
|
width: 14px;
|
||||||
|
height: 14px;
|
||||||
|
fill: #f5b301;
|
||||||
|
transition: transform 0.3s var(--ktx-ease), filter 0.3s var(--ktx-ease);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars:hover .ktx-stars-star {
|
||||||
|
transform: scale(1.18) rotate(-8deg);
|
||||||
|
filter: drop-shadow(0 1px 4px rgba(245, 179, 1, 0.55));
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars-count {
|
||||||
|
font-weight: 600;
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
color: var(--color-fd-foreground);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Skeleton shown only on the rare cold (uncached) fetch */
|
||||||
|
.ktx-stars--skeleton {
|
||||||
|
animation: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ktx-stars-skeleton-bar {
|
||||||
|
display: inline-block;
|
||||||
|
width: 26px;
|
||||||
|
height: 11px;
|
||||||
|
border-radius: 4px;
|
||||||
|
background: linear-gradient(
|
||||||
|
90deg,
|
||||||
|
var(--color-fd-muted) 25%,
|
||||||
|
color-mix(in oklch, var(--color-fd-muted-foreground) 28%, var(--color-fd-muted)) 50%,
|
||||||
|
var(--color-fd-muted) 75%
|
||||||
|
);
|
||||||
|
background-size: 200% 100%;
|
||||||
|
animation: ktx-stars-shimmer 1.4s ease-in-out infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes ktx-stars-shimmer {
|
||||||
|
from { background-position: 200% 0; }
|
||||||
|
to { background-position: -200% 0; }
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Compact on phones: drop the "Star" word, keep mark + count */
|
||||||
|
@media (max-width: 640px) {
|
||||||
|
.ktx-stars-text { display: none; }
|
||||||
|
.ktx-stars-seg { padding: 0 9px; }
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (prefers-reduced-motion: reduce) {
|
||||||
|
.ktx-stars { animation: none; transition: none; }
|
||||||
|
.ktx-stars:hover { transform: none; }
|
||||||
|
.ktx-stars:hover .ktx-stars-star { transform: none; }
|
||||||
|
.ktx-stars-skeleton-bar { animation: none; }
|
||||||
|
}
|
||||||
|
|
||||||
/* Dot grid */
|
/* Dot grid */
|
||||||
.dot-grid {
|
.dot-grid {
|
||||||
background-image: radial-gradient(
|
background-image: radial-gradient(
|
||||||
|
|
|
||||||
|
|
@ -1,22 +1,13 @@
|
||||||
import type { BaseLayoutProps } from "fumadocs-ui/layouts/shared";
|
import type { BaseLayoutProps } from "fumadocs-ui/layouts/shared";
|
||||||
import { GitHubIcon } from "@/components/github-icon";
|
|
||||||
import { Logo } from "@/components/logo";
|
import { Logo } from "@/components/logo";
|
||||||
import { SlackIcon } from "@/components/slack-icon";
|
import { SlackIcon } from "@/components/slack-icon";
|
||||||
|
|
||||||
export const baseOptions: BaseLayoutProps = {
|
export const baseOptions: BaseLayoutProps = {
|
||||||
nav: {
|
nav: {
|
||||||
title: <Logo />,
|
title: Logo,
|
||||||
transparentMode: "top",
|
transparentMode: "top",
|
||||||
},
|
},
|
||||||
links: [
|
links: [
|
||||||
{
|
|
||||||
type: "icon",
|
|
||||||
label: "GitHub",
|
|
||||||
icon: <GitHubIcon />,
|
|
||||||
text: "GitHub",
|
|
||||||
url: "https://github.com/kaelio/ktx",
|
|
||||||
external: true,
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
type: "icon",
|
type: "icon",
|
||||||
label: "Join the ktx Slack community",
|
label: "Join the ktx Slack community",
|
||||||
|
|
|
||||||
|
|
@ -305,8 +305,8 @@ export const runtimeEdges: Edge[] = [
|
||||||
sourceHandle: "to-context",
|
sourceHandle: "to-context",
|
||||||
target: "context",
|
target: "context",
|
||||||
targetHandle: "in",
|
targetHandle: "in",
|
||||||
type: "default",
|
type: "smoothstep",
|
||||||
label: "search",
|
label: "search + read",
|
||||||
...labelBg,
|
...labelBg,
|
||||||
style: edgeStyle,
|
style: edgeStyle,
|
||||||
markerStart: marker,
|
markerStart: marker,
|
||||||
|
|
@ -318,7 +318,7 @@ export const runtimeEdges: Edge[] = [
|
||||||
sourceHandle: "to-warehouse",
|
sourceHandle: "to-warehouse",
|
||||||
target: "warehouse",
|
target: "warehouse",
|
||||||
targetHandle: "in",
|
targetHandle: "in",
|
||||||
type: "default",
|
type: "smoothstep",
|
||||||
label: "read-only",
|
label: "read-only",
|
||||||
...labelBg,
|
...labelBg,
|
||||||
style: edgeStyle,
|
style: edgeStyle,
|
||||||
|
|
|
||||||
93
docs-site/components/github-stars.tsx
Normal file
93
docs-site/components/github-stars.tsx
Normal file
|
|
@ -0,0 +1,93 @@
|
||||||
|
import { Suspense } from "react";
|
||||||
|
import { GitHubIcon } from "@/components/github-icon";
|
||||||
|
|
||||||
|
const REPO = "kaelio/ktx";
|
||||||
|
const REPO_URL = `https://github.com/${REPO}`;
|
||||||
|
const API_URL = `https://api.github.com/repos/${REPO}`;
|
||||||
|
|
||||||
|
async function fetchStarCount(): Promise<number | null> {
|
||||||
|
try {
|
||||||
|
const res = await fetch(API_URL, {
|
||||||
|
headers: { Accept: "application/vnd.github+json" },
|
||||||
|
// Revalidate hourly. GitHub's unauthenticated REST limit is 60 req/h per
|
||||||
|
// IP, so a single cached server-side fetch keeps the count fresh while
|
||||||
|
// never exposing visitors to rate limits or layout shift.
|
||||||
|
next: { revalidate: 3600 },
|
||||||
|
});
|
||||||
|
if (!res.ok) return null;
|
||||||
|
const data = (await res.json()) as { stargazers_count?: unknown };
|
||||||
|
return typeof data.stargazers_count === "number"
|
||||||
|
? data.stargazers_count
|
||||||
|
: null;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Compact, GitHub-style count: 847 → "847", 1234 → "1.2k", 12345 → "12.3k". */
|
||||||
|
function formatStars(count: number): string {
|
||||||
|
if (count < 1000) return count.toLocaleString("en-US");
|
||||||
|
const thousands = count / 1000;
|
||||||
|
const rounded =
|
||||||
|
thousands >= 100 ? Math.round(thousands) : Math.round(thousands * 10) / 10;
|
||||||
|
return `${rounded}k`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function StarGlyph() {
|
||||||
|
return (
|
||||||
|
<svg className="ktx-stars-star" viewBox="0 0 24 24" aria-hidden="true">
|
||||||
|
<path d="M12 2.6l2.9 5.88 6.49.95-4.7 4.57 1.11 6.46L12 17.4l-5.8 3.06 1.11-6.46-4.7-4.57 6.49-.95z" />
|
||||||
|
</svg>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function StarsContent() {
|
||||||
|
const count = await fetchStarCount();
|
||||||
|
const label =
|
||||||
|
count === null
|
||||||
|
? "Star ktx on GitHub"
|
||||||
|
: `Star ktx on GitHub — ${count.toLocaleString("en-US")} stars`;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<a
|
||||||
|
href={REPO_URL}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
aria-label={label}
|
||||||
|
className="ktx-stars"
|
||||||
|
>
|
||||||
|
<span className="ktx-stars-seg ktx-stars-seg--label">
|
||||||
|
<GitHubIcon className="ktx-stars-gh" />
|
||||||
|
<span className="ktx-stars-text">Star</span>
|
||||||
|
</span>
|
||||||
|
{count !== null && (
|
||||||
|
<span className="ktx-stars-seg ktx-stars-seg--count">
|
||||||
|
<StarGlyph />
|
||||||
|
<span className="ktx-stars-count">{formatStars(count)}</span>
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</a>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function StarsSkeleton() {
|
||||||
|
return (
|
||||||
|
<span className="ktx-stars ktx-stars--skeleton" aria-hidden="true">
|
||||||
|
<span className="ktx-stars-seg ktx-stars-seg--label">
|
||||||
|
<GitHubIcon className="ktx-stars-gh" />
|
||||||
|
<span className="ktx-stars-text">Star</span>
|
||||||
|
</span>
|
||||||
|
<span className="ktx-stars-seg ktx-stars-seg--count">
|
||||||
|
<span className="ktx-stars-skeleton-bar" />
|
||||||
|
</span>
|
||||||
|
</span>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function GitHubStars() {
|
||||||
|
return (
|
||||||
|
<Suspense fallback={<StarsSkeleton />}>
|
||||||
|
<StarsContent />
|
||||||
|
</Suspense>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
@ -1,40 +1,56 @@
|
||||||
export function Logo() {
|
"use client";
|
||||||
|
|
||||||
|
import Link from "next/link";
|
||||||
|
|
||||||
|
const brandFont = {
|
||||||
|
fontFamily: "var(--font-display), var(--font-sans), sans-serif",
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
export function Logo({ href = "/", className }: { href?: string; className?: string }) {
|
||||||
return (
|
return (
|
||||||
<div className="flex items-center gap-3.5 group">
|
<div className={className}>
|
||||||
<div className="relative flex items-center justify-center transition-transform duration-300 ease-out group-hover:rotate-[-4deg]">
|
<div className="flex items-center gap-3.5 group">
|
||||||
<img
|
<Link href={href} aria-label="ktx documentation home" className="flex items-center no-underline">
|
||||||
src="/ktx/brand/ktx-mascot.svg"
|
<span className="relative flex items-center justify-center transition-transform duration-300 ease-out group-hover:rotate-[-4deg]">
|
||||||
alt=""
|
<img
|
||||||
aria-hidden="true"
|
src="/ktx/brand/ktx-mascot.svg"
|
||||||
className="h-20 w-20 object-contain block dark:hidden"
|
alt=""
|
||||||
/>
|
aria-hidden="true"
|
||||||
<img
|
className="h-20 w-20 object-contain block dark:hidden"
|
||||||
src="/ktx/brand/ktx-mascot-dark.svg"
|
/>
|
||||||
alt=""
|
<img
|
||||||
aria-hidden="true"
|
src="/ktx/brand/ktx-mascot-dark.svg"
|
||||||
className="h-20 w-20 object-contain hidden dark:block"
|
alt=""
|
||||||
/>
|
aria-hidden="true"
|
||||||
</div>
|
className="h-20 w-20 object-contain hidden dark:block"
|
||||||
<div className="flex flex-col items-start leading-none">
|
/>
|
||||||
|
</span>
|
||||||
|
</Link>
|
||||||
|
<div className="flex flex-col items-start leading-none">
|
||||||
|
<Link
|
||||||
|
href={href}
|
||||||
|
className="text-[42px] font-semibold text-fd-foreground tracking-tight no-underline"
|
||||||
|
style={brandFont}
|
||||||
|
>
|
||||||
|
ktx
|
||||||
|
</Link>
|
||||||
|
<a
|
||||||
|
href="https://www.kaelio.com"
|
||||||
|
target="_blank"
|
||||||
|
rel="noreferrer"
|
||||||
|
className="mt-1 whitespace-nowrap text-[13px] font-medium text-fd-muted-foreground/80 tracking-tight no-underline transition-colors hover:text-fd-foreground"
|
||||||
|
style={brandFont}
|
||||||
|
>
|
||||||
|
by Kaelio
|
||||||
|
</a>
|
||||||
|
</div>
|
||||||
<span
|
<span
|
||||||
className="text-[42px] font-semibold text-fd-foreground tracking-tight"
|
className="text-[19px] font-medium text-fd-muted-foreground/80 tracking-tight border-l border-fd-border pl-3 ml-1"
|
||||||
style={{ fontFamily: "var(--font-display), var(--font-sans), sans-serif" }}
|
style={brandFont}
|
||||||
>
|
>
|
||||||
ktx
|
Docs
|
||||||
</span>
|
|
||||||
<span
|
|
||||||
className="mt-1 whitespace-nowrap text-[13px] font-medium text-fd-muted-foreground/80 tracking-tight"
|
|
||||||
style={{ fontFamily: "var(--font-display), var(--font-sans), sans-serif" }}
|
|
||||||
>
|
|
||||||
by Kaelio
|
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<span
|
|
||||||
className="text-[19px] font-medium text-fd-muted-foreground/80 tracking-tight border-l border-fd-border pl-3 ml-1"
|
|
||||||
style={{ fontFamily: "var(--font-display), var(--font-sans), sans-serif" }}
|
|
||||||
>
|
|
||||||
Docs
|
|
||||||
</span>
|
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
576
docs-site/components/product-runtime.tsx
Normal file
576
docs-site/components/product-runtime.tsx
Normal file
|
|
@ -0,0 +1,576 @@
|
||||||
|
"use client";
|
||||||
|
|
||||||
|
import {
|
||||||
|
type Edge,
|
||||||
|
type EdgeProps,
|
||||||
|
getSmoothStepPath,
|
||||||
|
Handle,
|
||||||
|
MarkerType,
|
||||||
|
type Node,
|
||||||
|
type NodeProps,
|
||||||
|
Position,
|
||||||
|
} from "@xyflow/react";
|
||||||
|
|
||||||
|
import { FlowCanvas } from "./flow-canvas";
|
||||||
|
|
||||||
|
type AgentNodeData = {
|
||||||
|
title: string;
|
||||||
|
items: string[];
|
||||||
|
};
|
||||||
|
|
||||||
|
type HubNodeData = {
|
||||||
|
title: string;
|
||||||
|
badge: string;
|
||||||
|
rows: string[];
|
||||||
|
};
|
||||||
|
|
||||||
|
type TargetNodeData = {
|
||||||
|
accent: string;
|
||||||
|
title: string;
|
||||||
|
body: string;
|
||||||
|
rows: { text: string; color?: string; mono?: boolean }[];
|
||||||
|
badge?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
type AgentNode = Node<AgentNodeData, "agent">;
|
||||||
|
type HubNode = Node<HubNodeData, "hub">;
|
||||||
|
type TargetNode = Node<TargetNodeData, "target">;
|
||||||
|
type FlowNode = AgentNode | HubNode | TargetNode;
|
||||||
|
|
||||||
|
const AGENT_W = 252;
|
||||||
|
const AGENT_H = 96;
|
||||||
|
const HUB_W = 306;
|
||||||
|
const HUB_H = 190;
|
||||||
|
const TARGET_W = 268;
|
||||||
|
const TARGET_H = 148;
|
||||||
|
|
||||||
|
const CENTER_X = 470;
|
||||||
|
const ROW_AGENT_Y = 0;
|
||||||
|
const ROW_HUB_Y = 196;
|
||||||
|
const ROW_TARGET_Y = 488;
|
||||||
|
|
||||||
|
const AGENT_X = CENTER_X - AGENT_W / 2;
|
||||||
|
const HUB_X = CENTER_X - HUB_W / 2;
|
||||||
|
|
||||||
|
const TARGET_GAP_X = 38;
|
||||||
|
const TARGETS_TOTAL = TARGET_W * 2 + TARGET_GAP_X;
|
||||||
|
const TARGETS_START_X = CENTER_X - TARGETS_TOTAL / 2;
|
||||||
|
const CONTEXT_X = TARGETS_START_X;
|
||||||
|
const WAREHOUSE_X = TARGETS_START_X + TARGET_W + TARGET_GAP_X;
|
||||||
|
|
||||||
|
const EDGE_STROKE = "#94a3b8";
|
||||||
|
const CYCLE_STROKE = "#0e7490";
|
||||||
|
const EMERALD = "#059669";
|
||||||
|
const TEAL = "#0e7490";
|
||||||
|
|
||||||
|
const nodes: FlowNode[] = [
|
||||||
|
{
|
||||||
|
id: "agent",
|
||||||
|
type: "agent",
|
||||||
|
position: { x: AGENT_X, y: ROW_AGENT_Y },
|
||||||
|
data: {
|
||||||
|
title: "Your agent",
|
||||||
|
items: ["Claude Code", "Cursor", "Codex"],
|
||||||
|
},
|
||||||
|
draggable: false,
|
||||||
|
selectable: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "hub",
|
||||||
|
type: "hub",
|
||||||
|
position: { x: HUB_X, y: ROW_HUB_Y },
|
||||||
|
data: {
|
||||||
|
title: "ktx",
|
||||||
|
badge: "MCP + CLI",
|
||||||
|
rows: [
|
||||||
|
"Search wiki + semantic layer",
|
||||||
|
"Return approved metrics",
|
||||||
|
"Compile metrics → SQL",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
draggable: false,
|
||||||
|
selectable: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "context",
|
||||||
|
type: "target",
|
||||||
|
position: { x: CONTEXT_X, y: ROW_TARGET_Y },
|
||||||
|
data: {
|
||||||
|
accent: TEAL,
|
||||||
|
title: "Context layer",
|
||||||
|
body: "Approved definitions agents search before they answer.",
|
||||||
|
rows: [
|
||||||
|
{ text: "wiki/*.md", color: EMERALD, mono: true },
|
||||||
|
{ text: "semantic-layer/*.yaml", color: TEAL, mono: true },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
draggable: false,
|
||||||
|
selectable: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "warehouse",
|
||||||
|
type: "target",
|
||||||
|
position: { x: WAREHOUSE_X, y: ROW_TARGET_Y },
|
||||||
|
data: {
|
||||||
|
accent: "#334155",
|
||||||
|
title: "Database",
|
||||||
|
badge: "read-only",
|
||||||
|
body: "Runs the compiled SQL. ktx never writes to it.",
|
||||||
|
rows: [],
|
||||||
|
},
|
||||||
|
draggable: false,
|
||||||
|
selectable: false,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
const labelBg = {
|
||||||
|
labelBgPadding: [6, 3] as [number, number],
|
||||||
|
labelBgBorderRadius: 4,
|
||||||
|
labelStyle: {
|
||||||
|
fontSize: 13,
|
||||||
|
fontWeight: 600,
|
||||||
|
fill: "var(--color-fd-muted-foreground)",
|
||||||
|
},
|
||||||
|
labelBgStyle: {
|
||||||
|
fill: "var(--color-fd-background)",
|
||||||
|
stroke: "var(--color-fd-border)",
|
||||||
|
strokeWidth: 1,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const requestMarker = {
|
||||||
|
type: MarkerType.ArrowClosed,
|
||||||
|
color: EDGE_STROKE,
|
||||||
|
width: 16,
|
||||||
|
height: 16,
|
||||||
|
};
|
||||||
|
|
||||||
|
const flowEdges: Edge[] = [
|
||||||
|
{
|
||||||
|
id: "e-ask",
|
||||||
|
source: "agent",
|
||||||
|
sourceHandle: "ask",
|
||||||
|
target: "hub",
|
||||||
|
targetHandle: "ask",
|
||||||
|
type: "straight",
|
||||||
|
label: "ask",
|
||||||
|
...labelBg,
|
||||||
|
style: { stroke: EDGE_STROKE, strokeWidth: 1.5 },
|
||||||
|
markerEnd: requestMarker,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "e-answer",
|
||||||
|
source: "hub",
|
||||||
|
sourceHandle: "answer",
|
||||||
|
target: "agent",
|
||||||
|
targetHandle: "answer",
|
||||||
|
type: "straight",
|
||||||
|
label: "answer",
|
||||||
|
...labelBg,
|
||||||
|
style: { stroke: EDGE_STROKE, strokeWidth: 1.5 },
|
||||||
|
markerEnd: requestMarker,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "e-search",
|
||||||
|
source: "hub",
|
||||||
|
sourceHandle: "to-context",
|
||||||
|
target: "context",
|
||||||
|
targetHandle: "in",
|
||||||
|
type: "smoothstep",
|
||||||
|
label: "search + read",
|
||||||
|
...labelBg,
|
||||||
|
style: { stroke: CYCLE_STROKE, strokeWidth: 1.5 },
|
||||||
|
markerStart: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
|
||||||
|
markerEnd: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "e-readonly",
|
||||||
|
source: "hub",
|
||||||
|
sourceHandle: "to-warehouse",
|
||||||
|
target: "warehouse",
|
||||||
|
targetHandle: "in",
|
||||||
|
type: "smoothstep",
|
||||||
|
label: "read-only",
|
||||||
|
...labelBg,
|
||||||
|
style: { stroke: CYCLE_STROKE, strokeWidth: 1.5 },
|
||||||
|
markerStart: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
|
||||||
|
markerEnd: { type: MarkerType.ArrowClosed, color: CYCLE_STROKE, width: 14, height: 14 },
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
function AgentNodeView({ data }: NodeProps<AgentNode>) {
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
style={{ width: AGENT_W, height: AGENT_H }}
|
||||||
|
className="flex flex-col justify-center rounded-md border border-fd-border bg-fd-card px-3.5 py-2.5 shadow-sm"
|
||||||
|
>
|
||||||
|
<Handle
|
||||||
|
id="ask"
|
||||||
|
type="source"
|
||||||
|
position={Position.Bottom}
|
||||||
|
className="!opacity-0"
|
||||||
|
style={{ left: "35%" }}
|
||||||
|
/>
|
||||||
|
<Handle
|
||||||
|
id="answer"
|
||||||
|
type="target"
|
||||||
|
position={Position.Bottom}
|
||||||
|
className="!opacity-0"
|
||||||
|
style={{ left: "65%" }}
|
||||||
|
/>
|
||||||
|
<div className="flex items-center gap-2.5">
|
||||||
|
<span className="flex h-8 w-8 flex-none items-center justify-center rounded-full bg-fd-primary/15 text-fd-primary">
|
||||||
|
<svg
|
||||||
|
xmlns="http://www.w3.org/2000/svg"
|
||||||
|
width="18"
|
||||||
|
height="18"
|
||||||
|
viewBox="0 0 24 24"
|
||||||
|
fill="none"
|
||||||
|
stroke="currentColor"
|
||||||
|
strokeWidth="1.75"
|
||||||
|
strokeLinecap="round"
|
||||||
|
strokeLinejoin="round"
|
||||||
|
aria-hidden="true"
|
||||||
|
>
|
||||||
|
<rect x="3" y="6" width="18" height="12" rx="3" />
|
||||||
|
<circle cx="9" cy="12" r="1.25" fill="currentColor" stroke="none" />
|
||||||
|
<circle cx="15" cy="12" r="1.25" fill="currentColor" stroke="none" />
|
||||||
|
<path d="M12 3v3" />
|
||||||
|
</svg>
|
||||||
|
</span>
|
||||||
|
<p className="text-[17px] font-semibold leading-6 text-fd-foreground">
|
||||||
|
{data.title}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<div className="mt-2 flex flex-wrap gap-1.5">
|
||||||
|
{data.items.map((item) => (
|
||||||
|
<span
|
||||||
|
key={item}
|
||||||
|
className="rounded border border-fd-border bg-fd-background px-1.5 py-0.5 text-[12px] leading-5 text-fd-muted-foreground"
|
||||||
|
>
|
||||||
|
{item}
|
||||||
|
</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function HubNodeView({ data }: NodeProps<HubNode>) {
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
style={{ width: HUB_W, height: HUB_H }}
|
||||||
|
className="relative flex flex-col rounded-md border border-cyan-200/20 bg-[#0f1f23] px-4 py-3.5 text-white shadow-sm dark:bg-[#0b181b]"
|
||||||
|
>
|
||||||
|
<Handle
|
||||||
|
id="ask"
|
||||||
|
type="target"
|
||||||
|
position={Position.Top}
|
||||||
|
className="!opacity-0"
|
||||||
|
style={{ left: "37.5%" }}
|
||||||
|
/>
|
||||||
|
<Handle
|
||||||
|
id="answer"
|
||||||
|
type="source"
|
||||||
|
position={Position.Top}
|
||||||
|
className="!opacity-0"
|
||||||
|
style={{ left: "62.5%" }}
|
||||||
|
/>
|
||||||
|
<Handle
|
||||||
|
id="to-context"
|
||||||
|
type="source"
|
||||||
|
position={Position.Bottom}
|
||||||
|
className="!opacity-0"
|
||||||
|
style={{ left: "44%" }}
|
||||||
|
/>
|
||||||
|
<Handle
|
||||||
|
id="to-warehouse"
|
||||||
|
type="source"
|
||||||
|
position={Position.Bottom}
|
||||||
|
className="!opacity-0"
|
||||||
|
style={{ left: "56%" }}
|
||||||
|
/>
|
||||||
|
<div className="flex items-center gap-2.5">
|
||||||
|
<span className="flex h-7 w-7 flex-none items-center justify-center rounded-md bg-cyan-300/95 font-mono text-sm font-bold text-[#0b1c20]">
|
||||||
|
k
|
||||||
|
</span>
|
||||||
|
<span className="text-[19px] font-bold leading-6 text-white">
|
||||||
|
{data.title}
|
||||||
|
</span>
|
||||||
|
<span className="ml-1 rounded border border-cyan-200/30 bg-white/5 px-1.5 py-0.5 font-mono text-[11px] leading-5 text-cyan-100/85">
|
||||||
|
{data.badge}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div className="mt-3 flex flex-1 flex-col justify-center gap-2">
|
||||||
|
{data.rows.map((row) => (
|
||||||
|
<div key={row} className="flex items-center gap-2.5">
|
||||||
|
<span className="h-1.5 w-1.5 flex-none rounded-full bg-cyan-300/95" />
|
||||||
|
<span className="text-[14px] font-medium leading-5 text-cyan-50/90">
|
||||||
|
{row}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function TargetNodeView({ data }: NodeProps<TargetNode>) {
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
width: TARGET_W,
|
||||||
|
height: TARGET_H,
|
||||||
|
borderTop: `3px solid ${data.accent}`,
|
||||||
|
}}
|
||||||
|
className="overflow-hidden rounded-md border border-fd-border bg-fd-card px-3.5 py-3 shadow-sm"
|
||||||
|
>
|
||||||
|
<Handle id="in" type="target" position={Position.Top} className="!opacity-0" />
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<p className="text-[17px] font-semibold leading-6 text-fd-foreground">
|
||||||
|
{data.title}
|
||||||
|
</p>
|
||||||
|
{data.badge ? (
|
||||||
|
<span
|
||||||
|
className="rounded-full px-1.5 py-0.5 text-[11px] font-semibold leading-5"
|
||||||
|
style={{
|
||||||
|
color: data.accent,
|
||||||
|
background: "color-mix(in oklch, var(--color-fd-card) 86%, #64748b)",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{data.badge}
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
{data.rows.length > 0 ? (
|
||||||
|
<div className="mt-1 flex flex-col gap-0.5">
|
||||||
|
{data.rows.map((row) => (
|
||||||
|
<span
|
||||||
|
key={row.text}
|
||||||
|
className={
|
||||||
|
row.mono
|
||||||
|
? "font-mono text-[13px] font-semibold tracking-tight"
|
||||||
|
: "text-[12px] leading-4 text-fd-muted-foreground"
|
||||||
|
}
|
||||||
|
style={row.color ? { color: row.color } : undefined}
|
||||||
|
>
|
||||||
|
{row.text}
|
||||||
|
</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
<p className="mt-1.5 line-clamp-2 text-[13px] leading-[18px] text-fd-muted-foreground">
|
||||||
|
{data.body}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------- Particles ------------------------------- */
|
||||||
|
|
||||||
|
const PARTICLE_SPEED_PX_PER_SEC = 150;
|
||||||
|
const PARTICLE_MIN_DURATION_SEC = 5;
|
||||||
|
|
||||||
|
type Leg = {
|
||||||
|
sx: number;
|
||||||
|
sy: number;
|
||||||
|
sPos: Position;
|
||||||
|
tx: number;
|
||||||
|
ty: number;
|
||||||
|
tPos: Position;
|
||||||
|
};
|
||||||
|
|
||||||
|
const AGENT_ASK_X = AGENT_X + AGENT_W * 0.35;
|
||||||
|
const AGENT_ANSWER_X = AGENT_X + AGENT_W * 0.65;
|
||||||
|
const AGENT_BOTTOM_Y = ROW_AGENT_Y + AGENT_H;
|
||||||
|
const HUB_ASK_X = HUB_X + HUB_W * 0.375;
|
||||||
|
const HUB_ANSWER_X = HUB_X + HUB_W * 0.625;
|
||||||
|
const HUB_TO_CONTEXT_X = HUB_X + HUB_W * 0.44;
|
||||||
|
const HUB_TO_WAREHOUSE_X = HUB_X + HUB_W * 0.56;
|
||||||
|
const HUB_BOTTOM_Y = ROW_HUB_Y + HUB_H;
|
||||||
|
const CONTEXT_TOP_X = CONTEXT_X + TARGET_W / 2;
|
||||||
|
const WAREHOUSE_TOP_X = WAREHOUSE_X + TARGET_W / 2;
|
||||||
|
|
||||||
|
function buildCyclePath(spokeX: number, targetX: number): {
|
||||||
|
d: string;
|
||||||
|
length: number;
|
||||||
|
} {
|
||||||
|
const legs: Leg[] = [
|
||||||
|
// agent → hub (ask, down)
|
||||||
|
{ sx: AGENT_ASK_X, sy: AGENT_BOTTOM_Y, sPos: Position.Bottom, tx: HUB_ASK_X, ty: ROW_HUB_Y, tPos: Position.Top },
|
||||||
|
// through the hub to its spoke handle (down, drawn behind the hub)
|
||||||
|
{ sx: HUB_ASK_X, sy: ROW_HUB_Y, sPos: Position.Bottom, tx: spokeX, ty: HUB_BOTTOM_Y, tPos: Position.Top },
|
||||||
|
// hub → target (down)
|
||||||
|
{ sx: spokeX, sy: HUB_BOTTOM_Y, sPos: Position.Bottom, tx: targetX, ty: ROW_TARGET_Y, tPos: Position.Top },
|
||||||
|
// target → hub (up)
|
||||||
|
{ sx: targetX, sy: ROW_TARGET_Y, sPos: Position.Top, tx: spokeX, ty: HUB_BOTTOM_Y, tPos: Position.Bottom },
|
||||||
|
// through the hub to its answer handle (up, drawn behind the hub)
|
||||||
|
{ sx: spokeX, sy: HUB_BOTTOM_Y, sPos: Position.Top, tx: HUB_ANSWER_X, ty: ROW_HUB_Y, tPos: Position.Bottom },
|
||||||
|
// hub → agent (answer, up)
|
||||||
|
{ sx: HUB_ANSWER_X, sy: ROW_HUB_Y, sPos: Position.Top, tx: AGENT_ANSWER_X, ty: AGENT_BOTTOM_Y, tPos: Position.Bottom },
|
||||||
|
];
|
||||||
|
|
||||||
|
const segments = legs.map((leg) => {
|
||||||
|
const [segment] = getSmoothStepPath({
|
||||||
|
sourceX: leg.sx,
|
||||||
|
sourceY: leg.sy,
|
||||||
|
sourcePosition: leg.sPos,
|
||||||
|
targetX: leg.tx,
|
||||||
|
targetY: leg.ty,
|
||||||
|
targetPosition: leg.tPos,
|
||||||
|
});
|
||||||
|
return segment;
|
||||||
|
});
|
||||||
|
|
||||||
|
let d = segments[0];
|
||||||
|
for (let i = 1; i < segments.length; i += 1) {
|
||||||
|
d += ` ${segments[i].replace(/^M/, "L")}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const length = legs.reduce(
|
||||||
|
(sum, leg) => sum + Math.abs(leg.tx - leg.sx) + Math.abs(leg.ty - leg.sy),
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
|
||||||
|
return { d, length };
|
||||||
|
}
|
||||||
|
|
||||||
|
type ParticleEdgeData = {
|
||||||
|
d: string;
|
||||||
|
duration: number;
|
||||||
|
beginOffset: number;
|
||||||
|
color: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
type ParticleEdge = Edge<ParticleEdgeData, "particle">;
|
||||||
|
|
||||||
|
function ParticleEdgeView({ id, data }: EdgeProps<ParticleEdge>) {
|
||||||
|
if (!data) return null;
|
||||||
|
const pathId = `runtime-particle-path-${id}`;
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<path id={pathId} d={data.d} fill="none" stroke="none" pointerEvents="none" />
|
||||||
|
<g className="runtime-particle" style={{ color: data.color }}>
|
||||||
|
<circle r={7.5} fill="currentColor" opacity={0.16} />
|
||||||
|
<circle r={3.75} fill="currentColor" opacity={0.32} />
|
||||||
|
<circle r={2.1} fill="currentColor" />
|
||||||
|
<animateMotion
|
||||||
|
dur={`${data.duration.toFixed(2)}s`}
|
||||||
|
begin={`-${data.beginOffset.toFixed(2)}s`}
|
||||||
|
repeatCount="indefinite"
|
||||||
|
>
|
||||||
|
<mpath href={`#${pathId}`} />
|
||||||
|
</animateMotion>
|
||||||
|
</g>
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeCycleEdge(
|
||||||
|
id: string,
|
||||||
|
source: string,
|
||||||
|
spokeX: number,
|
||||||
|
targetX: number,
|
||||||
|
beginFraction: number,
|
||||||
|
): ParticleEdge {
|
||||||
|
const { d, length } = buildCyclePath(spokeX, targetX);
|
||||||
|
const duration = Math.max(
|
||||||
|
PARTICLE_MIN_DURATION_SEC,
|
||||||
|
length / PARTICLE_SPEED_PX_PER_SEC,
|
||||||
|
);
|
||||||
|
return {
|
||||||
|
id,
|
||||||
|
source,
|
||||||
|
target: source,
|
||||||
|
type: "particle",
|
||||||
|
data: { d, duration, beginOffset: duration * beginFraction, color: CYCLE_STROKE },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const particleEdges: ParticleEdge[] = [
|
||||||
|
makeCycleEdge("p-context", "context", HUB_TO_CONTEXT_X, CONTEXT_TOP_X, 0),
|
||||||
|
makeCycleEdge("p-warehouse", "warehouse", HUB_TO_WAREHOUSE_X, WAREHOUSE_TOP_X, 0.5),
|
||||||
|
];
|
||||||
|
|
||||||
|
const nodeTypes = {
|
||||||
|
agent: AgentNodeView,
|
||||||
|
hub: HubNodeView,
|
||||||
|
target: TargetNodeView,
|
||||||
|
};
|
||||||
|
|
||||||
|
const edgeTypes = {
|
||||||
|
particle: ParticleEdgeView,
|
||||||
|
};
|
||||||
|
|
||||||
|
const edges = [...flowEdges, ...particleEdges];
|
||||||
|
|
||||||
|
export function ProductRuntime() {
|
||||||
|
return (
|
||||||
|
<section
|
||||||
|
className="not-prose my-12 w-full max-w-full min-w-0 space-y-5"
|
||||||
|
aria-labelledby="runtime-title"
|
||||||
|
>
|
||||||
|
<div className="max-w-3xl">
|
||||||
|
<h2
|
||||||
|
id="runtime-title"
|
||||||
|
className="text-xl font-semibold tracking-normal text-fd-foreground sm:text-2xl"
|
||||||
|
style={{ fontFamily: "var(--font-display)" }}
|
||||||
|
>
|
||||||
|
How serving works
|
||||||
|
</h2>
|
||||||
|
<p className="mt-3 text-sm leading-6 text-fd-muted-foreground">
|
||||||
|
At runtime, agents reach ktx through MCP. ktx searches the context
|
||||||
|
layer, returns approved metrics, and compiles them into read-only SQL
|
||||||
|
the warehouse runs.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<article
|
||||||
|
className="max-w-full min-w-0 overflow-hidden rounded-lg border border-fd-border bg-fd-card shadow-sm"
|
||||||
|
aria-label="ktx serving flow from an agent request to a governed answer"
|
||||||
|
>
|
||||||
|
<div className="border-b border-fd-border bg-fd-muted/35 px-5 py-4">
|
||||||
|
<p className="text-xs font-semibold uppercase tracking-wide text-fd-primary">
|
||||||
|
Serving flow
|
||||||
|
</p>
|
||||||
|
<h3
|
||||||
|
className="mt-1 text-base font-semibold tracking-normal text-fd-foreground sm:text-lg"
|
||||||
|
style={{ fontFamily: "var(--font-display)" }}
|
||||||
|
>
|
||||||
|
From an agent request to a governed answer
|
||||||
|
</h3>
|
||||||
|
<p className="mt-2 max-w-3xl text-xs leading-5 text-fd-muted-foreground">
|
||||||
|
The agent asks in plain language. ktx is the only thing that touches
|
||||||
|
the context layer and the warehouse, and every database connection
|
||||||
|
is read-only.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<FlowCanvas
|
||||||
|
nodes={nodes}
|
||||||
|
edges={edges}
|
||||||
|
nodeTypes={nodeTypes}
|
||||||
|
edgeTypes={edgeTypes}
|
||||||
|
canvasStyle={{
|
||||||
|
height: "min(620px, 98vw)",
|
||||||
|
minHeight: 430,
|
||||||
|
}}
|
||||||
|
className="runtime-canvas"
|
||||||
|
fitViewOptions={{ padding: 0.06 }}
|
||||||
|
ariaLabel="ktx serving flow diagram"
|
||||||
|
/>
|
||||||
|
</article>
|
||||||
|
<style>{`
|
||||||
|
.runtime-canvas .runtime-particle {
|
||||||
|
pointer-events: none;
|
||||||
|
filter: drop-shadow(0 0 6px currentColor);
|
||||||
|
}
|
||||||
|
@media (prefers-reduced-motion: reduce) {
|
||||||
|
.runtime-canvas .runtime-particle {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
`}</style>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
---
|
|
||||||
title: Agent Instructions
|
|
||||||
description: Suggested instructions for coding assistants that need to read and cite ktx docs.
|
|
||||||
---
|
|
||||||
|
|
||||||
Use these instructions when a coding assistant needs to answer questions from the **ktx** documentation.
|
|
||||||
|
|
||||||
```text
|
|
||||||
When answering ktx docs questions:
|
|
||||||
|
|
||||||
1. Start with https://docs.kaelio.com/ktx/llms.txt.
|
|
||||||
2. Fetch the smallest relevant Markdown page from the index.
|
|
||||||
3. Prefer /docs/<path>.md over rendered HTML.
|
|
||||||
4. Use https://docs.kaelio.com/ktx/llms-full.txt only when the task needs broad docs context.
|
|
||||||
5. Quote commands exactly from docs pages.
|
|
||||||
6. If docs and local repository behavior disagree, say what differs and prefer local verified output for code changes.
|
|
||||||
```
|
|
||||||
|
|
||||||
## What this is for
|
|
||||||
|
|
||||||
This page is for documentation consumption only:
|
|
||||||
|
|
||||||
- answering questions about **ktx**
|
|
||||||
- finding the right docs page
|
|
||||||
- citing setup or CLI guidance
|
|
||||||
- helping an assistant avoid stale or invented commands
|
|
||||||
|
|
||||||
It does not describe local tool configuration.
|
|
||||||
|
|
||||||
## Minimal project prompt
|
|
||||||
|
|
||||||
```text
|
|
||||||
You are helping with ktx. Read https://docs.kaelio.com/ktx/llms.txt first, then fetch only the Markdown pages needed for the task. Do not scrape the rendered docs site when a .md route exists.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Repository prompt
|
|
||||||
|
|
||||||
```text
|
|
||||||
Before editing ktx docs, read /llms.txt and the affected .md docs pages. Keep AI Resources focused on docs consumption. After editing, verify /llms.txt, /llms-full.txt, and any changed .md routes.
|
|
||||||
```
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
---
|
|
||||||
title: Agent Quickstart
|
|
||||||
description: A task-first route for coding agents that need to understand ktx docs.
|
|
||||||
---
|
|
||||||
|
|
||||||
This page is for coding assistants reading or citing the **ktx** docs. It is intentionally limited to documentation lookup, docs navigation, and safe command discovery.
|
|
||||||
|
|
||||||
For Markdown endpoints, use [Markdown Access](/docs/ai-resources/markdown-access).
|
|
||||||
For reusable task prompts, use [Prompt Recipes](/docs/ai-resources/prompt-recipes).
|
|
||||||
To install **ktx** into an agent client, use [Agent Clients](/docs/integrations/agent-clients).
|
|
||||||
|
|
||||||
## First read
|
|
||||||
|
|
||||||
Agents should start with the smallest source that answers the task:
|
|
||||||
|
|
||||||
1. [`/llms.txt`](/llms.txt) - discover the docs and preferred entry points.
|
|
||||||
2. The relevant per-page Markdown URL, for example `/docs/getting-started/quickstart.md`.
|
|
||||||
3. [`/llms-full.txt`](/llms-full.txt) - use only when the task needs broad context across many pages.
|
|
||||||
|
|
||||||
## Task router
|
|
||||||
|
|
||||||
| User asks the agent to explain... | Read first | Then read |
|
|
||||||
|------------------------------------|------------|-----------|
|
|
||||||
| What **ktx** does | [Introduction](/docs/getting-started/introduction) | [The Context Layer](/docs/concepts/the-context-layer) |
|
|
||||||
| How to start from a checkout | [Quickstart](/docs/getting-started/quickstart) | [ktx setup](/docs/cli-reference/ktx-setup) |
|
|
||||||
| How to check project readiness | [ktx status](/docs/cli-reference/ktx-status) | [Quickstart](/docs/getting-started/quickstart) |
|
|
||||||
| How context gets built | [Building Context](/docs/guides/building-context) | [ktx ingest](/docs/cli-reference/ktx-ingest) |
|
|
||||||
| How semantic YAML works | [Writing Context](/docs/guides/writing-context) | [ktx sl](/docs/cli-reference/ktx-sl) |
|
|
||||||
| How machine-readable CLI output is shaped | [ktx sl](/docs/cli-reference/ktx-sl) | [ktx wiki](/docs/cli-reference/ktx-wiki) |
|
|
||||||
|
|
||||||
## Operating workflow
|
|
||||||
|
|
||||||
Use this workflow when the user asks an assistant to answer a **ktx** docs question:
|
|
||||||
|
|
||||||
1. Read [`/llms.txt`](/llms.txt).
|
|
||||||
2. Pick the smallest relevant `.md` page.
|
|
||||||
3. Use [`/llms-full.txt`](/llms-full.txt) only if the answer needs multiple sections of the docs.
|
|
||||||
4. Quote commands exactly from the docs page.
|
|
||||||
5. If a command affects a local project, ask the user before assuming credentials or live services are available.
|
|
||||||
|
|
||||||
## Docs lookup from a shell
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl https://docs.kaelio.com/ktx/llms.txt
|
|
||||||
curl https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
|
|
||||||
```
|
|
||||||
|
|
||||||
## Guardrails
|
|
||||||
|
|
||||||
- Do not invent CLI flags. Fetch the relevant CLI reference page.
|
|
||||||
- Do not scrape rendered HTML when a `.md` route exists.
|
|
||||||
- Do not assume docs lookup requires agent-client configuration.
|
|
||||||
- Do not include credentials or secrets in prompts, URLs, or copied docs snippets.
|
|
||||||
- When docs and local CLI behavior disagree, prefer the local CLI output and mention the mismatch.
|
|
||||||
|
|
@ -1,76 +0,0 @@
|
||||||
---
|
|
||||||
title: Markdown Access
|
|
||||||
description: Fetch ktx docs as llms.txt, llms-full.txt, or per-page Markdown.
|
|
||||||
---
|
|
||||||
|
|
||||||
**ktx** docs are available as plain Markdown so assistants do not need to parse the rendered HTML site.
|
|
||||||
|
|
||||||
## Index
|
|
||||||
|
|
||||||
Fetch the curated index:
|
|
||||||
|
|
||||||
```text
|
|
||||||
https://docs.kaelio.com/ktx/llms.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
Use this file to discover high-value pages, task-specific entry points, and Markdown URLs.
|
|
||||||
|
|
||||||
## Full corpus
|
|
||||||
|
|
||||||
Fetch the complete docs corpus:
|
|
||||||
|
|
||||||
```text
|
|
||||||
https://docs.kaelio.com/ktx/llms-full.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
Use this when an assistant needs broad context across setup, concepts, CLI reference, integrations, and troubleshooting. Prefer the smaller per-page Markdown route for narrow tasks.
|
|
||||||
|
|
||||||
## Per-page Markdown
|
|
||||||
|
|
||||||
Every docs page has a Markdown route:
|
|
||||||
|
|
||||||
```text
|
|
||||||
https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
|
|
||||||
https://docs.kaelio.com/ktx/docs/cli-reference/ktx-sl.md
|
|
||||||
https://docs.kaelio.com/ktx/docs/cli-reference/ktx-wiki.md
|
|
||||||
https://docs.kaelio.com/ktx/docs/guides/building-context.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Requests that ask for Markdown can also use the normal docs URL with `Accept: text/markdown`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -H "Accept: text/markdown" https://docs.kaelio.com/ktx/docs/getting-started/quickstart
|
|
||||||
```
|
|
||||||
|
|
||||||
## Recommended retrieval order
|
|
||||||
|
|
||||||
1. Fetch `/llms.txt`.
|
|
||||||
2. Select one or two relevant page Markdown URLs.
|
|
||||||
3. Fetch `/llms-full.txt` only when page-level docs are not enough.
|
|
||||||
|
|
||||||
## Output contract
|
|
||||||
|
|
||||||
Markdown responses are designed for agent consumption:
|
|
||||||
|
|
||||||
- Frontmatter is removed.
|
|
||||||
- Each page includes a title, description, canonical URL, and Markdown URL.
|
|
||||||
- Code blocks stay as code blocks.
|
|
||||||
- Tables stay as Markdown tables.
|
|
||||||
- Missing docs pages return a plain-text `404` instead of silently falling back to HTML.
|
|
||||||
|
|
||||||
## Page actions
|
|
||||||
|
|
||||||
Rendered docs pages include page-level actions near the title:
|
|
||||||
|
|
||||||
- **Copy MD** copies the generated Markdown for the current page.
|
|
||||||
- **View MD** opens the generated Markdown route.
|
|
||||||
- **Copy MDX** copies the source MDX for the current page.
|
|
||||||
|
|
||||||
## Common mistakes
|
|
||||||
|
|
||||||
| Mistake | Better path |
|
|
||||||
|---------|-------------|
|
|
||||||
| Scraping the HTML page for a docs answer | Fetch the `.md` route instead |
|
|
||||||
| Loading `/llms-full.txt` for a single CLI flag lookup | Fetch the relevant CLI reference page |
|
|
||||||
| Treating `/llms.txt` as complete documentation | Use it as an index, then fetch linked pages |
|
|
||||||
| Copying rendered text by hand | Use **Copy MD** or **Copy MDX** from the page actions |
|
|
||||||
|
|
@ -1,10 +0,0 @@
|
||||||
{
|
|
||||||
"title": "AI Resources",
|
|
||||||
"defaultOpen": true,
|
|
||||||
"pages": [
|
|
||||||
"agent-quickstart",
|
|
||||||
"markdown-access",
|
|
||||||
"agent-instructions",
|
|
||||||
"prompt-recipes"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
@ -1,55 +0,0 @@
|
||||||
---
|
|
||||||
title: Prompt Recipes
|
|
||||||
description: Copyable prompts for common ktx agent workflows.
|
|
||||||
---
|
|
||||||
|
|
||||||
Use these prompts when asking a coding assistant to work with **ktx**. Replace project names, connection ids, and business terms with your own values.
|
|
||||||
|
|
||||||
## Learn the docs
|
|
||||||
|
|
||||||
```text
|
|
||||||
Read https://docs.kaelio.com/ktx/llms.txt first. Then fetch only the ktx Markdown pages needed for this task. Do not scrape rendered HTML unless no Markdown route exists.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Set up a project
|
|
||||||
|
|
||||||
```text
|
|
||||||
Run npx skills add Kaelio/ktx --skill ktx and use the ktx skill to install
|
|
||||||
and configure ktx in this project.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Find a command
|
|
||||||
|
|
||||||
```text
|
|
||||||
Find the correct ktx command for this task: <task>. Start with /llms.txt, then fetch the smallest relevant CLI reference .md page. Quote the exact command and flags from the docs.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Explain setup
|
|
||||||
|
|
||||||
```text
|
|
||||||
Explain how to set up ktx for this repo. Read /docs/getting-started/quickstart.md and the relevant CLI reference pages. Summarize prerequisites, commands, generated files, and any credentials the user must provide manually.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Compare concepts
|
|
||||||
|
|
||||||
```text
|
|
||||||
Explain the difference between these ktx concepts: <concepts>. Start from /llms.txt, fetch the relevant concept and guide pages as Markdown, and answer with links to the source pages.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Review semantic changes
|
|
||||||
|
|
||||||
```text
|
|
||||||
Review the ktx semantic-layer and knowledge changes in this branch. Check that measures have clear definitions, joins use valid keys, hidden/internal columns are not exposed to agents, and validation passes. List concrete file and line issues first.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Copy exact docs source
|
|
||||||
|
|
||||||
```text
|
|
||||||
Open the relevant ktx docs page and use the page action to copy the generated Markdown or source MDX. Preserve code fences and tables exactly.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Update docs
|
|
||||||
|
|
||||||
```text
|
|
||||||
Update the ktx docs for agent readability. Keep AI Resources focused on docs consumption. After editing, verify /llms.txt, /llms-full.txt, and the affected .md routes.
|
|
||||||
```
|
|
||||||
|
|
@ -177,7 +177,9 @@ Slowest phase: reconciliation (2m 05s, 48% of wall time). 2 work units (1 failed
|
||||||
|
|
||||||
Work units run serially by default (`ingest.workUnits.maxConcurrency` is `1`);
|
Work units run serially by default (`ingest.workUnits.maxConcurrency` is `1`);
|
||||||
raise it in `ktx.yaml` if the profile shows the run is bound by serialized
|
raise it in `ktx.yaml` if the profile shows the run is bound by serialized
|
||||||
work-unit agent loops.
|
work-unit agent loops. If the provider reports an LLM rate limit, **ktx** shows
|
||||||
|
a transient wait message and temporarily reduces effective work-unit concurrency
|
||||||
|
according to `ingest.rateLimit`.
|
||||||
|
|
||||||
## Common errors
|
## Common errors
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -54,7 +54,6 @@ prompts.
|
||||||
| `--llm-backend <backend>` | LLM backend: `anthropic`, `vertex`, `claude-code`, or `codex` |
|
| `--llm-backend <backend>` | LLM backend: `anthropic`, `vertex`, `claude-code`, or `codex` |
|
||||||
| `--llm-backend claude-code` | Use the local Claude Code session for **ktx** LLM calls |
|
| `--llm-backend claude-code` | Use the local Claude Code session for **ktx** LLM calls |
|
||||||
| `--llm-backend codex` | Use local Codex authentication for **ktx** LLM calls |
|
| `--llm-backend codex` | Use local Codex authentication for **ktx** LLM calls |
|
||||||
| `--llm-model <model>` | LLM model ID or backend model alias to validate and save |
|
|
||||||
| `--anthropic-api-key-env <name>` | Environment variable containing the Anthropic API key |
|
| `--anthropic-api-key-env <name>` | Environment variable containing the Anthropic API key |
|
||||||
| `--anthropic-api-key-file <path>` | File containing the Anthropic API key |
|
| `--anthropic-api-key-file <path>` | File containing the Anthropic API key |
|
||||||
| `--vertex-project <project>` | Vertex AI project ID, `env:NAME`, or `file:/path` reference |
|
| `--vertex-project <project>` | Vertex AI project ID, `env:NAME`, or `file:/path` reference |
|
||||||
|
|
@ -64,13 +63,16 @@ prompts.
|
||||||
Choose only one Anthropic credential source. Anthropic credential flags are only
|
Choose only one Anthropic credential source. Anthropic credential flags are only
|
||||||
valid with the Anthropic backend; Vertex flags are only valid with the Vertex
|
valid with the Anthropic backend; Vertex flags are only valid with the Vertex
|
||||||
backend. The `claude-code` and `codex` backends use local authentication instead
|
backend. The `claude-code` and `codex` backends use local authentication instead
|
||||||
of Anthropic API key or Vertex flags. For Claude Code, `--llm-model` accepts
|
of Anthropic API key or Vertex flags. After you choose a backend, `ktx setup`
|
||||||
`sonnet`, `opus`, `haiku`, or a full Claude model ID. For Codex, `--llm-model`
|
writes that backend's per-role model preset to `ktx.yaml`. To change a model,
|
||||||
accepts `codex`, `default`, or a `gpt-*` / `codex-*` model ID such as
|
edit the matching `llm.models.<role>` value in `ktx.yaml`.
|
||||||
`gpt-5.5`; any other value is rejected before the auth probe. Run `codex` to
|
|
||||||
see the models available to your login, and pick a `gpt-*` / `codex-*` id from
|
With `--no-input`, `ktx setup` does not assume a default LLM provider, because
|
||||||
that list. Note that `*-codex` API-billing model IDs (for example
|
every backend needs credentials only you can supply. Pass `--llm-backend`
|
||||||
`gpt-5.3-codex`) are not available to ChatGPT-subscription logins.
|
explicitly. Note that `--target` selects the agent integration, not the LLM
|
||||||
|
provider: `ktx setup --target claude-code --no-input` still needs
|
||||||
|
`--llm-backend claude-code` to use your Claude subscription for **ktx** LLM
|
||||||
|
calls.
|
||||||
|
|
||||||
### Embeddings
|
### Embeddings
|
||||||
|
|
||||||
|
|
@ -198,14 +200,13 @@ ktx setup
|
||||||
# Run setup for a specific project directory
|
# Run setup for a specific project directory
|
||||||
ktx setup --project-dir ./analytics
|
ktx setup --project-dir ./analytics
|
||||||
|
|
||||||
# Use Claude Code with Opus for ktx LLM calls
|
# Use Claude Code for ktx LLM calls
|
||||||
ktx setup \
|
ktx setup \
|
||||||
--project-dir ./analytics \
|
--project-dir ./analytics \
|
||||||
--llm-backend claude-code \
|
--llm-backend claude-code
|
||||||
--llm-model opus
|
|
||||||
|
|
||||||
# Configure **ktx** to use local Codex authentication for LLM work
|
# Configure **ktx** to use local Codex authentication for LLM work
|
||||||
ktx setup --llm-backend codex --llm-model gpt-5.5 --no-input
|
ktx setup --llm-backend codex --no-input
|
||||||
```
|
```
|
||||||
|
|
||||||
When you choose `--llm-backend codex`, setup prints a warning if the public
|
When you choose `--llm-backend codex`, setup prints a warning if the public
|
||||||
|
|
@ -282,6 +283,7 @@ Use `ktx status` for repeatable readiness checks after setup exits.
|
||||||
|-------|-------|----------|
|
|-------|-------|----------|
|
||||||
| Setup resumes an unexpected project | `KTX_PROJECT_DIR` or nearest `ktx.yaml` points to another directory | Pass `--project-dir <path>` explicitly |
|
| Setup resumes an unexpected project | `KTX_PROJECT_DIR` or nearest `ktx.yaml` points to another directory | Pass `--project-dir <path>` explicitly |
|
||||||
| Setup cannot run in CI | Required values are missing and `--no-input` disables prompts | Provide the relevant automation flags or create a fixture `ktx.yaml` |
|
| Setup cannot run in CI | Required values are missing and `--no-input` disables prompts | Provide the relevant automation flags or create a fixture `ktx.yaml` |
|
||||||
|
| `Missing LLM backend: pass --llm-backend …` | `--no-input` setup ran without an LLM backend; `--target` does not select one | Pass `--llm-backend claude-code`, `codex`, `anthropic`, or `vertex` (with that backend's credential flags) |
|
||||||
| Provider health check fails | Provider key, model id, Vertex project, or Vertex location is invalid | Fix the `env:` or `file:` reference and rerun setup |
|
| Provider health check fails | Provider key, model id, Vertex project, or Vertex location is invalid | Fix the `env:` or `file:` reference and rerun setup |
|
||||||
| Python runtime is missing | The selected setup needs runtime-backed agent, query-history, Looker, or local embedding features | Accept the interactive prompt, rerun with `--yes`, or run the suggested `ktx admin runtime install` command |
|
| Python runtime is missing | The selected setup needs runtime-backed agent, query-history, Looker, or local embedding features | Accept the interactive prompt, rerun with `--yes`, or run the suggested `ktx admin runtime install` command |
|
||||||
| `--enable-query-history` is rejected | The selected database driver does not support query history | Use Postgres, BigQuery, or Snowflake, or rerun without query-history flags |
|
| `--enable-query-history` is rejected | The selected database driver does not support query history | Use Postgres, BigQuery, or Snowflake, or rerun without query-history flags |
|
||||||
|
|
|
||||||
|
|
@ -74,6 +74,56 @@ The public context-build entrypoint is `ktx ingest [connectionId]` or
|
||||||
| `-v`, `--version` | Show the CLI package name and version. |
|
| `-v`, `--version` | Show the CLI package name and version. |
|
||||||
| `-h`, `--help` | Show help for the current command. |
|
| `-h`, `--help` | Show help for the current command. |
|
||||||
|
|
||||||
|
## Update notices
|
||||||
|
|
||||||
|
> **Note:** The update notifier writes only to stderr and keeps command stdout
|
||||||
|
> unchanged.
|
||||||
|
|
||||||
|
When a newer package is available on your installed release channel, `ktx`
|
||||||
|
prints a short notice after the command finishes:
|
||||||
|
|
||||||
|
```text
|
||||||
|
↑ Update available: ktx 0.9.0 → 0.10.0
|
||||||
|
npm i -g @kaelio/ktx
|
||||||
|
```
|
||||||
|
|
||||||
|
Stable installs compare against the npm `latest` dist-tag.
|
||||||
|
Release-candidate installs compare against the `next` dist-tag and show:
|
||||||
|
|
||||||
|
```text
|
||||||
|
npm i -g @kaelio/ktx@next
|
||||||
|
```
|
||||||
|
|
||||||
|
The check is skipped for JSON output, CI, non-TTY stdout, and hidden completion
|
||||||
|
commands. To opt out explicitly, set any of these environment variables:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
KTX_NO_UPDATE_CHECK=1
|
||||||
|
NO_UPDATE_NOTIFIER=1
|
||||||
|
DO_NOT_TRACK=1
|
||||||
|
```
|
||||||
|
|
||||||
|
The `ktx` CLI prints one npm command because globally installed binaries don't
|
||||||
|
expose a reliable runtime package-manager signal. If you prefer another global
|
||||||
|
package manager, use the equivalent command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm add -g @kaelio/ktx
|
||||||
|
yarn global add @kaelio/ktx
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build-view star prompt
|
||||||
|
|
||||||
|
During an interactive context build, `ktx setup` and `ktx ingest` can show a dim
|
||||||
|
GitHub star reminder above the `Ctrl+C to stop` hint. **ktx** skips this prompt
|
||||||
|
for CI, non-TTY output, and `DO_NOT_TRACK=1`.
|
||||||
|
|
||||||
|
To suppress only this prompt while keeping other notices enabled, set:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
KTX_NO_STAR=1
|
||||||
|
```
|
||||||
|
|
||||||
## Project resolution
|
## Project resolution
|
||||||
|
|
||||||
Most commands are project-aware. Pass `--project-dir <path>` when scripting or
|
Most commands are project-aware. Pass `--project-dir <path>` when scripting or
|
||||||
|
|
|
||||||
111
docs-site/content/docs/community/ai-resources.mdx
Normal file
111
docs-site/content/docs/community/ai-resources.mdx
Normal file
|
|
@ -0,0 +1,111 @@
|
||||||
|
---
|
||||||
|
title: AI Resources
|
||||||
|
description: How coding agents read, cite, and act on the ktx docs - Markdown endpoints, a task router, and copy-paste prompts.
|
||||||
|
---
|
||||||
|
|
||||||
|
This page is for coding assistants that read or cite the **ktx** docs. It covers
|
||||||
|
the machine-readable endpoints, a task router for common questions, and
|
||||||
|
copy-paste prompts. It is scoped to documentation lookup and safe command
|
||||||
|
discovery - to wire **ktx** into an agent client, see
|
||||||
|
[Agent Clients](/docs/integrations/agent-clients).
|
||||||
|
|
||||||
|
## Markdown endpoints
|
||||||
|
|
||||||
|
**ktx** docs are available as plain Markdown so assistants never have to parse
|
||||||
|
the rendered HTML site.
|
||||||
|
|
||||||
|
- [`/llms.txt`](/llms.txt) - a curated index of high-value pages and agent entry
|
||||||
|
points. **Start here.**
|
||||||
|
- [`/llms-full.txt`](/llms-full.txt) - the entire docs corpus in one response.
|
||||||
|
Use only when a task needs broad context across many pages.
|
||||||
|
- **Per-page Markdown** - append `.md` to any docs URL:
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
|
||||||
|
https://docs.kaelio.com/ktx/docs/cli-reference/ktx-sl.md
|
||||||
|
https://docs.kaelio.com/ktx/docs/guides/building-context.md
|
||||||
|
```
|
||||||
|
|
||||||
|
A request for any docs URL with an `Accept: text/markdown` header returns the
|
||||||
|
same Markdown without the `.md` suffix:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -H "Accept: text/markdown" https://docs.kaelio.com/ktx/docs/getting-started/quickstart
|
||||||
|
```
|
||||||
|
|
||||||
|
Each Markdown response leads with the page title, description, canonical URL, and
|
||||||
|
Markdown URL; frontmatter is stripped; code blocks and tables are preserved; and
|
||||||
|
missing pages return a plain-text `404` instead of falling back to HTML. Rendered
|
||||||
|
pages also expose a **Copy as Markdown** action near the title.
|
||||||
|
|
||||||
|
### Retrieval order
|
||||||
|
|
||||||
|
1. Fetch [`/llms.txt`](/llms.txt).
|
||||||
|
2. Pick one or two relevant per-page `.md` URLs.
|
||||||
|
3. Fetch [`/llms-full.txt`](/llms-full.txt) only when page-level docs are not
|
||||||
|
enough.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl https://docs.kaelio.com/ktx/llms.txt
|
||||||
|
curl https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Task router
|
||||||
|
|
||||||
|
| User asks the agent to explain... | Read first | Then read |
|
||||||
|
|------------------------------------|------------|-----------|
|
||||||
|
| What **ktx** does | [Introduction](/docs/getting-started/introduction) | [The Context Layer](/docs/concepts/the-context-layer) |
|
||||||
|
| How to start from a checkout | [Quickstart](/docs/getting-started/quickstart) | [ktx setup](/docs/cli-reference/ktx-setup) |
|
||||||
|
| How to check project readiness | [ktx status](/docs/cli-reference/ktx-status) | [Quickstart](/docs/getting-started/quickstart) |
|
||||||
|
| How context gets built | [Building Context](/docs/guides/building-context) | [ktx ingest](/docs/cli-reference/ktx-ingest) |
|
||||||
|
| How semantic YAML works | [Writing Context](/docs/guides/writing-context) | [ktx sl](/docs/cli-reference/ktx-sl) |
|
||||||
|
| How machine-readable CLI output is shaped | [ktx sl](/docs/cli-reference/ktx-sl) | [ktx wiki](/docs/cli-reference/ktx-wiki) |
|
||||||
|
|
||||||
|
## Agent instructions
|
||||||
|
|
||||||
|
Paste this into a project or system prompt when an assistant needs to answer
|
||||||
|
from the **ktx** docs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
When answering ktx docs questions:
|
||||||
|
|
||||||
|
1. Start with https://docs.kaelio.com/ktx/llms.txt.
|
||||||
|
2. Fetch the smallest relevant Markdown page (append .md to its docs URL).
|
||||||
|
3. Prefer the .md route over rendered HTML.
|
||||||
|
4. Use https://docs.kaelio.com/ktx/llms-full.txt only when the task needs broad docs context.
|
||||||
|
5. Quote commands exactly from docs pages.
|
||||||
|
6. If docs and local CLI behavior disagree, say what differs and prefer local verified output.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prompts
|
||||||
|
|
||||||
|
Replace project names, connection ids, and business terms with your own values.
|
||||||
|
|
||||||
|
**Install and configure ktx in a project**
|
||||||
|
|
||||||
|
```text
|
||||||
|
Run npx skills add Kaelio/ktx --skill ktx and use the ktx skill to install and configure ktx
|
||||||
|
```
|
||||||
|
|
||||||
|
**Find the right command**
|
||||||
|
|
||||||
|
```text
|
||||||
|
Find the correct ktx command for this task: <task>. Start with /llms.txt, then fetch the smallest relevant CLI reference .md page. Quote the exact command and flags from the docs.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Review semantic changes**
|
||||||
|
|
||||||
|
```text
|
||||||
|
Review the ktx semantic-layer and wiki changes in this branch. Check that measures have clear definitions, joins use valid keys, hidden or internal columns are not exposed to agents, and validation passes. List concrete file and line issues first.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
|
||||||
|
- Do not invent CLI flags - fetch the relevant CLI reference page.
|
||||||
|
- Do not scrape rendered HTML when a `.md` route exists.
|
||||||
|
- Do not treat `/llms.txt` as complete documentation - use it as an index, then
|
||||||
|
fetch the linked pages.
|
||||||
|
- Do not include credentials or secrets in prompts, URLs, or copied docs
|
||||||
|
snippets.
|
||||||
|
- When docs and local CLI behavior disagree, prefer the local CLI output and
|
||||||
|
mention the mismatch.
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
{
|
{
|
||||||
"title": "Community",
|
"title": "Community & Resources",
|
||||||
"defaultOpen": true,
|
"defaultOpen": true,
|
||||||
"pages": ["support", "contributing", "telemetry"]
|
"pages": ["support", "contributing", "telemetry", "ai-resources"]
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -46,6 +46,33 @@ an operation errors, the detail we record is the error as your tools reported
|
||||||
it, which can include identifiers from your setup. If you'd rather send nothing
|
it, which can include identifiers from your setup. If you'd rather send nothing
|
||||||
at all, turn telemetry off using any of the options above.
|
at all, turn telemetry off using any of the options above.
|
||||||
|
|
||||||
|
## Error reports
|
||||||
|
|
||||||
|
When telemetry is enabled, **ktx** sends PostHog Error Tracking `$exception`
|
||||||
|
events for CLI and daemon exceptions. Error reports help group crashes and
|
||||||
|
handled failures into PostHog issues.
|
||||||
|
|
||||||
|
Error reports can include:
|
||||||
|
|
||||||
|
- Stack frames, including function names, local file paths, line numbers, and
|
||||||
|
SDK-provided source context.
|
||||||
|
- Error class names and raw error messages.
|
||||||
|
- Cause chains when the runtime exposes them.
|
||||||
|
- `source`, `handled`, and `fatal` diagnostic fields.
|
||||||
|
- Runtime version, OS, architecture, and CI fields.
|
||||||
|
- The hashed `projectId` when **ktx** knows the project.
|
||||||
|
|
||||||
|
Error reports never intentionally include:
|
||||||
|
|
||||||
|
- Secrets, credentials, API keys, tokens, cookies, signed URLs, or auth headers.
|
||||||
|
- Database URLs, connection strings, DSNs, raw argv, or raw environment values.
|
||||||
|
- SQL text, schema names, table names, or column names as explicit payload
|
||||||
|
properties.
|
||||||
|
- Customer row data.
|
||||||
|
- User prompt text or raw MCP arguments.
|
||||||
|
|
||||||
|
The same opt-out controls listed above disable error reports.
|
||||||
|
|
||||||
## Storage and retention
|
## Storage and retention
|
||||||
|
|
||||||
Telemetry is sent to PostHog, a third-party product-analytics service used by
|
Telemetry is sent to PostHog, a third-party product-analytics service used by
|
||||||
|
|
|
||||||
|
|
@ -377,6 +377,10 @@ llm:
|
||||||
models:
|
models:
|
||||||
default: claude-sonnet-4-6
|
default: claude-sonnet-4-6
|
||||||
triage: claude-haiku-4-5
|
triage: claude-haiku-4-5
|
||||||
|
candidateExtraction: claude-sonnet-4-6
|
||||||
|
curator: claude-opus-4-7
|
||||||
|
reconcile: claude-opus-4-7
|
||||||
|
repair: claude-haiku-4-5
|
||||||
promptCaching:
|
promptCaching:
|
||||||
enabled: true
|
enabled: true
|
||||||
systemTtl: 1h
|
systemTtl: 1h
|
||||||
|
|
@ -404,6 +408,11 @@ llm:
|
||||||
backend: codex
|
backend: codex
|
||||||
models:
|
models:
|
||||||
default: gpt-5.5
|
default: gpt-5.5
|
||||||
|
triage: gpt-5.5
|
||||||
|
candidateExtraction: gpt-5.5
|
||||||
|
curator: gpt-5.5
|
||||||
|
reconcile: gpt-5.5
|
||||||
|
repair: gpt-5.5
|
||||||
```
|
```
|
||||||
|
|
||||||
### Model roles
|
### Model roles
|
||||||
|
|
@ -452,6 +461,16 @@ ingest:
|
||||||
stepBudget: 40
|
stepBudget: 40
|
||||||
maxConcurrency: 2
|
maxConcurrency: 2
|
||||||
failureMode: continue
|
failureMode: continue
|
||||||
|
rateLimit:
|
||||||
|
enabled: true
|
||||||
|
throttleThreshold: 0.8
|
||||||
|
minConcurrencyUnderPressure: 1
|
||||||
|
maxWaitMs: 600000
|
||||||
|
retry:
|
||||||
|
maxAttempts: 6
|
||||||
|
baseDelayMs: 1000
|
||||||
|
maxDelayMs: 60000
|
||||||
|
jitter: true
|
||||||
```
|
```
|
||||||
|
|
||||||
### Adapters
|
### Adapters
|
||||||
|
|
@ -498,6 +517,24 @@ handles failures.
|
||||||
| `workUnits.maxConcurrency` | `int > 0` | `1` | How many work units run in parallel. |
|
| `workUnits.maxConcurrency` | `int > 0` | `1` | How many work units run in parallel. |
|
||||||
| `workUnits.failureMode` | `abort` \| `continue` | `continue` | `abort` stops the whole ingest run on the first failure; `continue` records it and keeps going. |
|
| `workUnits.failureMode` | `abort` \| `continue` | `continue` | `abort` stops the whole ingest run on the first failure; `continue` records it and keeps going. |
|
||||||
|
|
||||||
|
### Rate limits
|
||||||
|
|
||||||
|
`rateLimit` controls provider-neutral pacing for LLM calls during ingest. When a
|
||||||
|
provider reports a subscription window, retry-after delay, or HTTP 429,
|
||||||
|
**ktx** pauses new work-unit model calls, shows a transient wait in the CLI,
|
||||||
|
and reduces work-unit concurrency while the provider is under pressure.
|
||||||
|
|
||||||
|
| Field | Type | Default | Purpose |
|
||||||
|
|-------|------|---------|---------|
|
||||||
|
| `rateLimit.enabled` | `boolean` | `true` | Master switch for ingest LLM rate-limit pacing and visible waits. |
|
||||||
|
| `rateLimit.throttleThreshold` | `number between 0 and 1` | `0.8` | Fraction of a known provider window at which **ktx** starts reducing concurrency. |
|
||||||
|
| `rateLimit.minConcurrencyUnderPressure` | `int > 0` | `1` | Effective work-unit concurrency while a provider is under rate-limit pressure. |
|
||||||
|
| `rateLimit.maxWaitMs` | `int > 0` | unset | Caps how long a single provider-reset wait can last. This bounds each wait, not the whole run: after a capped wait elapses **ktx** retries and may pause again. Omit to wait until the provider's reset time. |
|
||||||
|
| `rateLimit.retry.maxAttempts` | `int > 0` | `6` | Maximum attempts for a single rate-limited LLM call before the failure surfaces (counts the first try). Also bounds how far opaque backoff grows for responses without a reset time or retry-after value. |
|
||||||
|
| `rateLimit.retry.baseDelayMs` | `int > 0` | `1000` | Initial opaque retry delay in milliseconds. |
|
||||||
|
| `rateLimit.retry.maxDelayMs` | `int > 0` | `60000` | Maximum opaque retry delay in milliseconds. |
|
||||||
|
| `rateLimit.retry.jitter` | `boolean` | `true` | Add jitter to opaque retry delays. |
|
||||||
|
|
||||||
## `scan`
|
## `scan`
|
||||||
|
|
||||||
`scan` configures how schema-level inputs become structured context:
|
`scan` configures how schema-level inputs become structured context:
|
||||||
|
|
@ -615,6 +652,11 @@ llm:
|
||||||
backend: claude-code
|
backend: claude-code
|
||||||
models:
|
models:
|
||||||
default: sonnet
|
default: sonnet
|
||||||
|
triage: haiku
|
||||||
|
candidateExtraction: sonnet
|
||||||
|
curator: opus
|
||||||
|
reconcile: opus
|
||||||
|
repair: haiku
|
||||||
ingest:
|
ingest:
|
||||||
adapters:
|
adapters:
|
||||||
- live-database
|
- live-database
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,7 @@ description: ktx is an open-source, self-improving context layer for data agents
|
||||||
---
|
---
|
||||||
|
|
||||||
import { ProductMechanics } from "@/components/product-mechanics";
|
import { ProductMechanics } from "@/components/product-mechanics";
|
||||||
|
import { ProductRuntime } from "@/components/product-runtime";
|
||||||
|
|
||||||
<div className="not-prose mb-10">
|
<div className="not-prose mb-10">
|
||||||
<div>
|
<div>
|
||||||
|
|
@ -23,7 +24,7 @@ import { ProductMechanics } from "@/components/product-mechanics";
|
||||||
>
|
>
|
||||||
Make analytics context usable by agents
|
Make analytics context usable by agents
|
||||||
</h1>
|
</h1>
|
||||||
<p className="mt-4 max-w-2xl text-lg text-fd-muted-foreground" style={{ lineHeight: '1.7' }}>
|
<p className="mt-4 max-w-full text-lg text-fd-muted-foreground" style={{ lineHeight: '1.7' }}>
|
||||||
{'ktx is an open-source context layer for data agents. It turns warehouse metadata, BI tool definitions, query history, docs, and approved metric definitions into reviewable files agents can search and execute.'}
|
{'ktx is an open-source context layer for data agents. It turns warehouse metadata, BI tool definitions, query history, docs, and approved metric definitions into reviewable files agents can search and execute.'}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
@ -59,6 +60,8 @@ serves that context to agents at runtime.
|
||||||
|
|
||||||
<ProductMechanics />
|
<ProductMechanics />
|
||||||
|
|
||||||
|
<ProductRuntime />
|
||||||
|
|
||||||
## Use it for
|
## Use it for
|
||||||
|
|
||||||
Use **ktx** when agents need more than raw database access. Agents can search wiki
|
Use **ktx** when agents need more than raw database access. Agents can search wiki
|
||||||
|
|
@ -92,8 +95,8 @@ best first step for users; contributor setup lives in the community docs.
|
||||||
<Card title="CLI Reference" href="/docs/cli-reference/ktx">
|
<Card title="CLI Reference" href="/docs/cli-reference/ktx">
|
||||||
Complete flag and subcommand reference for every **ktx** command.
|
Complete flag and subcommand reference for every **ktx** command.
|
||||||
</Card>
|
</Card>
|
||||||
<Card title="Agent Quickstart" href="/docs/ai-resources/agent-quickstart">
|
<Card title="AI Resources" href="/docs/community/ai-resources">
|
||||||
Machine-readable docs and agent-facing setup notes.
|
Machine-readable docs, a task router, and copy-paste agent prompts.
|
||||||
</Card>
|
</Card>
|
||||||
</Cards>
|
</Cards>
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -9,8 +9,8 @@ This guide takes a local analytics project from empty to agent-ready. You'll
|
||||||
install the CLI, run one guided setup command, and hand the context to a
|
install the CLI, run one guided setup command, and hand the context to a
|
||||||
coding assistant.
|
coding assistant.
|
||||||
|
|
||||||
If you're a coding assistant choosing a docs route, start with the
|
If you're a coding assistant choosing a docs route, start with
|
||||||
[Agent Quickstart](/docs/ai-resources/agent-quickstart) instead.
|
[AI Resources](/docs/community/ai-resources) instead.
|
||||||
|
|
||||||
<div
|
<div
|
||||||
className="not-prose my-8 overflow-hidden rounded-2xl border"
|
className="not-prose my-8 overflow-hidden rounded-2xl border"
|
||||||
|
|
@ -191,6 +191,12 @@ Install the published package globally:
|
||||||
npm install -g @kaelio/ktx
|
npm install -g @kaelio/ktx
|
||||||
```
|
```
|
||||||
|
|
||||||
|
To upgrade an existing install later, re-run with the `@latest` tag:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install -g @kaelio/ktx@latest
|
||||||
|
```
|
||||||
|
|
||||||
**ktx** is open source. If you'd like to hack on it or run from a local checkout,
|
**ktx** is open source. If you'd like to hack on it or run from a local checkout,
|
||||||
the source lives at [github.com/kaelio/ktx](https://github.com/kaelio/ktx) -
|
the source lives at [github.com/kaelio/ktx](https://github.com/kaelio/ktx) -
|
||||||
see [Contributing](/docs/community/contributing) to get set up.
|
see [Contributing](/docs/community/contributing) to get set up.
|
||||||
|
|
|
||||||
|
|
@ -43,7 +43,7 @@ Local-auth backends keep provider credentials out of `ktx.yaml`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ktx setup --llm-backend claude-code --no-input
|
ktx setup --llm-backend claude-code --no-input
|
||||||
ktx setup --llm-backend codex --llm-model gpt-5.5 --no-input
|
ktx setup --llm-backend codex --no-input
|
||||||
```
|
```
|
||||||
|
|
||||||
With `claude-code`, **ktx** agent loops can invoke only the **ktx** MCP tools
|
With `claude-code`, **ktx** agent loops can invoke only the **ktx** MCP tools
|
||||||
|
|
|
||||||
|
|
@ -30,19 +30,19 @@ llm:
|
||||||
default: sonnet
|
default: sonnet
|
||||||
triage: haiku
|
triage: haiku
|
||||||
candidateExtraction: sonnet
|
candidateExtraction: sonnet
|
||||||
curator: sonnet
|
curator: opus
|
||||||
reconcile: sonnet
|
reconcile: opus
|
||||||
repair: sonnet
|
repair: haiku
|
||||||
```
|
```
|
||||||
|
|
||||||
During setup, choose the backend interactively or pass the model in automation:
|
During setup, choose the backend interactively or pass it in automation:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ktx setup --llm-backend claude-code --llm-model opus --no-input
|
ktx setup --llm-backend claude-code --no-input
|
||||||
```
|
```
|
||||||
|
|
||||||
For Claude Code, `sonnet`, `opus`, and `haiku` map to **ktx** defaults. Full Claude
|
Setup writes `sonnet`, `haiku`, and `opus` aliases into `llm.models`. You can
|
||||||
model IDs are also accepted.
|
edit any role to another alias or a full Claude model ID after setup.
|
||||||
|
|
||||||
`claude-code` exposes only **ktx** MCP tools for the current agent loop. SDK init
|
`claude-code` exposes only **ktx** MCP tools for the current agent loop. SDK init
|
||||||
metadata may still list host slash commands, skills, and subagents; **ktx** does not
|
metadata may still list host slash commands, skills, and subagents; **ktx** does not
|
||||||
|
|
@ -59,12 +59,17 @@ llm:
|
||||||
backend: codex
|
backend: codex
|
||||||
models:
|
models:
|
||||||
default: gpt-5.5
|
default: gpt-5.5
|
||||||
|
triage: gpt-5.5
|
||||||
|
candidateExtraction: gpt-5.5
|
||||||
|
curator: gpt-5.5
|
||||||
|
reconcile: gpt-5.5
|
||||||
|
repair: gpt-5.5
|
||||||
```
|
```
|
||||||
|
|
||||||
Configure it non-interactively:
|
Configure it non-interactively:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ktx setup --llm-backend codex --llm-model gpt-5.5 --no-input
|
ktx setup --llm-backend codex --no-input
|
||||||
```
|
```
|
||||||
|
|
||||||
This is separate from Codex agent-client setup. `ktx setup --agents --target
|
This is separate from Codex agent-client setup. `ktx setup --agents --target
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,6 @@
|
||||||
"integrations",
|
"integrations",
|
||||||
"configuration",
|
"configuration",
|
||||||
"cli-reference",
|
"cli-reference",
|
||||||
"ai-resources",
|
|
||||||
"community"
|
"community"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -54,9 +54,7 @@ ktx provides semantic-layer files, warehouse scans, wiki pages, provenance, and
|
||||||
|
|
||||||
- Installable setup skill: run \`npx skills add Kaelio/ktx --skill ktx\` from
|
- Installable setup skill: run \`npx skills add Kaelio/ktx --skill ktx\` from
|
||||||
the project you want to configure.
|
the project you want to configure.
|
||||||
${link("/docs/ai-resources/agent-quickstart", "Agent Quickstart", "Task-first route for coding assistants using ktx")}
|
${link("/docs/community/ai-resources", "AI Resources", "How coding agents read, cite, and act on the ktx docs")}
|
||||||
${link("/docs/ai-resources/markdown-access", "Markdown Access", "Fetch ktx docs as llms.txt, llms-full.txt, or per-page Markdown")}
|
|
||||||
${link("/docs/ai-resources/agent-instructions", "Agent Instructions", "Suggested instructions for coding assistants that need to read and cite ktx docs")}
|
|
||||||
|
|
||||||
## Start Here
|
## Start Here
|
||||||
|
|
||||||
|
|
@ -67,7 +65,7 @@ ${link("/docs/guides/writing-context", "Writing Context", "Write semantic source
|
||||||
## Machine-Readable Documentation
|
## Machine-Readable Documentation
|
||||||
|
|
||||||
- [Full documentation](${absoluteUrl("/llms-full.txt")}): All docs pages in one plain-text markdown response
|
- [Full documentation](${absoluteUrl("/llms-full.txt")}): All docs pages in one plain-text markdown response
|
||||||
- [Markdown access guide](${absoluteUrl("/docs/ai-resources/markdown-access.md")}): How to fetch llms.txt, llms-full.txt, and per-page Markdown
|
- [AI Resources guide](${absoluteUrl("/docs/community/ai-resources.md")}): How agents fetch llms.txt, llms-full.txt, and per-page Markdown
|
||||||
- [Quickstart markdown](${absoluteUrl("/docs/getting-started/quickstart.md")}): Human setup walkthrough
|
- [Quickstart markdown](${absoluteUrl("/docs/getting-started/quickstart.md")}): Human setup walkthrough
|
||||||
- [Semantic-layer CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-sl.md")}): Semantic-layer commands and JSON output
|
- [Semantic-layer CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-sl.md")}): Semantic-layer commands and JSON output
|
||||||
- [Wiki CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-wiki.md")}): Wiki page commands and JSON output
|
- [Wiki CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-wiki.md")}): Wiki page commands and JSON output
|
||||||
|
|
@ -147,8 +145,8 @@ function absoluteUrl(path: string) {
|
||||||
|
|
||||||
function formatCategoryName(category: string) {
|
function formatCategoryName(category: string) {
|
||||||
const labels: Record<string, string> = {
|
const labels: Record<string, string> = {
|
||||||
"ai-resources": "AI Resources",
|
|
||||||
"cli-reference": "CLI Reference",
|
"cli-reference": "CLI Reference",
|
||||||
|
community: "Community & Resources",
|
||||||
};
|
};
|
||||||
|
|
||||||
if (labels[category]) {
|
if (labels[category]) {
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,36 @@ const config = {
|
||||||
};
|
};
|
||||||
},
|
},
|
||||||
async redirects() {
|
async redirects() {
|
||||||
|
// Alias-host canonicalization MUST come before the generic root/docs
|
||||||
|
// redirects below. Those generic rules have no host guard, so if they ran
|
||||||
|
// first they would inject a "/ktx" basePath into the path on the alias
|
||||||
|
// hosts, which the alias catch-alls would then prepend a second time —
|
||||||
|
// producing https://docs.kaelio.com/ktx/ktx/docs/... Redirects also run
|
||||||
|
// before beforeFiles rewrites, so the ktx.sh catch-all must exclude
|
||||||
|
// /stars* to let the stars dashboard rewrite proxy through.
|
||||||
return [
|
return [
|
||||||
|
{
|
||||||
|
source: "/slack",
|
||||||
|
has: [{ type: "host", value: "ktx.sh" }],
|
||||||
|
destination:
|
||||||
|
"https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ",
|
||||||
|
permanent: false,
|
||||||
|
basePath: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
source: "/:path*",
|
||||||
|
has: [{ type: "host", value: "docs.ktx.sh" }],
|
||||||
|
destination: "https://docs.kaelio.com/ktx/:path*",
|
||||||
|
permanent: true,
|
||||||
|
basePath: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
source: "/:path((?!stars(?:/|$)).*)",
|
||||||
|
has: [{ type: "host", value: "ktx.sh" }],
|
||||||
|
destination: "https://docs.kaelio.com/ktx/:path",
|
||||||
|
permanent: true,
|
||||||
|
basePath: false,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
source: "/",
|
source: "/",
|
||||||
destination: "/ktx/docs/getting-started/introduction",
|
destination: "/ktx/docs/getting-started/introduction",
|
||||||
|
|
@ -44,26 +73,30 @@ const config = {
|
||||||
basePath: false,
|
basePath: false,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
source: "/:path*",
|
// AI Resources collapsed from four pages to one and now lives under the
|
||||||
has: [{ type: "host", value: "docs.ktx.sh" }],
|
// Community & Resources section. Redirect the old top-level URL and the
|
||||||
destination: "https://docs.kaelio.com/ktx/:path*",
|
// retired per-page slugs to the new home. Redirects run before the .md
|
||||||
|
// rewrite, so the Markdown variants must be matched first and keep their
|
||||||
|
// .md suffix; otherwise a cached Markdown URL would 308 to the HTML page
|
||||||
|
// and break the agent Markdown contract.
|
||||||
|
source: "/docs/ai-resources.md",
|
||||||
|
destination: "/docs/community/ai-resources.md",
|
||||||
permanent: true,
|
permanent: true,
|
||||||
basePath: false,
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
source: "/slack",
|
source: "/docs/ai-resources/:slug([^/]+\\.md)",
|
||||||
has: [{ type: "host", value: "ktx.sh" }],
|
destination: "/docs/community/ai-resources.md",
|
||||||
destination:
|
permanent: true,
|
||||||
"https://join.slack.com/t/ktxcommunity/shared_invite/zt-3y9b44m1x-LVyNNJD5nwaZHq4XS29LMQ",
|
|
||||||
permanent: false,
|
|
||||||
basePath: false,
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
source: "/:path((?!stars(?:/|$)).*)",
|
source: "/docs/ai-resources",
|
||||||
has: [{ type: "host", value: "ktx.sh" }],
|
destination: "/docs/community/ai-resources",
|
||||||
destination: "https://docs.kaelio.com/ktx/:path",
|
permanent: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
source: "/docs/ai-resources/:slug",
|
||||||
|
destination: "/docs/community/ai-resources",
|
||||||
permanent: true,
|
permanent: true,
|
||||||
basePath: false,
|
|
||||||
},
|
},
|
||||||
];
|
];
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,8 @@ import assert from "node:assert/strict";
|
||||||
import { spawn } from "node:child_process";
|
import { spawn } from "node:child_process";
|
||||||
import { once } from "node:events";
|
import { once } from "node:events";
|
||||||
import { readFile, writeFile } from "node:fs/promises";
|
import { readFile, writeFile } from "node:fs/promises";
|
||||||
|
import http from "node:http";
|
||||||
|
import https from "node:https";
|
||||||
import { dirname, join } from "node:path";
|
import { dirname, join } from "node:path";
|
||||||
import { createServer } from "node:net";
|
import { createServer } from "node:net";
|
||||||
import { after, before, test } from "node:test";
|
import { after, before, test } from "node:test";
|
||||||
|
|
@ -100,6 +102,37 @@ after(async () => {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Node's fetch (undici) overwrites the Host header with the connection host,
|
||||||
|
// so the alias-host redirect rules never match. The low-level http(s) client
|
||||||
|
// sends Host verbatim, which is what the alias canonicalization keys off of.
|
||||||
|
function requestWithHost(hostHeader, path) {
|
||||||
|
const target = new URL(docsSiteUrl);
|
||||||
|
const client = target.protocol === "https:" ? https : http;
|
||||||
|
const port =
|
||||||
|
target.port || (target.protocol === "https:" ? "443" : "80");
|
||||||
|
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const request = client.request(
|
||||||
|
{
|
||||||
|
hostname: target.hostname,
|
||||||
|
port,
|
||||||
|
path,
|
||||||
|
method: "GET",
|
||||||
|
headers: { Host: hostHeader },
|
||||||
|
},
|
||||||
|
(response) => {
|
||||||
|
response.resume();
|
||||||
|
resolve({
|
||||||
|
status: response.statusCode,
|
||||||
|
location: response.headers.location,
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
request.on("error", reject);
|
||||||
|
request.end();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
test("/ktx/docs redirects to the docs introduction", async () => {
|
test("/ktx/docs redirects to the docs introduction", async () => {
|
||||||
const response = await fetch(`${docsSiteUrl}${docsBasePath}/docs`, {
|
const response = await fetch(`${docsSiteUrl}${docsBasePath}/docs`, {
|
||||||
redirect: "manual",
|
redirect: "manual",
|
||||||
|
|
@ -112,6 +145,53 @@ test("/ktx/docs redirects to the docs introduction", async () => {
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("retired AI Resources URLs redirect to the page under Community", async () => {
|
||||||
|
// The former top-level URL.
|
||||||
|
const bare = await fetch(
|
||||||
|
`${docsSiteUrl}${docsBasePath}/docs/ai-resources`,
|
||||||
|
{ redirect: "manual" },
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(bare.status, 308);
|
||||||
|
assert.equal(
|
||||||
|
bare.headers.get("location"),
|
||||||
|
`${docsBasePath}/docs/community/ai-resources`,
|
||||||
|
);
|
||||||
|
|
||||||
|
// A retired per-page slug.
|
||||||
|
const slug = await fetch(
|
||||||
|
`${docsSiteUrl}${docsBasePath}/docs/ai-resources/agent-quickstart`,
|
||||||
|
{ redirect: "manual" },
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(slug.status, 308);
|
||||||
|
assert.equal(
|
||||||
|
slug.headers.get("location"),
|
||||||
|
`${docsBasePath}/docs/community/ai-resources`,
|
||||||
|
);
|
||||||
|
|
||||||
|
// A retired per-page Markdown URL must stay Markdown: it has to redirect to
|
||||||
|
// the new .md route, not fall through to the HTML page.
|
||||||
|
const markdown = await fetch(
|
||||||
|
`${docsSiteUrl}${docsBasePath}/docs/ai-resources/agent-quickstart.md`,
|
||||||
|
{ redirect: "manual" },
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(markdown.status, 308);
|
||||||
|
assert.equal(
|
||||||
|
markdown.headers.get("location"),
|
||||||
|
`${docsBasePath}/docs/community/ai-resources.md`,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Following that redirect end to end must land on Markdown, not HTML.
|
||||||
|
const followed = await fetch(
|
||||||
|
`${docsSiteUrl}${docsBasePath}/docs/ai-resources/agent-quickstart.md`,
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(followed.status, 200);
|
||||||
|
assert.match(followed.headers.get("content-type") ?? "", /text\/markdown/);
|
||||||
|
});
|
||||||
|
|
||||||
test("/ redirects into the /ktx docs site", async () => {
|
test("/ redirects into the /ktx docs site", async () => {
|
||||||
const response = await fetch(`${docsSiteUrl}/`, {
|
const response = await fetch(`${docsSiteUrl}/`, {
|
||||||
redirect: "manual",
|
redirect: "manual",
|
||||||
|
|
@ -141,3 +221,51 @@ test("/ktx/api/search returns docs search results", async () => {
|
||||||
"search should return at least one docs result",
|
"search should return at least one docs result",
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("ktx.sh canonicalizes to a single /ktx basePath on the docs host", async () => {
|
||||||
|
const root = await requestWithHost("ktx.sh", "/");
|
||||||
|
assert.equal(root.status, 308);
|
||||||
|
assert.equal(root.location, "https://docs.kaelio.com/ktx/");
|
||||||
|
assert.ok(
|
||||||
|
!root.location.includes("/ktx/ktx"),
|
||||||
|
"the basePath must not be doubled",
|
||||||
|
);
|
||||||
|
|
||||||
|
const page = await requestWithHost(
|
||||||
|
"ktx.sh",
|
||||||
|
"/docs/getting-started/quickstart",
|
||||||
|
);
|
||||||
|
assert.equal(page.status, 308);
|
||||||
|
assert.equal(
|
||||||
|
page.location,
|
||||||
|
"https://docs.kaelio.com/ktx/docs/getting-started/quickstart",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("docs.ktx.sh canonicalizes to a single /ktx basePath on the docs host", async () => {
|
||||||
|
const root = await requestWithHost("docs.ktx.sh", "/");
|
||||||
|
assert.equal(root.status, 308);
|
||||||
|
assert.equal(root.location, "https://docs.kaelio.com/ktx");
|
||||||
|
assert.ok(
|
||||||
|
!root.location.includes("/ktx/ktx"),
|
||||||
|
"the basePath must not be doubled",
|
||||||
|
);
|
||||||
|
|
||||||
|
const page = await requestWithHost("docs.ktx.sh", "/llms.txt");
|
||||||
|
assert.equal(page.status, 308);
|
||||||
|
assert.equal(page.location, "https://docs.kaelio.com/ktx/llms.txt");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("ktx.sh keeps the /slack and /stars exceptions", async () => {
|
||||||
|
const slack = await requestWithHost("ktx.sh", "/slack");
|
||||||
|
assert.equal(slack.status, 307);
|
||||||
|
assert.match(slack.location, /^https:\/\/join\.slack\.com\//);
|
||||||
|
|
||||||
|
// /stars is proxied by a beforeFiles rewrite, so the apex catch-all must not
|
||||||
|
// canonicalize it to the docs host.
|
||||||
|
const stars = await requestWithHost("ktx.sh", "/stars");
|
||||||
|
assert.ok(
|
||||||
|
!(stars.location ?? "").startsWith("https://docs.kaelio.com"),
|
||||||
|
"the stars dashboard must not be redirected to the docs host",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -85,7 +85,7 @@ test("product mechanics component explains ingestion outputs", async () => {
|
||||||
"compile into SQL",
|
"compile into SQL",
|
||||||
'"use client"',
|
'"use client"',
|
||||||
"@xyflow/react",
|
"@xyflow/react",
|
||||||
"<ReactFlow",
|
"<FlowCanvas",
|
||||||
"getSmoothStepPath",
|
"getSmoothStepPath",
|
||||||
"animateMotion",
|
"animateMotion",
|
||||||
"mechanics-particle",
|
"mechanics-particle",
|
||||||
|
|
@ -97,21 +97,21 @@ test("product mechanics component explains ingestion outputs", async () => {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
assert.match(
|
// The ReactFlow canvas config lives in the shared FlowCanvas wrapper, which
|
||||||
component,
|
// product-mechanics renders. Assert the static read-only behavior there.
|
||||||
|
const flowCanvas = await readDocsFile("components/flow-canvas.tsx");
|
||||||
|
for (const guard of [
|
||||||
/nodesDraggable=\{false\}/,
|
/nodesDraggable=\{false\}/,
|
||||||
"ReactFlow canvas should disable node dragging",
|
/nodesConnectable=\{false\}/,
|
||||||
);
|
|
||||||
assert.match(
|
|
||||||
component,
|
|
||||||
/panOnDrag=\{false\}/,
|
|
||||||
"ReactFlow canvas should disable panning",
|
|
||||||
);
|
|
||||||
assert.match(
|
|
||||||
component,
|
|
||||||
/zoomOnScroll=\{false\}/,
|
/zoomOnScroll=\{false\}/,
|
||||||
"ReactFlow canvas should disable scroll zoom",
|
/elementsSelectable=\{false\}/,
|
||||||
);
|
]) {
|
||||||
|
assert.match(
|
||||||
|
flowCanvas,
|
||||||
|
guard,
|
||||||
|
`shared FlowCanvas should enforce static read-only behavior: ${guard}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
assert.doesNotMatch(component, /raw-sources/);
|
assert.doesNotMatch(component, /raw-sources/);
|
||||||
assert.doesNotMatch(component, /\.ktx/);
|
assert.doesNotMatch(component, /\.ktx/);
|
||||||
|
|
|
||||||
74
docs-site/tests/product-runtime-content.test.mjs
Normal file
74
docs-site/tests/product-runtime-content.test.mjs
Normal file
|
|
@ -0,0 +1,74 @@
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { dirname, join } from "node:path";
|
||||||
|
import { test } from "node:test";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
const docsSiteDir = join(dirname(fileURLToPath(import.meta.url)), "..");
|
||||||
|
|
||||||
|
async function readDocsFile(path) {
|
||||||
|
return readFile(join(docsSiteDir, path), "utf8");
|
||||||
|
}
|
||||||
|
|
||||||
|
test("docs introduction renders the serving phase after ingestion", async () => {
|
||||||
|
const introduction = await readDocsFile(
|
||||||
|
"content/docs/getting-started/introduction.mdx",
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.match(
|
||||||
|
introduction,
|
||||||
|
/import\s+\{\s*ProductRuntime\s*\}\s+from\s+"@\/components\/product-runtime";/,
|
||||||
|
);
|
||||||
|
assert.match(introduction, /<ProductRuntime\s*\/>/);
|
||||||
|
|
||||||
|
const mechanicsIndex = introduction.indexOf("<ProductMechanics />");
|
||||||
|
const runtimeIndex = introduction.indexOf("<ProductRuntime />");
|
||||||
|
const useCaseIndex = introduction.indexOf("## Use it for");
|
||||||
|
|
||||||
|
assert.ok(
|
||||||
|
runtimeIndex > mechanicsIndex,
|
||||||
|
"serving diagram should appear after the ingestion diagram",
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
runtimeIndex < useCaseIndex,
|
||||||
|
"serving diagram should appear before use-case sections",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("product runtime component explains the serving cycle", async () => {
|
||||||
|
const component = await readDocsFile("components/product-runtime.tsx");
|
||||||
|
|
||||||
|
for (const expectedText of [
|
||||||
|
"How serving works",
|
||||||
|
"Serving flow",
|
||||||
|
"From an agent request to a governed answer",
|
||||||
|
"Your agent",
|
||||||
|
"Claude Code",
|
||||||
|
"Cursor",
|
||||||
|
"Codex",
|
||||||
|
"Search wiki + semantic layer",
|
||||||
|
"Return approved metrics",
|
||||||
|
"Compile metrics → SQL",
|
||||||
|
"Context layer",
|
||||||
|
"Database",
|
||||||
|
"search + read",
|
||||||
|
"read-only",
|
||||||
|
"wiki/*.md",
|
||||||
|
"semantic-layer/*.yaml",
|
||||||
|
'"use client"',
|
||||||
|
"@xyflow/react",
|
||||||
|
"FlowCanvas",
|
||||||
|
"getSmoothStepPath",
|
||||||
|
"animateMotion",
|
||||||
|
"runtime-particle",
|
||||||
|
"buildCyclePath",
|
||||||
|
]) {
|
||||||
|
assert.ok(
|
||||||
|
component.includes(expectedText),
|
||||||
|
`component should include: ${expectedText}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.doesNotMatch(component, /raw-sources/);
|
||||||
|
assert.doesNotMatch(component, /<img/);
|
||||||
|
});
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "ktx-workspace",
|
"name": "ktx-workspace",
|
||||||
"version": "0.9.0",
|
"version": "0.11.0",
|
||||||
"description": "Workspace root for ktx packages",
|
"description": "Workspace root for ktx packages",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
|
|
@ -69,11 +69,6 @@
|
||||||
"typescript": "^6.0.3",
|
"typescript": "^6.0.3",
|
||||||
"yaml": "^2.9.0"
|
"yaml": "^2.9.0"
|
||||||
},
|
},
|
||||||
"pnpm": {
|
|
||||||
"onlyBuiltDependencies": [
|
|
||||||
"better-sqlite3"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"license": "Apache-2.0",
|
"license": "Apache-2.0",
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,11 @@
|
||||||
{
|
{
|
||||||
"name": "@kaelio/ktx",
|
"name": "@kaelio/ktx",
|
||||||
"version": "0.9.0",
|
"version": "0.11.0",
|
||||||
"description": "Standalone ktx context layer for data agents",
|
"description": "Standalone ktx context layer for data agents",
|
||||||
|
"author": {
|
||||||
|
"name": "Kaelio",
|
||||||
|
"url": "https://www.kaelio.com"
|
||||||
|
},
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=22.0.0"
|
"node": ">=22.0.0"
|
||||||
|
|
@ -47,6 +51,7 @@
|
||||||
"@ai-sdk/devtools": "0.0.18",
|
"@ai-sdk/devtools": "0.0.18",
|
||||||
"@ai-sdk/google-vertex": "^4.0.134",
|
"@ai-sdk/google-vertex": "^4.0.134",
|
||||||
"@anthropic-ai/claude-agent-sdk": "0.3.146",
|
"@anthropic-ai/claude-agent-sdk": "0.3.146",
|
||||||
|
"@clack/core": "1.3.1",
|
||||||
"@clack/prompts": "1.4.0",
|
"@clack/prompts": "1.4.0",
|
||||||
"@clickhouse/client": "^1.18.5",
|
"@clickhouse/client": "^1.18.5",
|
||||||
"@commander-js/extra-typings": "14.0.0",
|
"@commander-js/extra-typings": "14.0.0",
|
||||||
|
|
@ -72,6 +77,7 @@
|
||||||
"pg": "^8.21.0",
|
"pg": "^8.21.0",
|
||||||
"posthog-node": "^5.34.9",
|
"posthog-node": "^5.34.9",
|
||||||
"react": "^19.2.6",
|
"react": "^19.2.6",
|
||||||
|
"semver": "^7.8.1",
|
||||||
"simple-git": "3.36.0",
|
"simple-git": "3.36.0",
|
||||||
"snowflake-sdk": "^2.4.2",
|
"snowflake-sdk": "^2.4.2",
|
||||||
"yaml": "^2.9.0",
|
"yaml": "^2.9.0",
|
||||||
|
|
@ -85,6 +91,7 @@
|
||||||
"@types/node": "^25.9.1",
|
"@types/node": "^25.9.1",
|
||||||
"@types/pg": "^8.20.0",
|
"@types/pg": "^8.20.0",
|
||||||
"@types/react": "^19.2.15",
|
"@types/react": "^19.2.15",
|
||||||
|
"@types/semver": "^7.7.1",
|
||||||
"@vitest/coverage-v8": "^4.1.7",
|
"@vitest/coverage-v8": "^4.1.7",
|
||||||
"ajv": "8.20.0",
|
"ajv": "8.20.0",
|
||||||
"ink-testing-library": "^4.0.0",
|
"ink-testing-library": "^4.0.0",
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,30 @@ import type { KtxCliIo } from './cli-runtime.js';
|
||||||
|
|
||||||
const ESC = String.fromCharCode(0x1b);
|
const ESC = String.fromCharCode(0x1b);
|
||||||
|
|
||||||
|
export interface CliStyleEnv {
|
||||||
|
NO_COLOR?: string;
|
||||||
|
TERM?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
function ansiEnabled(env: CliStyleEnv = process.env): boolean {
|
||||||
|
return !env.NO_COLOR && env.TERM !== 'dumb';
|
||||||
|
}
|
||||||
|
|
||||||
|
function ansiColor(text: string, open: number, close: number, env?: CliStyleEnv): string {
|
||||||
|
if (!ansiEnabled(env)) {
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
return `${ESC}[${open}m${text}${ESC}[${close}m`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function dim(text: string, env?: CliStyleEnv): string {
|
||||||
|
return ansiColor(text, 2, 22, env);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function cyan(text: string, env?: CliStyleEnv): string {
|
||||||
|
return ansiColor(text, 36, 39, env);
|
||||||
|
}
|
||||||
|
|
||||||
export interface RailBufferedSource {
|
export interface RailBufferedSource {
|
||||||
stdoutText(): string;
|
stdoutText(): string;
|
||||||
stderrText(): string;
|
stderrText(): string;
|
||||||
|
|
@ -61,11 +85,11 @@ export function createClackSpinner(): KtxCliSpinner {
|
||||||
}
|
}
|
||||||
|
|
||||||
function magenta(text: string): string {
|
function magenta(text: string): string {
|
||||||
return `${ESC}[35m${text}${ESC}[39m`;
|
return ansiColor(text, 35, 39);
|
||||||
}
|
}
|
||||||
|
|
||||||
function red(text: string): string {
|
function red(text: string): string {
|
||||||
return `${ESC}[31m${text}${ESC}[39m`;
|
return ansiColor(text, 31, 39);
|
||||||
}
|
}
|
||||||
|
|
||||||
export function createStaticCliSpinner(io: KtxCliSpinnerIo): KtxCliSpinner {
|
export function createStaticCliSpinner(io: KtxCliSpinnerIo): KtxCliSpinner {
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,7 @@ import { existsSync } from 'node:fs';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
import { Command, type CommandUnknownOpts, InvalidArgumentError } from '@commander-js/extra-typings';
|
import { Command, type CommandUnknownOpts, InvalidArgumentError } from '@commander-js/extra-typings';
|
||||||
import type { KtxCliDeps, KtxCliIo, KtxCliPackageInfo } from './cli-runtime.js';
|
import type { KtxCliDeps, KtxCliIo, KtxCliPackageInfo } from './cli-runtime.js';
|
||||||
|
import { SLACK_HELP_FOOTER, writeErrorCommunityHint } from './community-cta.js';
|
||||||
import { registerCompletionCommands } from './commands/completion-commands.js';
|
import { registerCompletionCommands } from './commands/completion-commands.js';
|
||||||
import { registerConnectionCommands } from './commands/connection-commands.js';
|
import { registerConnectionCommands } from './commands/connection-commands.js';
|
||||||
import { registerIngestCommands } from './commands/ingest-commands.js';
|
import { registerIngestCommands } from './commands/ingest-commands.js';
|
||||||
|
|
@ -16,6 +17,7 @@ import { renderMissingProjectMessage } from './doctor.js';
|
||||||
import { findNearestKtxProjectDir, resolveKtxProjectDir } from './project-resolver.js';
|
import { findNearestKtxProjectDir, resolveKtxProjectDir } from './project-resolver.js';
|
||||||
import { profileMark, profileSpan } from './startup-profile.js';
|
import { profileMark, profileSpan } from './startup-profile.js';
|
||||||
import type { CommandOutcome } from './telemetry/index.js';
|
import type { CommandOutcome } from './telemetry/index.js';
|
||||||
|
import { prepareUpdateCheckNotice, type PrepareUpdateCheckNoticeOptions } from './update-check/update-check.js';
|
||||||
|
|
||||||
profileMark('module:cli-program');
|
profileMark('module:cli-program');
|
||||||
|
|
||||||
|
|
@ -39,6 +41,8 @@ interface KtxCommanderProgramOptions {
|
||||||
runInit: (args: { projectDir: string; force: boolean }, io: KtxCliIo) => Promise<number>;
|
runInit: (args: { projectDir: string; force: boolean }, io: KtxCliIo) => Promise<number>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type KtxCliUpdateCheckOptions = Pick<PrepareUpdateCheckNoticeOptions, 'env' | 'fetchDistTags' | 'homeDir' | 'now'>;
|
||||||
|
|
||||||
export interface BuildKtxProgramOptions {
|
export interface BuildKtxProgramOptions {
|
||||||
io: KtxCliIo;
|
io: KtxCliIo;
|
||||||
deps: KtxCliDeps;
|
deps: KtxCliDeps;
|
||||||
|
|
@ -47,6 +51,7 @@ export interface BuildKtxProgramOptions {
|
||||||
setExitCode?: (code: number) => void;
|
setExitCode?: (code: number) => void;
|
||||||
argv?: string[];
|
argv?: string[];
|
||||||
setTelemetryModule?: (telemetry: typeof import('./telemetry/index.js')) => void;
|
setTelemetryModule?: (telemetry: typeof import('./telemetry/index.js')) => void;
|
||||||
|
updateCheck?: KtxCliUpdateCheckOptions;
|
||||||
}
|
}
|
||||||
|
|
||||||
type CommanderExitLike = { exitCode: number; code: string; message: string };
|
type CommanderExitLike = { exitCode: number; code: string; message: string };
|
||||||
|
|
@ -254,6 +259,7 @@ function createBaseProgram(info: KtxCliPackageInfo, io: KtxCliIo): Command {
|
||||||
.helpOption('-h, --help', 'Show this help text')
|
.helpOption('-h, --help', 'Show this help text')
|
||||||
.configureHelp({ showGlobalOptions: true })
|
.configureHelp({ showGlobalOptions: true })
|
||||||
.showHelpAfterError()
|
.showHelpAfterError()
|
||||||
|
.addHelpText('after', `\n${SLACK_HELP_FOOTER}`)
|
||||||
.exitOverride()
|
.exitOverride()
|
||||||
.configureOutput({
|
.configureOutput({
|
||||||
writeOut: (chunk) => io.stdout.write(chunk),
|
writeOut: (chunk) => io.stdout.write(chunk),
|
||||||
|
|
@ -431,16 +437,29 @@ export function collectCommandFlagsPresent(command: CommandUnknownOpts): Record<
|
||||||
|
|
||||||
export function buildKtxProgram(options: BuildKtxProgramOptions): Command {
|
export function buildKtxProgram(options: BuildKtxProgramOptions): Command {
|
||||||
const program = createBaseProgram(options.packageInfo, options.io);
|
const program = createBaseProgram(options.packageInfo, options.io);
|
||||||
|
let pendingUpdateNotice: string | null = null;
|
||||||
|
|
||||||
program.hook('preAction', async (_thisCommand, actionCommand) => {
|
program.hook('preAction', async (_thisCommand, actionCommand) => {
|
||||||
// The hidden completion command must stay silent and side-effect free: skip
|
// The hidden completion command must stay silent and side-effect free: skip
|
||||||
// the telemetry notice, command span, and project checks entirely.
|
// the telemetry notice, command span, project checks, and update checks entirely.
|
||||||
if (commandPath(actionCommand as CommandPathNode).includes('__complete')) {
|
if (commandPath(actionCommand as CommandPathNode).includes('__complete')) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
const commandNode = actionCommand as CommandPathNode;
|
||||||
|
const updateCheck = await prepareUpdateCheckNotice({
|
||||||
|
io: options.io,
|
||||||
|
env: options.updateCheck?.env,
|
||||||
|
fetchDistTags: options.updateCheck?.fetchDistTags,
|
||||||
|
homeDir: options.updateCheck?.homeDir,
|
||||||
|
installedVersion: options.packageInfo.version,
|
||||||
|
now: options.updateCheck?.now,
|
||||||
|
commandOptions: commandOptions(commandNode),
|
||||||
|
});
|
||||||
|
pendingUpdateNotice = updateCheck.notice;
|
||||||
|
|
||||||
const telemetry = await import('./telemetry/index.js');
|
const telemetry = await import('./telemetry/index.js');
|
||||||
options.setTelemetryModule?.(telemetry);
|
options.setTelemetryModule?.(telemetry);
|
||||||
await telemetry.showTelemetryNoticeIfNeeded(options.io, options.packageInfo);
|
await telemetry.showTelemetryNoticeIfNeeded(options.io, options.packageInfo);
|
||||||
const commandNode = actionCommand as CommandPathNode;
|
|
||||||
const path = commandPath(commandNode);
|
const path = commandPath(commandNode);
|
||||||
const projectDir = resolveCommandProjectDir(commandNode);
|
const projectDir = resolveCommandProjectDir(commandNode);
|
||||||
const hasProject = ktxYamlExists(projectDir);
|
const hasProject = ktxYamlExists(projectDir);
|
||||||
|
|
@ -457,6 +476,13 @@ export function buildKtxProgram(options: BuildKtxProgramOptions): Command {
|
||||||
ensureProjectAvailable(options.io, commandNode);
|
ensureProjectAvailable(options.io, commandNode);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
program.hook('postAction', () => {
|
||||||
|
if (pendingUpdateNotice) {
|
||||||
|
options.io.stderr.write(pendingUpdateNotice);
|
||||||
|
pendingUpdateNotice = null;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
const context: KtxCliCommandContext = {
|
const context: KtxCliCommandContext = {
|
||||||
io: options.io,
|
io: options.io,
|
||||||
deps: options.deps,
|
deps: options.deps,
|
||||||
|
|
@ -529,7 +555,15 @@ export async function runCommanderKtxCli(
|
||||||
try {
|
try {
|
||||||
return await runBareInteractiveCommand(program, io, context);
|
return await runBareInteractiveCommand(program, io, context);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
const telemetry = await import('./telemetry/index.js');
|
||||||
|
await telemetry.reportException({
|
||||||
|
error,
|
||||||
|
context: { source: 'bare-interactive', handled: true, fatal: false },
|
||||||
|
packageInfo: info,
|
||||||
|
io,
|
||||||
|
});
|
||||||
io.stderr.write(`${formatCliError(error)}\n`);
|
io.stderr.write(`${formatCliError(error)}\n`);
|
||||||
|
writeErrorCommunityHint(io, 'error');
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -554,6 +588,7 @@ export async function runCommanderKtxCli(
|
||||||
exitCode = error.exitCode === 0 ? 0 : 1;
|
exitCode = error.exitCode === 0 ? 0 : 1;
|
||||||
} else {
|
} else {
|
||||||
io.stderr.write(`${formatCliError(error)}\n`);
|
io.stderr.write(`${formatCliError(error)}\n`);
|
||||||
|
writeErrorCommunityHint(io, 'error');
|
||||||
exitCode = 1;
|
exitCode = 1;
|
||||||
}
|
}
|
||||||
} finally {
|
} finally {
|
||||||
|
|
@ -563,6 +598,23 @@ export async function runCommanderKtxCli(
|
||||||
outcome: commandOutcomeForParseResult(parseError, exitCode),
|
outcome: commandOutcomeForParseResult(parseError, exitCode),
|
||||||
error: parseError,
|
error: parseError,
|
||||||
});
|
});
|
||||||
|
if (
|
||||||
|
parseError &&
|
||||||
|
!isCommanderExit(parseError) &&
|
||||||
|
!isKtxProjectMissingAbortError(parseError)
|
||||||
|
) {
|
||||||
|
await telemetryModule.reportException({
|
||||||
|
error: parseError,
|
||||||
|
context: {
|
||||||
|
source: completed?.commandPath.join(' ') ?? 'commander parseAsync',
|
||||||
|
handled: true,
|
||||||
|
fatal: false,
|
||||||
|
},
|
||||||
|
projectDir: completed?.projectGroupAttached ? completed.projectDir : undefined,
|
||||||
|
packageInfo: info,
|
||||||
|
io,
|
||||||
|
});
|
||||||
|
}
|
||||||
await telemetryModule.emitCompletedCommand({ completed, packageInfo: info, io });
|
await telemetryModule.emitCompletedCommand({ completed, packageInfo: info, io });
|
||||||
await telemetryModule.shutdownTelemetryEmitter();
|
await telemetryModule.shutdownTelemetryEmitter();
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -12,6 +12,7 @@ import type { KtxSqlArgs } from './sql.js';
|
||||||
import { profileMark, profileSpan } from './startup-profile.js';
|
import { profileMark, profileSpan } from './startup-profile.js';
|
||||||
import type { KtxTextIngestArgs } from './text-ingest.js';
|
import type { KtxTextIngestArgs } from './text-ingest.js';
|
||||||
import { assertCliVersion } from './release-version.js';
|
import { assertCliVersion } from './release-version.js';
|
||||||
|
import { writeErrorCommunityHint } from './community-cta.js';
|
||||||
|
|
||||||
profileMark('module:cli-runtime');
|
profileMark('module:cli-runtime');
|
||||||
|
|
||||||
|
|
@ -129,6 +130,54 @@ function installTelemetrySignalFlush(io: KtxCliIo, info: KtxCliPackageInfo): ()
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
|
export function createGlobalExceptionReporter(io: KtxCliIo, info: KtxCliPackageInfo) {
|
||||||
|
return async (source: 'uncaughtException' | 'unhandledRejection', error: unknown): Promise<void> => {
|
||||||
|
const { reportException, shutdownTelemetryEmitter } = await import('./telemetry/index.js');
|
||||||
|
await reportException({
|
||||||
|
error,
|
||||||
|
context: { source, handled: false, fatal: true },
|
||||||
|
io,
|
||||||
|
packageInfo: info,
|
||||||
|
immediate: true,
|
||||||
|
});
|
||||||
|
await shutdownTelemetryEmitter();
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
|
export function writeGlobalExceptionToStderr(io: KtxCliIo, error: unknown): void {
|
||||||
|
if (error instanceof Error && error.stack) {
|
||||||
|
io.stderr.write(`${error.stack}\n`);
|
||||||
|
} else {
|
||||||
|
io.stderr.write(`${String(error)}\n`);
|
||||||
|
}
|
||||||
|
writeErrorCommunityHint(io, 'crash');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function installGlobalExceptionHandlers(io: KtxCliIo, info: KtxCliPackageInfo): () => void {
|
||||||
|
const report = createGlobalExceptionReporter(io, info);
|
||||||
|
const handle = (source: 'uncaughtException' | 'unhandledRejection', error: unknown): void => {
|
||||||
|
void (async () => {
|
||||||
|
try {
|
||||||
|
await report(source, error);
|
||||||
|
} catch {
|
||||||
|
// Best-effort: preserve Node's process termination behavior.
|
||||||
|
}
|
||||||
|
writeGlobalExceptionToStderr(io, error);
|
||||||
|
process.exit(1);
|
||||||
|
})();
|
||||||
|
};
|
||||||
|
const onUncaught = (error: Error): void => handle('uncaughtException', error);
|
||||||
|
const onUnhandled = (reason: unknown): void => handle('unhandledRejection', reason);
|
||||||
|
process.on('uncaughtException', onUncaught);
|
||||||
|
process.on('unhandledRejection', onUnhandled);
|
||||||
|
return () => {
|
||||||
|
process.off('uncaughtException', onUncaught);
|
||||||
|
process.off('unhandledRejection', onUnhandled);
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
export async function runKtxCli(
|
export async function runKtxCli(
|
||||||
argv = process.argv.slice(2),
|
argv = process.argv.slice(2),
|
||||||
io: KtxCliIo = process,
|
io: KtxCliIo = process,
|
||||||
|
|
@ -141,11 +190,14 @@ export async function runKtxCli(
|
||||||
// Real-process entry only: flush telemetry if interrupted. Test/programmatic
|
// Real-process entry only: flush telemetry if interrupted. Test/programmatic
|
||||||
// callers pass their own `io`, so they never install process-level handlers.
|
// callers pass their own `io`, so they never install process-level handlers.
|
||||||
const removeSignalFlush = (io as unknown) === process ? installTelemetrySignalFlush(io, info) : undefined;
|
const removeSignalFlush = (io as unknown) === process ? installTelemetrySignalFlush(io, info) : undefined;
|
||||||
|
const removeGlobalExceptionHandlers =
|
||||||
|
(io as unknown) === process ? installGlobalExceptionHandlers(io, info) : undefined;
|
||||||
try {
|
try {
|
||||||
return await runCommanderKtxCli(argv, io, deps, info, {
|
return await runCommanderKtxCli(argv, io, deps, info, {
|
||||||
runInit: runInitForCommander,
|
runInit: runInitForCommander,
|
||||||
});
|
});
|
||||||
} finally {
|
} finally {
|
||||||
|
removeGlobalExceptionHandlers?.();
|
||||||
removeSignalFlush?.();
|
removeSignalFlush?.();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@ import { type Command, InvalidArgumentError, Option } from '@commander-js/extra-
|
||||||
import type { KtxCliCommandContext } from '../cli-program.js';
|
import type { KtxCliCommandContext } from '../cli-program.js';
|
||||||
import { resolveCommandProjectDir } from '../cli-program.js';
|
import { resolveCommandProjectDir } from '../cli-program.js';
|
||||||
import type { KtxSetupDatabaseDriver } from '../setup-databases.js';
|
import type { KtxSetupDatabaseDriver } from '../setup-databases.js';
|
||||||
import type { KtxSetupLlmBackend } from '../setup-models.js';
|
import { isKtxSetupLlmBackend, type KtxSetupLlmBackend } from '../setup-models.js';
|
||||||
import type { KtxSetupSourceType } from '../setup-sources.js';
|
import type { KtxSetupSourceType } from '../setup-sources.js';
|
||||||
|
|
||||||
async function runSetupArgs(
|
async function runSetupArgs(
|
||||||
|
|
@ -29,7 +29,7 @@ function embeddingBackend(value: string): 'openai' | 'sentence-transformers' {
|
||||||
}
|
}
|
||||||
|
|
||||||
function llmBackend(value: string): KtxSetupLlmBackend {
|
function llmBackend(value: string): KtxSetupLlmBackend {
|
||||||
if (value === 'anthropic' || value === 'vertex' || value === 'claude-code' || value === 'codex') {
|
if (isKtxSetupLlmBackend(value)) {
|
||||||
return value;
|
return value;
|
||||||
}
|
}
|
||||||
throw new InvalidArgumentError(`invalid choice '${value}'`);
|
throw new InvalidArgumentError(`invalid choice '${value}'`);
|
||||||
|
|
@ -95,7 +95,6 @@ function shouldShowSetupEntryMenu(
|
||||||
llmBackend?: KtxSetupLlmBackend;
|
llmBackend?: KtxSetupLlmBackend;
|
||||||
anthropicApiKeyEnv?: string;
|
anthropicApiKeyEnv?: string;
|
||||||
anthropicApiKeyFile?: string;
|
anthropicApiKeyFile?: string;
|
||||||
llmModel?: string;
|
|
||||||
vertexProject?: string;
|
vertexProject?: string;
|
||||||
vertexLocation?: string;
|
vertexLocation?: string;
|
||||||
skipLlm?: boolean;
|
skipLlm?: boolean;
|
||||||
|
|
@ -166,7 +165,6 @@ function shouldShowSetupEntryMenu(
|
||||||
'llmBackend',
|
'llmBackend',
|
||||||
'anthropicApiKeyEnv',
|
'anthropicApiKeyEnv',
|
||||||
'anthropicApiKeyFile',
|
'anthropicApiKeyFile',
|
||||||
'llmModel',
|
|
||||||
'vertexProject',
|
'vertexProject',
|
||||||
'vertexLocation',
|
'vertexLocation',
|
||||||
'skipLlm',
|
'skipLlm',
|
||||||
|
|
@ -229,7 +227,6 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
|
||||||
.addOption(
|
.addOption(
|
||||||
new Option('--anthropic-api-key-file <path>', 'File containing the Anthropic API key').hideHelp(),
|
new Option('--anthropic-api-key-file <path>', 'File containing the Anthropic API key').hideHelp(),
|
||||||
)
|
)
|
||||||
.addOption(new Option('--llm-model <model>', 'LLM model ID or backend model alias').hideHelp())
|
|
||||||
.addOption(new Option('--vertex-project <project>', 'Google Vertex AI project ID, env:NAME, or file:/path').hideHelp())
|
.addOption(new Option('--vertex-project <project>', 'Google Vertex AI project ID, env:NAME, or file:/path').hideHelp())
|
||||||
.addOption(new Option('--vertex-location <location>', 'Google Vertex AI location, env:NAME, or file:/path').hideHelp())
|
.addOption(new Option('--vertex-location <location>', 'Google Vertex AI location, env:NAME, or file:/path').hideHelp())
|
||||||
.addOption(new Option('--skip-llm', 'Leave LLM setup incomplete for now').hideHelp().default(false))
|
.addOption(new Option('--skip-llm', 'Leave LLM setup incomplete for now').hideHelp().default(false))
|
||||||
|
|
@ -406,6 +403,8 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
|
||||||
}
|
}
|
||||||
|
|
||||||
const resolvedAgentScope = options.local ? 'local' : options.global ? 'global' : 'project';
|
const resolvedAgentScope = options.local ? 'local' : options.global ? 'global' : 'project';
|
||||||
|
const debugEnabled =
|
||||||
|
((command.optsWithGlobals ? command.optsWithGlobals() : command.opts()) as { debug?: unknown }).debug === true;
|
||||||
await runSetupArgs(context, {
|
await runSetupArgs(context, {
|
||||||
command: 'run',
|
command: 'run',
|
||||||
projectDir: resolveCommandProjectDir(command),
|
projectDir: resolveCommandProjectDir(command),
|
||||||
|
|
@ -415,12 +414,12 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
|
||||||
agentScope: resolvedAgentScope,
|
agentScope: resolvedAgentScope,
|
||||||
skipAgents: options.skipAgents === true,
|
skipAgents: options.skipAgents === true,
|
||||||
inputMode: options.input === false ? 'disabled' : 'auto',
|
inputMode: options.input === false ? 'disabled' : 'auto',
|
||||||
|
...(debugEnabled ? { debug: true } : {}),
|
||||||
yes: options.yes === true,
|
yes: options.yes === true,
|
||||||
cliVersion: context.packageInfo.version,
|
cliVersion: context.packageInfo.version,
|
||||||
...(options.llmBackend ? { llmBackend: options.llmBackend } : {}),
|
...(options.llmBackend ? { llmBackend: options.llmBackend } : {}),
|
||||||
...(options.anthropicApiKeyEnv ? { anthropicApiKeyEnv: options.anthropicApiKeyEnv } : {}),
|
...(options.anthropicApiKeyEnv ? { anthropicApiKeyEnv: options.anthropicApiKeyEnv } : {}),
|
||||||
...(options.anthropicApiKeyFile ? { anthropicApiKeyFile: options.anthropicApiKeyFile } : {}),
|
...(options.anthropicApiKeyFile ? { anthropicApiKeyFile: options.anthropicApiKeyFile } : {}),
|
||||||
...(options.llmModel ? { llmModel: options.llmModel } : {}),
|
|
||||||
...(options.vertexProject ? { vertexProject: options.vertexProject } : {}),
|
...(options.vertexProject ? { vertexProject: options.vertexProject } : {}),
|
||||||
...(options.vertexLocation ? { vertexLocation: options.vertexLocation } : {}),
|
...(options.vertexLocation ? { vertexLocation: options.vertexLocation } : {}),
|
||||||
skipLlm: options.skipLlm === true,
|
skipLlm: options.skipLlm === true,
|
||||||
|
|
|
||||||
28
packages/cli/src/community-cta.ts
Normal file
28
packages/cli/src/community-cta.ts
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
import type { KtxCliIo } from './cli-runtime.js';
|
||||||
|
import { isWritableTtyOutput } from './io/tty.js';
|
||||||
|
import { dim } from './io/symbols.js';
|
||||||
|
import { SLACK_URL } from './links.js';
|
||||||
|
|
||||||
|
type ErrorCtaVariant = 'error' | 'crash';
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
|
export const SLACK_HELP_FOOTER = `Community & support: ${SLACK_URL}`;
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
|
export const SLACK_SETUP_NOTE = {
|
||||||
|
title: 'Community',
|
||||||
|
body: `Questions or feedback? Join the ktx Slack: ${SLACK_URL}`,
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
export function writeErrorCommunityHint(io: KtxCliIo, variant: ErrorCtaVariant): void {
|
||||||
|
if (!isWritableTtyOutput(io.stderr)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const line =
|
||||||
|
variant === 'crash'
|
||||||
|
? `This may be a bug - report it or ask in the ktx community: ${SLACK_URL}`
|
||||||
|
: `Stuck? The ktx community can help: ${SLACK_URL}`;
|
||||||
|
|
||||||
|
io.stderr.write(`${dim(line)}\n`);
|
||||||
|
}
|
||||||
|
|
@ -16,7 +16,8 @@ import { bold, dim, green, red, SYMBOLS } from './io/symbols.js';
|
||||||
import { createKtxCliScanConnector } from './local-scan-connectors.js';
|
import { createKtxCliScanConnector } from './local-scan-connectors.js';
|
||||||
import { profileMark } from './startup-profile.js';
|
import { profileMark } from './startup-profile.js';
|
||||||
import { isDemoConnection } from './telemetry/demo-detect.js';
|
import { isDemoConnection } from './telemetry/demo-detect.js';
|
||||||
import { emitTelemetryEvent } from './telemetry/index.js';
|
import { emitTelemetryEvent, reportException } from './telemetry/index.js';
|
||||||
|
import { collectTelemetryRedactionSecrets } from './telemetry/redaction-secrets.js';
|
||||||
import { formatErrorDetail, scrubErrorClass } from './telemetry/scrubber.js';
|
import { formatErrorDetail, scrubErrorClass } from './telemetry/scrubber.js';
|
||||||
|
|
||||||
profileMark('module:connection');
|
profileMark('module:connection');
|
||||||
|
|
@ -74,6 +75,12 @@ async function testNativeConnection(
|
||||||
}
|
}
|
||||||
const result = await connector.testConnection();
|
const result = await connector.testConnection();
|
||||||
if (!result.success) {
|
if (!result.success) {
|
||||||
|
// Re-throw the driver's original error so connection_test telemetry records
|
||||||
|
// its real class (e.g. ConnectionError) and code (e.g. ELOGIN) instead of
|
||||||
|
// collapsing every native failure to a generic Error with no code.
|
||||||
|
if (result.cause instanceof Error) {
|
||||||
|
throw result.cause;
|
||||||
|
}
|
||||||
throw new Error(result.error ?? 'connection test failed');
|
throw new Error(result.error ?? 'connection test failed');
|
||||||
}
|
}
|
||||||
return { driver: connector.driver };
|
return { driver: connector.driver };
|
||||||
|
|
@ -318,6 +325,21 @@ async function emitConnectionTest(input: {
|
||||||
...(errorDetail ? { errorDetail } : {}),
|
...(errorDetail ? { errorDetail } : {}),
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
if (input.error) {
|
||||||
|
await reportException({
|
||||||
|
error: input.error,
|
||||||
|
context: { source: 'connection test', handled: true, fatal: false },
|
||||||
|
projectDir: input.project.projectDir,
|
||||||
|
io: input.io,
|
||||||
|
redactionSecrets: await collectTelemetryRedactionSecrets({
|
||||||
|
project: input.project,
|
||||||
|
connectionId: input.connectionId,
|
||||||
|
includeLlm: false,
|
||||||
|
includeEmbeddings: false,
|
||||||
|
env: process.env,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function visualWidth(text: string): number {
|
function visualWidth(text: string): number {
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,9 @@ import { assertReadOnlySql, limitSqlForExecution } from '../../context/connectio
|
||||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||||
import {
|
import {
|
||||||
|
connectorTestFailure,
|
||||||
createKtxConnectorCapabilities,
|
createKtxConnectorCapabilities,
|
||||||
|
type KtxConnectorTestResult,
|
||||||
type KtxColumnSampleInput,
|
type KtxColumnSampleInput,
|
||||||
type KtxColumnSampleResult,
|
type KtxColumnSampleResult,
|
||||||
type KtxColumnStatsInput,
|
type KtxColumnStatsInput,
|
||||||
|
|
@ -320,7 +322,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
|
||||||
this.id = `bigquery:${options.connectionId}`;
|
this.id = `bigquery:${options.connectionId}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async testConnection(): Promise<{ success: boolean; error?: string }> {
|
async testConnection(): Promise<KtxConnectorTestResult> {
|
||||||
try {
|
try {
|
||||||
const client = this.getClient();
|
const client = this.getClient();
|
||||||
await client.getDatasets({ maxResults: 1 });
|
await client.getDatasets({ maxResults: 1 });
|
||||||
|
|
@ -329,7 +331,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
|
||||||
}
|
}
|
||||||
return { success: true };
|
return { success: true };
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return { success: false, error: error instanceof Error ? error.message : String(error) };
|
return connectorTestFailure(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
import { createClient } from '@clickhouse/client';
|
import { createClient } from '@clickhouse/client';
|
||||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
|
import { connectorTestFailure, createKtxConnectorCapabilities, type KtxConnectorTestResult, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { Agent as HttpsAgent } from 'node:https';
|
import { Agent as HttpsAgent } from 'node:https';
|
||||||
|
|
@ -317,12 +317,12 @@ export class KtxClickHouseScanConnector implements KtxScanConnector {
|
||||||
this.id = `clickhouse:${options.connectionId}`;
|
this.id = `clickhouse:${options.connectionId}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async testConnection(): Promise<{ success: boolean; error?: string }> {
|
async testConnection(): Promise<KtxConnectorTestResult> {
|
||||||
try {
|
try {
|
||||||
await this.query('SELECT 1');
|
await this.query('SELECT 1');
|
||||||
return { success: true };
|
return { success: true };
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return { success: false, error: error instanceof Error ? error.message : String(error) };
|
return connectorTestFailure(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,9 @@ import {
|
||||||
} from '../../context/scan/constraint-discovery.js';
|
} from '../../context/scan/constraint-discovery.js';
|
||||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||||
import {
|
import {
|
||||||
|
connectorTestFailure,
|
||||||
createKtxConnectorCapabilities,
|
createKtxConnectorCapabilities,
|
||||||
|
type KtxConnectorTestResult,
|
||||||
type KtxColumnSampleInput,
|
type KtxColumnSampleInput,
|
||||||
type KtxColumnSampleResult,
|
type KtxColumnSampleResult,
|
||||||
type KtxColumnStatsInput,
|
type KtxColumnStatsInput,
|
||||||
|
|
@ -157,6 +159,15 @@ interface MysqlDistinctValueRow extends RowDataPacket {
|
||||||
val: unknown;
|
val: unknown;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
interface MysqlStatsRow extends RowDataPacket {
|
||||||
|
column_name: string;
|
||||||
|
estimated_cardinality: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface KtxMysqlColumnStatisticsResult {
|
||||||
|
cardinalityByColumn: Map<string, number>;
|
||||||
|
}
|
||||||
|
|
||||||
class DefaultMysqlPoolFactory implements KtxMysqlPoolFactory {
|
class DefaultMysqlPoolFactory implements KtxMysqlPoolFactory {
|
||||||
createPool(config: KtxMysqlPoolConfig): KtxMysqlPool {
|
createPool(config: KtxMysqlPoolConfig): KtxMysqlPool {
|
||||||
return mysql.createPool(config) as Pool;
|
return mysql.createPool(config) as Pool;
|
||||||
|
|
@ -382,7 +393,7 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
|
||||||
readonly capabilities = createKtxConnectorCapabilities({
|
readonly capabilities = createKtxConnectorCapabilities({
|
||||||
tableSampling: true,
|
tableSampling: true,
|
||||||
columnSampling: true,
|
columnSampling: true,
|
||||||
columnStats: false,
|
columnStats: true,
|
||||||
readOnlySql: true,
|
readOnlySql: true,
|
||||||
nestedAnalysis: true,
|
nestedAnalysis: true,
|
||||||
formalForeignKeys: true,
|
formalForeignKeys: true,
|
||||||
|
|
@ -413,12 +424,12 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
|
||||||
this.id = `mysql:${options.connectionId}`;
|
this.id = `mysql:${options.connectionId}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async testConnection(): Promise<{ success: boolean; error?: string }> {
|
async testConnection(): Promise<KtxConnectorTestResult> {
|
||||||
try {
|
try {
|
||||||
await this.query('SELECT 1');
|
await this.query('SELECT 1');
|
||||||
return { success: true };
|
return { success: true };
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return { success: false, error: error instanceof Error ? error.message : String(error) };
|
return connectorTestFailure(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -560,8 +571,29 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
|
||||||
return { values, nullCount: null, distinctCount: null };
|
return { values, nullCount: null, distinctCount: null };
|
||||||
}
|
}
|
||||||
|
|
||||||
async columnStats(_input: KtxColumnStatsInput, _ctx: KtxScanContext): Promise<KtxColumnStatsResult | null> {
|
async columnStats(input: KtxColumnStatsInput, _ctx: KtxScanContext): Promise<KtxColumnStatsResult | null> {
|
||||||
return null;
|
const stats = await this.getColumnStatistics(input.table);
|
||||||
|
const value = stats?.cardinalityByColumn.get(input.column);
|
||||||
|
return value === undefined
|
||||||
|
? null
|
||||||
|
: { min: null, max: null, average: null, nullCount: null, distinctCount: value };
|
||||||
|
}
|
||||||
|
|
||||||
|
async getColumnStatistics(table: KtxTableRef): Promise<KtxMysqlColumnStatisticsResult | null> {
|
||||||
|
const schema = table.db ?? this.poolConfig.database;
|
||||||
|
const sql = this.dialect.generateColumnStatisticsQuery(schema, table.name);
|
||||||
|
if (!sql) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
const rows = await this.queryRaw<MysqlStatsRow>(sql);
|
||||||
|
const cardinalityByColumn = new Map<string, number>();
|
||||||
|
for (const row of rows) {
|
||||||
|
const cardinality = Number(row.estimated_cardinality);
|
||||||
|
if (Number.isFinite(cardinality) && cardinality >= 0) {
|
||||||
|
cardinalityByColumn.set(row.column_name, cardinality);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cardinalityByColumn.size > 0 ? { cardinalityByColumn } : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
async executeReadOnly(input: KtxMysqlReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {
|
async executeReadOnly(input: KtxMysqlReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {
|
||||||
|
|
|
||||||
|
|
@ -171,8 +171,18 @@ export class KtxMysqlDialect implements KtxDialect {
|
||||||
`;
|
`;
|
||||||
}
|
}
|
||||||
|
|
||||||
generateColumnStatisticsQuery(_schemaName: string, _tableName: string): string | null {
|
generateColumnStatisticsQuery(schemaName: string, tableName: string): string | null {
|
||||||
return null;
|
return `
|
||||||
|
SELECT
|
||||||
|
COLUMN_NAME AS column_name,
|
||||||
|
MAX(CARDINALITY) AS estimated_cardinality
|
||||||
|
FROM INFORMATION_SCHEMA.STATISTICS
|
||||||
|
WHERE TABLE_SCHEMA = '${schemaName.replace(/'/g, "''")}'
|
||||||
|
AND TABLE_NAME = '${tableName.replace(/'/g, "''")}'
|
||||||
|
AND CARDINALITY IS NOT NULL
|
||||||
|
AND SEQ_IN_INDEX = 1
|
||||||
|
GROUP BY COLUMN_NAME
|
||||||
|
`;
|
||||||
}
|
}
|
||||||
|
|
||||||
generateRandomizedCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
generateRandomizedCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,9 @@ import { assertReadOnlySql, limitSqlForExecution } from '../../context/connectio
|
||||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||||
import {
|
import {
|
||||||
|
connectorTestFailure,
|
||||||
createKtxConnectorCapabilities,
|
createKtxConnectorCapabilities,
|
||||||
|
type KtxConnectorTestResult,
|
||||||
type KtxColumnSampleInput,
|
type KtxColumnSampleInput,
|
||||||
type KtxColumnSampleResult,
|
type KtxColumnSampleResult,
|
||||||
type KtxColumnStatsInput,
|
type KtxColumnStatsInput,
|
||||||
|
|
@ -442,12 +444,12 @@ export class KtxPostgresScanConnector implements KtxScanConnector {
|
||||||
this.id = `postgres:${options.connectionId}`;
|
this.id = `postgres:${options.connectionId}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async testConnection(): Promise<{ success: boolean; error?: string }> {
|
async testConnection(): Promise<KtxConnectorTestResult> {
|
||||||
try {
|
try {
|
||||||
await this.query('SELECT 1');
|
await this.query('SELECT 1');
|
||||||
return { success: true };
|
return { success: true };
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return { success: false, error: error instanceof Error ? error.message : String(error) };
|
return connectorTestFailure(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,9 @@ import { assertReadOnlySql, limitSqlForExecution } from '../../context/connectio
|
||||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||||
import {
|
import {
|
||||||
|
connectorTestFailure,
|
||||||
createKtxConnectorCapabilities,
|
createKtxConnectorCapabilities,
|
||||||
|
type KtxConnectorTestResult,
|
||||||
type KtxColumnSampleInput,
|
type KtxColumnSampleInput,
|
||||||
type KtxColumnSampleResult,
|
type KtxColumnSampleResult,
|
||||||
type KtxColumnStatsInput,
|
type KtxColumnStatsInput,
|
||||||
|
|
@ -464,7 +466,7 @@ class SnowflakeSdkDriver implements KtxSnowflakeDriver {
|
||||||
await this.query('SELECT 1');
|
await this.query('SELECT 1');
|
||||||
return { success: true };
|
return { success: true };
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return { success: false, error: error instanceof Error ? error.message : String(error) };
|
return connectorTestFailure(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -573,7 +575,7 @@ export class KtxSnowflakeScanConnector implements KtxScanConnector {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async testConnection(): Promise<{ success: boolean; error?: string }> {
|
async testConnection(): Promise<KtxConnectorTestResult> {
|
||||||
return this.getDriver().test();
|
return this.getDriver().test();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ import { fileURLToPath } from 'node:url';
|
||||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import { normalizeQueryRows } from '../../context/connections/query-executor.js';
|
import { normalizeQueryRows } from '../../context/connections/query-executor.js';
|
||||||
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
import { connectorTestFailure, createKtxConnectorCapabilities, type KtxConnectorTestResult, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||||
|
|
||||||
export interface KtxSqliteConnectionConfig {
|
export interface KtxSqliteConnectionConfig {
|
||||||
|
|
@ -167,7 +167,7 @@ export class KtxSqliteScanConnector implements KtxScanConnector {
|
||||||
this.id = `sqlite:${options.connectionId}`;
|
this.id = `sqlite:${options.connectionId}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async testConnection(): Promise<{ success: boolean; error?: string }> {
|
async testConnection(): Promise<KtxConnectorTestResult> {
|
||||||
try {
|
try {
|
||||||
if (!existsSync(this.dbPath) || !statSync(this.dbPath).isFile()) {
|
if (!existsSync(this.dbPath) || !statSync(this.dbPath).isFile()) {
|
||||||
return { success: false, error: `File not found: ${this.dbPath}` };
|
return { success: false, error: `File not found: ${this.dbPath}` };
|
||||||
|
|
@ -175,7 +175,7 @@ export class KtxSqliteScanConnector implements KtxScanConnector {
|
||||||
this.database().prepare('SELECT 1').get();
|
this.database().prepare('SELECT 1').get();
|
||||||
return { success: true };
|
return { success: true };
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return { success: false, error: error instanceof Error ? error.message : String(error) };
|
return connectorTestFailure(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,9 @@ import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||||
import {
|
import {
|
||||||
|
connectorTestFailure,
|
||||||
createKtxConnectorCapabilities,
|
createKtxConnectorCapabilities,
|
||||||
|
type KtxConnectorTestResult,
|
||||||
type KtxColumnSampleInput,
|
type KtxColumnSampleInput,
|
||||||
type KtxColumnSampleResult,
|
type KtxColumnSampleResult,
|
||||||
type KtxColumnStatsInput,
|
type KtxColumnStatsInput,
|
||||||
|
|
@ -384,12 +386,12 @@ export class KtxSqlServerScanConnector implements KtxScanConnector {
|
||||||
this.id = `sqlserver:${options.connectionId}`;
|
this.id = `sqlserver:${options.connectionId}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
async testConnection(): Promise<{ success: boolean; error?: string }> {
|
async testConnection(): Promise<KtxConnectorTestResult> {
|
||||||
try {
|
try {
|
||||||
await this.query('SELECT 1');
|
await this.query('SELECT 1');
|
||||||
return { success: true };
|
return { success: true };
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return { success: false, error: error instanceof Error ? error.message : String(error) };
|
return connectorTestFailure(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -12,6 +12,13 @@ import { buildPublicIngestPlan, executePublicIngestTarget, publicProgressMessage
|
||||||
import { createAggregateProgressPort } from './progress-port-adapter.js';
|
import { createAggregateProgressPort } from './progress-port-adapter.js';
|
||||||
import { formatDuration } from './demo-metrics.js';
|
import { formatDuration } from './demo-metrics.js';
|
||||||
import { profileMark } from './startup-profile.js';
|
import { profileMark } from './startup-profile.js';
|
||||||
|
import {
|
||||||
|
isFreshStarCountCache,
|
||||||
|
readStarCountCache,
|
||||||
|
writeStarCountCache,
|
||||||
|
} from './star-prompt/cache.js';
|
||||||
|
import { fetchGitHubStarCount as defaultFetchGitHubStarCount } from './star-prompt/star-count.js';
|
||||||
|
import { renderStarPromptLine } from './star-prompt/star-line.js';
|
||||||
|
|
||||||
profileMark('module:context-build-view');
|
profileMark('module:context-build-view');
|
||||||
|
|
||||||
|
|
@ -79,6 +86,7 @@ export interface ContextBuildViewState {
|
||||||
frame: number;
|
frame: number;
|
||||||
startedAt: number | null;
|
startedAt: number | null;
|
||||||
totalElapsedMs: number;
|
totalElapsedMs: number;
|
||||||
|
starCount: number | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ContextBuildArgs {
|
export interface ContextBuildArgs {
|
||||||
|
|
@ -121,6 +129,8 @@ interface CompletedItemName {
|
||||||
interface ContextBuildRenderOptions {
|
interface ContextBuildRenderOptions {
|
||||||
styled?: boolean;
|
styled?: boolean;
|
||||||
showHint?: boolean;
|
showHint?: boolean;
|
||||||
|
showStarPrompt?: boolean;
|
||||||
|
columns?: number;
|
||||||
hintText?: string;
|
hintText?: string;
|
||||||
projectDir?: string;
|
projectDir?: string;
|
||||||
title?: string;
|
title?: string;
|
||||||
|
|
@ -138,6 +148,15 @@ export interface ContextBuildDeps {
|
||||||
now?: () => number;
|
now?: () => number;
|
||||||
onSourceProgress?: (sources: ContextBuildSourceProgressUpdate[]) => void;
|
onSourceProgress?: (sources: ContextBuildSourceProgressUpdate[]) => void;
|
||||||
sourceProgressThrottleMs?: number;
|
sourceProgressThrottleMs?: number;
|
||||||
|
fetchStarCount?: typeof defaultFetchGitHubStarCount;
|
||||||
|
starPromptEnv?: StarPromptEnv;
|
||||||
|
starPromptHomeDir?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface StarPromptEnv extends NodeJS.ProcessEnv {
|
||||||
|
CI?: string;
|
||||||
|
DO_NOT_TRACK?: string;
|
||||||
|
KTX_NO_STAR?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Rendering ---
|
// --- Rendering ---
|
||||||
|
|
@ -427,6 +446,14 @@ export function renderContextBuildView(
|
||||||
lines.push('');
|
lines.push('');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (options.showStarPrompt && hasActive) {
|
||||||
|
const starPrompt = renderStarPromptLine({
|
||||||
|
count: state.starCount,
|
||||||
|
columns: options.columns ?? 80,
|
||||||
|
});
|
||||||
|
lines.push(styled ? dim(starPrompt) : starPrompt);
|
||||||
|
}
|
||||||
|
|
||||||
if (options.showHint && hasActive) {
|
if (options.showHint && hasActive) {
|
||||||
const hintContent = options.hintText ?? 'Ctrl+C to stop';
|
const hintContent = options.hintText ?? 'Ctrl+C to stop';
|
||||||
const hint = ` ${hintContent}`;
|
const hint = ` ${hintContent}`;
|
||||||
|
|
@ -584,6 +611,7 @@ export function viewStateFromSourceProgress(
|
||||||
frame: 0,
|
frame: 0,
|
||||||
startedAt: startedAtMs ?? null,
|
startedAt: startedAtMs ?? null,
|
||||||
totalElapsedMs: startedAtMs ? now - startedAtMs : 0,
|
totalElapsedMs: startedAtMs ? now - startedAtMs : 0,
|
||||||
|
starCount: null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -631,6 +659,9 @@ export function createRepainter(io: KtxCliIo) {
|
||||||
hasPainted = true;
|
hasPainted = true;
|
||||||
lastCursorUpRows = cursorUpRowsAfterWrite(content);
|
lastCursorUpRows = cursorUpRowsAfterWrite(content);
|
||||||
},
|
},
|
||||||
|
columns() {
|
||||||
|
return terminalColumns();
|
||||||
|
},
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -806,6 +837,7 @@ export function initViewState(targets: KtxPublicIngestPlanTarget[]): ContextBuil
|
||||||
frame: 0,
|
frame: 0,
|
||||||
startedAt: null,
|
startedAt: null,
|
||||||
totalElapsedMs: 0,
|
totalElapsedMs: 0,
|
||||||
|
starCount: null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -817,6 +849,50 @@ function formatProgressDetail(
|
||||||
return `[${percent}%] ${publicProgressMessage(update.message, target)}`;
|
return `[${percent}%] ${publicProgressMessage(update.message, target)}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const STAR_COUNT_CACHE_TTL_MS = 24 * 60 * 60 * 1000;
|
||||||
|
|
||||||
|
function envFlag(value: string | undefined): boolean {
|
||||||
|
return value !== undefined && value !== '' && value !== '0' && value !== 'false';
|
||||||
|
}
|
||||||
|
|
||||||
|
function shouldSuppressStarPrompt(env: StarPromptEnv): boolean {
|
||||||
|
return envFlag(env.CI) || envFlag(env.DO_NOT_TRACK) || envFlag(env.KTX_NO_STAR);
|
||||||
|
}
|
||||||
|
|
||||||
|
function startStarPromptCountRefresh(input: {
|
||||||
|
fetchStarCount: typeof defaultFetchGitHubStarCount;
|
||||||
|
homeDir?: string;
|
||||||
|
now: () => number;
|
||||||
|
paint: () => void;
|
||||||
|
state: ContextBuildViewState;
|
||||||
|
}): void {
|
||||||
|
const cached = readStarCountCache({ homeDir: input.homeDir });
|
||||||
|
if (cached) {
|
||||||
|
input.state.starCount = cached.count;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isFreshStarCountCache(cached, new Date(input.now()), STAR_COUNT_CACHE_TTL_MS)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
void input.fetchStarCount()
|
||||||
|
.then((count) => {
|
||||||
|
if (typeof count !== 'number' || !Number.isFinite(count)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
input.state.starCount = count;
|
||||||
|
input.paint();
|
||||||
|
void writeStarCountCache(
|
||||||
|
{
|
||||||
|
count,
|
||||||
|
fetchedAt: new Date(input.now()).toISOString(),
|
||||||
|
},
|
||||||
|
{ homeDir: input.homeDir },
|
||||||
|
);
|
||||||
|
})
|
||||||
|
.catch(() => undefined);
|
||||||
|
}
|
||||||
|
|
||||||
export async function runContextBuild(
|
export async function runContextBuild(
|
||||||
project: KtxPublicIngestProject,
|
project: KtxPublicIngestProject,
|
||||||
args: ContextBuildArgs,
|
args: ContextBuildArgs,
|
||||||
|
|
@ -838,13 +914,31 @@ export async function runContextBuild(
|
||||||
state.startedAt = nowFn();
|
state.startedAt = nowFn();
|
||||||
|
|
||||||
const repainter = isTTY ? createRepainter(io) : null;
|
const repainter = isTTY ? createRepainter(io) : null;
|
||||||
|
const starPromptEnabled = repainter !== null && !shouldSuppressStarPrompt(deps.starPromptEnv ?? process.env);
|
||||||
const viewOpts = {
|
const viewOpts = {
|
||||||
styled: true,
|
styled: true,
|
||||||
projectDir: args.projectDir,
|
projectDir: args.projectDir,
|
||||||
notices: plan.notices ?? [],
|
notices: plan.notices ?? [],
|
||||||
warnings: plan.warnings,
|
warnings: plan.warnings,
|
||||||
};
|
};
|
||||||
const paint = (hint: boolean) => repainter?.paint(renderContextBuildView(state, { ...viewOpts, showHint: hint }));
|
const paint = (hint: boolean) =>
|
||||||
|
repainter?.paint(
|
||||||
|
renderContextBuildView(state, {
|
||||||
|
...viewOpts,
|
||||||
|
showHint: hint,
|
||||||
|
showStarPrompt: starPromptEnabled && hint,
|
||||||
|
columns: repainter.columns(),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
if (starPromptEnabled) {
|
||||||
|
startStarPromptCountRefresh({
|
||||||
|
fetchStarCount: deps.fetchStarCount ?? defaultFetchGitHubStarCount,
|
||||||
|
homeDir: deps.starPromptHomeDir,
|
||||||
|
now: nowFn,
|
||||||
|
paint: () => paint(true),
|
||||||
|
state,
|
||||||
|
});
|
||||||
|
}
|
||||||
paint(true);
|
paint(true);
|
||||||
|
|
||||||
let spinnerInterval: ReturnType<typeof setInterval> | null = null;
|
let spinnerInterval: ReturnType<typeof setInterval> | null = null;
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,6 @@ export interface KtxDriverRegistration {
|
||||||
readonly driver: KtxConnectionDriver;
|
readonly driver: KtxConnectionDriver;
|
||||||
readonly scopeConfigKey: KtxScopeConfigKey | null;
|
readonly scopeConfigKey: KtxScopeConfigKey | null;
|
||||||
readonly hasHistoricSqlReader: boolean;
|
readonly hasHistoricSqlReader: boolean;
|
||||||
readonly hasLocalQueryExecutor: boolean;
|
|
||||||
load(): Promise<KtxDriverConnectorModule>;
|
load(): Promise<KtxDriverConnectorModule>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -31,7 +30,6 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
|
||||||
driver: 'bigquery',
|
driver: 'bigquery',
|
||||||
scopeConfigKey: 'dataset_ids',
|
scopeConfigKey: 'dataset_ids',
|
||||||
hasHistoricSqlReader: true,
|
hasHistoricSqlReader: true,
|
||||||
hasLocalQueryExecutor: false,
|
|
||||||
load: async () => {
|
load: async () => {
|
||||||
const m = await import('../../connectors/bigquery/connector.js');
|
const m = await import('../../connectors/bigquery/connector.js');
|
||||||
return {
|
return {
|
||||||
|
|
@ -53,7 +51,6 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
|
||||||
driver: 'clickhouse',
|
driver: 'clickhouse',
|
||||||
scopeConfigKey: 'databases',
|
scopeConfigKey: 'databases',
|
||||||
hasHistoricSqlReader: false,
|
hasHistoricSqlReader: false,
|
||||||
hasLocalQueryExecutor: false,
|
|
||||||
load: async () => {
|
load: async () => {
|
||||||
const m = await import('../../connectors/clickhouse/connector.js');
|
const m = await import('../../connectors/clickhouse/connector.js');
|
||||||
return {
|
return {
|
||||||
|
|
@ -75,7 +72,6 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
|
||||||
driver: 'mysql',
|
driver: 'mysql',
|
||||||
scopeConfigKey: 'schemas',
|
scopeConfigKey: 'schemas',
|
||||||
hasHistoricSqlReader: false,
|
hasHistoricSqlReader: false,
|
||||||
hasLocalQueryExecutor: false,
|
|
||||||
load: async () => {
|
load: async () => {
|
||||||
const m = await import('../../connectors/mysql/connector.js');
|
const m = await import('../../connectors/mysql/connector.js');
|
||||||
return {
|
return {
|
||||||
|
|
@ -97,7 +93,6 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
|
||||||
driver: 'postgres',
|
driver: 'postgres',
|
||||||
scopeConfigKey: 'schemas',
|
scopeConfigKey: 'schemas',
|
||||||
hasHistoricSqlReader: true,
|
hasHistoricSqlReader: true,
|
||||||
hasLocalQueryExecutor: true,
|
|
||||||
load: async () => {
|
load: async () => {
|
||||||
const m = await import('../../connectors/postgres/connector.js');
|
const m = await import('../../connectors/postgres/connector.js');
|
||||||
return {
|
return {
|
||||||
|
|
@ -119,7 +114,6 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
|
||||||
driver: 'sqlite',
|
driver: 'sqlite',
|
||||||
scopeConfigKey: null,
|
scopeConfigKey: null,
|
||||||
hasHistoricSqlReader: false,
|
hasHistoricSqlReader: false,
|
||||||
hasLocalQueryExecutor: true,
|
|
||||||
load: async () => {
|
load: async () => {
|
||||||
const m = await import('../../connectors/sqlite/connector.js');
|
const m = await import('../../connectors/sqlite/connector.js');
|
||||||
return {
|
return {
|
||||||
|
|
@ -141,7 +135,6 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
|
||||||
driver: 'snowflake',
|
driver: 'snowflake',
|
||||||
scopeConfigKey: 'schema_names',
|
scopeConfigKey: 'schema_names',
|
||||||
hasHistoricSqlReader: true,
|
hasHistoricSqlReader: true,
|
||||||
hasLocalQueryExecutor: false,
|
|
||||||
load: async () => {
|
load: async () => {
|
||||||
const m = await import('../../connectors/snowflake/connector.js');
|
const m = await import('../../connectors/snowflake/connector.js');
|
||||||
return {
|
return {
|
||||||
|
|
@ -163,7 +156,6 @@ export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistrat
|
||||||
driver: 'sqlserver',
|
driver: 'sqlserver',
|
||||||
scopeConfigKey: 'schemas',
|
scopeConfigKey: 'schemas',
|
||||||
hasHistoricSqlReader: false,
|
hasHistoricSqlReader: false,
|
||||||
hasLocalQueryExecutor: false,
|
|
||||||
load: async () => {
|
load: async () => {
|
||||||
const m = await import('../../connectors/sqlserver/connector.js');
|
const m = await import('../../connectors/sqlserver/connector.js');
|
||||||
return {
|
return {
|
||||||
|
|
|
||||||
|
|
@ -1,59 +0,0 @@
|
||||||
import { driverRegistrations, getDriverRegistration } from './drivers.js';
|
|
||||||
import { createPostgresQueryExecutor } from './postgres-query-executor.js';
|
|
||||||
import type {
|
|
||||||
KtxSqlQueryExecutionInput,
|
|
||||||
KtxSqlQueryExecutionResult,
|
|
||||||
KtxSqlQueryExecutorPort,
|
|
||||||
} from './query-executor.js';
|
|
||||||
import { createSqliteQueryExecutor } from './sqlite-query-executor.js';
|
|
||||||
import type { KtxConnectionDriver } from '../scan/types.js';
|
|
||||||
|
|
||||||
export interface DefaultLocalQueryExecutorOptions {
|
|
||||||
postgres?: KtxSqlQueryExecutorPort;
|
|
||||||
sqlite?: KtxSqlQueryExecutorPort;
|
|
||||||
}
|
|
||||||
|
|
||||||
function driverFor(input: KtxSqlQueryExecutionInput): string {
|
|
||||||
return String(input.connection?.driver ?? '').toLowerCase();
|
|
||||||
}
|
|
||||||
|
|
||||||
function localExecutorMap(
|
|
||||||
options: DefaultLocalQueryExecutorOptions,
|
|
||||||
): Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> {
|
|
||||||
const wiredExecutors: Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> = {
|
|
||||||
postgres: options.postgres ?? createPostgresQueryExecutor(),
|
|
||||||
sqlite: options.sqlite ?? createSqliteQueryExecutor(),
|
|
||||||
};
|
|
||||||
|
|
||||||
const executors: Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> = {};
|
|
||||||
for (const registration of Object.values(driverRegistrations)) {
|
|
||||||
if (!registration.hasLocalQueryExecutor) continue;
|
|
||||||
const executor = wiredExecutors[registration.driver];
|
|
||||||
if (executor) {
|
|
||||||
executors[registration.driver] = executor;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return executors;
|
|
||||||
}
|
|
||||||
|
|
||||||
export function createDefaultLocalQueryExecutor(options: DefaultLocalQueryExecutorOptions = {}): KtxSqlQueryExecutorPort {
|
|
||||||
const executors = localExecutorMap(options);
|
|
||||||
|
|
||||||
return {
|
|
||||||
async execute(input: KtxSqlQueryExecutionInput): Promise<KtxSqlQueryExecutionResult> {
|
|
||||||
const driver = driverFor(input);
|
|
||||||
const registration = getDriverRegistration(driver);
|
|
||||||
if (!registration?.hasLocalQueryExecutor) {
|
|
||||||
throw new Error(`No local query executor is configured for driver "${input.connection?.driver ?? 'unknown'}".`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const executor = executors[registration.driver];
|
|
||||||
if (!executor) {
|
|
||||||
throw new Error(
|
|
||||||
`Local query executor flag is enabled for driver "${registration.driver}", but no executor factory is wired.`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
return executor.execute(input);
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
@ -1,78 +0,0 @@
|
||||||
import { Client, type ClientConfig } from 'pg';
|
|
||||||
import type {
|
|
||||||
KtxSqlQueryExecutionInput,
|
|
||||||
KtxSqlQueryExecutionResult,
|
|
||||||
KtxSqlQueryExecutorPort,
|
|
||||||
} from './query-executor.js';
|
|
||||||
import { limitSqlForExecution } from './read-only-sql.js';
|
|
||||||
|
|
||||||
interface PgClientLike {
|
|
||||||
connect(): Promise<unknown>;
|
|
||||||
query(input: string | { text: string; rowMode: 'array' }): Promise<{
|
|
||||||
fields: Array<{ name: string }>;
|
|
||||||
rows: unknown[][];
|
|
||||||
command: string;
|
|
||||||
rowCount: number | null;
|
|
||||||
}>;
|
|
||||||
end(): Promise<void>;
|
|
||||||
}
|
|
||||||
|
|
||||||
interface PostgresQueryExecutorOptions {
|
|
||||||
statementTimeoutMs?: number;
|
|
||||||
queryTimeoutMs?: number;
|
|
||||||
connectionTimeoutMs?: number;
|
|
||||||
clientFactory?: (config: ClientConfig) => PgClientLike;
|
|
||||||
}
|
|
||||||
|
|
||||||
function connectionDriver(input: KtxSqlQueryExecutionInput): string {
|
|
||||||
return String(input.connection?.driver ?? '').toLowerCase();
|
|
||||||
}
|
|
||||||
|
|
||||||
function createDefaultClient(config: ClientConfig): PgClientLike {
|
|
||||||
return new Client(config);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function createPostgresQueryExecutor(options: PostgresQueryExecutorOptions = {}): KtxSqlQueryExecutorPort {
|
|
||||||
const clientFactory = options.clientFactory ?? createDefaultClient;
|
|
||||||
return {
|
|
||||||
async execute(input: KtxSqlQueryExecutionInput): Promise<KtxSqlQueryExecutionResult> {
|
|
||||||
const driver = connectionDriver(input);
|
|
||||||
const connection = input.connection;
|
|
||||||
if (driver !== 'postgres') {
|
|
||||||
throw new Error(`Local Postgres execution cannot run driver "${connection?.driver ?? 'unknown'}".`);
|
|
||||||
}
|
|
||||||
if (typeof connection?.url !== 'string' || connection.url.trim().length === 0) {
|
|
||||||
throw new Error(`Local Postgres execution requires connections.${input.connectionId}.url.`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const client = clientFactory({
|
|
||||||
connectionString: connection.url,
|
|
||||||
statement_timeout: options.statementTimeoutMs ?? 30_000,
|
|
||||||
query_timeout: options.queryTimeoutMs ?? 35_000,
|
|
||||||
connectionTimeoutMillis: options.connectionTimeoutMs ?? 5_000,
|
|
||||||
application_name: 'ktx-local-query',
|
|
||||||
});
|
|
||||||
await client.connect();
|
|
||||||
try {
|
|
||||||
await client.query('BEGIN READ ONLY');
|
|
||||||
const result = await client.query({
|
|
||||||
text: limitSqlForExecution(input.sql, input.maxRows),
|
|
||||||
rowMode: 'array',
|
|
||||||
});
|
|
||||||
await client.query('COMMIT');
|
|
||||||
return {
|
|
||||||
headers: result.fields.map((field) => field.name),
|
|
||||||
rows: result.rows,
|
|
||||||
totalRows: result.rows.length,
|
|
||||||
command: result.command,
|
|
||||||
rowCount: result.rowCount,
|
|
||||||
};
|
|
||||||
} catch (error) {
|
|
||||||
await client.query('ROLLBACK').catch(() => undefined);
|
|
||||||
throw error;
|
|
||||||
} finally {
|
|
||||||
await client.end();
|
|
||||||
}
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
@ -8,7 +8,7 @@ export interface KtxSqlQueryExecutionInput {
|
||||||
maxRows?: number;
|
maxRows?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxSqlQueryExecutionResult {
|
interface KtxSqlQueryExecutionResult {
|
||||||
headers: string[];
|
headers: string[];
|
||||||
rows: unknown[][];
|
rows: unknown[][];
|
||||||
totalRows: number;
|
totalRows: number;
|
||||||
|
|
|
||||||
|
|
@ -1,92 +0,0 @@
|
||||||
import { isAbsolute, resolve } from 'node:path';
|
|
||||||
import { fileURLToPath } from 'node:url';
|
|
||||||
import Database from 'better-sqlite3';
|
|
||||||
import { readFileSync } from 'node:fs';
|
|
||||||
import { homedir } from 'node:os';
|
|
||||||
import type {
|
|
||||||
KtxSqlQueryExecutionInput,
|
|
||||||
KtxSqlQueryExecutionResult,
|
|
||||||
KtxSqlQueryExecutorPort,
|
|
||||||
} from './query-executor.js';
|
|
||||||
import { normalizeQueryRows } from './query-executor.js';
|
|
||||||
import { limitSqlForExecution } from './read-only-sql.js';
|
|
||||||
|
|
||||||
type SqliteConnectionConfig = Record<string, unknown> | undefined;
|
|
||||||
|
|
||||||
function connectionDriver(input: KtxSqlQueryExecutionInput): string {
|
|
||||||
return String(input.connection?.driver ?? '').toLowerCase();
|
|
||||||
}
|
|
||||||
|
|
||||||
function stringConfigValue(connection: SqliteConnectionConfig, key: string): string | undefined {
|
|
||||||
const value = connection?.[key];
|
|
||||||
return typeof value === 'string' && value.trim().length > 0 ? resolveStringReference(key, value.trim()) : undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
function resolveStringReference(key: string, value: string): string {
|
|
||||||
if (value.startsWith('env:')) {
|
|
||||||
return process.env[value.slice('env:'.length)] ?? '';
|
|
||||||
}
|
|
||||||
if (key !== 'url' && value.startsWith('file:')) {
|
|
||||||
const rawPath = value.slice('file:'.length);
|
|
||||||
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
|
|
||||||
return readFileSync(path, 'utf-8').trim();
|
|
||||||
}
|
|
||||||
return value;
|
|
||||||
}
|
|
||||||
|
|
||||||
function sqlitePathFromUrl(url: string): string {
|
|
||||||
if (url.startsWith('file:')) {
|
|
||||||
return fileURLToPath(url);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (url.startsWith('sqlite:')) {
|
|
||||||
const parsed = new URL(url);
|
|
||||||
if (parsed.pathname.length > 0) {
|
|
||||||
return decodeURIComponent(parsed.pathname);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return url;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** @internal */
|
|
||||||
export function sqliteDatabasePathFromConnection(input: KtxSqlQueryExecutionInput): string {
|
|
||||||
const driver = connectionDriver(input);
|
|
||||||
if (driver !== 'sqlite') {
|
|
||||||
throw new Error(`Local SQLite execution cannot run driver "${input.connection?.driver ?? 'unknown'}".`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const pathValue = stringConfigValue(input.connection, 'path');
|
|
||||||
const urlValue = stringConfigValue(input.connection, 'url');
|
|
||||||
if (!pathValue && !urlValue) {
|
|
||||||
throw new Error(
|
|
||||||
`Local SQLite execution requires connections.${input.connectionId}.path or connections.${input.connectionId}.url.`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
const candidate = pathValue ?? sqlitePathFromUrl(urlValue as string);
|
|
||||||
return isAbsolute(candidate) ? candidate : resolve(input.projectDir ?? process.cwd(), candidate);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function createSqliteQueryExecutor(): KtxSqlQueryExecutorPort {
|
|
||||||
return {
|
|
||||||
async execute(input: KtxSqlQueryExecutionInput): Promise<KtxSqlQueryExecutionResult> {
|
|
||||||
const sql = limitSqlForExecution(input.sql, input.maxRows);
|
|
||||||
const dbPath = sqliteDatabasePathFromConnection(input);
|
|
||||||
const db = new Database(dbPath, { readonly: true, fileMustExist: true });
|
|
||||||
try {
|
|
||||||
const statement = db.prepare(sql);
|
|
||||||
const rows = statement.all() as unknown[];
|
|
||||||
return {
|
|
||||||
headers: statement.columns().map((column) => column.name),
|
|
||||||
rows: normalizeQueryRows(rows),
|
|
||||||
totalRows: rows.length,
|
|
||||||
command: 'SELECT',
|
|
||||||
rowCount: rows.length,
|
|
||||||
};
|
|
||||||
} finally {
|
|
||||||
db.close();
|
|
||||||
}
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
|
||||||
39
packages/cli/src/context/core/abort.ts
Normal file
39
packages/cli/src/context/core/abort.ts
Normal file
|
|
@ -0,0 +1,39 @@
|
||||||
|
/** @internal */
|
||||||
|
export function createAbortError(message = 'Aborted'): DOMException {
|
||||||
|
return new DOMException(message, 'AbortError');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isAbortError(error: unknown): boolean {
|
||||||
|
if (error instanceof DOMException && error.name === 'AbortError') {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (!error || typeof error !== 'object') {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const record = error as { name?: unknown; code?: unknown };
|
||||||
|
return record.name === 'AbortError' || record.code === 'ABORT_ERR';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
|
export function throwIfAborted(signal?: AbortSignal): void {
|
||||||
|
if (signal?.aborted) {
|
||||||
|
throw createAbortError();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function linkAbortSignal(parent?: AbortSignal): { controller: AbortController; dispose: () => void } {
|
||||||
|
const controller = new AbortController();
|
||||||
|
if (!parent) {
|
||||||
|
return { controller, dispose: () => undefined };
|
||||||
|
}
|
||||||
|
if (parent.aborted) {
|
||||||
|
controller.abort(createAbortError());
|
||||||
|
return { controller, dispose: () => undefined };
|
||||||
|
}
|
||||||
|
const onAbort = () => controller.abort(createAbortError());
|
||||||
|
parent.addEventListener('abort', onAbort, { once: true });
|
||||||
|
return {
|
||||||
|
controller,
|
||||||
|
dispose: () => parent.removeEventListener('abort', onAbort),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
@ -24,6 +24,21 @@ function sanitizedGitEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEn
|
||||||
return sanitized;
|
return sanitized;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function createSimpleGit(baseDir: string): SimpleGit {
|
/**
|
||||||
return simpleGit({ baseDir, unsafe: { allowUnsafeAskPass: true } }).env(sanitizedGitEnv());
|
* Create a simple-git client scoped to `baseDir`. When an identity is provided, ktx's own
|
||||||
|
* commits carry it through the GIT_AUTHOR and GIT_COMMITTER environment variables instead of
|
||||||
|
* relying on repo-local or global git config. This keeps commits working when the project
|
||||||
|
* directory is an existing repo ktx did not create and the machine has no configured git
|
||||||
|
* identity (e.g. a fresh Mac with no ~/.gitconfig), without mutating the user's repo config.
|
||||||
|
* Explicit `--author` flags on individual commits still take precedence over GIT_AUTHOR_NAME.
|
||||||
|
*/
|
||||||
|
export function createSimpleGit(baseDir: string, identity?: { name: string; email: string }): SimpleGit {
|
||||||
|
const env = sanitizedGitEnv();
|
||||||
|
if (identity?.name && identity.email) {
|
||||||
|
env.GIT_AUTHOR_NAME = identity.name;
|
||||||
|
env.GIT_AUTHOR_EMAIL = identity.email;
|
||||||
|
env.GIT_COMMITTER_NAME = identity.name;
|
||||||
|
env.GIT_COMMITTER_EMAIL = identity.email;
|
||||||
|
}
|
||||||
|
return simpleGit({ baseDir, unsafe: { allowUnsafeAskPass: true } }).env(env);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
import { promises as fs } from 'node:fs';
|
import { promises as fs } from 'node:fs';
|
||||||
import { dirname, join } from 'node:path';
|
import { dirname, join } from 'node:path';
|
||||||
import type { SimpleGit } from 'simple-git';
|
import { CheckRepoActions, type SimpleGit } from 'simple-git';
|
||||||
import { noopLogger, resolveConfigDir, type KtxCoreConfig, type KtxLogger } from './config.js';
|
import { noopLogger, resolveConfigDir, type KtxCoreConfig, type KtxLogger } from './config.js';
|
||||||
import { createSimpleGit } from './git-env.js';
|
import { createSimpleGit } from './git-env.js';
|
||||||
|
|
||||||
|
|
@ -85,8 +85,12 @@ export class GitService {
|
||||||
await fs.mkdir(this.configDir, { recursive: true });
|
await fs.mkdir(this.configDir, { recursive: true });
|
||||||
this.logger.log(`Config directory ensured at: ${this.configDir}`);
|
this.logger.log(`Config directory ensured at: ${this.configDir}`);
|
||||||
|
|
||||||
// Initialize simple-git
|
// Initialize simple-git. Carry ktx's identity in the environment so commits succeed even
|
||||||
this.git = createSimpleGit(this.configDir);
|
// when this repo already exists and the machine has no configured git identity.
|
||||||
|
this.git = createSimpleGit(this.configDir, {
|
||||||
|
name: this.config.git.userName,
|
||||||
|
email: this.config.git.userEmail,
|
||||||
|
});
|
||||||
|
|
||||||
// Initialize git repository
|
// Initialize git repository
|
||||||
await this.initialize();
|
await this.initialize();
|
||||||
|
|
@ -94,14 +98,16 @@ export class GitService {
|
||||||
|
|
||||||
private async initialize(): Promise<void> {
|
private async initialize(): Promise<void> {
|
||||||
try {
|
try {
|
||||||
// Check if already initialized
|
// Adopt an existing repo ONLY when this directory is itself that repo's root.
|
||||||
const isRepo = await this.git.checkIsRepo();
|
// When it sits below an enclosing repo, a plain checkIsRepo() is true and ktx
|
||||||
|
// would silently piggyback on the enclosing tree — but every ktx relative path
|
||||||
|
// (file-store writes, session worktrees, squash-merges, reindex scans) assumes
|
||||||
|
// this directory IS the working-tree root. So treat "inside an enclosing repo"
|
||||||
|
// the same as "no repo" and initialize a dedicated repo rooted here.
|
||||||
|
const isRepoRoot = await this.git.checkIsRepo(CheckRepoActions.IS_REPO_ROOT);
|
||||||
|
|
||||||
if (!isRepo) {
|
if (!isRepoRoot) {
|
||||||
await this.git.init();
|
await this.git.init();
|
||||||
const gitConfig = this.config.git;
|
|
||||||
await this.git.addConfig('user.name', gitConfig.userName);
|
|
||||||
await this.git.addConfig('user.email', gitConfig.userEmail);
|
|
||||||
this.logger.log('Initialized git repository');
|
this.logger.log('Initialized git repository');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -125,7 +131,11 @@ export class GitService {
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.logger.error('Failed to initialize git repository', error);
|
this.logger.error('Failed to initialize git repository', error);
|
||||||
throw new Error('Failed to initialize git repository');
|
// Preserve the underlying git error: the generic message alone is undiagnosable in
|
||||||
|
// telemetry and unactionable for the user. The exception reporter walks `cause` and
|
||||||
|
// redacts secrets before send.
|
||||||
|
const detail = error instanceof Error ? error.message : String(error);
|
||||||
|
throw new Error(`Failed to initialize git repository: ${detail}`, { cause: error });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -899,7 +909,10 @@ export class GitService {
|
||||||
*/
|
*/
|
||||||
forWorktree(workdir: string): GitService {
|
forWorktree(workdir: string): GitService {
|
||||||
const scoped = new GitService(this.config, this.logger);
|
const scoped = new GitService(this.config, this.logger);
|
||||||
scoped.git = createSimpleGit(workdir);
|
scoped.git = createSimpleGit(workdir, {
|
||||||
|
name: this.config.git.userName,
|
||||||
|
email: this.config.git.userEmail,
|
||||||
|
});
|
||||||
scoped.configDir = workdir;
|
scoped.configDir = workdir;
|
||||||
return scoped;
|
return scoped;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -23,6 +23,7 @@ export interface QueryHistoryFilterProposal {
|
||||||
consideredRoleCount: number;
|
consideredRoleCount: number;
|
||||||
skipped: { reason: 'no-llm' | 'no-daemon' | 'no-in-scope-history' | 'user-block-present' } | null;
|
skipped: { reason: 'no-llm' | 'no-daemon' | 'no-in-scope-history' | 'user-block-present' } | null;
|
||||||
warnings: string[];
|
warnings: string[];
|
||||||
|
parseFailedTemplateIds: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ProposeQueryHistoryServiceAccountFiltersInput {
|
export interface ProposeQueryHistoryServiceAccountFiltersInput {
|
||||||
|
|
@ -74,7 +75,7 @@ const queryHistoryFilterAdjudicationSchema = z.object({
|
||||||
type QueryHistoryFilterAdjudication = z.infer<typeof queryHistoryFilterAdjudicationSchema>;
|
type QueryHistoryFilterAdjudication = z.infer<typeof queryHistoryFilterAdjudicationSchema>;
|
||||||
|
|
||||||
function emptyProposal(skipped: QueryHistoryFilterProposal['skipped'], warnings: string[] = []): QueryHistoryFilterProposal {
|
function emptyProposal(skipped: QueryHistoryFilterProposal['skipped'], warnings: string[] = []): QueryHistoryFilterProposal {
|
||||||
return { excludedRoles: [], consideredRoleCount: 0, skipped, warnings };
|
return { excludedRoles: [], consideredRoleCount: 0, skipped, warnings, parseFailedTemplateIds: [] };
|
||||||
}
|
}
|
||||||
|
|
||||||
function displayTableRef(ref: KtxTableRef): string {
|
function displayTableRef(ref: KtxTableRef): string {
|
||||||
|
|
@ -180,6 +181,7 @@ export async function proposeQueryHistoryServiceAccountFilters(
|
||||||
const windowDays = 'windowDays' in config ? config.windowDays : 90;
|
const windowDays = 'windowDays' in config ? config.windowDays : 90;
|
||||||
const windowStart = new Date(now.getTime() - windowDays * 24 * 60 * 60 * 1000);
|
const windowStart = new Date(now.getTime() - windowDays * 24 * 60 * 60 * 1000);
|
||||||
const warnings: string[] = [];
|
const warnings: string[] = [];
|
||||||
|
const parseFailedTemplateIds: string[] = [];
|
||||||
const snapshot: AggregatedTemplate[] = [];
|
const snapshot: AggregatedTemplate[] = [];
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
|
@ -212,7 +214,7 @@ export async function proposeQueryHistoryServiceAccountFilters(
|
||||||
for (const template of snapshot) {
|
for (const template of snapshot) {
|
||||||
const parsed = analysis.get(template.templateId);
|
const parsed = analysis.get(template.templateId);
|
||||||
if (!parsed || parsed.error) {
|
if (!parsed || parsed.error) {
|
||||||
warnings.push(`query_history_filter_picker_parse_failed:${template.templateId}`);
|
parseFailedTemplateIds.push(template.templateId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const tablesTouched = [...new Map(parsed.tablesTouched.map((ref) => [tableRefKey(ref), ref])).values()]
|
const tablesTouched = [...new Map(parsed.tablesTouched.map((ref) => [tableRefKey(ref), ref])).values()]
|
||||||
|
|
@ -236,6 +238,7 @@ export async function proposeQueryHistoryServiceAccountFilters(
|
||||||
consideredRoleCount: records.length,
|
consideredRoleCount: records.length,
|
||||||
skipped: { reason: 'no-in-scope-history' },
|
skipped: { reason: 'no-in-scope-history' },
|
||||||
warnings,
|
warnings,
|
||||||
|
parseFailedTemplateIds,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -256,6 +259,7 @@ export async function proposeQueryHistoryServiceAccountFilters(
|
||||||
...warnings,
|
...warnings,
|
||||||
`query_history_filter_picker_llm_failed:${error instanceof Error ? error.message : String(error)}`,
|
`query_history_filter_picker_llm_failed:${error instanceof Error ? error.message : String(error)}`,
|
||||||
],
|
],
|
||||||
|
parseFailedTemplateIds,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -274,5 +278,6 @@ export async function proposeQueryHistoryServiceAccountFilters(
|
||||||
consideredRoleCount: records.length,
|
consideredRoleCount: records.length,
|
||||||
skipped: input.userServiceAccountsPresent ? { reason: 'user-block-present' } : null,
|
skipped: input.userServiceAccountsPresent ? { reason: 'user-block-present' } : null,
|
||||||
warnings,
|
warnings,
|
||||||
|
parseFailedTemplateIds,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -39,7 +39,7 @@ export interface CuratorPaginationInput {
|
||||||
buildUserPrompt: (input: CuratorPaginationPromptInput) => string;
|
buildUserPrompt: (input: CuratorPaginationPromptInput) => string;
|
||||||
buildToolSet: (passNumber: number) => KtxRuntimeToolSet;
|
buildToolSet: (passNumber: number) => KtxRuntimeToolSet;
|
||||||
getReconciliationActions: () => MemoryAction[];
|
getReconciliationActions: () => MemoryAction[];
|
||||||
onStepFinish?: (info: { passNumber: number; stepIndex: number; stepBudget: number }) => void;
|
abortSignal?: AbortSignal;
|
||||||
}
|
}
|
||||||
|
|
||||||
interface CuratorPaginationResult extends ReconciliationOutcome {
|
interface CuratorPaginationResult extends ReconciliationOutcome {
|
||||||
|
|
@ -243,10 +243,7 @@ export class CuratorPaginationService implements CuratorPaginationPort {
|
||||||
sourceKey: params.input.sourceKey,
|
sourceKey: params.input.sourceKey,
|
||||||
jobId: params.input.jobId,
|
jobId: params.input.jobId,
|
||||||
forceRun: params.forceRun,
|
forceRun: params.forceRun,
|
||||||
onStepFinish: params.input.onStepFinish
|
abortSignal: params.input.abortSignal,
|
||||||
? ({ stepIndex, stepBudget }) =>
|
|
||||||
params.input.onStepFinish?.({ passNumber: params.passNumber, stepIndex, stepBudget })
|
|
||||||
: undefined,
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,7 @@ export interface RepairFinalGateFailureInput {
|
||||||
repairKind: FinalGateRepairKind;
|
repairKind: FinalGateRepairKind;
|
||||||
maxAttempts?: number;
|
maxAttempts?: number;
|
||||||
stepBudget?: number;
|
stepBudget?: number;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
}
|
}
|
||||||
|
|
||||||
const readRepairFileSchema = z.object({
|
const readRepairFileSchema = z.object({
|
||||||
|
|
@ -200,6 +201,7 @@ export async function repairFinalGateFailure(
|
||||||
jobId: input.trace.context.jobId,
|
jobId: input.trace.context.jobId,
|
||||||
repairKind: input.repairKind,
|
repairKind: input.repairKind,
|
||||||
},
|
},
|
||||||
|
abortSignal: input.abortSignal,
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@ import { dirname, join } from 'node:path';
|
||||||
import pLimit from 'p-limit';
|
import pLimit from 'p-limit';
|
||||||
import { z } from 'zod';
|
import { z } from 'zod';
|
||||||
import { type KtxLogger, noopLogger } from '../../context/core/config.js';
|
import { type KtxLogger, noopLogger } from '../../context/core/config.js';
|
||||||
|
import type { RateLimitWaitState } from '../../context/llm/rate-limit-governor.js';
|
||||||
import { createRuntimeToolDescriptorFromAiTool } from '../../context/llm/runtime-tools.js';
|
import { createRuntimeToolDescriptorFromAiTool } from '../../context/llm/runtime-tools.js';
|
||||||
import type { KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
|
import type { KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
|
||||||
import type { CaptureSession, MemoryAction } from '../../context/memory/types.js';
|
import type { CaptureSession, MemoryAction } from '../../context/memory/types.js';
|
||||||
|
|
@ -219,6 +220,10 @@ export class IngestBundleRunner {
|
||||||
}
|
}
|
||||||
|
|
||||||
async run(job: IngestBundleJob, ctx?: IngestJobContext): Promise<IngestBundleResult> {
|
async run(job: IngestBundleJob, ctx?: IngestJobContext): Promise<IngestBundleResult> {
|
||||||
|
const unsubscribeRateLimitGovernor = this.subscribeRateLimitGovernor({
|
||||||
|
trace: this.createTrace(job),
|
||||||
|
memoryFlow: ctx?.memoryFlow,
|
||||||
|
});
|
||||||
const key = job.connectionId;
|
const key = job.connectionId;
|
||||||
const previous = this.chainByConnection.get(key);
|
const previous = this.chainByConnection.get(key);
|
||||||
if (previous) {
|
if (previous) {
|
||||||
|
|
@ -241,10 +246,72 @@ export class IngestBundleRunner {
|
||||||
ctx?.memoryFlow?.finish('error', [sanitizeMemoryFlowError(error)]);
|
ctx?.memoryFlow?.finish('error', [sanitizeMemoryFlowError(error)]);
|
||||||
throw error;
|
throw error;
|
||||||
} finally {
|
} finally {
|
||||||
|
unsubscribeRateLimitGovernor();
|
||||||
await this.maybeEmitIngestProfile(job.jobId);
|
await this.maybeEmitIngestProfile(job.jobId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private formatRateLimitWait(
|
||||||
|
state: Extract<RateLimitWaitState, { kind: 'wait_tick' | 'wait_started' | 'wait_finished' }>,
|
||||||
|
): string {
|
||||||
|
const seconds = Math.ceil(state.remainingMs / 1_000);
|
||||||
|
const minutes = Math.floor(seconds / 60);
|
||||||
|
const remainder = seconds % 60;
|
||||||
|
const duration = minutes > 0 ? `${minutes}m${String(remainder).padStart(2, '0')}s` : `${seconds}s`;
|
||||||
|
const type = state.rateLimitType ? ` ${state.rateLimitType}` : '';
|
||||||
|
return `Rate-limited (${state.provider}${type}); resuming in ${duration}; Ctrl+C to stop`;
|
||||||
|
}
|
||||||
|
|
||||||
|
private subscribeRateLimitGovernor(input: {
|
||||||
|
trace: IngestTraceWriter;
|
||||||
|
memoryFlow?: MemoryFlowEventSink;
|
||||||
|
}): () => void {
|
||||||
|
const governor = this.deps.settings.rateLimitGovernor;
|
||||||
|
if (!governor) {
|
||||||
|
return () => undefined;
|
||||||
|
}
|
||||||
|
return governor.subscribe((state: RateLimitWaitState) => {
|
||||||
|
if (state.kind === 'rate_limit_observed') {
|
||||||
|
void input.trace.event('info', 'rate_limit', 'rate_limit_observed', { ...state });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (state.kind === 'concurrency_adjusted') {
|
||||||
|
void input.trace.event('info', 'rate_limit', 'concurrency_adjusted', { ...state });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
void input.trace.event('info', 'rate_limit', state.kind, { ...state });
|
||||||
|
if (state.kind === 'wait_tick' || state.kind === 'wait_started') {
|
||||||
|
input.memoryFlow?.emit({
|
||||||
|
type: 'rate_limit_wait',
|
||||||
|
provider: state.provider,
|
||||||
|
...(state.rateLimitType ? { rateLimitType: state.rateLimitType } : {}),
|
||||||
|
resumeAtMs: state.resumeAtMs,
|
||||||
|
remainingMs: state.remainingMs,
|
||||||
|
});
|
||||||
|
input.memoryFlow?.emit({
|
||||||
|
type: 'stage_progress',
|
||||||
|
stage: 'integration',
|
||||||
|
percent: 50,
|
||||||
|
message: this.formatRateLimitWait(state),
|
||||||
|
transient: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private async withRateLimitWorkSlot<T>(abortSignal: AbortSignal | undefined, fn: () => Promise<T>): Promise<T> {
|
||||||
|
const governor = this.deps.settings.rateLimitGovernor;
|
||||||
|
if (!governor) {
|
||||||
|
return fn();
|
||||||
|
}
|
||||||
|
const release = await governor.acquireWorkSlot(abortSignal);
|
||||||
|
try {
|
||||||
|
return await fn();
|
||||||
|
} finally {
|
||||||
|
release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* When profiling is enabled — via the `KTX_PROFILE_INGEST` env var or the
|
* When profiling is enabled — via the `KTX_PROFILE_INGEST` env var or the
|
||||||
* `ingest.profile` config setting — read the job's trace + tool transcripts
|
* `ingest.profile` config setting — read the job's trace + tool transcripts
|
||||||
|
|
@ -872,13 +939,13 @@ export class IngestBundleRunner {
|
||||||
workUnitSettings: { maxConcurrency: number; stepBudget: number; failureMode: 'abort' | 'continue' };
|
workUnitSettings: { maxConcurrency: number; stepBudget: number; failureMode: 'abort' | 'continue' };
|
||||||
transcriptDir: string;
|
transcriptDir: string;
|
||||||
transcriptSummaries: Map<string, MutableToolTranscriptSummary>;
|
transcriptSummaries: Map<string, MutableToolTranscriptSummary>;
|
||||||
recordTranscriptEntry(path: string): (entry: ToolCallLogEntry) => void;
|
recordTranscriptEntry(path: string): (entry: ToolCallLogEntry) => MutableToolTranscriptSummary;
|
||||||
stageIndex: StageIndex;
|
stageIndex: StageIndex;
|
||||||
includeContextEvidenceTools: boolean;
|
includeContextEvidenceTools: boolean;
|
||||||
currentTableExists(tableRef: string): Promise<boolean>;
|
currentTableExists(tableRef: string): Promise<boolean>;
|
||||||
memoryFlow?: MemoryFlowEventSink;
|
memoryFlow?: MemoryFlowEventSink;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
wuSkillNames: string[];
|
wuSkillNames: string[];
|
||||||
onStepFinish?: (info: { stepIndex: number; stepBudget: number }) => void;
|
|
||||||
}): Promise<WorkUnitOutcome> {
|
}): Promise<WorkUnitOutcome> {
|
||||||
const session: CaptureSession = {
|
const session: CaptureSession = {
|
||||||
userId: 'system',
|
userId: 'system',
|
||||||
|
|
@ -982,7 +1049,6 @@ export class IngestBundleRunner {
|
||||||
type: 'work_unit_started',
|
type: 'work_unit_started',
|
||||||
unitKey: input.wu.unitKey,
|
unitKey: input.wu.unitKey,
|
||||||
skills: input.wuSkillNames,
|
skills: input.wuSkillNames,
|
||||||
stepBudget: input.workUnitSettings.stepBudget,
|
|
||||||
});
|
});
|
||||||
return executeWorkUnit(
|
return executeWorkUnit(
|
||||||
{
|
{
|
||||||
|
|
@ -1006,8 +1072,10 @@ export class IngestBundleRunner {
|
||||||
slIndex: input.slIndex,
|
slIndex: input.slIndex,
|
||||||
priorProvenance: input.priorProvenance,
|
priorProvenance: input.priorProvenance,
|
||||||
}),
|
}),
|
||||||
buildToolSet: (wuInner) =>
|
buildToolSet: (wuInner) => {
|
||||||
wrapToolsWithLogger(
|
const transcriptPath = join(input.transcriptDir, `${wuInner.unitKey}.jsonl`);
|
||||||
|
const record = input.recordTranscriptEntry(transcriptPath);
|
||||||
|
return wrapToolsWithLogger(
|
||||||
buildWuToolSet({
|
buildWuToolSet({
|
||||||
sourceKey: input.job.sourceKey,
|
sourceKey: input.job.sourceKey,
|
||||||
stagedDir: input.stagedDir,
|
stagedDir: input.stagedDir,
|
||||||
|
|
@ -1016,10 +1084,23 @@ export class IngestBundleRunner {
|
||||||
emitUnmappedFallbackTool: wuEmitUnmappedFallbackTool,
|
emitUnmappedFallbackTool: wuEmitUnmappedFallbackTool,
|
||||||
toolsetTools: wuToolset.toRuntimeTools(wuToolContext),
|
toolsetTools: wuToolset.toRuntimeTools(wuToolContext),
|
||||||
}),
|
}),
|
||||||
join(input.transcriptDir, `${wuInner.unitKey}.jsonl`),
|
transcriptPath,
|
||||||
wuInner.unitKey,
|
wuInner.unitKey,
|
||||||
{ onEntry: input.recordTranscriptEntry(join(input.transcriptDir, `${wuInner.unitKey}.jsonl`)) },
|
{
|
||||||
),
|
// Drive the live HUD heartbeat from real tool calls: each invocation
|
||||||
|
// ticks the running per-unit count. This is an observed signal, not a
|
||||||
|
// re-derived turn count, so it can never overshoot a budget.
|
||||||
|
onEntry: (entry) => {
|
||||||
|
const summary = record(entry);
|
||||||
|
input.memoryFlow?.emit({
|
||||||
|
type: 'work_unit_step',
|
||||||
|
unitKey: wuInner.unitKey,
|
||||||
|
toolCalls: summary.toolCallCount,
|
||||||
|
});
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
},
|
||||||
captureSession: session,
|
captureSession: session,
|
||||||
sessionActions,
|
sessionActions,
|
||||||
modelRole: 'candidateExtraction',
|
modelRole: 'candidateExtraction',
|
||||||
|
|
@ -1028,7 +1109,7 @@ export class IngestBundleRunner {
|
||||||
connectionId: input.job.connectionId,
|
connectionId: input.job.connectionId,
|
||||||
jobId: input.job.jobId,
|
jobId: input.job.jobId,
|
||||||
toolFailureCount: (unitKey) => input.transcriptSummaries.get(unitKey)?.fatalErrorCount ?? 0,
|
toolFailureCount: (unitKey) => input.transcriptSummaries.get(unitKey)?.fatalErrorCount ?? 0,
|
||||||
onStepFinish: input.onStepFinish,
|
abortSignal: input.abortSignal,
|
||||||
},
|
},
|
||||||
input.wu,
|
input.wu,
|
||||||
);
|
);
|
||||||
|
|
@ -1097,11 +1178,12 @@ export class IngestBundleRunner {
|
||||||
const transcriptDir = this.deps.storage.resolveTranscriptDir(job.jobId);
|
const transcriptDir = this.deps.storage.resolveTranscriptDir(job.jobId);
|
||||||
const recordTranscriptEntry =
|
const recordTranscriptEntry =
|
||||||
(path: string) =>
|
(path: string) =>
|
||||||
(entry: ToolCallLogEntry): void => {
|
(entry: ToolCallLogEntry): MutableToolTranscriptSummary => {
|
||||||
const current =
|
const current =
|
||||||
transcriptSummaries.get(entry.wuKey) ?? createMutableToolTranscriptSummary(entry.wuKey, path);
|
transcriptSummaries.get(entry.wuKey) ?? createMutableToolTranscriptSummary(entry.wuKey, path);
|
||||||
recordToolTranscriptEntry(current, entry);
|
recordToolTranscriptEntry(current, entry);
|
||||||
transcriptSummaries.set(entry.wuKey, current);
|
transcriptSummaries.set(entry.wuKey, current);
|
||||||
|
return current;
|
||||||
};
|
};
|
||||||
const overrideReport = await this.loadOverrideReport(job);
|
const overrideReport = await this.loadOverrideReport(job);
|
||||||
|
|
||||||
|
|
@ -1524,7 +1606,8 @@ export class IngestBundleRunner {
|
||||||
try {
|
try {
|
||||||
await Promise.all(
|
await Promise.all(
|
||||||
workUnits.map((wu, index) =>
|
workUnits.map((wu, index) =>
|
||||||
limitWorkUnit(async () => {
|
limitWorkUnit(() =>
|
||||||
|
this.withRateLimitWorkSlot(ctx?.abortSignal, async () => {
|
||||||
const outcome = await runIsolatedWorkUnit({
|
const outcome = await runIsolatedWorkUnit({
|
||||||
unitIndex: index,
|
unitIndex: index,
|
||||||
ingestionBaseSha,
|
ingestionBaseSha,
|
||||||
|
|
@ -1532,6 +1615,7 @@ export class IngestBundleRunner {
|
||||||
patchDir,
|
patchDir,
|
||||||
trace: runTrace,
|
trace: runTrace,
|
||||||
workUnit: wu,
|
workUnit: wu,
|
||||||
|
abortSignal: ctx?.abortSignal,
|
||||||
afterSuccess: (child) => copyTransientIngestEvidence(child.workdir, sessionWorktree.workdir),
|
afterSuccess: (child) => copyTransientIngestEvidence(child.workdir, sessionWorktree.workdir),
|
||||||
run: async (child) => {
|
run: async (child) => {
|
||||||
const scopedWikiService = this.deps.wikiService.forWorktree(child.workdir);
|
const scopedWikiService = this.deps.wikiService.forWorktree(child.workdir);
|
||||||
|
|
@ -1565,11 +1649,9 @@ export class IngestBundleRunner {
|
||||||
includeContextEvidenceTools: adapter.evidenceIndexing === 'documents' && !!contextReport,
|
includeContextEvidenceTools: adapter.evidenceIndexing === 'documents' && !!contextReport,
|
||||||
currentTableExists: (tableRef) =>
|
currentTableExists: (tableRef) =>
|
||||||
this.tableRefExistsInSemanticLayer(scopedSemanticLayerService, slConnectionIds, tableRef),
|
this.tableRefExistsInSemanticLayer(scopedSemanticLayerService, slConnectionIds, tableRef),
|
||||||
|
abortSignal: ctx?.abortSignal,
|
||||||
memoryFlow,
|
memoryFlow,
|
||||||
wuSkillNames,
|
wuSkillNames,
|
||||||
onStepFinish: ({ stepIndex, stepBudget }) => {
|
|
||||||
memoryFlow?.emit({ type: 'work_unit_step', unitKey: wu.unitKey, stepIndex, stepBudget });
|
|
||||||
},
|
|
||||||
});
|
});
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
@ -1594,7 +1676,8 @@ export class IngestBundleRunner {
|
||||||
completedWorkUnits / workUnits.length,
|
completedWorkUnits / workUnits.length,
|
||||||
`${completedWorkUnits} of ${workUnits.length} work units complete`,
|
`${completedWorkUnits} of ${workUnits.length} work units complete`,
|
||||||
);
|
);
|
||||||
}),
|
}),
|
||||||
|
),
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
|
@ -1693,6 +1776,7 @@ export class IngestBundleRunner {
|
||||||
reason: context.reason,
|
reason: context.reason,
|
||||||
maxAttempts: 1,
|
maxAttempts: 1,
|
||||||
stepBudget: 12,
|
stepBudget: 12,
|
||||||
|
abortSignal: ctx?.abortSignal,
|
||||||
});
|
});
|
||||||
emitStageProgress(
|
emitStageProgress(
|
||||||
'integration',
|
'integration',
|
||||||
|
|
@ -1714,6 +1798,7 @@ export class IngestBundleRunner {
|
||||||
repairKind: 'patch_semantic_gate',
|
repairKind: 'patch_semantic_gate',
|
||||||
maxAttempts: 1,
|
maxAttempts: 1,
|
||||||
stepBudget: 16,
|
stepBudget: 16,
|
||||||
|
abortSignal: ctx?.abortSignal,
|
||||||
});
|
});
|
||||||
emitStageProgress(
|
emitStageProgress(
|
||||||
'integration',
|
'integration',
|
||||||
|
|
@ -1938,6 +2023,45 @@ export class IngestBundleRunner {
|
||||||
let curatorWarnings: string[] = [];
|
let curatorWarnings: string[] = [];
|
||||||
let reconcileOutcome: Awaited<ReturnType<typeof runReconciliationStage4>>;
|
let reconcileOutcome: Awaited<ReturnType<typeof runReconciliationStage4>>;
|
||||||
|
|
||||||
|
// Reconcile shares the work-unit liveness model: the HUD heartbeat is driven
|
||||||
|
// by real tool calls (a monotonic, observed count), not a re-derived turn
|
||||||
|
// counter. The soft cap only paces the phase progress bar; it is never shown
|
||||||
|
// to the user, so it cannot read as a misleading "X/Y" fraction.
|
||||||
|
const reconcileTranscriptPath = join(transcriptDir, 'reconcile.jsonl');
|
||||||
|
const reconcileProgressSoftCap = 40;
|
||||||
|
const buildReconcileToolSetWithHeartbeat = (): KtxRuntimeToolSet => {
|
||||||
|
const record = recordTranscriptEntry(reconcileTranscriptPath);
|
||||||
|
return wrapToolsWithLogger(
|
||||||
|
buildReconcileToolSet({
|
||||||
|
loadSkillTool: rcLoadSkill,
|
||||||
|
stageListTool: rcStageListTool,
|
||||||
|
stageDiffTool: rcStageDiffTool,
|
||||||
|
evictionListTool: rcEvictionListTool,
|
||||||
|
emitConflictResolutionTool: rcEmitConflictResolutionTool,
|
||||||
|
emitEvictionDecisionTool: rcEmitEvictionDecisionTool,
|
||||||
|
emitArtifactResolutionTool: rcEmitArtifactResolutionTool,
|
||||||
|
emitUnmappedFallbackTool: rcEmitUnmappedFallbackTool,
|
||||||
|
readRawSpanTool: rcRawSpanTool,
|
||||||
|
toolsetTools: rcToolset.toRuntimeTools(rcToolContext),
|
||||||
|
}),
|
||||||
|
reconcileTranscriptPath,
|
||||||
|
'reconcile',
|
||||||
|
{
|
||||||
|
onEntry: (entry) => {
|
||||||
|
const summary = record(entry);
|
||||||
|
if (!stage4) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const label = `Reconciling results · ${summary.toolCallCount} action${
|
||||||
|
summary.toolCallCount === 1 ? '' : 's'
|
||||||
|
}`;
|
||||||
|
emitStageProgress('reconciliation', 85, label, { transient: true });
|
||||||
|
void stage4.updateProgress(Math.min(0.95, summary.toolCallCount / reconcileProgressSoftCap), label);
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
const reconcileStartedAt = Date.now();
|
const reconcileStartedAt = Date.now();
|
||||||
const reconcileMode = contextReport && this.deps.curatorPagination ? 'curator' : 'single';
|
const reconcileMode = contextReport && this.deps.curatorPagination ? 'curator' : 'single';
|
||||||
if (contextReport && this.deps.curatorPagination) {
|
if (contextReport && this.deps.curatorPagination) {
|
||||||
|
|
@ -1960,39 +2084,9 @@ export class IngestBundleRunner {
|
||||||
}),
|
}),
|
||||||
buildUserPrompt: ({ summary, items, runState }) =>
|
buildUserPrompt: ({ summary, items, runState }) =>
|
||||||
buildReconcileUserPrompt(stageIndex, eviction, { summary, items }, reconcileNotes, runState),
|
buildReconcileUserPrompt(stageIndex, eviction, { summary, items }, reconcileNotes, runState),
|
||||||
buildToolSet: (_passNumber) =>
|
buildToolSet: (_passNumber) => buildReconcileToolSetWithHeartbeat(),
|
||||||
wrapToolsWithLogger(
|
|
||||||
buildReconcileToolSet({
|
|
||||||
loadSkillTool: rcLoadSkill,
|
|
||||||
stageListTool: rcStageListTool,
|
|
||||||
stageDiffTool: rcStageDiffTool,
|
|
||||||
evictionListTool: rcEvictionListTool,
|
|
||||||
emitConflictResolutionTool: rcEmitConflictResolutionTool,
|
|
||||||
emitEvictionDecisionTool: rcEmitEvictionDecisionTool,
|
|
||||||
emitArtifactResolutionTool: rcEmitArtifactResolutionTool,
|
|
||||||
emitUnmappedFallbackTool: rcEmitUnmappedFallbackTool,
|
|
||||||
readRawSpanTool: rcRawSpanTool,
|
|
||||||
toolsetTools: rcToolset.toRuntimeTools(rcToolContext),
|
|
||||||
}),
|
|
||||||
join(transcriptDir, 'reconcile.jsonl'),
|
|
||||||
'reconcile',
|
|
||||||
{ onEntry: recordTranscriptEntry(join(transcriptDir, 'reconcile.jsonl')) },
|
|
||||||
),
|
|
||||||
getReconciliationActions: () => reconcileActions,
|
getReconciliationActions: () => reconcileActions,
|
||||||
onStepFinish: stage4
|
abortSignal: ctx?.abortSignal,
|
||||||
? ({ passNumber, stepIndex, stepBudget }) => {
|
|
||||||
emitStageProgress(
|
|
||||||
'reconciliation',
|
|
||||||
85,
|
|
||||||
`Reconciling results: pass ${passNumber} step ${stepIndex}/${stepBudget}`,
|
|
||||||
{ transient: true },
|
|
||||||
);
|
|
||||||
void stage4.updateProgress(
|
|
||||||
stepIndex / stepBudget,
|
|
||||||
`Reconciling results · pass ${passNumber} step ${stepIndex}`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
: undefined,
|
|
||||||
});
|
});
|
||||||
curatorReport = curatorOutcome.report;
|
curatorReport = curatorOutcome.report;
|
||||||
curatorWarnings = curatorOutcome.warnings;
|
curatorWarnings = curatorOutcome.warnings;
|
||||||
|
|
@ -2015,37 +2109,13 @@ export class IngestBundleRunner {
|
||||||
canonicalPins: relevantCanonicalPins,
|
canonicalPins: relevantCanonicalPins,
|
||||||
}),
|
}),
|
||||||
buildUserPrompt: (idx, ev) => buildReconcileUserPrompt(idx, ev, undefined, reconcileNotes),
|
buildUserPrompt: (idx, ev) => buildReconcileUserPrompt(idx, ev, undefined, reconcileNotes),
|
||||||
buildToolSet: () =>
|
buildToolSet: () => buildReconcileToolSetWithHeartbeat(),
|
||||||
wrapToolsWithLogger(
|
|
||||||
buildReconcileToolSet({
|
|
||||||
loadSkillTool: rcLoadSkill,
|
|
||||||
stageListTool: rcStageListTool,
|
|
||||||
stageDiffTool: rcStageDiffTool,
|
|
||||||
evictionListTool: rcEvictionListTool,
|
|
||||||
emitConflictResolutionTool: rcEmitConflictResolutionTool,
|
|
||||||
emitEvictionDecisionTool: rcEmitEvictionDecisionTool,
|
|
||||||
emitArtifactResolutionTool: rcEmitArtifactResolutionTool,
|
|
||||||
emitUnmappedFallbackTool: rcEmitUnmappedFallbackTool,
|
|
||||||
readRawSpanTool: rcRawSpanTool,
|
|
||||||
toolsetTools: rcToolset.toRuntimeTools(rcToolContext),
|
|
||||||
}),
|
|
||||||
join(transcriptDir, 'reconcile.jsonl'),
|
|
||||||
'reconcile',
|
|
||||||
{ onEntry: recordTranscriptEntry(join(transcriptDir, 'reconcile.jsonl')) },
|
|
||||||
),
|
|
||||||
modelRole: 'reconcile',
|
modelRole: 'reconcile',
|
||||||
stepBudget: 60,
|
stepBudget: 60,
|
||||||
sourceKey: job.sourceKey,
|
sourceKey: job.sourceKey,
|
||||||
jobId: job.jobId,
|
jobId: job.jobId,
|
||||||
force: !!overrideReport,
|
force: !!overrideReport,
|
||||||
onStepFinish: stage4
|
abortSignal: ctx?.abortSignal,
|
||||||
? ({ stepIndex, stepBudget }) => {
|
|
||||||
emitStageProgress('reconciliation', 85, `Reconciling results: step ${stepIndex}/${stepBudget}`, {
|
|
||||||
transient: true,
|
|
||||||
});
|
|
||||||
void stage4.updateProgress(stepIndex / stepBudget, `Reconciling results · step ${stepIndex}`);
|
|
||||||
}
|
|
||||||
: undefined,
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
await runTrace.event(
|
await runTrace.event(
|
||||||
|
|
@ -2470,6 +2540,7 @@ export class IngestBundleRunner {
|
||||||
repairKind: 'final_artifact_gate',
|
repairKind: 'final_artifact_gate',
|
||||||
maxAttempts: 1,
|
maxAttempts: 1,
|
||||||
stepBudget: 16,
|
stepBudget: 16,
|
||||||
|
abortSignal: ctx?.abortSignal,
|
||||||
});
|
});
|
||||||
|
|
||||||
isolatedDiffSummary.gateRepairAttempts += gateRepair.attempts;
|
isolatedDiffSummary.gateRepairAttempts += gateRepair.attempts;
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,7 @@ export interface ResolveTextualConflictInput {
|
||||||
reason: string;
|
reason: string;
|
||||||
maxAttempts?: number;
|
maxAttempts?: number;
|
||||||
stepBudget?: number;
|
stepBudget?: number;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
}
|
}
|
||||||
|
|
||||||
const readIntegrationFileSchema = z.object({
|
const readIntegrationFileSchema = z.object({
|
||||||
|
|
@ -208,6 +209,7 @@ export async function resolveTextualConflict(
|
||||||
jobId: input.trace.context.jobId,
|
jobId: input.trace.context.jobId,
|
||||||
unitKey: input.unitKey,
|
unitKey: input.unitKey,
|
||||||
},
|
},
|
||||||
|
abortSignal: input.abortSignal,
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -14,6 +14,7 @@ export interface RunIsolatedWorkUnitInput {
|
||||||
patchDir: string;
|
patchDir: string;
|
||||||
trace: IngestTraceWriter;
|
trace: IngestTraceWriter;
|
||||||
workUnit: WorkUnit;
|
workUnit: WorkUnit;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
run(child: IngestSessionWorktree): Promise<WorkUnitOutcome>;
|
run(child: IngestSessionWorktree): Promise<WorkUnitOutcome>;
|
||||||
afterSuccess?(child: IngestSessionWorktree): Promise<void>;
|
afterSuccess?(child: IngestSessionWorktree): Promise<void>;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -12,6 +12,7 @@ import type { KtxSemanticLayerComputePort } from '../../context/daemon/semantic-
|
||||||
import { createRuntimeToolDescriptorFromAiTool } from '../../context/llm/runtime-tools.js';
|
import { createRuntimeToolDescriptorFromAiTool } from '../../context/llm/runtime-tools.js';
|
||||||
import { createLocalKtxLlmRuntimeFromConfig } from '../../context/llm/local-config.js';
|
import { createLocalKtxLlmRuntimeFromConfig } from '../../context/llm/local-config.js';
|
||||||
import { KtxIngestEmbeddingPortAdapter } from '../../context/llm/embedding-port.js';
|
import { KtxIngestEmbeddingPortAdapter } from '../../context/llm/embedding-port.js';
|
||||||
|
import { createRateLimitGovernorConfig, RateLimitGovernor } from '../../context/llm/rate-limit-governor.js';
|
||||||
import { RuntimeAgentRunner, type AgentRunnerPort, type KtxLlmRuntimePort, type KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
|
import { RuntimeAgentRunner, type AgentRunnerPort, type KtxLlmRuntimePort, type KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
|
||||||
import type { KtxEmbeddingProvider } from '../../llm/types.js';
|
import type { KtxEmbeddingProvider } from '../../llm/types.js';
|
||||||
import type { KtxLocalProject } from '../../context/project/project.js';
|
import type { KtxLocalProject } from '../../context/project/project.js';
|
||||||
|
|
@ -614,12 +615,12 @@ function localIngestLlmProviderGuardMessage(projectDir: string): string {
|
||||||
'ktx ingest requires llm.provider.backend: anthropic, vertex, gateway, claude-code, or codex, or an injected agentRunner.',
|
'ktx ingest requires llm.provider.backend: anthropic, vertex, gateway, claude-code, or codex, or an injected agentRunner.',
|
||||||
'Configure a local Claude Code/Codex session or API-backed LLM, then rerun ingest:',
|
'Configure a local Claude Code/Codex session or API-backed LLM, then rerun ingest:',
|
||||||
` ktx setup --project-dir ${projectDir} --llm-backend claude-code --no-input`,
|
` ktx setup --project-dir ${projectDir} --llm-backend claude-code --no-input`,
|
||||||
` ktx setup --project-dir ${projectDir} --llm-backend codex --llm-model gpt-5.5 --no-input`,
|
` ktx setup --project-dir ${projectDir} --llm-backend codex --no-input`,
|
||||||
` ktx setup --project-dir ${projectDir} --llm-backend anthropic --anthropic-api-key-env ANTHROPIC_API_KEY --llm-model claude-sonnet-4-6 --no-input`,
|
` ktx setup --project-dir ${projectDir} --llm-backend anthropic --anthropic-api-key-env ANTHROPIC_API_KEY --no-input`,
|
||||||
].join('\n');
|
].join('\n');
|
||||||
}
|
}
|
||||||
|
|
||||||
function resolveAgentRunner(options: CreateLocalBundleIngestRuntimeOptions): {
|
function resolveAgentRunner(options: CreateLocalBundleIngestRuntimeOptions, rateLimitGovernor: RateLimitGovernor): {
|
||||||
agentRunner: AgentRunnerPort;
|
agentRunner: AgentRunnerPort;
|
||||||
llmRuntime?: KtxLlmRuntimePort;
|
llmRuntime?: KtxLlmRuntimePort;
|
||||||
} {
|
} {
|
||||||
|
|
@ -628,6 +629,7 @@ function resolveAgentRunner(options: CreateLocalBundleIngestRuntimeOptions): {
|
||||||
(options.createLlmRuntime ?? createLocalKtxLlmRuntimeFromConfig)(options.project.config.llm, {
|
(options.createLlmRuntime ?? createLocalKtxLlmRuntimeFromConfig)(options.project.config.llm, {
|
||||||
projectDir: options.project.projectDir,
|
projectDir: options.project.projectDir,
|
||||||
env: process.env,
|
env: process.env,
|
||||||
|
rateLimitGovernor,
|
||||||
}) ??
|
}) ??
|
||||||
undefined;
|
undefined;
|
||||||
|
|
||||||
|
|
@ -677,7 +679,13 @@ export function createLocalBundleIngestRuntime(
|
||||||
const knowledgeIndex = new LocalKnowledgeIndex(options.project, embedding);
|
const knowledgeIndex = new LocalKnowledgeIndex(options.project, embedding);
|
||||||
const knowledgeEvents = new NoopKnowledgeEventPort();
|
const knowledgeEvents = new NoopKnowledgeEventPort();
|
||||||
const wikiService = new KnowledgeWikiService(rootFileStore, embedding, knowledgeIndex, options.project.git, logger);
|
const wikiService = new KnowledgeWikiService(rootFileStore, embedding, knowledgeIndex, options.project.git, logger);
|
||||||
const { agentRunner, llmRuntime } = resolveAgentRunner(options);
|
const rateLimitGovernor = new RateLimitGovernor(
|
||||||
|
createRateLimitGovernorConfig({
|
||||||
|
...options.project.config.ingest.rateLimit,
|
||||||
|
maxConcurrency: options.project.config.ingest.workUnits.maxConcurrency,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const { agentRunner, llmRuntime } = resolveAgentRunner(options, rateLimitGovernor);
|
||||||
const promptService = new PromptService({ promptsDir, partials: [], logger });
|
const promptService = new PromptService({ promptsDir, partials: [], logger });
|
||||||
const storage = new LocalIngestStorage(options.project);
|
const storage = new LocalIngestStorage(options.project);
|
||||||
const registry = registerAdapters(options.adapters);
|
const registry = registerAdapters(options.adapters);
|
||||||
|
|
@ -717,6 +725,7 @@ export function createLocalBundleIngestRuntime(
|
||||||
workUnitMaxConcurrency: options.project.config.ingest.workUnits.maxConcurrency,
|
workUnitMaxConcurrency: options.project.config.ingest.workUnits.maxConcurrency,
|
||||||
workUnitStepBudget: options.project.config.ingest.workUnits.stepBudget,
|
workUnitStepBudget: options.project.config.ingest.workUnits.stepBudget,
|
||||||
workUnitFailureMode: options.project.config.ingest.workUnits.failureMode,
|
workUnitFailureMode: options.project.config.ingest.workUnits.failureMode,
|
||||||
|
rateLimitGovernor,
|
||||||
profileIngest: options.project.config.ingest.profile,
|
profileIngest: options.project.config.ingest.profile,
|
||||||
ingestTraceLevel: ingestTraceLevelFromEnv(),
|
ingestTraceLevel: ingestTraceLevelFromEnv(),
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@ import { cp, mkdir, rm } from 'node:fs/promises';
|
||||||
import { isAbsolute, resolve } from 'node:path';
|
import { isAbsolute, resolve } from 'node:path';
|
||||||
import type { KtxSqlQueryExecutorPort } from '../../context/connections/query-executor.js';
|
import type { KtxSqlQueryExecutorPort } from '../../context/connections/query-executor.js';
|
||||||
import type { KtxLogger } from '../../context/core/config.js';
|
import type { KtxLogger } from '../../context/core/config.js';
|
||||||
|
import { createAbortError, isAbortError } from '../../context/core/abort.js';
|
||||||
import type { KtxSemanticLayerComputePort } from '../../context/daemon/semantic-layer-compute.js';
|
import type { KtxSemanticLayerComputePort } from '../../context/daemon/semantic-layer-compute.js';
|
||||||
import type { AgentRunnerPort, KtxLlmRuntimePort } from '../../context/llm/runtime-port.js';
|
import type { AgentRunnerPort, KtxLlmRuntimePort } from '../../context/llm/runtime-port.js';
|
||||||
import type { KtxLocalProject } from '../../context/project/project.js';
|
import type { KtxLocalProject } from '../../context/project/project.js';
|
||||||
|
|
@ -36,6 +37,7 @@ export interface RunLocalIngestOptions {
|
||||||
queryExecutor?: KtxSqlQueryExecutorPort;
|
queryExecutor?: KtxSqlQueryExecutorPort;
|
||||||
logger?: KtxLogger;
|
logger?: KtxLogger;
|
||||||
embeddingProvider?: import('../../llm/types.js').KtxEmbeddingProvider | null;
|
embeddingProvider?: import('../../llm/types.js').KtxEmbeddingProvider | null;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface LocalIngestResult {
|
export interface LocalIngestResult {
|
||||||
|
|
@ -123,10 +125,11 @@ function findAdapter(adapters: SourceAdapter[], source: string): SourceAdapter {
|
||||||
return adapter;
|
return adapter;
|
||||||
}
|
}
|
||||||
|
|
||||||
function localJobContext(jobId: string, memoryFlow?: MemoryFlowEventSink): IngestJobContext {
|
function localJobContext(jobId: string, memoryFlow?: MemoryFlowEventSink, abortSignal?: AbortSignal): IngestJobContext {
|
||||||
return {
|
return {
|
||||||
jobId,
|
jobId,
|
||||||
...(memoryFlow ? { memoryFlow } : {}),
|
...(memoryFlow ? { memoryFlow } : {}),
|
||||||
|
...(abortSignal ? { abortSignal } : {}),
|
||||||
startPhase() {
|
startPhase() {
|
||||||
return new LocalIngestPhase();
|
return new LocalIngestPhase();
|
||||||
},
|
},
|
||||||
|
|
@ -158,6 +161,7 @@ async function runScheduledPullJob(options: {
|
||||||
queryExecutor?: KtxSqlQueryExecutorPort;
|
queryExecutor?: KtxSqlQueryExecutorPort;
|
||||||
logger?: KtxLogger;
|
logger?: KtxLogger;
|
||||||
embeddingProvider?: import('../../llm/types.js').KtxEmbeddingProvider | null;
|
embeddingProvider?: import('../../llm/types.js').KtxEmbeddingProvider | null;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
}): Promise<LocalIngestResult> {
|
}): Promise<LocalIngestResult> {
|
||||||
const runtime = createLocalBundleIngestRuntime(options);
|
const runtime = createLocalBundleIngestRuntime(options);
|
||||||
const jobId = options.jobId ?? runtime.nextJobId();
|
const jobId = options.jobId ?? runtime.nextJobId();
|
||||||
|
|
@ -169,7 +173,7 @@ async function runScheduledPullJob(options: {
|
||||||
trigger: options.trigger ?? 'manual_resync',
|
trigger: options.trigger ?? 'manual_resync',
|
||||||
bundleRef: { kind: 'scheduled_pull', config: options.pullConfig },
|
bundleRef: { kind: 'scheduled_pull', config: options.pullConfig },
|
||||||
},
|
},
|
||||||
localJobContext(jobId, options.memoryFlow),
|
localJobContext(jobId, options.memoryFlow, options.abortSignal),
|
||||||
);
|
);
|
||||||
const report = await runtime.store.findByJobId(jobId);
|
const report = await runtime.store.findByJobId(jobId);
|
||||||
if (!report) {
|
if (!report) {
|
||||||
|
|
@ -212,6 +216,7 @@ export async function runLocalIngest(options: RunLocalIngestOptions): Promise<Lo
|
||||||
queryExecutor: options.queryExecutor,
|
queryExecutor: options.queryExecutor,
|
||||||
logger: options.logger,
|
logger: options.logger,
|
||||||
embeddingProvider: options.embeddingProvider,
|
embeddingProvider: options.embeddingProvider,
|
||||||
|
abortSignal: options.abortSignal,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -223,7 +228,7 @@ export async function runLocalIngest(options: RunLocalIngestOptions): Promise<Lo
|
||||||
trigger: options.trigger ?? (options.sourceDir ? 'upload' : 'manual_resync'),
|
trigger: options.trigger ?? (options.sourceDir ? 'upload' : 'manual_resync'),
|
||||||
bundleRef,
|
bundleRef,
|
||||||
},
|
},
|
||||||
localJobContext(jobId, options.memoryFlow),
|
localJobContext(jobId, options.memoryFlow, options.abortSignal),
|
||||||
);
|
);
|
||||||
const report = await runtime.store.findByJobId(jobId);
|
const report = await runtime.store.findByJobId(jobId);
|
||||||
if (!report) {
|
if (!report) {
|
||||||
|
|
@ -362,6 +367,9 @@ export async function runLocalMetabaseIngest(
|
||||||
|
|
||||||
const children: LocalMetabaseFanoutChild[] = [];
|
const children: LocalMetabaseFanoutChild[] = [];
|
||||||
for (const childPlan of childPlans) {
|
for (const childPlan of childPlans) {
|
||||||
|
if (options.abortSignal?.aborted) {
|
||||||
|
throw createAbortError();
|
||||||
|
}
|
||||||
const targetConnectionId = safeSegment('target connection id', childPlan.targetConnectionId);
|
const targetConnectionId = safeSegment('target connection id', childPlan.targetConnectionId);
|
||||||
if (!options.project.config.connections[targetConnectionId]) {
|
if (!options.project.config.connections[targetConnectionId]) {
|
||||||
throw new Error(`Target connection "${targetConnectionId}" is not configured in ktx.yaml`);
|
throw new Error(`Target connection "${targetConnectionId}" is not configured in ktx.yaml`);
|
||||||
|
|
@ -391,8 +399,12 @@ export async function runLocalMetabaseIngest(
|
||||||
queryExecutor: options.queryExecutor,
|
queryExecutor: options.queryExecutor,
|
||||||
logger: options.logger,
|
logger: options.logger,
|
||||||
embeddingProvider: options.embeddingProvider,
|
embeddingProvider: options.embeddingProvider,
|
||||||
|
abortSignal: options.abortSignal,
|
||||||
});
|
});
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
if (isAbortError(error)) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
child = await recordLocalMetabaseChildFailure({
|
child = await recordLocalMetabaseChildFailure({
|
||||||
project: options.project,
|
project: options.project,
|
||||||
jobId: childJobId,
|
jobId: childJobId,
|
||||||
|
|
|
||||||
|
|
@ -174,7 +174,7 @@ export function ingestReportToMemoryFlowReplay(
|
||||||
|
|
||||||
const actions = allReportActions(report);
|
const actions = allReportActions(report);
|
||||||
const workUnitEvents: MemoryFlowEvent[] = report.body.workUnits.flatMap((workUnit) => [
|
const workUnitEvents: MemoryFlowEvent[] = report.body.workUnits.flatMap((workUnit) => [
|
||||||
{ type: 'work_unit_started', unitKey: workUnit.unitKey, skills: [], stepBudget: 0 } satisfies MemoryFlowEvent,
|
{ type: 'work_unit_started', unitKey: workUnit.unitKey, skills: [] } satisfies MemoryFlowEvent,
|
||||||
...workUnit.actions.map(
|
...workUnit.actions.map(
|
||||||
(action): MemoryFlowEvent => ({
|
(action): MemoryFlowEvent => ({
|
||||||
type: 'candidate_action',
|
type: 'candidate_action',
|
||||||
|
|
|
||||||
|
|
@ -70,17 +70,22 @@ const memoryFlowEventSchema = z.discriminatedUnion('type', [
|
||||||
message: z.string().min(1),
|
message: z.string().min(1),
|
||||||
transient: z.boolean().optional(),
|
transient: z.boolean().optional(),
|
||||||
}),
|
}),
|
||||||
|
eventSchema({
|
||||||
|
type: z.literal('rate_limit_wait'),
|
||||||
|
provider: z.string(),
|
||||||
|
rateLimitType: z.string().optional(),
|
||||||
|
resumeAtMs: z.number().int().nonnegative(),
|
||||||
|
remainingMs: z.number().int().nonnegative(),
|
||||||
|
}),
|
||||||
eventSchema({
|
eventSchema({
|
||||||
type: z.literal('work_unit_started'),
|
type: z.literal('work_unit_started'),
|
||||||
unitKey: z.string().min(1),
|
unitKey: z.string().min(1),
|
||||||
skills: z.array(z.string().min(1)),
|
skills: z.array(z.string().min(1)),
|
||||||
stepBudget: z.number().int().min(0),
|
|
||||||
}),
|
}),
|
||||||
eventSchema({
|
eventSchema({
|
||||||
type: z.literal('work_unit_step'),
|
type: z.literal('work_unit_step'),
|
||||||
unitKey: z.string().min(1),
|
unitKey: z.string().min(1),
|
||||||
stepIndex: z.number().int().min(0),
|
toolCalls: z.number().int().min(0),
|
||||||
stepBudget: z.number().int().min(0),
|
|
||||||
}),
|
}),
|
||||||
eventSchema({
|
eventSchema({
|
||||||
type: z.literal('candidate_action'),
|
type: z.literal('candidate_action'),
|
||||||
|
|
|
||||||
|
|
@ -60,17 +60,22 @@ type MemoryFlowEventPayload =
|
||||||
message: string;
|
message: string;
|
||||||
transient?: boolean;
|
transient?: boolean;
|
||||||
}
|
}
|
||||||
|
| {
|
||||||
|
type: 'rate_limit_wait';
|
||||||
|
provider: string;
|
||||||
|
rateLimitType?: string;
|
||||||
|
resumeAtMs: number;
|
||||||
|
remainingMs: number;
|
||||||
|
}
|
||||||
| {
|
| {
|
||||||
type: 'work_unit_started';
|
type: 'work_unit_started';
|
||||||
unitKey: string;
|
unitKey: string;
|
||||||
skills: string[];
|
skills: string[];
|
||||||
stepBudget: number;
|
|
||||||
}
|
}
|
||||||
| {
|
| {
|
||||||
type: 'work_unit_step';
|
type: 'work_unit_step';
|
||||||
unitKey: string;
|
unitKey: string;
|
||||||
stepIndex: number;
|
toolCalls: number;
|
||||||
stepBudget: number;
|
|
||||||
}
|
}
|
||||||
| {
|
| {
|
||||||
type: 'candidate_action';
|
type: 'candidate_action';
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@ import type { KtxFileStorePort } from '../../context/core/file-store.js';
|
||||||
import type { KtxLogger } from '../../context/core/config.js';
|
import type { KtxLogger } from '../../context/core/config.js';
|
||||||
import type { SessionOutcome } from '../../context/core/session-worktree.service.js';
|
import type { SessionOutcome } from '../../context/core/session-worktree.service.js';
|
||||||
import type { AgentRunnerPort, KtxLlmRuntimePort, KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
|
import type { AgentRunnerPort, KtxLlmRuntimePort, KtxRuntimeToolSet } from '../../context/llm/runtime-port.js';
|
||||||
|
import type { RateLimitGovernor } from '../llm/rate-limit-governor.js';
|
||||||
import type { MemoryAction, MemoryKnowledgeSlRefsPort } from '../../context/memory/types.js';
|
import type { MemoryAction, MemoryKnowledgeSlRefsPort } from '../../context/memory/types.js';
|
||||||
import type { PromptService } from '../../context/prompts/prompt.service.js';
|
import type { PromptService } from '../../context/prompts/prompt.service.js';
|
||||||
import type { SkillsRegistryService } from '../../context/skills/skills-registry.service.js';
|
import type { SkillsRegistryService } from '../../context/skills/skills-registry.service.js';
|
||||||
|
|
@ -144,6 +145,7 @@ interface IngestSettingsPort {
|
||||||
workUnitMaxConcurrency?: number;
|
workUnitMaxConcurrency?: number;
|
||||||
workUnitStepBudget?: number;
|
workUnitStepBudget?: number;
|
||||||
workUnitFailureMode?: 'abort' | 'continue';
|
workUnitFailureMode?: 'abort' | 'continue';
|
||||||
|
rateLimitGovernor?: RateLimitGovernor;
|
||||||
/** Print a timing breakdown to stderr at the end of each run (config-driven; see also KTX_PROFILE_INGEST). `'json'` emits the raw structured profile. */
|
/** Print a timing breakdown to stderr at the end of each run (config-driven; see also KTX_PROFILE_INGEST). `'json'` emits the raw structured profile. */
|
||||||
profileIngest?: boolean | 'json';
|
profileIngest?: boolean | 'json';
|
||||||
ingestTraceLevel?: IngestTraceLevel;
|
ingestTraceLevel?: IngestTraceLevel;
|
||||||
|
|
@ -322,7 +324,7 @@ export interface CuratorPaginationPort {
|
||||||
}) => string;
|
}) => string;
|
||||||
buildToolSet: (passNumber: number) => KtxRuntimeToolSet;
|
buildToolSet: (passNumber: number) => KtxRuntimeToolSet;
|
||||||
getReconciliationActions: () => MemoryAction[];
|
getReconciliationActions: () => MemoryAction[];
|
||||||
onStepFinish?: (info: { passNumber: number; stepIndex: number; stepBudget: number }) => void;
|
abortSignal?: AbortSignal;
|
||||||
}): Promise<ReconciliationOutcome & { report: CuratorPaginationReport; warnings: string[] }>;
|
}): Promise<ReconciliationOutcome & { report: CuratorPaginationReport; warnings: string[] }>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
import type { KtxModelRole } from '../../../llm/types.js';
|
import type { KtxModelRole } from '../../../llm/types.js';
|
||||||
|
import { isAbortError } from '../../core/abort.js';
|
||||||
import type { AgentRunnerPort, KtxRuntimeToolSet, RunLoopMetrics } from '../../../context/llm/runtime-port.js';
|
import type { AgentRunnerPort, KtxRuntimeToolSet, RunLoopMetrics } from '../../../context/llm/runtime-port.js';
|
||||||
import type { CaptureSession, MemoryAction } from '../../../context/memory/types.js';
|
import type { CaptureSession, MemoryAction } from '../../../context/memory/types.js';
|
||||||
import { listTouchedSlSources, type TouchedSlSource } from '../../../context/tools/touched-sl-sources.js';
|
import { listTouchedSlSources, type TouchedSlSource } from '../../../context/tools/touched-sl-sources.js';
|
||||||
|
|
@ -27,7 +28,7 @@ export interface WorkUnitExecutionDeps {
|
||||||
sourceKey: string;
|
sourceKey: string;
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
jobId: string;
|
jobId: string;
|
||||||
onStepFinish?: (info: { stepIndex: number; stepBudget: number }) => void;
|
abortSignal?: AbortSignal;
|
||||||
toolFailureCount?: (unitKey: string) => number;
|
toolFailureCount?: (unitKey: string) => number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -105,9 +106,12 @@ export async function executeWorkUnit(deps: WorkUnitExecutionDeps, wu: WorkUnit)
|
||||||
unitKey: wu.unitKey,
|
unitKey: wu.unitKey,
|
||||||
jobId: deps.jobId,
|
jobId: deps.jobId,
|
||||||
},
|
},
|
||||||
onStepFinish: deps.onStepFinish,
|
abortSignal: deps.abortSignal,
|
||||||
});
|
});
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
if (isAbortError(error)) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
return failWithResetFromCurrentHead(error instanceof Error ? error.message : String(error));
|
return failWithResetFromCurrentHead(error instanceof Error ? error.message : String(error));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ export interface ReconciliationContext {
|
||||||
sourceKey: string;
|
sourceKey: string;
|
||||||
jobId: string;
|
jobId: string;
|
||||||
force?: boolean;
|
force?: boolean;
|
||||||
onStepFinish?: (info: { stepIndex: number; stepBudget: number }) => void;
|
abortSignal?: AbortSignal;
|
||||||
forceRun?: boolean;
|
forceRun?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -39,7 +39,7 @@ export async function runReconciliationStage4(ctx: ReconciliationContext): Promi
|
||||||
toolSet: ctx.buildToolSet(),
|
toolSet: ctx.buildToolSet(),
|
||||||
stepBudget: ctx.stepBudget,
|
stepBudget: ctx.stepBudget,
|
||||||
telemetryTags: { operationName: 'ingest-bundle-reconcile', source: ctx.sourceKey, jobId: ctx.jobId },
|
telemetryTags: { operationName: 'ingest-bundle-reconcile', source: ctx.sourceKey, jobId: ctx.jobId },
|
||||||
onStepFinish: ctx.onStepFinish,
|
abortSignal: ctx.abortSignal,
|
||||||
});
|
});
|
||||||
return { skipped: false, stopReason: run.stopReason, error: run.error, ...(run.metrics ? { metrics: run.metrics } : {}) };
|
return { skipped: false, stopReason: run.stopReason, error: run.error, ...(run.metrics ? { metrics: run.metrics } : {}) };
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -220,5 +220,6 @@ export interface IngestJobPhase {
|
||||||
export interface IngestJobContext {
|
export interface IngestJobContext {
|
||||||
jobId: string;
|
jobId: string;
|
||||||
memoryFlow?: MemoryFlowEventSink;
|
memoryFlow?: MemoryFlowEventSink;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
startPhase(weight: number): IngestJobPhase;
|
startPhase(weight: number): IngestJobPhase;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,9 @@ import type { KtxLlmProvider } from '../../llm/types.js';
|
||||||
import { generateText, Output, stepCountIs, type FlexibleSchema, type TelemetrySettings, type ToolSet } from 'ai';
|
import { generateText, Output, stepCountIs, type FlexibleSchema, type TelemetrySettings, type ToolSet } from 'ai';
|
||||||
import type { z } from 'zod';
|
import type { z } from 'zod';
|
||||||
import { noopLogger, type KtxLogger } from '../../context/core/config.js';
|
import { noopLogger, type KtxLogger } from '../../context/core/config.js';
|
||||||
|
import { isAbortError } from '../core/abort.js';
|
||||||
import { summarizeKtxLlmDebugRequest, type KtxLlmDebugRequestRecorder } from './debug-request-recorder.js';
|
import { summarizeKtxLlmDebugRequest, type KtxLlmDebugRequestRecorder } from './debug-request-recorder.js';
|
||||||
|
import type { RateLimitGovernor, RateLimitProvider, RateLimitSignal } from './rate-limit-governor.js';
|
||||||
import { createAiSdkToolSet } from './runtime-tools.js';
|
import { createAiSdkToolSet } from './runtime-tools.js';
|
||||||
import type {
|
import type {
|
||||||
KtxGenerateObjectInput,
|
KtxGenerateObjectInput,
|
||||||
|
|
@ -40,12 +42,129 @@ export interface AiSdkKtxLlmRuntimeDeps {
|
||||||
telemetry?: AgentTelemetryPort;
|
telemetry?: AgentTelemetryPort;
|
||||||
logger?: KtxLogger;
|
logger?: KtxLogger;
|
||||||
debugRequestRecorder?: KtxLlmDebugRequestRecorder;
|
debugRequestRecorder?: KtxLlmDebugRequestRecorder;
|
||||||
|
rateLimitGovernor?: Pick<RateLimitGovernor, 'waitForReady' | 'report' | 'maxRetryAttempts'>;
|
||||||
}
|
}
|
||||||
|
|
||||||
function hasTools(tools: Record<string, unknown>): boolean {
|
function hasTools(tools: Record<string, unknown>): boolean {
|
||||||
return Object.keys(tools).length > 0;
|
return Object.keys(tools).length > 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function modelProviderName(model: unknown): RateLimitProvider {
|
||||||
|
const provider = (model as { provider?: string }).provider ?? '';
|
||||||
|
return provider.includes('vertex') || provider.includes('google') ? 'vertex' : 'anthropic-api';
|
||||||
|
}
|
||||||
|
|
||||||
|
interface HeaderLimitPair {
|
||||||
|
limit: string;
|
||||||
|
remaining: string;
|
||||||
|
rateLimitType: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const RATE_LIMIT_HEADER_PAIRS: HeaderLimitPair[] = [
|
||||||
|
{
|
||||||
|
limit: 'anthropic-ratelimit-requests-limit',
|
||||||
|
remaining: 'anthropic-ratelimit-requests-remaining',
|
||||||
|
rateLimitType: 'rpm',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
limit: 'anthropic-ratelimit-tokens-limit',
|
||||||
|
remaining: 'anthropic-ratelimit-tokens-remaining',
|
||||||
|
rateLimitType: 'tpm',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
limit: 'anthropic-ratelimit-input-tokens-limit',
|
||||||
|
remaining: 'anthropic-ratelimit-input-tokens-remaining',
|
||||||
|
rateLimitType: 'itpm',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
limit: 'anthropic-ratelimit-output-tokens-limit',
|
||||||
|
remaining: 'anthropic-ratelimit-output-tokens-remaining',
|
||||||
|
rateLimitType: 'otpm',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
limit: 'x-ratelimit-limit-requests',
|
||||||
|
remaining: 'x-ratelimit-remaining-requests',
|
||||||
|
rateLimitType: 'rpm',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
limit: 'x-ratelimit-limit-tokens',
|
||||||
|
remaining: 'x-ratelimit-remaining-tokens',
|
||||||
|
rateLimitType: 'tpm',
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
function normalizeHeaders(headers: unknown): Record<string, string> {
|
||||||
|
if (!headers || typeof headers !== 'object') {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
const get = (headers as { get?: unknown }).get;
|
||||||
|
if (typeof get === 'function') {
|
||||||
|
const out: Record<string, string> = {};
|
||||||
|
for (const pair of RATE_LIMIT_HEADER_PAIRS) {
|
||||||
|
const limit = get.call(headers, pair.limit);
|
||||||
|
const remaining = get.call(headers, pair.remaining);
|
||||||
|
if (typeof limit === 'string') out[pair.limit] = limit;
|
||||||
|
if (typeof remaining === 'string') out[pair.remaining] = remaining;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
return Object.fromEntries(
|
||||||
|
Object.entries(headers as Record<string, unknown>)
|
||||||
|
.filter((entry): entry is [string, string | number] => typeof entry[1] === 'string' || typeof entry[1] === 'number')
|
||||||
|
.map(([key, value]) => [key.toLowerCase(), String(value)]),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function numericHeader(headers: Record<string, string>, key: string): number | undefined {
|
||||||
|
const value = Number(headers[key]);
|
||||||
|
return Number.isFinite(value) && value >= 0 ? value : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function utilizationForPair(headers: Record<string, string>, pair: HeaderLimitPair): number | undefined {
|
||||||
|
const limit = numericHeader(headers, pair.limit);
|
||||||
|
const remaining = numericHeader(headers, pair.remaining);
|
||||||
|
if (limit === undefined || remaining === undefined || limit <= 0) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return 1 - Math.min(limit, remaining) / limit;
|
||||||
|
}
|
||||||
|
|
||||||
|
function aiSdkHeaderRateLimitSignal(provider: RateLimitProvider, result: unknown): RateLimitSignal | undefined {
|
||||||
|
const headers = normalizeHeaders((result as { response?: { headers?: unknown } }).response?.headers);
|
||||||
|
let best: { utilization: number; rateLimitType: string } | undefined;
|
||||||
|
for (const pair of RATE_LIMIT_HEADER_PAIRS) {
|
||||||
|
const utilization = utilizationForPair(headers, pair);
|
||||||
|
if (utilization === undefined) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!best || utilization > best.utilization) {
|
||||||
|
best = { utilization, rateLimitType: pair.rateLimitType };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!best) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
provider,
|
||||||
|
status: 'allowed',
|
||||||
|
rateLimitType: best.rateLimitType,
|
||||||
|
utilization: Number(best.utilization.toFixed(4)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function retryAfterMs(error: unknown): number | undefined {
|
||||||
|
const value = (error as { retryAfter?: unknown }).retryAfter;
|
||||||
|
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
||||||
|
return value < 1_000 ? value * 1_000 : value;
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isAiSdkRateLimitError(error: unknown): boolean {
|
||||||
|
const record = error as { name?: string; statusCode?: number; status?: number };
|
||||||
|
return record.name === 'TooManyRequestsError' || record.statusCode === 429 || record.status === 429;
|
||||||
|
}
|
||||||
|
|
||||||
export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
private readonly logger: KtxLogger;
|
private readonly logger: KtxLogger;
|
||||||
|
|
||||||
|
|
@ -53,6 +172,41 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
this.logger = deps.logger ?? noopLogger;
|
this.logger = deps.logger ?? noopLogger;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private async generateTextWithRateLimitRetry<T>(
|
||||||
|
provider: RateLimitProvider,
|
||||||
|
abortSignal: AbortSignal | undefined,
|
||||||
|
run: () => Promise<T>,
|
||||||
|
): Promise<T> {
|
||||||
|
// maxRetryAttempts() returns 1 when no governor is present or pacing is
|
||||||
|
// disabled, so a 429 throws immediately instead of hammering the provider
|
||||||
|
// with no backoff; the AI SDK's own maxRetries still handles transient 429s.
|
||||||
|
const maxAttempts = this.deps.rateLimitGovernor?.maxRetryAttempts() ?? 1;
|
||||||
|
let attempt = 0;
|
||||||
|
while (true) {
|
||||||
|
await this.deps.rateLimitGovernor?.waitForReady(abortSignal);
|
||||||
|
try {
|
||||||
|
const result = await run();
|
||||||
|
const signal = aiSdkHeaderRateLimitSignal(provider, result);
|
||||||
|
if (signal) {
|
||||||
|
this.deps.rateLimitGovernor?.report(signal);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
} catch (error) {
|
||||||
|
if (isAbortError(error) || !isAiSdkRateLimitError(error) || attempt >= maxAttempts - 1) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
attempt += 1;
|
||||||
|
const retryAfter = retryAfterMs(error);
|
||||||
|
this.deps.rateLimitGovernor?.report({
|
||||||
|
provider,
|
||||||
|
status: 'rejected',
|
||||||
|
rateLimitType: 'http_429',
|
||||||
|
...(retryAfter !== undefined ? { retryAfterMs: retryAfter } : {}),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async generateText(input: KtxGenerateTextInput): Promise<string> {
|
async generateText(input: KtxGenerateTextInput): Promise<string> {
|
||||||
const model = this.deps.llmProvider.getModel(input.role);
|
const model = this.deps.llmProvider.getModel(input.role);
|
||||||
if ((model as { provider?: string }).provider === 'deterministic') {
|
if ((model as { provider?: string }).provider === 'deterministic') {
|
||||||
|
|
@ -67,12 +221,13 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
});
|
});
|
||||||
const split = splitKtxSystemMessages(built.messages);
|
const split = splitKtxSystemMessages(built.messages);
|
||||||
const startedAt = Date.now();
|
const startedAt = Date.now();
|
||||||
const result = await generateText({
|
const request = {
|
||||||
model,
|
model,
|
||||||
temperature: input.temperature ?? 0,
|
temperature: input.temperature ?? 0,
|
||||||
...(split.system ? { system: split.system } : {}),
|
...(split.system ? { system: split.system } : {}),
|
||||||
messages: split.messages,
|
messages: split.messages,
|
||||||
tools: built.tools as ToolSet,
|
tools: built.tools as ToolSet,
|
||||||
|
...(input.abortSignal ? { abortSignal: input.abortSignal } : {}),
|
||||||
...(hasTools(tools)
|
...(hasTools(tools)
|
||||||
? {
|
? {
|
||||||
experimental_repairToolCall: this.deps.llmProvider.repairToolCallHandler({
|
experimental_repairToolCall: this.deps.llmProvider.repairToolCallHandler({
|
||||||
|
|
@ -80,7 +235,8 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
: {}),
|
: {}),
|
||||||
});
|
};
|
||||||
|
const result = await this.generateTextWithRateLimitRetry(modelProviderName(model), input.abortSignal, () => generateText(request));
|
||||||
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: toLlmTokenUsage(result.totalUsage ?? result.usage) });
|
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: toLlmTokenUsage(result.totalUsage ?? result.usage) });
|
||||||
if (typeof result.text !== 'string') {
|
if (typeof result.text !== 'string') {
|
||||||
throw new Error('KTX LLM text generation returned no text');
|
throw new Error('KTX LLM text generation returned no text');
|
||||||
|
|
@ -101,12 +257,13 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
});
|
});
|
||||||
const split = splitKtxSystemMessages(built.messages);
|
const split = splitKtxSystemMessages(built.messages);
|
||||||
const startedAt = Date.now();
|
const startedAt = Date.now();
|
||||||
const result = await generateText({
|
const request = {
|
||||||
model,
|
model,
|
||||||
temperature: input.temperature ?? 0,
|
temperature: input.temperature ?? 0,
|
||||||
...(split.system ? { system: split.system } : {}),
|
...(split.system ? { system: split.system } : {}),
|
||||||
messages: split.messages,
|
messages: split.messages,
|
||||||
tools: built.tools as ToolSet,
|
tools: built.tools as ToolSet,
|
||||||
|
...(input.abortSignal ? { abortSignal: input.abortSignal } : {}),
|
||||||
...(hasTools(tools)
|
...(hasTools(tools)
|
||||||
? {
|
? {
|
||||||
experimental_repairToolCall: this.deps.llmProvider.repairToolCallHandler({
|
experimental_repairToolCall: this.deps.llmProvider.repairToolCallHandler({
|
||||||
|
|
@ -115,7 +272,8 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
}
|
}
|
||||||
: {}),
|
: {}),
|
||||||
output: Output.object({ schema: input.schema as unknown as FlexibleSchema<TOutput> }),
|
output: Output.object({ schema: input.schema as unknown as FlexibleSchema<TOutput> }),
|
||||||
});
|
};
|
||||||
|
const result = await this.generateTextWithRateLimitRetry(modelProviderName(model), input.abortSignal, () => generateText(request));
|
||||||
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: toLlmTokenUsage(result.totalUsage ?? result.usage) });
|
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: toLlmTokenUsage(result.totalUsage ?? result.usage) });
|
||||||
if (result.output == null) {
|
if (result.output == null) {
|
||||||
throw new Error('KTX LLM object generation returned no output');
|
throw new Error('KTX LLM object generation returned no output');
|
||||||
|
|
@ -152,7 +310,7 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
|
|
||||||
const result = await generateText({
|
const request = {
|
||||||
model,
|
model,
|
||||||
temperature: 0,
|
temperature: 0,
|
||||||
stopWhen: stepCountIs(params.stepBudget),
|
stopWhen: stepCountIs(params.stepBudget),
|
||||||
|
|
@ -163,23 +321,15 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
...(promptMessages.system ? { system: promptMessages.system } : {}),
|
...(promptMessages.system ? { system: promptMessages.system } : {}),
|
||||||
messages: promptMessages.messages,
|
messages: promptMessages.messages,
|
||||||
tools: built.tools as ToolSet,
|
tools: built.tools as ToolSet,
|
||||||
onStepFinish: async () => {
|
...(params.abortSignal ? { abortSignal: params.abortSignal } : {}),
|
||||||
|
// Count model round-trips locally for metrics. `stepCountIs(stepBudget)`
|
||||||
|
// caps the loop, so this counter never exceeds the budget.
|
||||||
|
onStepFinish: () => {
|
||||||
stepIndex += 1;
|
stepIndex += 1;
|
||||||
stepBoundariesMs.push(Date.now() - startedAt);
|
stepBoundariesMs.push(Date.now() - startedAt);
|
||||||
if (!params.onStepFinish) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
await params.onStepFinish({ stepIndex, stepBudget: params.stepBudget });
|
|
||||||
} catch (err) {
|
|
||||||
this.logger.warn(
|
|
||||||
`[agent-runner] onStepFinish callback threw; ignoring: ${
|
|
||||||
err instanceof Error ? err.message : String(err)
|
|
||||||
}`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
},
|
},
|
||||||
});
|
};
|
||||||
|
const result = await this.generateTextWithRateLimitRetry(modelProviderName(model), params.abortSignal, () => generateText(request));
|
||||||
return {
|
return {
|
||||||
stopReason: 'natural',
|
stopReason: 'natural',
|
||||||
metrics: {
|
metrics: {
|
||||||
|
|
@ -190,6 +340,9 @@ export class AiSdkKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
if (isAbortError(error)) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
const err = error instanceof Error ? error : new Error(String(error));
|
const err = error instanceof Error ? error : new Error(String(error));
|
||||||
this.logger.warn(`[agent-runner] loop failed: ${err.message}`);
|
this.logger.warn(`[agent-runner] loop failed: ${err.message}`);
|
||||||
return {
|
return {
|
||||||
|
|
|
||||||
|
|
@ -6,9 +6,10 @@ import {
|
||||||
type SDKResultMessage,
|
type SDKResultMessage,
|
||||||
} from '@anthropic-ai/claude-agent-sdk';
|
} from '@anthropic-ai/claude-agent-sdk';
|
||||||
import { z } from 'zod';
|
import { z } from 'zod';
|
||||||
import { noopLogger, type KtxLogger } from '../../context/core/config.js';
|
import { createAbortError, isAbortError, throwIfAborted } from '../core/abort.js';
|
||||||
import { createKtxClaudeCodeEnv } from './claude-code-env.js';
|
import { createKtxClaudeCodeEnv } from './claude-code-env.js';
|
||||||
import { resolveClaudeCodeModel } from './claude-code-models.js';
|
import { resolveClaudeCodeModel } from './claude-code-models.js';
|
||||||
|
import type { RateLimitGovernor, RateLimitSignal } from './rate-limit-governor.js';
|
||||||
import { createClaudeSdkTools, mcpToolIds } from './runtime-tools.js';
|
import { createClaudeSdkTools, mcpToolIds } from './runtime-tools.js';
|
||||||
import type {
|
import type {
|
||||||
KtxGenerateObjectInput,
|
KtxGenerateObjectInput,
|
||||||
|
|
@ -21,7 +22,16 @@ import type {
|
||||||
RunLoopStopReason,
|
RunLoopStopReason,
|
||||||
} from './runtime-port.js';
|
} from './runtime-port.js';
|
||||||
|
|
||||||
type QueryFn = (params: Parameters<typeof defaultQuery>[0]) => AsyncIterable<SDKMessage>;
|
type QueryResult = AsyncIterable<SDKMessage> & {
|
||||||
|
interrupt?: () => void | Promise<void>;
|
||||||
|
};
|
||||||
|
|
||||||
|
type QueryFn = (params: Parameters<typeof defaultQuery>[0]) => QueryResult;
|
||||||
|
|
||||||
|
interface ClaudeQueryOutcome {
|
||||||
|
result: SDKResultMessage;
|
||||||
|
rejectedRateLimitSignal?: RateLimitSignal;
|
||||||
|
}
|
||||||
|
|
||||||
function claudeTokenUsage(result: SDKResultMessage): LlmTokenUsage {
|
function claudeTokenUsage(result: SDKResultMessage): LlmTokenUsage {
|
||||||
const usage = (result as { usage?: { input_tokens?: number; output_tokens?: number } }).usage;
|
const usage = (result as { usage?: { input_tokens?: number; output_tokens?: number } }).usage;
|
||||||
|
|
@ -42,7 +52,7 @@ export interface ClaudeCodeKtxLlmRuntimeDeps {
|
||||||
modelSlots: { default: string } & Partial<Record<string, string>>;
|
modelSlots: { default: string } & Partial<Record<string, string>>;
|
||||||
query?: QueryFn;
|
query?: QueryFn;
|
||||||
env?: NodeJS.ProcessEnv;
|
env?: NodeJS.ProcessEnv;
|
||||||
logger?: KtxLogger;
|
rateLimitGovernor?: Pick<RateLimitGovernor, 'waitForReady' | 'report' | 'maxRetryAttempts'>;
|
||||||
}
|
}
|
||||||
|
|
||||||
const BUILTIN_TOOLS = [
|
const BUILTIN_TOOLS = [
|
||||||
|
|
@ -73,22 +83,6 @@ function isResult(message: SDKMessage): message is SDKResultMessage {
|
||||||
return message.type === 'result';
|
return message.type === 'result';
|
||||||
}
|
}
|
||||||
|
|
||||||
// Skip emissions the SDK does not count toward `num_turns`: `pause_turn` continuations and
|
|
||||||
// errored partials (e.g. `max_output_tokens`) it retries internally. Without this, the
|
|
||||||
// runtime's step counter outruns `maxTurns` and the HUD renders e.g. `step 69/40`.
|
|
||||||
function countsAsAssistantTurn(message: SDKMessage): boolean {
|
|
||||||
if (message.type !== 'assistant' || message.parent_tool_use_id !== null) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
if (message.error !== undefined) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
if (message.message.stop_reason === 'pause_turn') {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
function resultError(result: SDKResultMessage): Error | undefined {
|
function resultError(result: SDKResultMessage): Error | undefined {
|
||||||
if (result.subtype === 'success') {
|
if (result.subtype === 'success') {
|
||||||
return undefined;
|
return undefined;
|
||||||
|
|
@ -157,6 +151,74 @@ function expectedMcpServerNames(tools: KtxRuntimeToolSet | undefined): Set<strin
|
||||||
return tools && Object.keys(tools).length > 0 ? new Set([KTX_MCP_SERVER_NAME]) : new Set();
|
return tools && Object.keys(tools).length > 0 ? new Set([KTX_MCP_SERVER_NAME]) : new Set();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const CLAUDE_RATE_LIMIT_ERROR_MARKERS = /\b429\b|rate limit|too many requests|quota exceeded|overloaded|max_retries/i;
|
||||||
|
|
||||||
|
function normalizeClaudeResetAtMs(value: unknown): number | undefined {
|
||||||
|
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
|
||||||
|
return Math.round(value < 10_000_000_000 ? value * 1_000 : value);
|
||||||
|
}
|
||||||
|
if (typeof value === 'string') {
|
||||||
|
const numeric = Number(value);
|
||||||
|
if (Number.isFinite(numeric) && numeric > 0) {
|
||||||
|
return normalizeClaudeResetAtMs(numeric);
|
||||||
|
}
|
||||||
|
const parsed = Date.parse(value);
|
||||||
|
return Number.isFinite(parsed) ? parsed : undefined;
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isClaudeRateLimitResult(result: SDKResultMessage, rejectedSignal: RateLimitSignal | undefined): boolean {
|
||||||
|
const error = resultError(result);
|
||||||
|
if (!error) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (rejectedSignal?.status === 'rejected') {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
const resultDetails = result as {
|
||||||
|
stop_reason?: unknown;
|
||||||
|
terminal_reason?: unknown;
|
||||||
|
errors?: unknown[];
|
||||||
|
};
|
||||||
|
const details = [
|
||||||
|
error.message,
|
||||||
|
resultDetails.stop_reason,
|
||||||
|
resultDetails.terminal_reason,
|
||||||
|
...(resultDetails.errors ?? []),
|
||||||
|
]
|
||||||
|
.filter((value): value is string => typeof value === 'string' && value.length > 0)
|
||||||
|
.join('\n');
|
||||||
|
return CLAUDE_RATE_LIMIT_ERROR_MARKERS.test(details);
|
||||||
|
}
|
||||||
|
|
||||||
|
function claudeRateLimitSignal(message: SDKMessage): RateLimitSignal | null {
|
||||||
|
const record = message as unknown as Record<string, unknown>;
|
||||||
|
if (record.type === 'rate_limit_event') {
|
||||||
|
const info = record.rate_limit_info as Record<string, unknown> | undefined;
|
||||||
|
if (!info) return null;
|
||||||
|
const rawStatus = typeof info.status === 'string' ? info.status : 'allowed';
|
||||||
|
const resetAtMs = normalizeClaudeResetAtMs(info.resetsAt);
|
||||||
|
return {
|
||||||
|
provider: 'claude-subscription',
|
||||||
|
status: rawStatus === 'rejected' ? 'rejected' : rawStatus === 'allowed_warning' ? 'warning' : 'allowed',
|
||||||
|
...(resetAtMs !== undefined ? { resetAtMs } : {}),
|
||||||
|
...(typeof info.rateLimitType === 'string' ? { rateLimitType: info.rateLimitType } : {}),
|
||||||
|
...(typeof info.utilization === 'number' ? { utilization: info.utilization } : {}),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (record.subtype === 'api_retry' || record.type === 'api_retry') {
|
||||||
|
const retryDelayMs = typeof record.retry_delay_ms === 'number' ? record.retry_delay_ms : undefined;
|
||||||
|
return {
|
||||||
|
provider: 'claude-subscription',
|
||||||
|
status: 'warning',
|
||||||
|
...(retryDelayMs !== undefined ? { retryAfterMs: retryDelayMs } : {}),
|
||||||
|
rateLimitType: 'api_retry',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
function managedMcpSettings(serverNames: string[]): NonNullable<Options['managedSettings']> {
|
function managedMcpSettings(serverNames: string[]): NonNullable<Options['managedSettings']> {
|
||||||
return {
|
return {
|
||||||
allowManagedMcpServersOnly: true,
|
allowManagedMcpServersOnly: true,
|
||||||
|
|
@ -216,31 +278,67 @@ async function collectResult(params: {
|
||||||
options: Options;
|
options: Options;
|
||||||
allowedToolIds: Set<string>;
|
allowedToolIds: Set<string>;
|
||||||
expectedMcpServerNames: Set<string>;
|
expectedMcpServerNames: Set<string>;
|
||||||
onAssistantTurn?: () => Promise<void>;
|
rateLimitGovernor?: Pick<RateLimitGovernor, 'waitForReady' | 'report' | 'maxRetryAttempts'>;
|
||||||
}): Promise<SDKResultMessage> {
|
abortSignal?: AbortSignal;
|
||||||
|
}): Promise<ClaudeQueryOutcome> {
|
||||||
let result: SDKResultMessage | undefined;
|
let result: SDKResultMessage | undefined;
|
||||||
for await (const message of params.query({ prompt: params.prompt, options: params.options })) {
|
let rejectedRateLimitSignal: RateLimitSignal | undefined;
|
||||||
assertInitIsolation(message, params.allowedToolIds, params.expectedMcpServerNames);
|
throwIfAborted(params.abortSignal);
|
||||||
if (countsAsAssistantTurn(message)) {
|
await params.rateLimitGovernor?.waitForReady(params.abortSignal);
|
||||||
await params.onAssistantTurn?.();
|
throwIfAborted(params.abortSignal);
|
||||||
}
|
const queryResult = params.query({ prompt: params.prompt, options: params.options });
|
||||||
if (isResult(message)) {
|
const onAbort = () => {
|
||||||
result = message;
|
void Promise.resolve(queryResult.interrupt?.()).catch(() => undefined);
|
||||||
|
};
|
||||||
|
params.abortSignal?.addEventListener('abort', onAbort, { once: true });
|
||||||
|
try {
|
||||||
|
for await (const message of queryResult) {
|
||||||
|
throwIfAborted(params.abortSignal);
|
||||||
|
const rateLimitSignal = claudeRateLimitSignal(message);
|
||||||
|
if (rateLimitSignal) {
|
||||||
|
if (rateLimitSignal.status === 'rejected') {
|
||||||
|
rejectedRateLimitSignal = rateLimitSignal;
|
||||||
|
}
|
||||||
|
params.rateLimitGovernor?.report(rateLimitSignal);
|
||||||
|
}
|
||||||
|
assertInitIsolation(message, params.allowedToolIds, params.expectedMcpServerNames);
|
||||||
|
if (isResult(message)) {
|
||||||
|
result = message;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
} finally {
|
||||||
|
params.abortSignal?.removeEventListener('abort', onAbort);
|
||||||
|
}
|
||||||
|
if (params.abortSignal?.aborted) {
|
||||||
|
throw createAbortError();
|
||||||
}
|
}
|
||||||
if (!result) {
|
if (!result) {
|
||||||
throw new Error('Claude Code query returned no result message');
|
throw new Error('Claude Code query returned no result message');
|
||||||
}
|
}
|
||||||
return result;
|
return {
|
||||||
|
result,
|
||||||
|
...(rejectedRateLimitSignal ? { rejectedRateLimitSignal } : {}),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function collectResultWithRateLimitRetry(params: Parameters<typeof collectResult>[0]): Promise<SDKResultMessage> {
|
||||||
|
// maxRetryAttempts() returns 1 when no governor is present or pacing is
|
||||||
|
// disabled, so a rate-limited result surfaces without an extra query; the
|
||||||
|
// Claude Code SDK applies its own backoff for transient rejections.
|
||||||
|
const maxAttempts = params.rateLimitGovernor?.maxRetryAttempts() ?? 1;
|
||||||
|
for (let attempt = 0; ; attempt += 1) {
|
||||||
|
const outcome = await collectResult(params);
|
||||||
|
if (!isClaudeRateLimitResult(outcome.result, outcome.rejectedRateLimitSignal) || attempt >= maxAttempts - 1) {
|
||||||
|
return outcome.result;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
|
export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
private readonly runQuery: QueryFn;
|
private readonly runQuery: QueryFn;
|
||||||
private readonly logger: KtxLogger;
|
|
||||||
|
|
||||||
constructor(private readonly deps: ClaudeCodeKtxLlmRuntimeDeps) {
|
constructor(private readonly deps: ClaudeCodeKtxLlmRuntimeDeps) {
|
||||||
this.runQuery = deps.query ?? defaultQuery;
|
this.runQuery = deps.query ?? defaultQuery;
|
||||||
this.logger = deps.logger ?? noopLogger;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async generateText(input: KtxGenerateTextInput): Promise<string> {
|
async generateText(input: KtxGenerateTextInput): Promise<string> {
|
||||||
|
|
@ -252,12 +350,14 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
tools: input.tools,
|
tools: input.tools,
|
||||||
});
|
});
|
||||||
const startedAt = Date.now();
|
const startedAt = Date.now();
|
||||||
const result = await collectResult({
|
const result = await collectResultWithRateLimitRetry({
|
||||||
query: this.runQuery,
|
query: this.runQuery,
|
||||||
prompt: [input.system, input.prompt].filter(Boolean).join('\n\n'),
|
prompt: [input.system, input.prompt].filter(Boolean).join('\n\n'),
|
||||||
options,
|
options,
|
||||||
allowedToolIds: new Set(mcpToolIds(input.tools ?? {})),
|
allowedToolIds: new Set(mcpToolIds(input.tools ?? {})),
|
||||||
expectedMcpServerNames: expectedMcpServerNames(input.tools),
|
expectedMcpServerNames: expectedMcpServerNames(input.tools),
|
||||||
|
rateLimitGovernor: this.deps.rateLimitGovernor,
|
||||||
|
abortSignal: input.abortSignal,
|
||||||
});
|
});
|
||||||
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: claudeTokenUsage(result) });
|
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: claudeTokenUsage(result) });
|
||||||
const error = resultError(result);
|
const error = resultError(result);
|
||||||
|
|
@ -289,12 +389,14 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
outputFormat: { type: 'json_schema' as const, schema: jsonSchema(input.schema as z.ZodType) },
|
outputFormat: { type: 'json_schema' as const, schema: jsonSchema(input.schema as z.ZodType) },
|
||||||
};
|
};
|
||||||
const startedAt = Date.now();
|
const startedAt = Date.now();
|
||||||
const result = await collectResult({
|
const result = await collectResultWithRateLimitRetry({
|
||||||
query: this.runQuery,
|
query: this.runQuery,
|
||||||
prompt: [input.system, input.prompt].filter(Boolean).join('\n\n'),
|
prompt: [input.system, input.prompt].filter(Boolean).join('\n\n'),
|
||||||
options,
|
options,
|
||||||
allowedToolIds: new Set([...mcpToolIds(input.tools ?? {}), STRUCTURED_OUTPUT_TOOL_NAME]),
|
allowedToolIds: new Set([...mcpToolIds(input.tools ?? {}), STRUCTURED_OUTPUT_TOOL_NAME]),
|
||||||
expectedMcpServerNames: expectedMcpServerNames(input.tools),
|
expectedMcpServerNames: expectedMcpServerNames(input.tools),
|
||||||
|
rateLimitGovernor: this.deps.rateLimitGovernor,
|
||||||
|
abortSignal: input.abortSignal,
|
||||||
});
|
});
|
||||||
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: claudeTokenUsage(result) });
|
input.onMetrics?.({ totalMs: Date.now() - startedAt, usage: claudeTokenUsage(result) });
|
||||||
const error = resultError(result);
|
const error = resultError(result);
|
||||||
|
|
@ -308,9 +410,7 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
}
|
}
|
||||||
|
|
||||||
async runAgentLoop(params: RunLoopParams): Promise<RunLoopResult> {
|
async runAgentLoop(params: RunLoopParams): Promise<RunLoopResult> {
|
||||||
let stepIndex = 0;
|
|
||||||
const startedAt = Date.now();
|
const startedAt = Date.now();
|
||||||
const stepBoundariesMs: number[] = [];
|
|
||||||
try {
|
try {
|
||||||
const options = baseOptions({
|
const options = baseOptions({
|
||||||
projectDir: this.deps.projectDir,
|
projectDir: this.deps.projectDir,
|
||||||
|
|
@ -319,28 +419,14 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
maxTurns: params.stepBudget,
|
maxTurns: params.stepBudget,
|
||||||
tools: params.toolSet,
|
tools: params.toolSet,
|
||||||
});
|
});
|
||||||
const result = await collectResult({
|
const result = await collectResultWithRateLimitRetry({
|
||||||
query: this.runQuery,
|
query: this.runQuery,
|
||||||
prompt: params.userPrompt,
|
prompt: params.userPrompt,
|
||||||
options: { ...options, systemPrompt: params.systemPrompt },
|
options: { ...options, systemPrompt: params.systemPrompt },
|
||||||
allowedToolIds: new Set(mcpToolIds(params.toolSet)),
|
allowedToolIds: new Set(mcpToolIds(params.toolSet)),
|
||||||
expectedMcpServerNames: expectedMcpServerNames(params.toolSet),
|
expectedMcpServerNames: expectedMcpServerNames(params.toolSet),
|
||||||
onAssistantTurn: async () => {
|
rateLimitGovernor: this.deps.rateLimitGovernor,
|
||||||
stepIndex += 1;
|
abortSignal: params.abortSignal,
|
||||||
stepBoundariesMs.push(Date.now() - startedAt);
|
|
||||||
if (!params.onStepFinish) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
await params.onStepFinish({ stepIndex, stepBudget: params.stepBudget });
|
|
||||||
} catch (error) {
|
|
||||||
this.logger.warn(
|
|
||||||
`[claude-code-runner] onStepFinish callback threw; ignoring: ${
|
|
||||||
error instanceof Error ? error.message : String(error)
|
|
||||||
}`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
},
|
|
||||||
});
|
});
|
||||||
const stopReason = mapClaudeCodeStopReason(result);
|
const stopReason = mapClaudeCodeStopReason(result);
|
||||||
const error = resultError(result);
|
const error = resultError(result);
|
||||||
|
|
@ -349,17 +435,24 @@ export class ClaudeCodeKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
...(stopReason === 'error' && error ? { error } : {}),
|
...(stopReason === 'error' && error ? { error } : {}),
|
||||||
metrics: {
|
metrics: {
|
||||||
totalMs: Date.now() - startedAt,
|
totalMs: Date.now() - startedAt,
|
||||||
stepCount: stepIndex,
|
// Authoritative turn count from the SDK result. The runtime no longer
|
||||||
stepBoundariesMs,
|
// re-derives a per-turn counter: it could not match the SDK's `num_turns`
|
||||||
|
// and overshot `maxTurns` (the source of the misleading `step 70/40`).
|
||||||
|
// Per-step boundaries require that counter and are not consumed anywhere.
|
||||||
|
stepCount: result.num_turns,
|
||||||
|
stepBoundariesMs: [],
|
||||||
usage: claudeTokenUsage(result),
|
usage: claudeTokenUsage(result),
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
if (isAbortError(error)) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
const err = error instanceof Error ? error : new Error(String(error));
|
const err = error instanceof Error ? error : new Error(String(error));
|
||||||
return {
|
return {
|
||||||
stopReason: 'error',
|
stopReason: 'error',
|
||||||
error: err,
|
error: err,
|
||||||
metrics: { totalMs: Date.now() - startedAt, stepCount: stepIndex, stepBoundariesMs, usage: {} },
|
metrics: { totalMs: Date.now() - startedAt, stepCount: 0, stepBoundariesMs: [], usage: {} },
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -388,7 +481,7 @@ export async function runClaudeCodeAuthProbe(input: {
|
||||||
env: input.env,
|
env: input.env,
|
||||||
maxTurns: 1,
|
maxTurns: 1,
|
||||||
});
|
});
|
||||||
const result = await collectResult({
|
const result = await collectResultWithRateLimitRetry({
|
||||||
query: input.query ?? defaultQuery,
|
query: input.query ?? defaultQuery,
|
||||||
prompt: 'Reply with exactly: ok',
|
prompt: 'Reply with exactly: ok',
|
||||||
options,
|
options,
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import { z } from 'zod';
|
import { z } from 'zod';
|
||||||
import { noopLogger, type KtxLogger } from '../core/config.js';
|
import { isAbortError, linkAbortSignal } from '../core/abort.js';
|
||||||
import { isCompletedAgentStep, summarizeCodexExecEvents, type CodexExecEventSummary } from './codex-exec-events.js';
|
import { isCompletedAgentStep, summarizeCodexExecEvents, type CodexExecEventSummary } from './codex-exec-events.js';
|
||||||
import {
|
import {
|
||||||
startCodexRuntimeMcpServer,
|
startCodexRuntimeMcpServer,
|
||||||
|
|
@ -8,6 +8,7 @@ import {
|
||||||
import { resolveCodexModel } from './codex-models.js';
|
import { resolveCodexModel } from './codex-models.js';
|
||||||
import { buildCodexRuntimeConfig } from './codex-runtime-config.js';
|
import { buildCodexRuntimeConfig } from './codex-runtime-config.js';
|
||||||
import { CodexSdkCliRunner, type CodexSdkRunner } from './codex-sdk-runner.js';
|
import { CodexSdkCliRunner, type CodexSdkRunner } from './codex-sdk-runner.js';
|
||||||
|
import type { RateLimitGovernor } from './rate-limit-governor.js';
|
||||||
import type {
|
import type {
|
||||||
KtxGenerateObjectInput,
|
KtxGenerateObjectInput,
|
||||||
KtxGenerateTextInput,
|
KtxGenerateTextInput,
|
||||||
|
|
@ -23,7 +24,7 @@ export interface CodexKtxLlmRuntimeDeps {
|
||||||
modelSlots: { default: string } & Partial<Record<string, string>>;
|
modelSlots: { default: string } & Partial<Record<string, string>>;
|
||||||
runner?: CodexSdkRunner;
|
runner?: CodexSdkRunner;
|
||||||
startMcpServer?: (input: { projectDir: string; toolSet: KtxRuntimeToolSet }) => Promise<CodexRuntimeMcpServerHandle>;
|
startMcpServer?: (input: { projectDir: string; toolSet: KtxRuntimeToolSet }) => Promise<CodexRuntimeMcpServerHandle>;
|
||||||
logger?: KtxLogger;
|
rateLimitGovernor?: Pick<RateLimitGovernor, 'waitForReady' | 'report' | 'maxRetryAttempts'>;
|
||||||
}
|
}
|
||||||
|
|
||||||
function modelForRole(modelSlots: CodexKtxLlmRuntimeDeps['modelSlots'], role: string): string {
|
function modelForRole(modelSlots: CodexKtxLlmRuntimeDeps['modelSlots'], role: string): string {
|
||||||
|
|
@ -37,7 +38,6 @@ function promptWithSystem(system: string | undefined, prompt: string): string {
|
||||||
interface CollectCodexEventsOptions {
|
interface CollectCodexEventsOptions {
|
||||||
stepBudget?: number;
|
stepBudget?: number;
|
||||||
abortController?: AbortController;
|
abortController?: AbortController;
|
||||||
onStep?: (stepIndex: number) => void | Promise<void>;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
interface CollectCodexEventsResult {
|
interface CollectCodexEventsResult {
|
||||||
|
|
@ -55,8 +55,8 @@ function isTurnCompleted(event: unknown): boolean {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Drains the Codex stream once, emitting a step as each agent action completes
|
* Drains the Codex stream once, counting each completed agent action so the
|
||||||
* so callers see live progress and the step budget is enforced mid-run. Every
|
* step budget is enforced mid-run. Every
|
||||||
* completed agent-action item counts (see {@link isCompletedAgentStep}), so
|
* completed agent-action item counts (see {@link isCompletedAgentStep}), so
|
||||||
* built-in `command_execution` steps decrement the budget the same as
|
* built-in `command_execution` steps decrement the budget the same as
|
||||||
* `mcp_tool_call`s. A turn that produced no actions still counts as one step,
|
* `mcp_tool_call`s. A turn that produced no actions still counts as one step,
|
||||||
|
|
@ -90,7 +90,6 @@ async function collectEvents(
|
||||||
}
|
}
|
||||||
|
|
||||||
completedSteps += 1;
|
completedSteps += 1;
|
||||||
await options.onStep?.(completedSteps);
|
|
||||||
if (isActionStep && options.stepBudget !== undefined && completedSteps >= options.stepBudget) {
|
if (isActionStep && options.stepBudget !== undefined && completedSteps >= options.stepBudget) {
|
||||||
budgetExceeded = true;
|
budgetExceeded = true;
|
||||||
options.abortController?.abort();
|
options.abortController?.abort();
|
||||||
|
|
@ -159,13 +158,48 @@ function runtimeToolNames(toolSet: KtxRuntimeToolSet | undefined): string[] {
|
||||||
return Object.values(toolSet ?? {}).map((descriptor) => descriptor.name);
|
return Object.values(toolSet ?? {}).map((descriptor) => descriptor.name);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const CODEX_RATE_LIMIT_MARKERS = /\b429\b|rate limit|too many requests|quota exceeded|temporarily overloaded/i;
|
||||||
|
|
||||||
|
function isCodexRateLimitError(error: Error | undefined): boolean {
|
||||||
|
return !!error && CODEX_RATE_LIMIT_MARKERS.test(error.message);
|
||||||
|
}
|
||||||
|
|
||||||
export class CodexKtxLlmRuntime implements KtxLlmRuntimePort {
|
export class CodexKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
private readonly runner: CodexSdkRunner;
|
private readonly runner: CodexSdkRunner;
|
||||||
private readonly logger: KtxLogger;
|
|
||||||
|
|
||||||
constructor(private readonly deps: CodexKtxLlmRuntimeDeps) {
|
constructor(private readonly deps: CodexKtxLlmRuntimeDeps) {
|
||||||
this.runner = deps.runner ?? new CodexSdkCliRunner();
|
this.runner = deps.runner ?? new CodexSdkCliRunner();
|
||||||
this.logger = deps.logger ?? noopLogger;
|
}
|
||||||
|
|
||||||
|
private async runWithRateLimitRetry<T>(
|
||||||
|
abortSignal: AbortSignal | undefined,
|
||||||
|
run: () => Promise<T>,
|
||||||
|
getError: (result: T) => Error | undefined,
|
||||||
|
): Promise<T> {
|
||||||
|
// maxRetryAttempts() returns 1 when no governor is present or pacing is
|
||||||
|
// disabled, so an opaque rate-limit failure surfaces on the first attempt
|
||||||
|
// instead of being retried with no backoff.
|
||||||
|
const maxAttempts = this.deps.rateLimitGovernor?.maxRetryAttempts() ?? 1;
|
||||||
|
for (let attempt = 0; ; attempt += 1) {
|
||||||
|
await this.deps.rateLimitGovernor?.waitForReady(abortSignal);
|
||||||
|
const lastAttempt = attempt >= maxAttempts - 1;
|
||||||
|
try {
|
||||||
|
const result = await run();
|
||||||
|
const error = getError(result);
|
||||||
|
if (!isCodexRateLimitError(error) || lastAttempt) {
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
if (isAbortError(error)) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
const err = error instanceof Error ? error : new Error(String(error));
|
||||||
|
if (!isCodexRateLimitError(err) || lastAttempt) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this.deps.rateLimitGovernor?.report({ provider: 'codex', status: 'rejected', rateLimitType: 'opaque' });
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async generateText(input: KtxGenerateTextInput): Promise<string> {
|
async generateText(input: KtxGenerateTextInput): Promise<string> {
|
||||||
|
|
@ -190,18 +224,26 @@ export class CodexKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
}
|
}
|
||||||
: {}),
|
: {}),
|
||||||
});
|
});
|
||||||
const collected = await collectEvents(
|
const result = await this.runWithRateLimitRetry(
|
||||||
await this.runner.runStreamed({
|
input.abortSignal,
|
||||||
projectDir: this.deps.projectDir,
|
async () => {
|
||||||
model,
|
const collected = await collectEvents(
|
||||||
prompt: promptWithSystem(input.system, input.prompt),
|
await this.runner.runStreamed({
|
||||||
configOverrides: config.configOverrides,
|
projectDir: this.deps.projectDir,
|
||||||
env: config.env,
|
model,
|
||||||
}),
|
prompt: promptWithSystem(input.system, input.prompt),
|
||||||
|
configOverrides: config.configOverrides,
|
||||||
|
env: config.env,
|
||||||
|
...(input.abortSignal ? { signal: input.abortSignal } : {}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const summary = summarizeCodexExecEvents(collected.events, { startedAt });
|
||||||
|
return { collected, summary };
|
||||||
|
},
|
||||||
|
({ collected, summary }) => summaryError(summary, collected.streamError),
|
||||||
);
|
);
|
||||||
const summary = summarizeCodexExecEvents(collected.events, { startedAt });
|
input.onMetrics?.(metrics(result.summary, startedAt));
|
||||||
input.onMetrics?.(metrics(summary, startedAt));
|
return assertSuccessfulText(result.summary, result.collected.streamError);
|
||||||
return assertSuccessfulText(summary, collected.streamError);
|
|
||||||
} finally {
|
} finally {
|
||||||
await mcp?.close();
|
await mcp?.close();
|
||||||
}
|
}
|
||||||
|
|
@ -231,19 +273,27 @@ export class CodexKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
}
|
}
|
||||||
: {}),
|
: {}),
|
||||||
});
|
});
|
||||||
const collected = await collectEvents(
|
const result = await this.runWithRateLimitRetry(
|
||||||
await this.runner.runStreamed({
|
input.abortSignal,
|
||||||
projectDir: this.deps.projectDir,
|
async () => {
|
||||||
model,
|
const collected = await collectEvents(
|
||||||
prompt: promptWithSystem(input.system, input.prompt),
|
await this.runner.runStreamed({
|
||||||
configOverrides: config.configOverrides,
|
projectDir: this.deps.projectDir,
|
||||||
env: config.env,
|
model,
|
||||||
outputSchema: z.toJSONSchema(input.schema, { target: 'draft-7' }) as Record<string, unknown>,
|
prompt: promptWithSystem(input.system, input.prompt),
|
||||||
}),
|
configOverrides: config.configOverrides,
|
||||||
|
env: config.env,
|
||||||
|
outputSchema: z.toJSONSchema(input.schema, { target: 'draft-7' }) as Record<string, unknown>,
|
||||||
|
...(input.abortSignal ? { signal: input.abortSignal } : {}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const summary = summarizeCodexExecEvents(collected.events, { startedAt });
|
||||||
|
return { collected, summary };
|
||||||
|
},
|
||||||
|
({ collected, summary }) => summaryError(summary, collected.streamError),
|
||||||
);
|
);
|
||||||
const summary = summarizeCodexExecEvents(collected.events, { startedAt });
|
input.onMetrics?.(metrics(result.summary, startedAt));
|
||||||
input.onMetrics?.(metrics(summary, startedAt));
|
return parseStructuredOutput(input.schema, assertSuccessfulText(result.summary, result.collected.streamError));
|
||||||
return parseStructuredOutput(input.schema, assertSuccessfulText(summary, collected.streamError));
|
|
||||||
} finally {
|
} finally {
|
||||||
await mcp?.close();
|
await mcp?.close();
|
||||||
}
|
}
|
||||||
|
|
@ -272,41 +322,50 @@ export class CodexKtxLlmRuntime implements KtxLlmRuntimePort {
|
||||||
}
|
}
|
||||||
: {}),
|
: {}),
|
||||||
});
|
});
|
||||||
const abortController = new AbortController();
|
const result = await this.runWithRateLimitRetry(
|
||||||
const onStep = async (stepIndex: number): Promise<void> => {
|
params.abortSignal,
|
||||||
try {
|
async () => {
|
||||||
await params.onStepFinish?.({ stepIndex, stepBudget: params.stepBudget });
|
const linked = linkAbortSignal(params.abortSignal);
|
||||||
} catch (error) {
|
const abortController = linked.controller;
|
||||||
this.logger.warn(
|
try {
|
||||||
`[codex-runner] onStepFinish callback threw; ignoring: ${error instanceof Error ? error.message : String(error)}`,
|
const collected = await collectEvents(
|
||||||
);
|
await this.runner.runStreamed({
|
||||||
}
|
projectDir: this.deps.projectDir,
|
||||||
};
|
model,
|
||||||
const collected = await collectEvents(
|
prompt: promptWithSystem(params.systemPrompt, params.userPrompt),
|
||||||
await this.runner.runStreamed({
|
configOverrides: config.configOverrides,
|
||||||
projectDir: this.deps.projectDir,
|
env: config.env,
|
||||||
model,
|
signal: abortController.signal,
|
||||||
prompt: promptWithSystem(params.systemPrompt, params.userPrompt),
|
}),
|
||||||
configOverrides: config.configOverrides,
|
{ stepBudget: params.stepBudget, abortController },
|
||||||
env: config.env,
|
);
|
||||||
signal: abortController.signal,
|
const summary = summarizeCodexExecEvents(collected.events, { startedAt });
|
||||||
}),
|
return { collected, summary };
|
||||||
{ stepBudget: params.stepBudget, abortController, onStep },
|
} finally {
|
||||||
|
linked.dispose();
|
||||||
|
}
|
||||||
|
},
|
||||||
|
({ collected, summary }) => summaryError(summary, collected.streamError),
|
||||||
);
|
);
|
||||||
const summary = summarizeCodexExecEvents(collected.events, { startedAt });
|
const error = summaryError(result.summary, result.collected.streamError);
|
||||||
const error = summaryError(summary, collected.streamError);
|
if (isAbortError(error)) {
|
||||||
const stopReason = collected.budgetExceeded ? 'budget' : error ? 'error' : summary.stopReason;
|
throw error;
|
||||||
|
}
|
||||||
|
const stopReason = result.collected.budgetExceeded ? 'budget' : error ? 'error' : result.summary.stopReason;
|
||||||
return {
|
return {
|
||||||
stopReason,
|
stopReason,
|
||||||
...(stopReason === 'error' && error ? { error } : {}),
|
...(stopReason === 'error' && error ? { error } : {}),
|
||||||
metrics: {
|
metrics: {
|
||||||
totalMs: Date.now() - startedAt,
|
totalMs: Date.now() - startedAt,
|
||||||
usage: summary.usage,
|
usage: result.summary.usage,
|
||||||
stepCount: summary.stepCount,
|
stepCount: result.summary.stepCount,
|
||||||
stepBoundariesMs: summary.stepBoundariesMs,
|
stepBoundariesMs: result.summary.stepBoundariesMs,
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
if (isAbortError(error)) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
const err = error instanceof Error ? error : new Error(String(error));
|
const err = error instanceof Error ? error : new Error(String(error));
|
||||||
return {
|
return {
|
||||||
stopReason: 'error',
|
stopReason: 'error',
|
||||||
|
|
|
||||||
|
|
@ -6,16 +6,28 @@ import type { KtxProjectEmbeddingConfig, KtxProjectLlmConfig } from '../project/
|
||||||
import { AiSdkKtxLlmRuntime } from './ai-sdk-runtime.js';
|
import { AiSdkKtxLlmRuntime } from './ai-sdk-runtime.js';
|
||||||
import { ClaudeCodeKtxLlmRuntime } from './claude-code-runtime.js';
|
import { ClaudeCodeKtxLlmRuntime } from './claude-code-runtime.js';
|
||||||
import { CodexKtxLlmRuntime } from './codex-runtime.js';
|
import { CodexKtxLlmRuntime } from './codex-runtime.js';
|
||||||
|
import type { RateLimitGovernor } from './rate-limit-governor.js';
|
||||||
import type { KtxLlmRuntimePort } from './runtime-port.js';
|
import type { KtxLlmRuntimePort } from './runtime-port.js';
|
||||||
|
|
||||||
|
type ClaudeCodeRuntimeDeps = ConstructorParameters<typeof ClaudeCodeKtxLlmRuntime>[0] & {
|
||||||
|
rateLimitGovernor?: RateLimitGovernor;
|
||||||
|
};
|
||||||
|
type CodexRuntimeDeps = ConstructorParameters<typeof CodexKtxLlmRuntime>[0] & {
|
||||||
|
rateLimitGovernor?: RateLimitGovernor;
|
||||||
|
};
|
||||||
|
type AiSdkRuntimeDeps = ConstructorParameters<typeof AiSdkKtxLlmRuntime>[0] & {
|
||||||
|
rateLimitGovernor?: RateLimitGovernor;
|
||||||
|
};
|
||||||
|
|
||||||
interface LocalConfigDeps {
|
interface LocalConfigDeps {
|
||||||
env?: NodeJS.ProcessEnv;
|
env?: NodeJS.ProcessEnv;
|
||||||
projectDir?: string;
|
projectDir?: string;
|
||||||
|
rateLimitGovernor?: RateLimitGovernor;
|
||||||
createKtxLlmProvider?: typeof createKtxLlmProvider;
|
createKtxLlmProvider?: typeof createKtxLlmProvider;
|
||||||
createKtxEmbeddingProvider?: typeof createKtxEmbeddingProvider;
|
createKtxEmbeddingProvider?: typeof createKtxEmbeddingProvider;
|
||||||
createClaudeCodeRuntime?: (deps: ConstructorParameters<typeof ClaudeCodeKtxLlmRuntime>[0]) => KtxLlmRuntimePort;
|
createClaudeCodeRuntime?: (deps: ClaudeCodeRuntimeDeps) => KtxLlmRuntimePort;
|
||||||
createCodexRuntime?: (deps: ConstructorParameters<typeof CodexKtxLlmRuntime>[0]) => KtxLlmRuntimePort;
|
createCodexRuntime?: (deps: CodexRuntimeDeps) => KtxLlmRuntimePort;
|
||||||
createAiSdkRuntime?: (deps: { llmProvider: KtxLlmProvider }) => KtxLlmRuntimePort;
|
createAiSdkRuntime?: (deps: AiSdkRuntimeDeps) => KtxLlmRuntimePort;
|
||||||
}
|
}
|
||||||
|
|
||||||
function resolveOptional(value: string | undefined, env: NodeJS.ProcessEnv): string | undefined {
|
function resolveOptional(value: string | undefined, env: NodeJS.ProcessEnv): string | undefined {
|
||||||
|
|
@ -129,6 +141,7 @@ export function createLocalKtxLlmRuntimeFromConfig(
|
||||||
projectDir,
|
projectDir,
|
||||||
modelSlots: resolved.modelSlots,
|
modelSlots: resolved.modelSlots,
|
||||||
env: deps.env,
|
env: deps.env,
|
||||||
|
rateLimitGovernor: deps.rateLimitGovernor,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
if (resolved.backend === 'codex') {
|
if (resolved.backend === 'codex') {
|
||||||
|
|
@ -139,10 +152,14 @@ export function createLocalKtxLlmRuntimeFromConfig(
|
||||||
return (deps.createCodexRuntime ?? ((runtimeDeps) => new CodexKtxLlmRuntime(runtimeDeps)))({
|
return (deps.createCodexRuntime ?? ((runtimeDeps) => new CodexKtxLlmRuntime(runtimeDeps)))({
|
||||||
projectDir,
|
projectDir,
|
||||||
modelSlots: resolved.modelSlots,
|
modelSlots: resolved.modelSlots,
|
||||||
|
rateLimitGovernor: deps.rateLimitGovernor,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
const llmProvider = (deps.createKtxLlmProvider ?? createKtxLlmProvider)(resolved);
|
const llmProvider = (deps.createKtxLlmProvider ?? createKtxLlmProvider)(resolved);
|
||||||
return (deps.createAiSdkRuntime ?? ((runtimeDeps) => new AiSdkKtxLlmRuntime(runtimeDeps)))({ llmProvider });
|
return (deps.createAiSdkRuntime ?? ((runtimeDeps) => new AiSdkKtxLlmRuntime(runtimeDeps)))({
|
||||||
|
llmProvider,
|
||||||
|
rateLimitGovernor: deps.rateLimitGovernor,
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
export function resolveLocalKtxEmbeddingConfig(
|
export function resolveLocalKtxEmbeddingConfig(
|
||||||
|
|
|
||||||
387
packages/cli/src/context/llm/rate-limit-governor.ts
Normal file
387
packages/cli/src/context/llm/rate-limit-governor.ts
Normal file
|
|
@ -0,0 +1,387 @@
|
||||||
|
import { createAbortError, throwIfAborted } from '../core/abort.js';
|
||||||
|
|
||||||
|
export type RateLimitProvider = 'claude-subscription' | 'anthropic-api' | 'vertex' | 'codex';
|
||||||
|
type RateLimitSignalStatus = 'allowed' | 'warning' | 'rejected';
|
||||||
|
|
||||||
|
export interface RateLimitSignal {
|
||||||
|
provider: RateLimitProvider;
|
||||||
|
status: RateLimitSignalStatus;
|
||||||
|
resetAtMs?: number;
|
||||||
|
retryAfterMs?: number;
|
||||||
|
utilization?: number;
|
||||||
|
rateLimitType?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RateLimitRetryConfig {
|
||||||
|
maxAttempts: number;
|
||||||
|
baseDelayMs: number;
|
||||||
|
maxDelayMs: number;
|
||||||
|
jitter: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RateLimitGovernorConfig {
|
||||||
|
enabled: boolean;
|
||||||
|
maxConcurrency: number;
|
||||||
|
throttleThreshold: number;
|
||||||
|
minConcurrencyUnderPressure: number;
|
||||||
|
maxWaitMs?: number;
|
||||||
|
waitStateTickMs: number;
|
||||||
|
retry: RateLimitRetryConfig;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type RateLimitWaitState =
|
||||||
|
| {
|
||||||
|
kind: 'rate_limit_observed';
|
||||||
|
provider: RateLimitProvider;
|
||||||
|
status: RateLimitSignalStatus;
|
||||||
|
rateLimitType?: string;
|
||||||
|
resetAtMs?: number;
|
||||||
|
retryAfterMs?: number;
|
||||||
|
utilization?: number;
|
||||||
|
}
|
||||||
|
| {
|
||||||
|
kind: 'concurrency_adjusted';
|
||||||
|
provider: RateLimitProvider;
|
||||||
|
from: number;
|
||||||
|
to: number;
|
||||||
|
reason: string;
|
||||||
|
rateLimitType?: string;
|
||||||
|
utilization?: number;
|
||||||
|
}
|
||||||
|
| {
|
||||||
|
kind: 'wait_started' | 'wait_tick' | 'wait_finished';
|
||||||
|
provider: RateLimitProvider;
|
||||||
|
rateLimitType?: string;
|
||||||
|
resumeAtMs: number;
|
||||||
|
remainingMs: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
export interface RateLimitGovernorDeps {
|
||||||
|
now?: () => number;
|
||||||
|
sleep?: (ms: number, signal?: AbortSignal) => Promise<void>;
|
||||||
|
random?: () => number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type RateLimitRelease = () => void;
|
||||||
|
type Subscriber = (state: RateLimitWaitState) => void;
|
||||||
|
|
||||||
|
const defaultSleep = (ms: number, signal?: AbortSignal): Promise<void> =>
|
||||||
|
new Promise((resolve, reject) => {
|
||||||
|
if (signal?.aborted) {
|
||||||
|
reject(createAbortError());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const timeout = setTimeout(resolve, ms);
|
||||||
|
signal?.addEventListener(
|
||||||
|
'abort',
|
||||||
|
() => {
|
||||||
|
clearTimeout(timeout);
|
||||||
|
reject(createAbortError());
|
||||||
|
},
|
||||||
|
{ once: true },
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
export function createRateLimitGovernorConfig(
|
||||||
|
input: Partial<RateLimitGovernorConfig> & { retry?: Partial<RateLimitRetryConfig> } = {},
|
||||||
|
): RateLimitGovernorConfig {
|
||||||
|
return {
|
||||||
|
enabled: input.enabled ?? true,
|
||||||
|
maxConcurrency: input.maxConcurrency ?? 1,
|
||||||
|
throttleThreshold: input.throttleThreshold ?? 0.8,
|
||||||
|
minConcurrencyUnderPressure: input.minConcurrencyUnderPressure ?? 1,
|
||||||
|
...(input.maxWaitMs !== undefined ? { maxWaitMs: input.maxWaitMs } : {}),
|
||||||
|
waitStateTickMs: input.waitStateTickMs ?? 1_000,
|
||||||
|
retry: {
|
||||||
|
maxAttempts: input.retry?.maxAttempts ?? 6,
|
||||||
|
baseDelayMs: input.retry?.baseDelayMs ?? 1_000,
|
||||||
|
maxDelayMs: input.retry?.maxDelayMs ?? 60_000,
|
||||||
|
jitter: input.retry?.jitter ?? true,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export class RateLimitGovernor {
|
||||||
|
private readonly now: () => number;
|
||||||
|
private readonly sleep: (ms: number, signal?: AbortSignal) => Promise<void>;
|
||||||
|
private readonly random: () => number;
|
||||||
|
private readonly subscribers = new Set<Subscriber>();
|
||||||
|
private waiters: Array<() => void> = [];
|
||||||
|
private active = 0;
|
||||||
|
private effectiveLimit: number;
|
||||||
|
private pausedUntilMs: number | null = null;
|
||||||
|
private pausedProvider: RateLimitProvider | null = null;
|
||||||
|
private pausedRateLimitType: string | undefined;
|
||||||
|
private pausedTickMs: number | null = null;
|
||||||
|
private opaqueAttempts = new Map<RateLimitProvider, number>();
|
||||||
|
private pauseGeneration = 0;
|
||||||
|
private visibleWaitAbort: AbortController | null = null;
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private readonly config: RateLimitGovernorConfig,
|
||||||
|
deps: RateLimitGovernorDeps = {},
|
||||||
|
) {
|
||||||
|
this.now = deps.now ?? Date.now;
|
||||||
|
this.sleep = deps.sleep ?? defaultSleep;
|
||||||
|
this.random = deps.random ?? Math.random;
|
||||||
|
this.effectiveLimit = Math.max(1, config.maxConcurrency);
|
||||||
|
}
|
||||||
|
|
||||||
|
currentLimit(): number {
|
||||||
|
return this.config.enabled ? this.effectiveLimit : this.config.maxConcurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Total attempts a runtime should make for a single rate-limited LLM call,
|
||||||
|
* including the first try. Returns 1 (no outer retry) when pacing is disabled:
|
||||||
|
* the outer retry loop only exists to cooperate with this governor's pause, so
|
||||||
|
* without active pacing there is no backoff to apply and the backend's own
|
||||||
|
* retry handles transient rejections.
|
||||||
|
*/
|
||||||
|
maxRetryAttempts(): number {
|
||||||
|
return this.config.enabled ? Math.max(1, this.config.retry.maxAttempts) : 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
activeSlots(): number {
|
||||||
|
return this.active;
|
||||||
|
}
|
||||||
|
|
||||||
|
subscribe(cb: Subscriber): () => void {
|
||||||
|
this.subscribers.add(cb);
|
||||||
|
if (this.pausedUntilMs !== null) {
|
||||||
|
this.startVisibleWaitTicker();
|
||||||
|
}
|
||||||
|
return () => {
|
||||||
|
this.subscribers.delete(cb);
|
||||||
|
if (this.subscribers.size === 0) {
|
||||||
|
this.stopVisibleWaitTicker();
|
||||||
|
this.wakeWaiters();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
report(signal: RateLimitSignal): void {
|
||||||
|
if (!this.config.enabled) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.emit({
|
||||||
|
kind: 'rate_limit_observed',
|
||||||
|
provider: signal.provider,
|
||||||
|
status: signal.status,
|
||||||
|
...(signal.rateLimitType ? { rateLimitType: signal.rateLimitType } : {}),
|
||||||
|
...(signal.resetAtMs !== undefined ? { resetAtMs: signal.resetAtMs } : {}),
|
||||||
|
...(signal.retryAfterMs !== undefined ? { retryAfterMs: signal.retryAfterMs } : {}),
|
||||||
|
...(signal.utilization !== undefined ? { utilization: signal.utilization } : {}),
|
||||||
|
});
|
||||||
|
|
||||||
|
if (signal.status === 'rejected') {
|
||||||
|
this.applyPause(signal);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (signal.status === 'warning' || (signal.utilization ?? 0) >= this.config.throttleThreshold) {
|
||||||
|
this.adjustLimit(Math.max(1, this.config.minConcurrencyUnderPressure), signal, 'provider pressure');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.opaqueAttempts.delete(signal.provider);
|
||||||
|
if ((signal.utilization ?? 0) < this.config.throttleThreshold) {
|
||||||
|
this.adjustLimit(Math.max(1, this.config.maxConcurrency), signal, 'provider recovered');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async waitForReady(signal?: AbortSignal): Promise<void> {
|
||||||
|
throwIfAborted(signal);
|
||||||
|
if (!this.config.enabled) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
await this.waitForPause(signal);
|
||||||
|
throwIfAborted(signal);
|
||||||
|
}
|
||||||
|
|
||||||
|
async acquireWorkSlot(signal?: AbortSignal): Promise<RateLimitRelease> {
|
||||||
|
throwIfAborted(signal);
|
||||||
|
if (!this.config.enabled) {
|
||||||
|
this.active += 1;
|
||||||
|
return () => {
|
||||||
|
this.active -= 1;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
throwIfAborted(signal);
|
||||||
|
await this.waitForPause(signal);
|
||||||
|
throwIfAborted(signal);
|
||||||
|
if (this.active < this.effectiveLimit) {
|
||||||
|
this.active += 1;
|
||||||
|
let released = false;
|
||||||
|
return () => {
|
||||||
|
if (released) return;
|
||||||
|
released = true;
|
||||||
|
this.active -= 1;
|
||||||
|
this.wakeWaiters();
|
||||||
|
};
|
||||||
|
}
|
||||||
|
await this.waitForSlot(signal);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private applyPause(signal: RateLimitSignal): void {
|
||||||
|
const resumeAtMs = this.resumeAtMsFor(signal);
|
||||||
|
const boundedResumeAtMs =
|
||||||
|
this.config.maxWaitMs === undefined ? resumeAtMs : Math.min(resumeAtMs, this.now() + this.config.maxWaitMs);
|
||||||
|
if (this.pausedUntilMs === null || boundedResumeAtMs > this.pausedUntilMs) {
|
||||||
|
this.pausedUntilMs = boundedResumeAtMs;
|
||||||
|
this.pausedProvider = signal.provider;
|
||||||
|
this.pausedRateLimitType = signal.rateLimitType;
|
||||||
|
this.pausedTickMs = signal.rateLimitType === 'opaque' ? Math.max(1, boundedResumeAtMs - this.now()) : null;
|
||||||
|
this.emitWait('wait_started');
|
||||||
|
this.startVisibleWaitTicker();
|
||||||
|
this.wakeWaiters();
|
||||||
|
}
|
||||||
|
this.adjustLimit(Math.max(1, this.config.minConcurrencyUnderPressure), signal, 'provider rejected');
|
||||||
|
}
|
||||||
|
|
||||||
|
private resumeAtMsFor(signal: RateLimitSignal): number {
|
||||||
|
if (signal.resetAtMs !== undefined) {
|
||||||
|
return signal.resetAtMs;
|
||||||
|
}
|
||||||
|
if (signal.retryAfterMs !== undefined) {
|
||||||
|
return this.now() + signal.retryAfterMs;
|
||||||
|
}
|
||||||
|
const attempts = this.opaqueAttempts.get(signal.provider) ?? 0;
|
||||||
|
this.opaqueAttempts.set(signal.provider, Math.min(attempts + 1, this.config.retry.maxAttempts));
|
||||||
|
const base = Math.min(
|
||||||
|
this.config.retry.maxDelayMs,
|
||||||
|
this.config.retry.baseDelayMs * 2 ** Math.min(attempts, this.config.retry.maxAttempts - 1),
|
||||||
|
);
|
||||||
|
const jitterMultiplier = this.config.retry.jitter ? 0.75 + this.random() * 0.5 : 1;
|
||||||
|
return this.now() + Math.round(base * jitterMultiplier);
|
||||||
|
}
|
||||||
|
|
||||||
|
private adjustLimit(to: number, signal: RateLimitSignal, reason: string): void {
|
||||||
|
const bounded = Math.max(1, Math.min(this.config.maxConcurrency, to));
|
||||||
|
if (bounded === this.effectiveLimit) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const from = this.effectiveLimit;
|
||||||
|
this.effectiveLimit = bounded;
|
||||||
|
this.emit({
|
||||||
|
kind: 'concurrency_adjusted',
|
||||||
|
provider: signal.provider,
|
||||||
|
from,
|
||||||
|
to: bounded,
|
||||||
|
reason,
|
||||||
|
...(signal.rateLimitType ? { rateLimitType: signal.rateLimitType } : {}),
|
||||||
|
...(signal.utilization !== undefined ? { utilization: signal.utilization } : {}),
|
||||||
|
});
|
||||||
|
this.wakeWaiters();
|
||||||
|
}
|
||||||
|
|
||||||
|
private startVisibleWaitTicker(): void {
|
||||||
|
if (this.subscribers.size === 0 || this.pausedUntilMs === null) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.stopVisibleWaitTicker();
|
||||||
|
const generation = (this.pauseGeneration += 1);
|
||||||
|
const controller = new AbortController();
|
||||||
|
this.visibleWaitAbort = controller;
|
||||||
|
void this.runVisibleWaitTicker(generation, controller.signal).catch(() => undefined);
|
||||||
|
}
|
||||||
|
|
||||||
|
private stopVisibleWaitTicker(): void {
|
||||||
|
this.visibleWaitAbort?.abort();
|
||||||
|
this.visibleWaitAbort = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async runVisibleWaitTicker(generation: number, signal: AbortSignal): Promise<void> {
|
||||||
|
while (!signal.aborted && generation === this.pauseGeneration && this.pausedUntilMs !== null) {
|
||||||
|
const remainingMs = this.pausedUntilMs - this.now();
|
||||||
|
if (remainingMs <= 0) {
|
||||||
|
this.finishPause(generation);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.emitWait('wait_tick');
|
||||||
|
await this.sleep(Math.min(this.pausedTickMs ?? this.config.waitStateTickMs, remainingMs), signal);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private finishPause(generation?: number): void {
|
||||||
|
if (generation !== undefined && generation !== this.pauseGeneration) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.emitWait('wait_finished');
|
||||||
|
this.pausedUntilMs = null;
|
||||||
|
this.pausedProvider = null;
|
||||||
|
this.pausedRateLimitType = undefined;
|
||||||
|
this.pausedTickMs = null;
|
||||||
|
this.stopVisibleWaitTicker();
|
||||||
|
this.wakeWaiters();
|
||||||
|
}
|
||||||
|
|
||||||
|
private async waitForPause(signal?: AbortSignal): Promise<void> {
|
||||||
|
throwIfAborted(signal);
|
||||||
|
while (this.pausedUntilMs !== null) {
|
||||||
|
const remainingMs = this.pausedUntilMs - this.now();
|
||||||
|
if (remainingMs <= 0) {
|
||||||
|
this.finishPause();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (this.visibleWaitAbort !== null) {
|
||||||
|
await this.waitForSlot(signal);
|
||||||
|
} else {
|
||||||
|
await this.sleep(Math.min(this.pausedTickMs ?? this.config.waitStateTickMs, remainingMs), signal);
|
||||||
|
}
|
||||||
|
throwIfAborted(signal);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private waitForSlot(signal?: AbortSignal): Promise<void> {
|
||||||
|
if (signal?.aborted) {
|
||||||
|
return Promise.reject(createAbortError());
|
||||||
|
}
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const wake = () => {
|
||||||
|
cleanup();
|
||||||
|
resolve();
|
||||||
|
};
|
||||||
|
const onAbort = () => {
|
||||||
|
cleanup();
|
||||||
|
reject(createAbortError());
|
||||||
|
};
|
||||||
|
const cleanup = () => {
|
||||||
|
this.waiters = this.waiters.filter((candidate) => candidate !== wake);
|
||||||
|
signal?.removeEventListener('abort', onAbort);
|
||||||
|
};
|
||||||
|
this.waiters.push(wake);
|
||||||
|
signal?.addEventListener('abort', onAbort, { once: true });
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private wakeWaiters(): void {
|
||||||
|
const waiters = this.waiters;
|
||||||
|
this.waiters = [];
|
||||||
|
for (const waiter of waiters) {
|
||||||
|
waiter();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private emitWait(kind: Extract<RateLimitWaitState['kind'], 'wait_started' | 'wait_tick' | 'wait_finished'>): void {
|
||||||
|
if (this.pausedUntilMs === null || this.pausedProvider === null) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.emit({
|
||||||
|
kind,
|
||||||
|
provider: this.pausedProvider,
|
||||||
|
...(this.pausedRateLimitType ? { rateLimitType: this.pausedRateLimitType } : {}),
|
||||||
|
resumeAtMs: this.pausedUntilMs,
|
||||||
|
remainingMs: Math.max(0, this.pausedUntilMs - this.now()),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private emit(state: RateLimitWaitState): void {
|
||||||
|
for (const subscriber of this.subscribers) {
|
||||||
|
subscriber(state);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -17,12 +17,6 @@ export type KtxRuntimeToolSet = Record<string, KtxRuntimeToolDescriptor>;
|
||||||
|
|
||||||
export type RunLoopStopReason = 'budget' | 'natural' | 'error';
|
export type RunLoopStopReason = 'budget' | 'natural' | 'error';
|
||||||
|
|
||||||
/** @internal */
|
|
||||||
export interface RunLoopStepInfo {
|
|
||||||
stepIndex: number;
|
|
||||||
stepBudget: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface LlmTokenUsage {
|
export interface LlmTokenUsage {
|
||||||
inputTokens?: number;
|
inputTokens?: number;
|
||||||
outputTokens?: number;
|
outputTokens?: number;
|
||||||
|
|
@ -48,7 +42,7 @@ export interface RunLoopParams {
|
||||||
toolSet: KtxRuntimeToolSet;
|
toolSet: KtxRuntimeToolSet;
|
||||||
stepBudget: number;
|
stepBudget: number;
|
||||||
telemetryTags: Record<string, string>;
|
telemetryTags: Record<string, string>;
|
||||||
onStepFinish?: (info: RunLoopStepInfo) => void | Promise<void>;
|
abortSignal?: AbortSignal;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface RunLoopResult {
|
export interface RunLoopResult {
|
||||||
|
|
@ -64,6 +58,7 @@ export interface KtxGenerateTextInput {
|
||||||
tools?: KtxRuntimeToolSet;
|
tools?: KtxRuntimeToolSet;
|
||||||
temperature?: number;
|
temperature?: number;
|
||||||
onMetrics?: (metrics: { totalMs: number; usage: LlmTokenUsage }) => void;
|
onMetrics?: (metrics: { totalMs: number; usage: LlmTokenUsage }) => void;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxGenerateObjectInput<TOutput, TSchema extends z.ZodType<TOutput>> {
|
export interface KtxGenerateObjectInput<TOutput, TSchema extends z.ZodType<TOutput>> {
|
||||||
|
|
@ -74,6 +69,7 @@ export interface KtxGenerateObjectInput<TOutput, TSchema extends z.ZodType<TOutp
|
||||||
temperature?: number;
|
temperature?: number;
|
||||||
schema: TSchema;
|
schema: TSchema;
|
||||||
onMetrics?: (metrics: { totalMs: number; usage: LlmTokenUsage }) => void;
|
onMetrics?: (metrics: { totalMs: number; usage: LlmTokenUsage }) => void;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxLlmRuntimePort {
|
export interface KtxLlmRuntimePort {
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,13 @@ import type { ToolAnnotations } from '@modelcontextprotocol/sdk/types.js';
|
||||||
import { z } from 'zod';
|
import { z } from 'zod';
|
||||||
import type { KtxCliIo } from '../../cli-runtime.js';
|
import type { KtxCliIo } from '../../cli-runtime.js';
|
||||||
import type { MemoryAgentInput } from '../../context/memory/types.js';
|
import type { MemoryAgentInput } from '../../context/memory/types.js';
|
||||||
import { emitTelemetryEvent, mcpTelemetrySampleRate, shouldEmitMcpTelemetry } from '../../telemetry/index.js';
|
import {
|
||||||
|
emitTelemetryEvent,
|
||||||
|
mcpTelemetrySampleRate,
|
||||||
|
reportException,
|
||||||
|
shouldEmitMcpTelemetry,
|
||||||
|
} from '../../telemetry/index.js';
|
||||||
|
import { collectTelemetryRedactionSecrets } from '../../telemetry/redaction-secrets.js';
|
||||||
import { scrubErrorClass } from '../../telemetry/scrubber.js';
|
import { scrubErrorClass } from '../../telemetry/scrubber.js';
|
||||||
import type {
|
import type {
|
||||||
KtxMcpClientInfo,
|
KtxMcpClientInfo,
|
||||||
|
|
@ -518,11 +524,26 @@ function registerParsedTool<TSchema extends z.ZodType>(
|
||||||
},
|
},
|
||||||
schema: TSchema,
|
schema: TSchema,
|
||||||
handler: (input: z.infer<TSchema>, context?: KtxMcpToolHandlerContext) => Promise<KtxMcpToolResult>,
|
handler: (input: z.infer<TSchema>, context?: KtxMcpToolHandlerContext) => Promise<KtxMcpToolResult>,
|
||||||
|
telemetry?: { projectDir?: string; io?: KtxCliIo },
|
||||||
): void {
|
): void {
|
||||||
server.registerTool(name, config, async (input, context) => {
|
server.registerTool(name, config, async (input, context) => {
|
||||||
try {
|
try {
|
||||||
return await handler(schema.parse(input), context);
|
return await handler(schema.parse(input), context);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
if (telemetry?.io) {
|
||||||
|
await reportException({
|
||||||
|
error,
|
||||||
|
context: { source: `mcp:${name}`, handled: true, fatal: false },
|
||||||
|
projectDir: telemetry.projectDir,
|
||||||
|
io: telemetry.io,
|
||||||
|
redactionSecrets: await collectTelemetryRedactionSecrets({
|
||||||
|
projectDir: telemetry.projectDir,
|
||||||
|
includeLlm: true,
|
||||||
|
includeEmbeddings: true,
|
||||||
|
env: process.env,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
}
|
||||||
return jsonErrorToolResult(formatToolError(error));
|
return jsonErrorToolResult(formatToolError(error));
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
@ -571,6 +592,20 @@ function instrumentMcpServer(
|
||||||
}
|
}
|
||||||
return result;
|
return result;
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
if (telemetry.io) {
|
||||||
|
await reportException({
|
||||||
|
error,
|
||||||
|
context: { source: `mcp:${name}`, handled: true, fatal: false },
|
||||||
|
projectDir: telemetry.projectDir,
|
||||||
|
io: telemetry.io,
|
||||||
|
redactionSecrets: await collectTelemetryRedactionSecrets({
|
||||||
|
projectDir: telemetry.projectDir,
|
||||||
|
includeLlm: true,
|
||||||
|
includeEmbeddings: true,
|
||||||
|
env: process.env,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
}
|
||||||
if (telemetry.io && telemetry.projectDir && shouldEmitMcpTelemetry()) {
|
if (telemetry.io && telemetry.projectDir && shouldEmitMcpTelemetry()) {
|
||||||
const errorClass = scrubErrorClass(error);
|
const errorClass = scrubErrorClass(error);
|
||||||
await emitTelemetryEvent({
|
await emitTelemetryEvent({
|
||||||
|
|
@ -596,6 +631,7 @@ function instrumentMcpServer(
|
||||||
|
|
||||||
export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void {
|
export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void {
|
||||||
const { ports, userContext } = deps;
|
const { ports, userContext } = deps;
|
||||||
|
const toolTelemetry = { projectDir: deps.projectDir, io: deps.io };
|
||||||
const server = instrumentMcpServer(deps.server, {
|
const server = instrumentMcpServer(deps.server, {
|
||||||
projectDir: deps.projectDir,
|
projectDir: deps.projectDir,
|
||||||
io: deps.io,
|
io: deps.io,
|
||||||
|
|
@ -616,6 +652,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
},
|
},
|
||||||
connectionListSchema,
|
connectionListSchema,
|
||||||
async () => jsonToolResult({ connections: await connections.list() }),
|
async () => jsonToolResult({ connections: await connections.list() }),
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -640,6 +677,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
limit: input.limit,
|
limit: input.limit,
|
||||||
}),
|
}),
|
||||||
),
|
),
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
|
|
||||||
registerParsedTool(
|
registerParsedTool(
|
||||||
|
|
@ -657,6 +695,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
const page = await knowledge.read({ userId: userContext.userId, key: input.key });
|
const page = await knowledge.read({ userId: userContext.userId, key: input.key });
|
||||||
return page ? jsonToolResult(page) : jsonErrorToolResult(`Wiki page "${input.key}" was not found.`);
|
return page ? jsonToolResult(page) : jsonErrorToolResult(`Wiki page "${input.key}" was not found.`);
|
||||||
},
|
},
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -679,6 +718,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
? jsonToolResult(source)
|
? jsonToolResult(source)
|
||||||
: jsonErrorToolResult(`Semantic-layer source "${input.sourceName}" was not found.`);
|
: jsonErrorToolResult(`Semantic-layer source "${input.sourceName}" was not found.`);
|
||||||
},
|
},
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
|
|
||||||
registerParsedTool(
|
registerParsedTool(
|
||||||
|
|
@ -711,6 +751,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
);
|
);
|
||||||
return jsonToolResult(projectSlQueryResult(result, input.include));
|
return jsonToolResult(projectSlQueryResult(result, input.include));
|
||||||
},
|
},
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -728,6 +769,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
},
|
},
|
||||||
entityDetailsSchema,
|
entityDetailsSchema,
|
||||||
async (input) => jsonToolResult(await entityDetails.read(input)),
|
async (input) => jsonToolResult(await entityDetails.read(input)),
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -745,6 +787,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
},
|
},
|
||||||
dictionarySearchSchema,
|
dictionarySearchSchema,
|
||||||
async (input) => jsonToolResult(await dictionarySearch.search(input)),
|
async (input) => jsonToolResult(await dictionarySearch.search(input)),
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -762,6 +805,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
},
|
},
|
||||||
discoverDataSchema,
|
discoverDataSchema,
|
||||||
async (input) => jsonToolResult({ refs: await discover.search(input) }),
|
async (input) => jsonToolResult({ refs: await discover.search(input) }),
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -791,6 +835,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
},
|
},
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -818,6 +863,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
};
|
};
|
||||||
return jsonToolResult(await memoryIngest.ingest(ingestInput));
|
return jsonToolResult(await memoryIngest.ingest(ingestInput));
|
||||||
},
|
},
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
|
|
||||||
registerParsedTool(
|
registerParsedTool(
|
||||||
|
|
@ -835,6 +881,7 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void
|
||||||
const status = await memoryIngest.status(input.runId);
|
const status = await memoryIngest.status(input.runId);
|
||||||
return status ? jsonToolResult(status) : jsonErrorToolResult(`Memory ingest run "${input.runId}" was not found.`);
|
return status ? jsonToolResult(status) : jsonErrorToolResult(`Memory ingest run "${input.runId}" was not found.`);
|
||||||
},
|
},
|
||||||
|
toolTelemetry,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -100,6 +100,44 @@ const workUnitsSchema = z
|
||||||
})
|
})
|
||||||
.describe('Concurrency and failure handling for ingest work units.');
|
.describe('Concurrency and failure handling for ingest work units.');
|
||||||
|
|
||||||
|
const ingestRateLimitRetrySchema = z
|
||||||
|
.strictObject({
|
||||||
|
maxAttempts: z
|
||||||
|
.int()
|
||||||
|
.positive()
|
||||||
|
.default(6)
|
||||||
|
.describe(
|
||||||
|
'Maximum attempts for a single rate-limited LLM call before the failure surfaces, counting the first try. Also bounds how far opaque backoff grows for providers that do not expose a reset time.',
|
||||||
|
),
|
||||||
|
baseDelayMs: z.int().positive().default(1_000).describe('Initial opaque retry delay in milliseconds.'),
|
||||||
|
maxDelayMs: z.int().positive().default(60_000).describe('Maximum opaque retry delay in milliseconds.'),
|
||||||
|
jitter: z.boolean().default(true).describe('When true, apply bounded jitter to opaque retry delays.'),
|
||||||
|
})
|
||||||
|
.describe('Retry policy for rate-limit responses that do not include a reset time or retry-after value.');
|
||||||
|
|
||||||
|
const ingestRateLimitSchema = z
|
||||||
|
.strictObject({
|
||||||
|
enabled: z.boolean().default(true).describe('Master switch for ingest LLM rate-limit pacing and visible waits.'),
|
||||||
|
throttleThreshold: z
|
||||||
|
.number()
|
||||||
|
.min(0)
|
||||||
|
.max(1)
|
||||||
|
.default(0.8)
|
||||||
|
.describe('Provider utilization at or above which ingest throttles new work-unit starts.'),
|
||||||
|
minConcurrencyUnderPressure: z
|
||||||
|
.int()
|
||||||
|
.positive()
|
||||||
|
.default(1)
|
||||||
|
.describe('Effective work-unit concurrency while a provider is under rate-limit pressure.'),
|
||||||
|
maxWaitMs: z
|
||||||
|
.int()
|
||||||
|
.positive()
|
||||||
|
.optional()
|
||||||
|
.describe('Optional cap on a single provider reset wait. Omit to wait indefinitely until the provider reset time.'),
|
||||||
|
retry: ingestRateLimitRetrySchema.prefault({}).describe('Opaque retry policy for providers without reset hints.'),
|
||||||
|
})
|
||||||
|
.describe('Rate-limit pacing and wait policy for ingest LLM calls.');
|
||||||
|
|
||||||
const ingestSchema = z
|
const ingestSchema = z
|
||||||
.strictObject({
|
.strictObject({
|
||||||
adapters: z
|
adapters: z
|
||||||
|
|
@ -110,6 +148,7 @@ const ingestSchema = z
|
||||||
.prefault({ backend: 'none' })
|
.prefault({ backend: 'none' })
|
||||||
.describe('Embedding configuration used when ingest adapters need to embed documents.'),
|
.describe('Embedding configuration used when ingest adapters need to embed documents.'),
|
||||||
workUnits: workUnitsSchema.prefault({}).describe('Concurrency and failure handling for ingest work units.'),
|
workUnits: workUnitsSchema.prefault({}).describe('Concurrency and failure handling for ingest work units.'),
|
||||||
|
rateLimit: ingestRateLimitSchema.prefault({}).describe('LLM rate-limit pacing and visible-wait policy for ingest.'),
|
||||||
profile: z
|
profile: z
|
||||||
.union([z.boolean(), z.literal('json')])
|
.union([z.boolean(), z.literal('json')])
|
||||||
.default(false)
|
.default(false)
|
||||||
|
|
|
||||||
|
|
@ -303,9 +303,29 @@ export interface KtxTableListEntry {
|
||||||
kind: 'table' | 'view';
|
kind: 'table' | 'view';
|
||||||
}
|
}
|
||||||
|
|
||||||
interface KtxConnectorTestResult {
|
export interface KtxConnectorTestResult {
|
||||||
success: boolean;
|
success: boolean;
|
||||||
error?: string;
|
error?: string;
|
||||||
|
/**
|
||||||
|
* The original error thrown by the driver, preserved unflattened so the
|
||||||
|
* connection-test path can re-throw it. Keeping the real error object lets
|
||||||
|
* telemetry record the driver's actual error class (e.g. `ConnectionError`)
|
||||||
|
* and `.code` (e.g. `ELOGIN`) instead of collapsing every failure to `Error`.
|
||||||
|
*/
|
||||||
|
cause?: unknown;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Single source of truth for a failed connector test result. Captures the
|
||||||
|
* driver's message for display while preserving the original error as `cause`
|
||||||
|
* so callers can surface its real class and code.
|
||||||
|
*/
|
||||||
|
export function connectorTestFailure(error: unknown): KtxConnectorTestResult {
|
||||||
|
return {
|
||||||
|
success: false,
|
||||||
|
error: error instanceof Error ? error.message : String(error),
|
||||||
|
cause: error,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxScanConnector {
|
export interface KtxScanConnector {
|
||||||
|
|
|
||||||
|
|
@ -50,13 +50,6 @@ function humanizeIdentifier(value: string): string {
|
||||||
.toLowerCase();
|
.toLowerCase();
|
||||||
}
|
}
|
||||||
|
|
||||||
function formatCount(count: number, singular: string, plural = `${singular}s`): string | null {
|
|
||||||
if (count <= 0) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
return `${count} ${count === 1 ? singular : plural}`;
|
|
||||||
}
|
|
||||||
|
|
||||||
function sourceFallback(source: Record<string, unknown>, sourceName: string): string {
|
function sourceFallback(source: Record<string, unknown>, sourceName: string): string {
|
||||||
const table = cleanText(source.table);
|
const table = cleanText(source.table);
|
||||||
const sql = cleanText(source.sql);
|
const sql = cleanText(source.sql);
|
||||||
|
|
@ -66,15 +59,10 @@ function sourceFallback(source: Record<string, unknown>, sourceName: string): st
|
||||||
if (sql) {
|
if (sql) {
|
||||||
return `Semantic-layer source for ${sourceName} backed by curated SQL.`;
|
return `Semantic-layer source for ${sourceName} backed by curated SQL.`;
|
||||||
}
|
}
|
||||||
|
// Measure/segment/column counts are rendered live from the body at list/read
|
||||||
const counts = [
|
// time, so baking them into stored prose freezes a derived value that drifts
|
||||||
formatCount(Array.isArray(source.measures) ? source.measures.length : 0, 'measure'),
|
// as the source later gains measures. Keep the auto fallback count-free.
|
||||||
formatCount(Array.isArray(source.segments) ? source.segments.length : 0, 'segment'),
|
return `Semantic-layer overlay for ${sourceName}.`;
|
||||||
formatCount(Array.isArray(source.columns) ? source.columns.length : 0, 'computed column'),
|
|
||||||
].filter((item): item is string => Boolean(item));
|
|
||||||
return counts.length > 0
|
|
||||||
? `Semantic-layer overlay for ${sourceName} defining ${counts.join(', ')}.`
|
|
||||||
: `Semantic-layer overlay for ${sourceName}.`;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
function columnFallback(column: Record<string, unknown>, sourceName: string): string {
|
function columnFallback(column: Record<string, unknown>, sourceName: string): string {
|
||||||
|
|
|
||||||
|
|
@ -15,8 +15,6 @@ interface DemoMetricsTuning {
|
||||||
interface DemoMetricsSnapshot {
|
interface DemoMetricsSnapshot {
|
||||||
elapsedMs: number;
|
elapsedMs: number;
|
||||||
etaMs: number | null;
|
etaMs: number | null;
|
||||||
agentSteps: number;
|
|
||||||
agentStepBudget: number;
|
|
||||||
toolCalls: number;
|
toolCalls: number;
|
||||||
workUnitsStarted: number;
|
workUnitsStarted: number;
|
||||||
workUnitsFinished: number;
|
workUnitsFinished: number;
|
||||||
|
|
@ -37,18 +35,6 @@ function eventsOf<T extends MemoryFlowEvent['type']>(
|
||||||
return events.filter((event): event is Extract<MemoryFlowEvent, { type: T }> => event.type === type);
|
return events.filter((event): event is Extract<MemoryFlowEvent, { type: T }> => event.type === type);
|
||||||
}
|
}
|
||||||
|
|
||||||
function maxAgentStep(events: MemoryFlowEvent[]): { step: number; budget: number } {
|
|
||||||
const steps = eventsOf(events, 'work_unit_step');
|
|
||||||
const started = eventsOf(events, 'work_unit_started');
|
|
||||||
const stepIndex = steps.reduce((max, event) => Math.max(max, event.stepIndex), 0);
|
|
||||||
const stepBudget = Math.max(
|
|
||||||
0,
|
|
||||||
...steps.map((event) => event.stepBudget),
|
|
||||||
...started.map((event) => event.stepBudget),
|
|
||||||
);
|
|
||||||
return { step: stepIndex, budget: stepBudget };
|
|
||||||
}
|
|
||||||
|
|
||||||
function totalToolCalls(input: MemoryFlowReplayInput): number {
|
function totalToolCalls(input: MemoryFlowReplayInput): number {
|
||||||
return input.details.transcripts.reduce((total, transcript) => total + transcript.toolCallCount, 0);
|
return input.details.transcripts.reduce((total, transcript) => total + transcript.toolCallCount, 0);
|
||||||
}
|
}
|
||||||
|
|
@ -96,11 +82,10 @@ export function buildDemoMetrics(
|
||||||
const nowMs = (options.now ?? Date.now)();
|
const nowMs = (options.now ?? Date.now)();
|
||||||
const elapsedMs = elapsedMsFromEvents(input.events, nowMs);
|
const elapsedMs = elapsedMsFromEvents(input.events, nowMs);
|
||||||
|
|
||||||
const { step, budget } = maxAgentStep(input.events);
|
|
||||||
const toolCalls = totalToolCalls(input);
|
const toolCalls = totalToolCalls(input);
|
||||||
const progress = workUnitProgress(input);
|
const progress = workUnitProgress(input);
|
||||||
const finishedCount = eventsOf(input.events, 'work_unit_finished').length;
|
const finishedCount = eventsOf(input.events, 'work_unit_finished').length;
|
||||||
const stepDriver = Math.max(step, toolCalls, finishedCount * 4);
|
const stepDriver = Math.max(toolCalls, finishedCount * 4);
|
||||||
|
|
||||||
const inputTokens = stepDriver * inputTokensPerStep;
|
const inputTokens = stepDriver * inputTokensPerStep;
|
||||||
const outputTokens = stepDriver * outputTokensPerStep;
|
const outputTokens = stepDriver * outputTokensPerStep;
|
||||||
|
|
@ -113,8 +98,6 @@ export function buildDemoMetrics(
|
||||||
return {
|
return {
|
||||||
elapsedMs,
|
elapsedMs,
|
||||||
etaMs: estimateEtaMs(elapsedMs, progress.finished, progress.total, input.status),
|
etaMs: estimateEtaMs(elapsedMs, progress.finished, progress.total, input.status),
|
||||||
agentSteps: step,
|
|
||||||
agentStepBudget: budget,
|
|
||||||
toolCalls,
|
toolCalls,
|
||||||
workUnitsStarted: progress.started,
|
workUnitsStarted: progress.started,
|
||||||
workUnitsFinished: progress.finished,
|
workUnitsFinished: progress.finished,
|
||||||
|
|
|
||||||
|
|
@ -78,6 +78,7 @@ export interface KtxIngestDeps {
|
||||||
readReportFile?: typeof readIngestReportSnapshotFile;
|
readReportFile?: typeof readIngestReportSnapshotFile;
|
||||||
renderStoredMemoryFlow?: typeof renderMemoryFlowTui;
|
renderStoredMemoryFlow?: typeof renderMemoryFlowTui;
|
||||||
startLiveMemoryFlow?: typeof startLiveMemoryFlowTui;
|
startLiveMemoryFlow?: typeof startLiveMemoryFlowTui;
|
||||||
|
abortSignal?: AbortSignal;
|
||||||
env?: NodeJS.ProcessEnv;
|
env?: NodeJS.ProcessEnv;
|
||||||
localIngestOptions?: Pick<
|
localIngestOptions?: Pick<
|
||||||
RunLocalIngestOptions,
|
RunLocalIngestOptions,
|
||||||
|
|
@ -93,6 +94,23 @@ export interface KtxIngestDeps {
|
||||||
runtimeIo?: KtxIngestIo;
|
runtimeIo?: KtxIngestIo;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function createCliAbortSignal(): { signal: AbortSignal; dispose: () => void } {
|
||||||
|
const controller = new AbortController();
|
||||||
|
let interrupted = false;
|
||||||
|
const onSigint = () => {
|
||||||
|
if (interrupted) {
|
||||||
|
process.exit(130);
|
||||||
|
}
|
||||||
|
interrupted = true;
|
||||||
|
controller.abort(new DOMException('Aborted', 'AbortError'));
|
||||||
|
};
|
||||||
|
process.on('SIGINT', onSigint);
|
||||||
|
return {
|
||||||
|
signal: controller.signal,
|
||||||
|
dispose: () => process.off('SIGINT', onSigint),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
const REPORT_SOURCE_LABELS = new Map<string, string>([
|
const REPORT_SOURCE_LABELS = new Map<string, string>([
|
||||||
['live-database', 'Database schema'],
|
['live-database', 'Database schema'],
|
||||||
['historic-sql', 'Query history'],
|
['historic-sql', 'Query history'],
|
||||||
|
|
@ -364,6 +382,12 @@ function plainIngestEventProgress(
|
||||||
message: event.message,
|
message: event.message,
|
||||||
...(event.transient !== undefined ? { transient: event.transient } : {}),
|
...(event.transient !== undefined ? { transient: event.transient } : {}),
|
||||||
};
|
};
|
||||||
|
case 'rate_limit_wait':
|
||||||
|
return {
|
||||||
|
percent: 50,
|
||||||
|
message: `Rate-limited (${event.provider}${event.rateLimitType ? ` ${event.rateLimitType}` : ''}); resuming in ${Math.ceil(event.remainingMs / 1_000)}s`,
|
||||||
|
transient: true,
|
||||||
|
};
|
||||||
case 'work_unit_started': {
|
case 'work_unit_started': {
|
||||||
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
||||||
const ordinal = workUnitOrdinalThrough(snapshot, eventIndex, event.unitKey);
|
const ordinal = workUnitOrdinalThrough(snapshot, eventIndex, event.unitKey);
|
||||||
|
|
@ -374,9 +398,8 @@ function plainIngestEventProgress(
|
||||||
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
||||||
const completed = completedWorkUnitCountThrough(snapshot, eventIndex);
|
const completed = completedWorkUnitCountThrough(snapshot, eventIndex);
|
||||||
const active = activeWorkUnitCountThrough(snapshot, eventIndex);
|
const active = activeWorkUnitCountThrough(snapshot, eventIndex);
|
||||||
const stepFraction = event.stepBudget > 0 ? Math.min(1, event.stepIndex / event.stepBudget) : 0;
|
const percent = total > 0 ? 55 + Math.ceil((completed / total) * 25) : 55;
|
||||||
const percent = total > 0 ? 55 + Math.ceil(((completed + stepFraction) / total) * 25) : 55;
|
const latest = `${event.unitKey} · ${pluralize(event.toolCalls, 'action')}`;
|
||||||
const latest = `${event.unitKey} step ${event.stepIndex}/${event.stepBudget}`;
|
|
||||||
return {
|
return {
|
||||||
percent,
|
percent,
|
||||||
message: `Processing tasks: ${completed}/${total} complete, ${active} active; latest ${latest}`,
|
message: `Processing tasks: ${completed}/${total} complete, ${active} active; latest ${latest}`,
|
||||||
|
|
@ -750,6 +773,8 @@ export async function runKtxIngest(
|
||||||
);
|
);
|
||||||
plainProgress?.start();
|
plainProgress?.start();
|
||||||
structuredProgress?.start();
|
structuredProgress?.start();
|
||||||
|
const cliAbort = deps.abortSignal ? null : createCliAbortSignal();
|
||||||
|
const abortSignal = deps.abortSignal ?? cliAbort?.signal;
|
||||||
let result: LocalMetabaseFanoutResult;
|
let result: LocalMetabaseFanoutResult;
|
||||||
try {
|
try {
|
||||||
result = await executeMetabaseFanout({
|
result = await executeMetabaseFanout({
|
||||||
|
|
@ -763,6 +788,7 @@ export async function runKtxIngest(
|
||||||
embeddingProvider,
|
embeddingProvider,
|
||||||
...(memoryFlow ? { memoryFlow } : {}),
|
...(memoryFlow ? { memoryFlow } : {}),
|
||||||
...(progress ? { progress } : {}),
|
...(progress ? { progress } : {}),
|
||||||
|
...(abortSignal ? { abortSignal } : {}),
|
||||||
});
|
});
|
||||||
plainProgress?.flush();
|
plainProgress?.flush();
|
||||||
if (args.outputMode === 'json') {
|
if (args.outputMode === 'json') {
|
||||||
|
|
@ -772,6 +798,7 @@ export async function runKtxIngest(
|
||||||
}
|
}
|
||||||
} finally {
|
} finally {
|
||||||
plainProgress?.flush();
|
plainProgress?.flush();
|
||||||
|
cliAbort?.dispose();
|
||||||
}
|
}
|
||||||
return result.status === 'all_failed' ? 1 : 0;
|
return result.status === 'all_failed' ? 1 : 0;
|
||||||
}
|
}
|
||||||
|
|
@ -820,6 +847,8 @@ export async function runKtxIngest(
|
||||||
|
|
||||||
plainProgress?.start();
|
plainProgress?.start();
|
||||||
structuredProgress?.start();
|
structuredProgress?.start();
|
||||||
|
const cliAbort = deps.abortSignal ? null : createCliAbortSignal();
|
||||||
|
const abortSignal = deps.abortSignal ?? cliAbort?.signal;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const result = await executeLocalIngest({
|
const result = await executeLocalIngest({
|
||||||
|
|
@ -836,6 +865,7 @@ export async function runKtxIngest(
|
||||||
embeddingProvider,
|
embeddingProvider,
|
||||||
...(args.debugLlmRequestFile ? { llmDebugRequestFile: args.debugLlmRequestFile } : {}),
|
...(args.debugLlmRequestFile ? { llmDebugRequestFile: args.debugLlmRequestFile } : {}),
|
||||||
...(memoryFlow ? { memoryFlow } : {}),
|
...(memoryFlow ? { memoryFlow } : {}),
|
||||||
|
...(abortSignal ? { abortSignal } : {}),
|
||||||
});
|
});
|
||||||
if (shouldUseLiveViz && memoryFlow) {
|
if (shouldUseLiveViz && memoryFlow) {
|
||||||
latestMemoryFlowSnapshot = finalRunMemoryFlowInput(memoryFlow.snapshot(), result.report);
|
latestMemoryFlowSnapshot = finalRunMemoryFlowInput(memoryFlow.snapshot(), result.report);
|
||||||
|
|
@ -854,6 +884,7 @@ export async function runKtxIngest(
|
||||||
} finally {
|
} finally {
|
||||||
plainProgress?.flush();
|
plainProgress?.flush();
|
||||||
liveTui?.close();
|
liveTui?.close();
|
||||||
|
cliAbort?.dispose();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,8 @@ const unicode = detectUnicodeSupport();
|
||||||
export const SYMBOLS = {
|
export const SYMBOLS = {
|
||||||
middot: unicode ? '·' : '-',
|
middot: unicode ? '·' : '-',
|
||||||
emDash: unicode ? '—' : '--',
|
emDash: unicode ? '—' : '--',
|
||||||
|
star: unicode ? '★' : '*',
|
||||||
|
rightArrow: unicode ? '→' : '->',
|
||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
export function dim(text: string): string {
|
export function dim(text: string): string {
|
||||||
|
|
|
||||||
17
packages/cli/src/io/tty.ts
Normal file
17
packages/cli/src/io/tty.ts
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
import type { Writable } from 'node:stream';
|
||||||
|
|
||||||
|
import type { KtxCliIo } from '../cli-runtime.js';
|
||||||
|
|
||||||
|
type KtxCliOutput = (KtxCliIo['stdout'] | KtxCliIo['stderr']) & {
|
||||||
|
isTTY?: boolean;
|
||||||
|
columns?: number;
|
||||||
|
on?: unknown;
|
||||||
|
};
|
||||||
|
|
||||||
|
export function isWritableTtyOutput(output: KtxCliOutput): output is KtxCliOutput & Writable {
|
||||||
|
return (
|
||||||
|
(output as { isTTY?: unknown }).isTTY === true &&
|
||||||
|
typeof (output as { on?: unknown }).on === 'function' &&
|
||||||
|
typeof (output as { columns?: unknown }).columns !== 'undefined'
|
||||||
|
);
|
||||||
|
}
|
||||||
1
packages/cli/src/links.ts
Normal file
1
packages/cli/src/links.ts
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
export const SLACK_URL = 'https://ktx.sh/slack';
|
||||||
|
|
@ -139,31 +139,21 @@ function sourceDescription(input: MemoryFlowReplayInput): SourceInfo {
|
||||||
return { type: info.type, name: conn, sourceCount: count, itemNounPlural: info.plural, readingVerb: info.verb, ingestDescription: info.description };
|
return { type: info.type, name: conn, sourceCount: count, itemNounPlural: info.plural, readingVerb: info.verb, ingestDescription: info.description };
|
||||||
}
|
}
|
||||||
|
|
||||||
function activeWorkUnits(
|
function activeWorkUnits(input: MemoryFlowReplayInput): string[] {
|
||||||
input: MemoryFlowReplayInput,
|
|
||||||
): Array<{ unitKey: string; stepIndex: number; stepBudget: number }> {
|
|
||||||
const finishedKeys = new Set<string>();
|
const finishedKeys = new Set<string>();
|
||||||
const unitMap = new Map<string, { stepIndex: number; stepBudget: number }>();
|
|
||||||
|
|
||||||
for (const e of input.events) {
|
for (const e of input.events) {
|
||||||
if (e.type === 'work_unit_started') {
|
|
||||||
unitMap.set(e.unitKey, { stepIndex: 0, stepBudget: e.stepBudget });
|
|
||||||
}
|
|
||||||
if (e.type === 'work_unit_step') {
|
|
||||||
const existing = unitMap.get(e.unitKey);
|
|
||||||
if (existing) {
|
|
||||||
existing.stepIndex = e.stepIndex;
|
|
||||||
existing.stepBudget = e.stepBudget;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (e.type === 'work_unit_finished') finishedKeys.add(e.unitKey);
|
if (e.type === 'work_unit_finished') finishedKeys.add(e.unitKey);
|
||||||
}
|
}
|
||||||
|
|
||||||
const result: Array<{ unitKey: string; stepIndex: number; stepBudget: number }> = [];
|
const active: string[] = [];
|
||||||
for (const [unitKey, data] of unitMap) {
|
const seen = new Set<string>();
|
||||||
if (!finishedKeys.has(unitKey)) result.push({ unitKey, ...data });
|
for (const e of input.events) {
|
||||||
|
if (e.type === 'work_unit_started' && !finishedKeys.has(e.unitKey) && !seen.has(e.unitKey)) {
|
||||||
|
seen.add(e.unitKey);
|
||||||
|
active.push(e.unitKey);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return result;
|
return active;
|
||||||
}
|
}
|
||||||
|
|
||||||
function queuedWorkUnits(input: MemoryFlowReplayInput): string[] {
|
function queuedWorkUnits(input: MemoryFlowReplayInput): string[] {
|
||||||
|
|
|
||||||
|
|
@ -23,7 +23,8 @@ import type { KtxScanArgs, KtxScanDeps } from './scan.js';
|
||||||
import type { KtxTableRef } from './context/scan/types.js';
|
import type { KtxTableRef } from './context/scan/types.js';
|
||||||
import { profileMark } from './startup-profile.js';
|
import { profileMark } from './startup-profile.js';
|
||||||
import { isDemoConnection } from './telemetry/demo-detect.js';
|
import { isDemoConnection } from './telemetry/demo-detect.js';
|
||||||
import { emitProjectStackSnapshot, emitTelemetryEvent } from './telemetry/index.js';
|
import { emitProjectStackSnapshot, emitTelemetryEvent, reportException } from './telemetry/index.js';
|
||||||
|
import { collectTelemetryRedactionSecrets } from './telemetry/redaction-secrets.js';
|
||||||
import { formatErrorDetail } from './telemetry/scrubber.js';
|
import { formatErrorDetail } from './telemetry/scrubber.js';
|
||||||
|
|
||||||
profileMark('module:public-ingest');
|
profileMark('module:public-ingest');
|
||||||
|
|
@ -1119,30 +1120,63 @@ export async function runKtxPublicIngest(
|
||||||
feature,
|
feature,
|
||||||
});
|
});
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
await reportException({
|
||||||
|
error,
|
||||||
|
context: { source: 'ingest runtime', handled: true, fatal: false },
|
||||||
|
projectDir: args.projectDir,
|
||||||
|
io,
|
||||||
|
redactionSecrets: await collectTelemetryRedactionSecrets({
|
||||||
|
project,
|
||||||
|
projectDir: args.projectDir,
|
||||||
|
connectionId: args.targetConnectionId,
|
||||||
|
includeLlm: true,
|
||||||
|
includeEmbeddings: true,
|
||||||
|
env: deps.env ?? process.env,
|
||||||
|
}),
|
||||||
|
});
|
||||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
const { runContextBuild } = await import('./context-build-view.js');
|
const { runContextBuild } = await import('./context-build-view.js');
|
||||||
const contextBuild = deps.runContextBuild ?? runContextBuild;
|
const contextBuild = deps.runContextBuild ?? runContextBuild;
|
||||||
const result = await contextBuild(
|
try {
|
||||||
project,
|
const result = await contextBuild(
|
||||||
{
|
project,
|
||||||
|
{
|
||||||
|
projectDir: args.projectDir,
|
||||||
|
...(args.targetConnectionId ? { targetConnectionId: args.targetConnectionId } : {}),
|
||||||
|
all: args.all,
|
||||||
|
entrypoint: 'ingest',
|
||||||
|
inputMode: args.inputMode,
|
||||||
|
...(args.queryHistory ? { queryHistory: args.queryHistory } : {}),
|
||||||
|
...(args.queryHistoryWindowDays !== undefined ? { queryHistoryWindowDays: args.queryHistoryWindowDays } : {}),
|
||||||
|
...(args.scanMode ? { scanMode: args.scanMode } : {}),
|
||||||
|
...(args.detectRelationships !== undefined ? { detectRelationships: args.detectRelationships } : {}),
|
||||||
|
...(args.cliVersion ? { cliVersion: args.cliVersion } : {}),
|
||||||
|
...(args.runtimeInstallPolicy ? { runtimeInstallPolicy: args.runtimeInstallPolicy } : {}),
|
||||||
|
},
|
||||||
|
io,
|
||||||
|
);
|
||||||
|
return result.exitCode;
|
||||||
|
} catch (error) {
|
||||||
|
await reportException({
|
||||||
|
error,
|
||||||
|
context: { source: 'ingest context-build', handled: true, fatal: false },
|
||||||
projectDir: args.projectDir,
|
projectDir: args.projectDir,
|
||||||
...(args.targetConnectionId ? { targetConnectionId: args.targetConnectionId } : {}),
|
io,
|
||||||
all: args.all,
|
redactionSecrets: await collectTelemetryRedactionSecrets({
|
||||||
entrypoint: 'ingest',
|
project,
|
||||||
inputMode: args.inputMode,
|
projectDir: args.projectDir,
|
||||||
...(args.queryHistory ? { queryHistory: args.queryHistory } : {}),
|
connectionId: args.targetConnectionId,
|
||||||
...(args.queryHistoryWindowDays !== undefined ? { queryHistoryWindowDays: args.queryHistoryWindowDays } : {}),
|
includeLlm: true,
|
||||||
...(args.scanMode ? { scanMode: args.scanMode } : {}),
|
includeEmbeddings: true,
|
||||||
...(args.detectRelationships !== undefined ? { detectRelationships: args.detectRelationships } : {}),
|
env: deps.env ?? process.env,
|
||||||
...(args.cliVersion ? { cliVersion: args.cliVersion } : {}),
|
}),
|
||||||
...(args.runtimeInstallPolicy ? { runtimeInstallPolicy: args.runtimeInstallPolicy } : {}),
|
});
|
||||||
},
|
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||||
io,
|
return 1;
|
||||||
);
|
}
|
||||||
return result.exitCode;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const plan = buildPublicIngestPlan(project, args);
|
const plan = buildPublicIngestPlan(project, args);
|
||||||
|
|
|
||||||
93
packages/cli/src/reveal-password-prompt.ts
Normal file
93
packages/cli/src/reveal-password-prompt.ts
Normal file
|
|
@ -0,0 +1,93 @@
|
||||||
|
import { styleText } from 'node:util';
|
||||||
|
import { PasswordPrompt, type PasswordOptions } from '@clack/core';
|
||||||
|
import { S_BAR, S_BAR_END, S_PASSWORD_MASK, settings, symbol } from '@clack/prompts';
|
||||||
|
|
||||||
|
// How many trailing characters of a pasted secret to leave visible so the user
|
||||||
|
// can confirm what landed (e.g. `••••••a1b2`). Kept small on purpose.
|
||||||
|
const REVEAL_TAIL_COUNT = 4;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Mask every character of `userInput` except the last `tail`, but only reveal the
|
||||||
|
* tail once the secret is long enough that the hidden portion still dominates
|
||||||
|
* (`length > tail * 2`). Short secrets stay fully masked so we never expose most
|
||||||
|
* of a small value. The returned string keeps the same code-unit length as the
|
||||||
|
* input so clack's cursor slicing in `userInputWithCursor` stays aligned.
|
||||||
|
*
|
||||||
|
* @internal
|
||||||
|
*/
|
||||||
|
export function maskRevealingTail(userInput: string, maskChar: string, tail: number): string {
|
||||||
|
const revealLength = userInput.length > tail * 2 ? tail : 0;
|
||||||
|
const hiddenLength = userInput.length - revealLength;
|
||||||
|
return maskChar.repeat(hiddenLength) + userInput.slice(hiddenLength);
|
||||||
|
}
|
||||||
|
|
||||||
|
class RevealTailPasswordPrompt extends PasswordPrompt {
|
||||||
|
readonly #maskChar: string;
|
||||||
|
readonly #tail: number;
|
||||||
|
|
||||||
|
constructor(options: PasswordOptions & { tail: number }) {
|
||||||
|
super(options);
|
||||||
|
this.#maskChar = options.mask ?? S_PASSWORD_MASK;
|
||||||
|
this.#tail = options.tail;
|
||||||
|
}
|
||||||
|
|
||||||
|
override get masked(): string {
|
||||||
|
return maskRevealingTail(this.userInput, this.#maskChar, this.#tail);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reproduces the @clack/prompts password frame (pinned to the installed version)
|
||||||
|
// so this prompt is visually identical to every other setup prompt; the only
|
||||||
|
// behavioral change is the tail-revealing `masked` getter above.
|
||||||
|
function renderPasswordFrame(prompt: Omit<PasswordPrompt, 'prompt'>, message: string): string {
|
||||||
|
const withGuide = settings.withGuide;
|
||||||
|
const title = `${withGuide ? `${styleText('gray', S_BAR)}\n` : ''}${symbol(prompt.state)} ${message}\n`;
|
||||||
|
const masked = prompt.masked;
|
||||||
|
switch (prompt.state) {
|
||||||
|
case 'error': {
|
||||||
|
const bar = withGuide ? `${styleText('yellow', S_BAR)} ` : '';
|
||||||
|
const end = withGuide ? `${styleText('yellow', S_BAR_END)} ` : '';
|
||||||
|
return `${title.trim()}\n${bar}${masked}\n${end}${styleText('yellow', prompt.error)}\n`;
|
||||||
|
}
|
||||||
|
case 'submit': {
|
||||||
|
const bar = withGuide ? `${styleText('gray', S_BAR)} ` : '';
|
||||||
|
return `${title}${bar}${masked ? styleText('dim', masked) : ''}`;
|
||||||
|
}
|
||||||
|
case 'cancel': {
|
||||||
|
const bar = withGuide ? `${styleText('gray', S_BAR)} ` : '';
|
||||||
|
const body = masked ? styleText(['strikethrough', 'dim'], masked) : '';
|
||||||
|
return `${title}${bar}${body}${masked && withGuide ? `\n${styleText('gray', S_BAR)}` : ''}`;
|
||||||
|
}
|
||||||
|
default: {
|
||||||
|
const bar = withGuide ? `${styleText('cyan', S_BAR)} ` : '';
|
||||||
|
const end = withGuide ? styleText('cyan', S_BAR_END) : '';
|
||||||
|
return `${title}${bar}${prompt.userInputWithCursor}\n${end}\n`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RevealPasswordOptions {
|
||||||
|
message: string;
|
||||||
|
mask?: string;
|
||||||
|
tail?: number;
|
||||||
|
validate?: PasswordOptions['validate'];
|
||||||
|
signal?: AbortSignal;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Drop-in replacement for clack's `password()` that reveals the last few
|
||||||
|
* characters of the entered value while typing. Resolves to the raw value or the
|
||||||
|
* clack cancel symbol, matching `password()`'s contract.
|
||||||
|
*/
|
||||||
|
export function revealPassword(options: RevealPasswordOptions): Promise<string | symbol> {
|
||||||
|
const prompt = new RevealTailPasswordPrompt({
|
||||||
|
mask: options.mask ?? S_PASSWORD_MASK,
|
||||||
|
tail: options.tail ?? REVEAL_TAIL_COUNT,
|
||||||
|
validate: options.validate,
|
||||||
|
signal: options.signal,
|
||||||
|
render() {
|
||||||
|
return renderPasswordFrame(this, options.message);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return prompt.prompt() as Promise<string | symbol>;
|
||||||
|
}
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
import type { KtxProgressPort, KtxScanMode, KtxScanReport, KtxScanWarning } from './context/scan/types.js';
|
import type { KtxProgressPort, KtxScanMode, KtxScanReport, KtxScanWarning } from './context/scan/types.js';
|
||||||
import { runLocalScan } from './context/scan/local-scan.js';
|
import { runLocalScan } from './context/scan/local-scan.js';
|
||||||
import { loadKtxProject } from './context/project/project.js';
|
import { loadKtxProject, type KtxLocalProject } from './context/project/project.js';
|
||||||
import { getKtxCliPackageInfo } from './cli-runtime.js';
|
import { getKtxCliPackageInfo } from './cli-runtime.js';
|
||||||
import { resolveProjectEmbeddingProvider } from './embedding-resolution.js';
|
import { resolveProjectEmbeddingProvider } from './embedding-resolution.js';
|
||||||
import type { KtxCliIo } from './index.js';
|
import type { KtxCliIo } from './index.js';
|
||||||
|
|
@ -8,7 +8,8 @@ import { createKtxCliLocalIngestAdapters } from './local-adapters.js';
|
||||||
import { createKtxCliScanConnector } from './local-scan-connectors.js';
|
import { createKtxCliScanConnector } from './local-scan-connectors.js';
|
||||||
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
|
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
|
||||||
import { profileMark } from './startup-profile.js';
|
import { profileMark } from './startup-profile.js';
|
||||||
import { emitTelemetryEvent } from './telemetry/index.js';
|
import { emitTelemetryEvent, reportException } from './telemetry/index.js';
|
||||||
|
import { collectTelemetryRedactionSecrets } from './telemetry/redaction-secrets.js';
|
||||||
import { formatErrorDetail, scrubErrorClass } from './telemetry/scrubber.js';
|
import { formatErrorDetail, scrubErrorClass } from './telemetry/scrubber.js';
|
||||||
|
|
||||||
profileMark('module:scan');
|
profileMark('module:scan');
|
||||||
|
|
@ -322,8 +323,9 @@ export function createCliScanProgress(
|
||||||
|
|
||||||
export async function runKtxScan(args: KtxScanArgs, io: KtxCliIo = process, deps: KtxScanDeps = {}): Promise<number> {
|
export async function runKtxScan(args: KtxScanArgs, io: KtxCliIo = process, deps: KtxScanDeps = {}): Promise<number> {
|
||||||
const startedAt = performance.now();
|
const startedAt = performance.now();
|
||||||
|
let project: KtxLocalProject | undefined;
|
||||||
try {
|
try {
|
||||||
const project = await loadKtxProject({ projectDir: args.projectDir });
|
project = await loadKtxProject({ projectDir: args.projectDir });
|
||||||
const resolveEmbeddingProvider = deps.resolveEmbeddingProvider ?? resolveProjectEmbeddingProvider;
|
const resolveEmbeddingProvider = deps.resolveEmbeddingProvider ?? resolveProjectEmbeddingProvider;
|
||||||
const resolution = await resolveEmbeddingProvider(project, {
|
const resolution = await resolveEmbeddingProvider(project, {
|
||||||
mode: 'ensure',
|
mode: 'ensure',
|
||||||
|
|
@ -397,6 +399,20 @@ export async function runKtxScan(args: KtxScanArgs, io: KtxCliIo = process, deps
|
||||||
...(errorDetail ? { errorDetail } : {}),
|
...(errorDetail ? { errorDetail } : {}),
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
await reportException({
|
||||||
|
error,
|
||||||
|
context: { source: 'scan run', handled: true, fatal: false },
|
||||||
|
projectDir: args.projectDir,
|
||||||
|
io,
|
||||||
|
redactionSecrets: await collectTelemetryRedactionSecrets({
|
||||||
|
project,
|
||||||
|
projectDir: args.projectDir,
|
||||||
|
connectionId: args.connectionId,
|
||||||
|
includeLlm: true,
|
||||||
|
includeEmbeddings: true,
|
||||||
|
env: process.env,
|
||||||
|
}),
|
||||||
|
});
|
||||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
import { existsSync } from 'node:fs';
|
import { existsSync } from 'node:fs';
|
||||||
import { mkdir, readFile, rm, writeFile } from 'node:fs/promises';
|
import { mkdir, readFile, rm, writeFile } from 'node:fs/promises';
|
||||||
import { dirname, join, relative, resolve } from 'node:path';
|
import { dirname, join, relative, resolve } from 'node:path';
|
||||||
import type { Writable } from 'node:stream';
|
|
||||||
import { fileURLToPath } from 'node:url';
|
import { fileURLToPath } from 'node:url';
|
||||||
import { styleText } from 'node:util';
|
import { styleText } from 'node:util';
|
||||||
import { log, outro } from '@clack/prompts';
|
import { log, outro } from '@clack/prompts';
|
||||||
|
|
@ -11,6 +10,7 @@ import { serializeKtxProjectConfig } from './context/project/config.js';
|
||||||
import { strToU8, zipSync } from 'fflate';
|
import { strToU8, zipSync } from 'fflate';
|
||||||
import type { KtxCliIo } from './cli-runtime.js';
|
import type { KtxCliIo } from './cli-runtime.js';
|
||||||
import { errorMessage, writePrefixedLines } from './clack.js';
|
import { errorMessage, writePrefixedLines } from './clack.js';
|
||||||
|
import { isWritableTtyOutput } from './io/tty.js';
|
||||||
import {
|
import {
|
||||||
createKtxSetupPromptAdapter,
|
createKtxSetupPromptAdapter,
|
||||||
createKtxSetupUiAdapter,
|
createKtxSetupUiAdapter,
|
||||||
|
|
@ -84,14 +84,6 @@ interface KtxCliLauncher {
|
||||||
args: string[];
|
args: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
function isWritableTtyOutput(output: KtxCliIo['stdout']): output is KtxCliIo['stdout'] & Writable {
|
|
||||||
return (
|
|
||||||
output.isTTY === true &&
|
|
||||||
typeof (output as { on?: unknown }).on === 'function' &&
|
|
||||||
typeof (output as { columns?: unknown }).columns !== 'undefined'
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function writeSetupInfo(io: KtxCliIo, message: string): void {
|
function writeSetupInfo(io: KtxCliIo, message: string): void {
|
||||||
if (isWritableTtyOutput(io.stdout)) {
|
if (isWritableTtyOutput(io.stdout)) {
|
||||||
log.info(message, { output: io.stdout });
|
log.info(message, { output: io.stdout });
|
||||||
|
|
|
||||||
|
|
@ -73,6 +73,7 @@ export type KtxSetupDatabaseDriver =
|
||||||
export interface KtxSetupDatabasesArgs {
|
export interface KtxSetupDatabasesArgs {
|
||||||
projectDir: string;
|
projectDir: string;
|
||||||
inputMode: 'auto' | 'disabled';
|
inputMode: 'auto' | 'disabled';
|
||||||
|
debug?: boolean;
|
||||||
yes?: boolean;
|
yes?: boolean;
|
||||||
cliVersion?: string;
|
cliVersion?: string;
|
||||||
runtimeInstallPolicy?: KtxManagedPythonInstallPolicy;
|
runtimeInstallPolicy?: KtxManagedPythonInstallPolicy;
|
||||||
|
|
@ -1626,7 +1627,12 @@ function hasServiceAccountsBlock(connection: KtxProjectConnectionConfig | undefi
|
||||||
return 'serviceAccounts' in filters;
|
return 'serviceAccounts' in filters;
|
||||||
}
|
}
|
||||||
|
|
||||||
function printQueryHistoryFilterProposal(io: KtxCliIo, proposal: QueryHistoryFilterProposal): void {
|
function printQueryHistoryFilterProposal(io: KtxCliIo, proposal: QueryHistoryFilterProposal, debug = false): void {
|
||||||
|
if (debug && proposal.parseFailedTemplateIds.length > 0) {
|
||||||
|
io.stderr.write(
|
||||||
|
`[debug] query-history filter picker could not parse ${proposal.parseFailedTemplateIds.length} template(s): ${proposal.parseFailedTemplateIds.join(', ')}\n`,
|
||||||
|
);
|
||||||
|
}
|
||||||
if (proposal.excludedRoles.length === 0) {
|
if (proposal.excludedRoles.length === 0) {
|
||||||
if (proposal.skipped?.reason === 'no-llm') {
|
if (proposal.skipped?.reason === 'no-llm') {
|
||||||
io.stdout.write('│ Query-history filter picker skipped: no LLM is configured.\n');
|
io.stdout.write('│ Query-history filter picker skipped: no LLM is configured.\n');
|
||||||
|
|
@ -1635,6 +1641,12 @@ function printQueryHistoryFilterProposal(io: KtxCliIo, proposal: QueryHistoryFil
|
||||||
} else if (proposal.skipped?.reason === 'no-in-scope-history') {
|
} else if (proposal.skipped?.reason === 'no-in-scope-history') {
|
||||||
io.stdout.write('│ Query-history filter picker found no in-scope service-account exclusions.\n');
|
io.stdout.write('│ Query-history filter picker found no in-scope service-account exclusions.\n');
|
||||||
}
|
}
|
||||||
|
if (proposal.parseFailedTemplateIds.length > 0) {
|
||||||
|
const count = proposal.parseFailedTemplateIds.length;
|
||||||
|
io.stdout.write(
|
||||||
|
`│ Skipped ${count} query template${count === 1 ? '' : 's'} ktx could not parse (run with --debug to list them).\n`,
|
||||||
|
);
|
||||||
|
}
|
||||||
for (const warning of proposal.warnings) {
|
for (const warning of proposal.warnings) {
|
||||||
io.stdout.write(`│ ! ${warning}\n`);
|
io.stdout.write(`│ ! ${warning}\n`);
|
||||||
}
|
}
|
||||||
|
|
@ -1727,12 +1739,17 @@ async function maybeProposeQueryHistoryFilters(input: {
|
||||||
deps: input.deps,
|
deps: input.deps,
|
||||||
});
|
});
|
||||||
if (!llmRuntime && !input.deps.queryHistoryFilterPicker) {
|
if (!llmRuntime && !input.deps.queryHistoryFilterPicker) {
|
||||||
printQueryHistoryFilterProposal(input.io, {
|
printQueryHistoryFilterProposal(
|
||||||
excludedRoles: [],
|
input.io,
|
||||||
consideredRoleCount: 0,
|
{
|
||||||
skipped: { reason: 'no-llm' },
|
excludedRoles: [],
|
||||||
warnings: [],
|
consideredRoleCount: 0,
|
||||||
});
|
skipped: { reason: 'no-llm' },
|
||||||
|
warnings: [],
|
||||||
|
parseFailedTemplateIds: [],
|
||||||
|
},
|
||||||
|
input.args.debug === true,
|
||||||
|
);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1773,7 +1790,19 @@ async function maybeProposeQueryHistoryFilters(input: {
|
||||||
userServiceAccountsPresent,
|
userServiceAccountsPresent,
|
||||||
});
|
});
|
||||||
|
|
||||||
printQueryHistoryFilterProposal(input.io, proposal);
|
printQueryHistoryFilterProposal(input.io, proposal, input.args.debug === true);
|
||||||
|
await emitTelemetryEvent({
|
||||||
|
name: 'query_history_filter_completed',
|
||||||
|
projectDir: input.projectDir,
|
||||||
|
io: input.io,
|
||||||
|
fields: {
|
||||||
|
dialect,
|
||||||
|
consideredRoleCount: proposal.consideredRoleCount,
|
||||||
|
excludedRoleCount: proposal.excludedRoles.length,
|
||||||
|
parseFailedCount: proposal.parseFailedTemplateIds.length,
|
||||||
|
outcome: 'ok',
|
||||||
|
},
|
||||||
|
});
|
||||||
if (proposal.skipped?.reason === 'user-block-present') {
|
if (proposal.skipped?.reason === 'user-block-present') {
|
||||||
input.io.stdout.write('│ Existing query-history service-account filters left unchanged.\n');
|
input.io.stdout.write('│ Existing query-history service-account filters left unchanged.\n');
|
||||||
return;
|
return;
|
||||||
|
|
|
||||||
|
|
@ -259,6 +259,7 @@ async function runDemoContextReplay(
|
||||||
frame: 0,
|
frame: 0,
|
||||||
startedAt: Date.now(),
|
startedAt: Date.now(),
|
||||||
totalElapsedMs: 0,
|
totalElapsedMs: 0,
|
||||||
|
starCount: null,
|
||||||
};
|
};
|
||||||
|
|
||||||
const allTargets = [...allPrimary, ...allContext];
|
const allTargets = [...allPrimary, ...allContext];
|
||||||
|
|
|
||||||
|
|
@ -222,8 +222,8 @@ async function chooseCredentialRef(
|
||||||
const choice = await prompts.select({
|
const choice = await prompts.select({
|
||||||
message: `How should KTX find your ${embeddingBackendDisplayName(backend)} embedding API key?`,
|
message: `How should KTX find your ${embeddingBackendDisplayName(backend)} embedding API key?`,
|
||||||
options: [
|
options: [
|
||||||
{ value: 'env', label: `Use ${defaultEnv} from the environment` },
|
|
||||||
{ value: 'paste', label: 'Paste a key and save it as a local secret file' },
|
{ value: 'paste', label: 'Paste a key and save it as a local secret file' },
|
||||||
|
{ value: 'env', label: `Use ${defaultEnv} from the environment` },
|
||||||
{ value: 'back', label: 'Back' },
|
{ value: 'back', label: 'Back' },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ import { resolveKtxConfigReference } from './context/core/config-reference.js';
|
||||||
import { type KtxProjectConfig, type KtxProjectLlmConfig, serializeKtxProjectConfig } from './context/project/config.js';
|
import { type KtxProjectConfig, type KtxProjectLlmConfig, serializeKtxProjectConfig } from './context/project/config.js';
|
||||||
import { loadKtxProject } from './context/project/project.js';
|
import { loadKtxProject } from './context/project/project.js';
|
||||||
import { markKtxSetupStateStepComplete } from './context/project/setup-config.js';
|
import { markKtxSetupStateStepComplete } from './context/project/setup-config.js';
|
||||||
import type { KtxLlmConfig } from './llm/types.js';
|
import { type KtxModelRole, KTX_MODEL_ROLES, type KtxLlmConfig } from './llm/types.js';
|
||||||
import { type KtxLlmHealthCheckResult, runKtxLlmHealthCheck } from './llm/model-health.js';
|
import { type KtxLlmHealthCheckResult, runKtxLlmHealthCheck } from './llm/model-health.js';
|
||||||
import {
|
import {
|
||||||
formatClaudeCodePromptCachingWarning,
|
formatClaudeCodePromptCachingWarning,
|
||||||
|
|
@ -37,7 +37,6 @@ export interface KtxSetupModelArgs {
|
||||||
llmBackend?: KtxSetupLlmBackend;
|
llmBackend?: KtxSetupLlmBackend;
|
||||||
anthropicApiKeyEnv?: string;
|
anthropicApiKeyEnv?: string;
|
||||||
anthropicApiKeyFile?: string;
|
anthropicApiKeyFile?: string;
|
||||||
llmModel?: string;
|
|
||||||
vertexProject?: string;
|
vertexProject?: string;
|
||||||
vertexLocation?: string;
|
vertexLocation?: string;
|
||||||
forcePrompt?: boolean;
|
forcePrompt?: boolean;
|
||||||
|
|
@ -52,14 +51,27 @@ export type KtxSetupModelResult =
|
||||||
| { status: 'missing-input'; projectDir: string }
|
| { status: 'missing-input'; projectDir: string }
|
||||||
| { status: 'failed'; projectDir: string };
|
| { status: 'failed'; projectDir: string };
|
||||||
|
|
||||||
/** @internal */
|
// Single source of truth for the LLM backends a user can pick during setup.
|
||||||
export interface AnthropicModelChoice {
|
// The CLI arg parser, the interactive prompt, and the missing-backend error all
|
||||||
id: string;
|
// derive from this list, so adding a backend is one edit. Order is the prompt's
|
||||||
label: string;
|
// preference order (subscription backends first).
|
||||||
recommended: boolean;
|
const KTX_SETUP_LLM_BACKENDS = ['claude-code', 'codex', 'anthropic', 'vertex'] as const;
|
||||||
|
export type KtxSetupLlmBackend = (typeof KTX_SETUP_LLM_BACKENDS)[number];
|
||||||
|
|
||||||
|
/** Validates a raw CLI or prompt value against the setup-selectable LLM backends. */
|
||||||
|
export function isKtxSetupLlmBackend(value: string): value is KtxSetupLlmBackend {
|
||||||
|
return KTX_SETUP_LLM_BACKENDS.some((backend) => backend === value);
|
||||||
}
|
}
|
||||||
|
|
||||||
export type KtxSetupLlmBackend = 'anthropic' | 'vertex' | 'claude-code' | 'codex';
|
// Display labels for the interactive provider prompt. The Record key type forces
|
||||||
|
// every backend to carry a label, so adding one to KTX_SETUP_LLM_BACKENDS fails
|
||||||
|
// to compile until its prompt option exists here.
|
||||||
|
const KTX_SETUP_LLM_BACKEND_LABELS: Record<KtxSetupLlmBackend, string> = {
|
||||||
|
'claude-code': 'Claude subscription (Pro/Max)',
|
||||||
|
codex: 'Codex subscription',
|
||||||
|
anthropic: 'Anthropic API key',
|
||||||
|
vertex: 'Google Vertex AI for Anthropic Claude',
|
||||||
|
};
|
||||||
|
|
||||||
/** @internal */
|
/** @internal */
|
||||||
export interface KtxSetupModelPromptAdapter {
|
export interface KtxSetupModelPromptAdapter {
|
||||||
|
|
@ -76,9 +88,7 @@ export interface KtxSetupModelPromptAdapter {
|
||||||
|
|
||||||
export interface KtxSetupModelDeps {
|
export interface KtxSetupModelDeps {
|
||||||
env?: NodeJS.ProcessEnv;
|
env?: NodeJS.ProcessEnv;
|
||||||
fetch?: typeof fetch;
|
|
||||||
prompts?: KtxSetupModelPromptAdapter;
|
prompts?: KtxSetupModelPromptAdapter;
|
||||||
listModels?: (apiKey: string) => Promise<AnthropicModelChoice[]>;
|
|
||||||
healthCheck?: (config: KtxLlmConfig) => Promise<KtxLlmHealthCheckResult>;
|
healthCheck?: (config: KtxLlmConfig) => Promise<KtxLlmHealthCheckResult>;
|
||||||
claudeCodeAuthProbe?: (input: {
|
claudeCodeAuthProbe?: (input: {
|
||||||
projectDir: string;
|
projectDir: string;
|
||||||
|
|
@ -91,94 +101,72 @@ export interface KtxSetupModelDeps {
|
||||||
spinner?: () => KtxCliSpinner;
|
spinner?: () => KtxCliSpinner;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @internal */
|
|
||||||
export const BUNDLED_ANTHROPIC_MODELS: AnthropicModelChoice[] = [
|
|
||||||
{ id: 'claude-sonnet-4-6', label: 'Claude Sonnet 4.6', recommended: true },
|
|
||||||
{ id: 'claude-opus-4-6', label: 'Claude Opus 4.6', recommended: false },
|
|
||||||
{ id: 'claude-haiku-4-5', label: 'Claude Haiku 4.5', recommended: false },
|
|
||||||
];
|
|
||||||
|
|
||||||
const VERTEX_ANTHROPIC_MODELS: AnthropicModelChoice[] = [
|
|
||||||
{ id: 'claude-opus-4-7', label: 'Claude Opus 4.7', recommended: false },
|
|
||||||
{ id: 'claude-sonnet-4-6', label: 'Claude Sonnet 4.6', recommended: false },
|
|
||||||
{ id: 'claude-opus-4-6', label: 'Claude Opus 4.6', recommended: false },
|
|
||||||
{ id: 'claude-opus-4-5', label: 'Claude Opus 4.5', recommended: false },
|
|
||||||
{ id: 'claude-haiku-4-5', label: 'Claude Haiku 4.5', recommended: false },
|
|
||||||
{ id: 'claude-sonnet-4-5', label: 'Claude Sonnet 4.5', recommended: false },
|
|
||||||
{ id: 'claude-opus-4-1', label: 'Claude Opus 4.1', recommended: false },
|
|
||||||
];
|
|
||||||
|
|
||||||
const CLAUDE_CODE_MODELS: AnthropicModelChoice[] = [
|
|
||||||
{ id: 'sonnet', label: 'Claude Sonnet', recommended: true },
|
|
||||||
{ id: 'opus', label: 'Claude Opus', recommended: false },
|
|
||||||
{ id: 'haiku', label: 'Claude Haiku', recommended: false },
|
|
||||||
];
|
|
||||||
|
|
||||||
// Curated Codex models from OpenAI's current lineup that work under both
|
|
||||||
// ChatGPT-account (subscription) and API-key auth. Intentionally omitted:
|
|
||||||
// the `*-codex` ids (e.g. gpt-5.3-codex, gpt-5.2-codex) are API-key-only and
|
|
||||||
// fail on ChatGPT-account auth, and gpt-5.3-codex-spark is a ChatGPT-Pro-only
|
|
||||||
// research preview. Codex resolves real availability per account at runtime
|
|
||||||
// (its binary remote-fetches the model list), so this is a convenience
|
|
||||||
// shortlist only — the manual-entry option accepts any id your account's
|
|
||||||
// `codex` picker exposes, and the auth probe reports an unsupported choice.
|
|
||||||
const CODEX_MODELS: AnthropicModelChoice[] = [
|
|
||||||
{ id: 'gpt-5.5', label: 'GPT-5.5', recommended: true },
|
|
||||||
{ id: 'gpt-5.4', label: 'GPT-5.4', recommended: false },
|
|
||||||
{ id: 'gpt-5.4-mini', label: 'GPT-5.4 mini', recommended: false },
|
|
||||||
];
|
|
||||||
|
|
||||||
const HIDDEN_ANTHROPIC_MODEL_PATTERNS = [
|
|
||||||
/^claude-sonnet-4$/i,
|
|
||||||
/^claude-opus-4$/i,
|
|
||||||
/^Claude Sonnet 4$/i,
|
|
||||||
/^Claude Opus 4$/i,
|
|
||||||
];
|
|
||||||
|
|
||||||
const ANTHROPIC_CREDENTIAL_PROMPT_CONTEXT =
|
const ANTHROPIC_CREDENTIAL_PROMPT_CONTEXT =
|
||||||
'KTX uses the key to verify Anthropic model access now and to run ingest agents that turn schemas, SQL, ' +
|
'KTX uses the key to verify Anthropic model access now and to run ingest agents that turn schemas, SQL, ' +
|
||||||
'BI metadata, and docs into semantic-layer sources and wiki context. ktx.yaml stores an env: or file: ' +
|
'BI metadata, and docs into semantic-layer sources and wiki context. ktx.yaml stores an env: or file: ' +
|
||||||
'reference, not the raw key.';
|
'reference, not the raw key.';
|
||||||
|
|
||||||
const ANTHROPIC_MODEL_PROMPT_CONTEXT =
|
|
||||||
'KTX uses this as the default model for ingest agents that turn schemas, SQL, BI metadata, and docs ' +
|
|
||||||
'into semantic-layer sources and wiki context.';
|
|
||||||
|
|
||||||
const VERTEX_PROJECT_PROMPT_CONTEXT =
|
const VERTEX_PROJECT_PROMPT_CONTEXT =
|
||||||
'KTX stores the selected Google Cloud project ID in ktx.yaml and uses Application Default Credentials for ' +
|
'KTX stores the selected Google Cloud project ID in ktx.yaml and uses Application Default Credentials for ' +
|
||||||
'access. Project visibility depends on the signed-in Google account and organization permissions.';
|
'access. Project visibility depends on the signed-in Google account and organization permissions.';
|
||||||
const DEFAULT_VERTEX_LOCATION = 'us-east5';
|
const DEFAULT_VERTEX_LOCATION = 'us-east5';
|
||||||
|
|
||||||
|
type KtxSetupModelPreset = Record<KtxModelRole, string>;
|
||||||
|
|
||||||
|
const ANTHROPIC_PRESET = {
|
||||||
|
default: 'claude-sonnet-4-6',
|
||||||
|
triage: 'claude-haiku-4-5',
|
||||||
|
candidateExtraction: 'claude-sonnet-4-6',
|
||||||
|
curator: 'claude-opus-4-7',
|
||||||
|
reconcile: 'claude-opus-4-7',
|
||||||
|
repair: 'claude-haiku-4-5',
|
||||||
|
} satisfies KtxSetupModelPreset;
|
||||||
|
|
||||||
|
const CLAUDE_CODE_PRESET = {
|
||||||
|
default: 'sonnet',
|
||||||
|
triage: 'haiku',
|
||||||
|
candidateExtraction: 'sonnet',
|
||||||
|
curator: 'opus',
|
||||||
|
reconcile: 'opus',
|
||||||
|
repair: 'haiku',
|
||||||
|
} satisfies KtxSetupModelPreset;
|
||||||
|
|
||||||
|
const CODEX_PRESET = {
|
||||||
|
default: DEFAULT_CODEX_MODEL,
|
||||||
|
triage: DEFAULT_CODEX_MODEL,
|
||||||
|
candidateExtraction: DEFAULT_CODEX_MODEL,
|
||||||
|
curator: DEFAULT_CODEX_MODEL,
|
||||||
|
reconcile: DEFAULT_CODEX_MODEL,
|
||||||
|
repair: DEFAULT_CODEX_MODEL,
|
||||||
|
} satisfies KtxSetupModelPreset;
|
||||||
|
|
||||||
|
const MODEL_PRESETS = {
|
||||||
|
anthropic: ANTHROPIC_PRESET,
|
||||||
|
vertex: ANTHROPIC_PRESET,
|
||||||
|
'claude-code': CLAUDE_CODE_PRESET,
|
||||||
|
codex: CODEX_PRESET,
|
||||||
|
} satisfies Record<KtxSetupLlmBackend, KtxSetupModelPreset>;
|
||||||
|
|
||||||
|
function presetForBackend(backend: KtxSetupLlmBackend): KtxSetupModelPreset {
|
||||||
|
return MODEL_PRESETS[backend];
|
||||||
|
}
|
||||||
|
|
||||||
const execFileAsync = promisify(execFile);
|
const execFileAsync = promisify(execFile);
|
||||||
|
|
||||||
type AnthropicModelDiscoveryErrorReason = 'authentication' | 'http' | 'empty-response';
|
|
||||||
|
|
||||||
class AnthropicModelDiscoveryError extends Error {
|
|
||||||
constructor(
|
|
||||||
message: string,
|
|
||||||
public readonly reason: AnthropicModelDiscoveryErrorReason,
|
|
||||||
public readonly status?: number,
|
|
||||||
) {
|
|
||||||
super(message);
|
|
||||||
this.name = 'AnthropicModelDiscoveryError';
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function isAnthropicModelAuthenticationError(error: unknown): error is AnthropicModelDiscoveryError {
|
|
||||||
return error instanceof AnthropicModelDiscoveryError && error.reason === 'authentication';
|
|
||||||
}
|
|
||||||
|
|
||||||
function isSelectableAnthropicModel(model: AnthropicModelChoice): boolean {
|
|
||||||
return !HIDDEN_ANTHROPIC_MODEL_PATTERNS.some((pattern) => pattern.test(model.id) || pattern.test(model.label));
|
|
||||||
}
|
|
||||||
|
|
||||||
type ChooseModelResult =
|
|
||||||
| { status: 'ready'; model: string }
|
|
||||||
| { status: 'back' | 'missing-input' | 'invalid-credential' };
|
|
||||||
|
|
||||||
type ChooseBackendResult =
|
type ChooseBackendResult =
|
||||||
| { status: 'ready'; backend: KtxSetupLlmBackend; prompted: boolean }
|
| { status: 'ready'; backend: KtxSetupLlmBackend; prompted: boolean }
|
||||||
| { status: 'back' };
|
| { status: 'back' }
|
||||||
|
| { status: 'missing-input' };
|
||||||
|
|
||||||
|
// Non-interactive setup cannot pick a provider safely: every backend needs
|
||||||
|
// something the user must supply (an API key, gcloud ADC, or a logged-in local
|
||||||
|
// CLI), so there is no credential-free default to fall back to. Name the hidden
|
||||||
|
// --llm-backend flag and its choices here instead, mirroring how the other
|
||||||
|
// automation errors guide users to the flag they need.
|
||||||
|
const MISSING_LLM_BACKEND_MESSAGE =
|
||||||
|
`Missing LLM backend: pass --llm-backend with one of ${KTX_SETUP_LLM_BACKENDS.join(', ')}. ` +
|
||||||
|
'claude-code and codex use local CLI authentication; anthropic also needs --anthropic-api-key-env or ' +
|
||||||
|
'--anthropic-api-key-file, and vertex also needs --vertex-project.';
|
||||||
|
|
||||||
type VertexConfigChoice =
|
type VertexConfigChoice =
|
||||||
| {
|
| {
|
||||||
|
|
@ -234,47 +222,6 @@ async function defaultListGcloudProjects(): Promise<GcloudProjectChoice[]> {
|
||||||
.filter((project): project is GcloudProjectChoice => Boolean(project));
|
.filter((project): project is GcloudProjectChoice => Boolean(project));
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @internal */
|
|
||||||
export async function fetchAnthropicModels(
|
|
||||||
apiKey: string,
|
|
||||||
fetchFn: typeof fetch = fetch,
|
|
||||||
): Promise<AnthropicModelChoice[]> {
|
|
||||||
const response = await fetchFn('https://api.anthropic.com/v1/models?limit=1000', {
|
|
||||||
headers: {
|
|
||||||
'anthropic-version': '2023-06-01',
|
|
||||||
'x-api-key': apiKey,
|
|
||||||
},
|
|
||||||
});
|
|
||||||
if (!response.ok) {
|
|
||||||
if (response.status === 401 || response.status === 403) {
|
|
||||||
throw new AnthropicModelDiscoveryError(
|
|
||||||
`Anthropic model discovery failed with HTTP ${response.status}`,
|
|
||||||
'authentication',
|
|
||||||
response.status,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
throw new AnthropicModelDiscoveryError(
|
|
||||||
`Anthropic model discovery failed with HTTP ${response.status}`,
|
|
||||||
'http',
|
|
||||||
response.status,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
const body = (await response.json()) as { data?: Array<{ id?: unknown; display_name?: unknown; type?: unknown }> };
|
|
||||||
const models = (body.data ?? [])
|
|
||||||
.map((item) => ({
|
|
||||||
id: typeof item.id === 'string' ? item.id : '',
|
|
||||||
label: typeof item.display_name === 'string' ? item.display_name : typeof item.id === 'string' ? item.id : '',
|
|
||||||
recommended: false,
|
|
||||||
}))
|
|
||||||
.filter((item) => item.id.startsWith('claude-'))
|
|
||||||
.filter(isSelectableAnthropicModel);
|
|
||||||
if (models.length === 0) {
|
|
||||||
throw new AnthropicModelDiscoveryError('Anthropic model discovery returned no Claude models', 'empty-response');
|
|
||||||
}
|
|
||||||
const recommendedIndex = models.findIndex((item) => item.id.includes('sonnet'));
|
|
||||||
return models.map((item, index) => ({ ...item, recommended: index === Math.max(recommendedIndex, 0) }));
|
|
||||||
}
|
|
||||||
|
|
||||||
export function isKtxSetupLlmConfigReady(config: KtxProjectLlmConfig): boolean {
|
export function isKtxSetupLlmConfigReady(config: KtxProjectLlmConfig): boolean {
|
||||||
let resolved: KtxLlmConfig | null;
|
let resolved: KtxLlmConfig | null;
|
||||||
try {
|
try {
|
||||||
|
|
@ -309,12 +256,12 @@ function buildProjectLlmConfig(
|
||||||
| { backend: 'vertex'; vertex: { project?: string; location: string } }
|
| { backend: 'vertex'; vertex: { project?: string; location: string } }
|
||||||
| { backend: 'claude-code' }
|
| { backend: 'claude-code' }
|
||||||
| { backend: 'codex' },
|
| { backend: 'codex' },
|
||||||
model: string,
|
models: KtxSetupModelPreset,
|
||||||
): KtxProjectLlmConfig {
|
): KtxProjectLlmConfig {
|
||||||
if (provider.backend === 'claude-code') {
|
if (provider.backend === 'claude-code') {
|
||||||
return {
|
return {
|
||||||
provider: { backend: 'claude-code' },
|
provider: { backend: 'claude-code' },
|
||||||
models: { ...existing.models, default: model },
|
models,
|
||||||
promptCaching: existing.promptCaching,
|
promptCaching: existing.promptCaching,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
@ -322,7 +269,7 @@ function buildProjectLlmConfig(
|
||||||
if (provider.backend === 'codex') {
|
if (provider.backend === 'codex') {
|
||||||
return {
|
return {
|
||||||
provider: { backend: 'codex' },
|
provider: { backend: 'codex' },
|
||||||
models: { ...existing.models, default: model },
|
models,
|
||||||
promptCaching: existing.promptCaching,
|
promptCaching: existing.promptCaching,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
@ -333,7 +280,7 @@ function buildProjectLlmConfig(
|
||||||
backend: 'vertex',
|
backend: 'vertex',
|
||||||
vertex: provider.vertex,
|
vertex: provider.vertex,
|
||||||
},
|
},
|
||||||
models: { ...existing.models, default: model },
|
models,
|
||||||
promptCaching: { ...(existing.promptCaching ?? {}), enabled: true, vertexFallbackTo5m: true },
|
promptCaching: { ...(existing.promptCaching ?? {}), enabled: true, vertexFallbackTo5m: true },
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
@ -343,7 +290,7 @@ function buildProjectLlmConfig(
|
||||||
backend: 'anthropic',
|
backend: 'anthropic',
|
||||||
anthropic: { api_key: provider.credentialRef },
|
anthropic: { api_key: provider.credentialRef },
|
||||||
},
|
},
|
||||||
models: { ...existing.models, default: model },
|
models,
|
||||||
promptCaching: { ...(existing.promptCaching ?? {}), enabled: true },
|
promptCaching: { ...(existing.promptCaching ?? {}), enabled: true },
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
@ -470,8 +417,8 @@ async function chooseCredentialRef(
|
||||||
const choice = await prompts.select({
|
const choice = await prompts.select({
|
||||||
message: `How should KTX find your Anthropic API key?\n\n${ANTHROPIC_CREDENTIAL_PROMPT_CONTEXT}`,
|
message: `How should KTX find your Anthropic API key?\n\n${ANTHROPIC_CREDENTIAL_PROMPT_CONTEXT}`,
|
||||||
options: [
|
options: [
|
||||||
{ value: 'env', label: 'Use ANTHROPIC_API_KEY from the environment' },
|
|
||||||
{ value: 'paste', label: 'Paste a key and save it as a local secret file' },
|
{ value: 'paste', label: 'Paste a key and save it as a local secret file' },
|
||||||
|
{ value: 'env', label: 'Use ANTHROPIC_API_KEY from the environment' },
|
||||||
{ value: 'back', label: 'Back' },
|
{ value: 'back', label: 'Back' },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
|
|
@ -514,16 +461,12 @@ function requestedBackend(args: KtxSetupModelArgs): KtxSetupLlmBackend | undefin
|
||||||
if (args.vertexProject || args.vertexLocation) {
|
if (args.vertexProject || args.vertexLocation) {
|
||||||
return 'vertex';
|
return 'vertex';
|
||||||
}
|
}
|
||||||
if (args.anthropicApiKeyEnv || args.anthropicApiKeyFile || args.llmModel) {
|
if (args.anthropicApiKeyEnv || args.anthropicApiKeyFile) {
|
||||||
return 'anthropic';
|
return 'anthropic';
|
||||||
}
|
}
|
||||||
return undefined;
|
return undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
function requestedModel(args: KtxSetupModelArgs): string | undefined {
|
|
||||||
return args.llmModel;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function chooseBackend(
|
async function chooseBackend(
|
||||||
args: KtxSetupModelArgs,
|
args: KtxSetupModelArgs,
|
||||||
io: KtxCliIo,
|
io: KtxCliIo,
|
||||||
|
|
@ -534,7 +477,8 @@ async function chooseBackend(
|
||||||
return { status: 'ready', backend: explicit, prompted: false };
|
return { status: 'ready', backend: explicit, prompted: false };
|
||||||
}
|
}
|
||||||
if (args.inputMode === 'disabled') {
|
if (args.inputMode === 'disabled') {
|
||||||
return { status: 'ready', backend: 'anthropic', prompted: false };
|
io.stderr.write(`${MISSING_LLM_BACKEND_MESSAGE}\n`);
|
||||||
|
return { status: 'missing-input' };
|
||||||
}
|
}
|
||||||
|
|
||||||
const prompts = deps.prompts ?? createPromptAdapter();
|
const prompts = deps.prompts ?? createPromptAdapter();
|
||||||
|
|
@ -546,21 +490,20 @@ async function chooseBackend(
|
||||||
const choice = await prompts.select({
|
const choice = await prompts.select({
|
||||||
message: 'Which LLM provider should KTX use?',
|
message: 'Which LLM provider should KTX use?',
|
||||||
options: [
|
options: [
|
||||||
{ value: 'claude-code', label: 'Claude subscription (Pro/Max)' },
|
...KTX_SETUP_LLM_BACKENDS.map((backend) => ({ value: backend, label: KTX_SETUP_LLM_BACKEND_LABELS[backend] })),
|
||||||
{ value: 'codex', label: 'Codex subscription' },
|
|
||||||
{ value: 'anthropic', label: 'Anthropic API key' },
|
|
||||||
{ value: 'vertex', label: 'Google Vertex AI for Anthropic Claude' },
|
|
||||||
{ value: 'back', label: 'Back' },
|
{ value: 'back', label: 'Back' },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
if (choice === 'back') {
|
if (choice === 'back') {
|
||||||
return { status: 'back' };
|
return { status: 'back' };
|
||||||
}
|
}
|
||||||
return {
|
if (isKtxSetupLlmBackend(choice)) {
|
||||||
status: 'ready',
|
return { status: 'ready', backend: choice, prompted: true };
|
||||||
backend: choice === 'vertex' || choice === 'claude-code' || choice === 'codex' ? choice : 'anthropic',
|
}
|
||||||
prompted: true,
|
// Options are derived from KTX_SETUP_LLM_BACKENDS, so the only other value is
|
||||||
};
|
// 'back' (handled above). Treat any unexpected value as a cancel rather than
|
||||||
|
// silently assuming a provider.
|
||||||
|
return { status: 'back' };
|
||||||
}
|
}
|
||||||
|
|
||||||
function resolveProvidedVertexRef(
|
function resolveProvidedVertexRef(
|
||||||
|
|
@ -774,187 +717,6 @@ async function chooseVertexConfig(
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
async function chooseModel(
|
|
||||||
args: KtxSetupModelArgs,
|
|
||||||
credentialValue: string,
|
|
||||||
io: KtxCliIo,
|
|
||||||
deps: KtxSetupModelDeps,
|
|
||||||
): Promise<ChooseModelResult> {
|
|
||||||
const providedModel = requestedModel(args);
|
|
||||||
if (providedModel) {
|
|
||||||
return { status: 'ready', model: providedModel };
|
|
||||||
}
|
|
||||||
if (args.inputMode === 'disabled') {
|
|
||||||
io.stderr.write('Missing LLM model: pass --llm-model.\n');
|
|
||||||
return { status: 'missing-input' };
|
|
||||||
}
|
|
||||||
|
|
||||||
let models: AnthropicModelChoice[];
|
|
||||||
try {
|
|
||||||
models = deps.listModels
|
|
||||||
? await deps.listModels(credentialValue)
|
|
||||||
: await fetchAnthropicModels(credentialValue, deps.fetch);
|
|
||||||
} catch (error) {
|
|
||||||
if (isAnthropicModelAuthenticationError(error)) {
|
|
||||||
const statusSuffix = error.status ? ` (HTTP ${error.status})` : '';
|
|
||||||
io.stderr.write(`Anthropic API key is invalid or unauthorized${statusSuffix}. Check the key and try again.\n`);
|
|
||||||
return { status: 'invalid-credential' };
|
|
||||||
}
|
|
||||||
io.stderr.write(
|
|
||||||
'Could not fetch live Anthropic models. Showing bundled defaults. Setup will still test the selected model before saving it.\n',
|
|
||||||
);
|
|
||||||
models = BUNDLED_ANTHROPIC_MODELS;
|
|
||||||
}
|
|
||||||
|
|
||||||
const selectableModels = models.filter(isSelectableAnthropicModel);
|
|
||||||
const prompts = deps.prompts ?? createPromptAdapter();
|
|
||||||
const modelOptions = [
|
|
||||||
...selectableModels.map((model) => ({
|
|
||||||
value: model.id,
|
|
||||||
label: model.label || model.id,
|
|
||||||
...(model.recommended ? { hint: 'recommended' } : {}),
|
|
||||||
})),
|
|
||||||
{ value: 'manual', label: 'Enter a model ID manually' },
|
|
||||||
{ value: 'back', label: 'Back' },
|
|
||||||
];
|
|
||||||
const choice = await prompts.autocomplete({
|
|
||||||
message: `Which Anthropic model should KTX use?\n\n${ANTHROPIC_MODEL_PROMPT_CONTEXT}`,
|
|
||||||
placeholder: 'Type to search models',
|
|
||||||
options: modelOptions,
|
|
||||||
});
|
|
||||||
if (choice === 'back') {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
if (choice === 'manual') {
|
|
||||||
const manual = await prompts.text({
|
|
||||||
message: withTextInputNavigation('Anthropic model ID'),
|
|
||||||
placeholder: selectableModels.find((model) => model.recommended)?.id ?? selectableModels[0]?.id,
|
|
||||||
});
|
|
||||||
if (manual === undefined) {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
return manual.trim() ? { status: 'ready', model: manual.trim() } : { status: 'missing-input' };
|
|
||||||
}
|
|
||||||
return { status: 'ready', model: choice };
|
|
||||||
}
|
|
||||||
|
|
||||||
async function chooseVertexModel(args: KtxSetupModelArgs, io: KtxCliIo, deps: KtxSetupModelDeps): Promise<ChooseModelResult> {
|
|
||||||
const providedModel = requestedModel(args);
|
|
||||||
if (providedModel) {
|
|
||||||
return { status: 'ready', model: providedModel };
|
|
||||||
}
|
|
||||||
if (args.inputMode === 'disabled') {
|
|
||||||
io.stderr.write('Missing LLM model: pass --llm-model.\n');
|
|
||||||
return { status: 'missing-input' };
|
|
||||||
}
|
|
||||||
|
|
||||||
const selectableModels = VERTEX_ANTHROPIC_MODELS.filter(isSelectableAnthropicModel);
|
|
||||||
const prompts = deps.prompts ?? createPromptAdapter();
|
|
||||||
const choice = await prompts.autocomplete({
|
|
||||||
message: `Which Anthropic model should KTX use?\n\n${ANTHROPIC_MODEL_PROMPT_CONTEXT}`,
|
|
||||||
placeholder: 'Type to search models',
|
|
||||||
options: [
|
|
||||||
...selectableModels.map((model) => ({
|
|
||||||
value: model.id,
|
|
||||||
label: model.label || model.id,
|
|
||||||
...(model.recommended ? { hint: 'recommended' } : {}),
|
|
||||||
})),
|
|
||||||
{ value: 'manual', label: 'Enter a model ID manually' },
|
|
||||||
{ value: 'back', label: 'Back' },
|
|
||||||
],
|
|
||||||
});
|
|
||||||
if (choice === 'back') {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
if (choice === 'manual') {
|
|
||||||
const manual = await prompts.text({
|
|
||||||
message: withTextInputNavigation('Anthropic model ID'),
|
|
||||||
placeholder: selectableModels.find((model) => model.recommended)?.id ?? selectableModels[0]?.id,
|
|
||||||
});
|
|
||||||
if (manual === undefined) {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
return manual.trim() ? { status: 'ready', model: manual.trim() } : { status: 'missing-input' };
|
|
||||||
}
|
|
||||||
return { status: 'ready', model: choice };
|
|
||||||
}
|
|
||||||
|
|
||||||
async function chooseClaudeCodeModel(args: KtxSetupModelArgs, deps: KtxSetupModelDeps): Promise<ChooseModelResult> {
|
|
||||||
const providedModel = requestedModel(args);
|
|
||||||
if (providedModel) {
|
|
||||||
return { status: 'ready', model: providedModel };
|
|
||||||
}
|
|
||||||
if (args.inputMode === 'disabled') {
|
|
||||||
return { status: 'ready', model: 'sonnet' };
|
|
||||||
}
|
|
||||||
|
|
||||||
const prompts = deps.prompts ?? createPromptAdapter();
|
|
||||||
const choice = await prompts.select({
|
|
||||||
message: `Which Claude Code model should KTX use?\n\n${ANTHROPIC_MODEL_PROMPT_CONTEXT}`,
|
|
||||||
options: [
|
|
||||||
...CLAUDE_CODE_MODELS.map((model) => ({
|
|
||||||
value: model.id,
|
|
||||||
label: model.label,
|
|
||||||
...(model.recommended ? { hint: 'recommended' } : {}),
|
|
||||||
})),
|
|
||||||
{ value: 'manual', label: 'Enter a Claude Code model ID manually' },
|
|
||||||
{ value: 'back', label: 'Back' },
|
|
||||||
],
|
|
||||||
});
|
|
||||||
if (choice === 'back') {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
if (choice === 'manual') {
|
|
||||||
const manual = await prompts.text({
|
|
||||||
message: withTextInputNavigation('Claude Code model ID'),
|
|
||||||
placeholder: CLAUDE_CODE_MODELS.find((model) => model.recommended)?.id ?? CLAUDE_CODE_MODELS[0]?.id,
|
|
||||||
});
|
|
||||||
if (manual === undefined) {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
return manual.trim() ? { status: 'ready', model: manual.trim() } : { status: 'missing-input' };
|
|
||||||
}
|
|
||||||
return { status: 'ready', model: choice };
|
|
||||||
}
|
|
||||||
|
|
||||||
async function chooseCodexModel(args: KtxSetupModelArgs, deps: KtxSetupModelDeps): Promise<ChooseModelResult> {
|
|
||||||
const providedModel = requestedModel(args);
|
|
||||||
if (providedModel) {
|
|
||||||
return { status: 'ready', model: providedModel };
|
|
||||||
}
|
|
||||||
if (args.inputMode === 'disabled') {
|
|
||||||
return { status: 'ready', model: DEFAULT_CODEX_MODEL };
|
|
||||||
}
|
|
||||||
|
|
||||||
const prompts = deps.prompts ?? createPromptAdapter();
|
|
||||||
const choice = await prompts.select({
|
|
||||||
message: `Which Codex model should KTX use?\n\n${ANTHROPIC_MODEL_PROMPT_CONTEXT}`,
|
|
||||||
options: [
|
|
||||||
...CODEX_MODELS.map((model) => ({
|
|
||||||
value: model.id,
|
|
||||||
label: model.label,
|
|
||||||
...(model.recommended ? { hint: 'recommended' } : {}),
|
|
||||||
})),
|
|
||||||
{ value: 'manual', label: 'Enter a Codex model ID manually' },
|
|
||||||
{ value: 'back', label: 'Back' },
|
|
||||||
],
|
|
||||||
});
|
|
||||||
if (choice === 'back') {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
if (choice === 'manual') {
|
|
||||||
const manual = await prompts.text({
|
|
||||||
message: withTextInputNavigation('Codex model ID'),
|
|
||||||
placeholder: CODEX_MODELS.find((model) => model.recommended)?.id ?? CODEX_MODELS[0]?.id,
|
|
||||||
});
|
|
||||||
if (manual === undefined) {
|
|
||||||
return { status: 'back' };
|
|
||||||
}
|
|
||||||
return manual.trim() ? { status: 'ready', model: manual.trim() } : { status: 'missing-input' };
|
|
||||||
}
|
|
||||||
return { status: 'ready', model: choice };
|
|
||||||
}
|
|
||||||
|
|
||||||
async function persistLlmConfig(
|
async function persistLlmConfig(
|
||||||
projectDir: string,
|
projectDir: string,
|
||||||
provider:
|
provider:
|
||||||
|
|
@ -962,12 +724,12 @@ async function persistLlmConfig(
|
||||||
| { backend: 'vertex'; vertex: { project?: string; location: string } }
|
| { backend: 'vertex'; vertex: { project?: string; location: string } }
|
||||||
| { backend: 'claude-code' }
|
| { backend: 'claude-code' }
|
||||||
| { backend: 'codex' },
|
| { backend: 'codex' },
|
||||||
model: string,
|
models: KtxSetupModelPreset,
|
||||||
): Promise<void> {
|
): Promise<void> {
|
||||||
const project = await loadKtxProject({ projectDir });
|
const project = await loadKtxProject({ projectDir });
|
||||||
const config = {
|
const config = {
|
||||||
...project.config,
|
...project.config,
|
||||||
llm: buildProjectLlmConfig(project.config.llm, provider, model),
|
llm: buildProjectLlmConfig(project.config.llm, provider, models),
|
||||||
scan: {
|
scan: {
|
||||||
...project.config.scan,
|
...project.config.scan,
|
||||||
enrichment: {
|
enrichment: {
|
||||||
|
|
@ -990,6 +752,61 @@ function buildInteractiveRetryArgs(args: KtxSetupModelArgs, backend?: KtxSetupLl
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type PresetModelValidationResult = { ok: true } | { ok: false; message: string };
|
||||||
|
|
||||||
|
function distinctPresetModels(preset: KtxSetupModelPreset): string[] {
|
||||||
|
const models: string[] = [];
|
||||||
|
const seen = new Set<string>();
|
||||||
|
for (const role of KTX_MODEL_ROLES) {
|
||||||
|
const model = preset[role];
|
||||||
|
if (!seen.has(model)) {
|
||||||
|
seen.add(model);
|
||||||
|
models.push(model);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return models;
|
||||||
|
}
|
||||||
|
|
||||||
|
function rolesUsingModel(preset: KtxSetupModelPreset, model: string): KtxModelRole[] {
|
||||||
|
return KTX_MODEL_ROLES.filter((role) => preset[role] === model);
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatPresetFallbackWarning(roles: KtxModelRole[], unavailableModel: string, anchorModel: string): string {
|
||||||
|
return `LLM model ${unavailableModel} is unavailable for ${roles.join(', ')}; using ${anchorModel} for those roles.`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function validatePresetModels(
|
||||||
|
preset: KtxSetupModelPreset,
|
||||||
|
validateModel: (model: string) => Promise<PresetModelValidationResult>,
|
||||||
|
io: KtxCliIo,
|
||||||
|
): Promise<{ status: 'ready'; models: KtxSetupModelPreset } | { status: 'failed'; message: string }> {
|
||||||
|
const anchorModel = preset.default;
|
||||||
|
const degraded = { ...preset };
|
||||||
|
const models = distinctPresetModels(preset);
|
||||||
|
|
||||||
|
const anchorResult = await validateModel(anchorModel);
|
||||||
|
if (!anchorResult.ok) {
|
||||||
|
return { status: 'failed', message: anchorResult.message };
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const model of models) {
|
||||||
|
if (model === anchorModel) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const result = await validateModel(model);
|
||||||
|
if (result.ok) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const affectedRoles = rolesUsingModel(degraded, model);
|
||||||
|
for (const role of affectedRoles) {
|
||||||
|
degraded[role] = anchorModel;
|
||||||
|
}
|
||||||
|
io.stderr.write(`${formatPresetFallbackWarning(affectedRoles, model, anchorModel)}\n`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return { status: 'ready', models: degraded };
|
||||||
|
}
|
||||||
|
|
||||||
export async function runKtxSetupAnthropicModelStep(
|
export async function runKtxSetupAnthropicModelStep(
|
||||||
args: KtxSetupModelArgs,
|
args: KtxSetupModelArgs,
|
||||||
io: KtxCliIo,
|
io: KtxCliIo,
|
||||||
|
|
@ -1007,7 +824,6 @@ export async function runKtxSetupAnthropicModelStep(
|
||||||
!args.llmBackend &&
|
!args.llmBackend &&
|
||||||
!args.anthropicApiKeyEnv &&
|
!args.anthropicApiKeyEnv &&
|
||||||
!args.anthropicApiKeyFile &&
|
!args.anthropicApiKeyFile &&
|
||||||
!args.llmModel &&
|
|
||||||
!args.vertexProject &&
|
!args.vertexProject &&
|
||||||
!args.vertexLocation
|
!args.vertexLocation
|
||||||
) {
|
) {
|
||||||
|
|
@ -1038,94 +854,74 @@ export async function runKtxSetupAnthropicModelStep(
|
||||||
return { status: vertex.status, projectDir: args.projectDir };
|
return { status: vertex.status, projectDir: args.projectDir };
|
||||||
}
|
}
|
||||||
|
|
||||||
const model = await chooseVertexModel(backendArgs, io, deps);
|
const preset = presetForBackend('vertex');
|
||||||
if (model.status === 'back' && !backendArgs.vertexLocation) {
|
const validation = await validatePresetModels(
|
||||||
|
preset,
|
||||||
|
async (model) =>
|
||||||
|
runLlmHealthCheckWithProgress(
|
||||||
|
buildVertexHealthConfig(vertex.values, model),
|
||||||
|
'Vertex AI',
|
||||||
|
model,
|
||||||
|
healthCheck,
|
||||||
|
deps,
|
||||||
|
),
|
||||||
|
io,
|
||||||
|
);
|
||||||
|
if (validation.status !== 'ready') {
|
||||||
|
io.stderr.write(
|
||||||
|
`Vertex AI Anthropic model health check failed: ${formatVertexHealthFailure(validation.message, vertex.values)}\n`,
|
||||||
|
);
|
||||||
|
if (args.inputMode === 'disabled') {
|
||||||
|
return { status: 'failed', projectDir: args.projectDir };
|
||||||
|
}
|
||||||
|
io.stderr.write('Choose a different Vertex AI project or location, or Back.\n');
|
||||||
attemptArgs = buildInteractiveRetryArgs(args, backendChoice.backend);
|
attemptArgs = buildInteractiveRetryArgs(args, backendChoice.backend);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (model.status === 'invalid-credential') {
|
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
if (model.status !== 'ready') {
|
|
||||||
return { status: model.status, projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
|
|
||||||
const health = await runLlmHealthCheckWithProgress(
|
await persistLlmConfig(args.projectDir, { backend: 'vertex', vertex: vertex.refs }, validation.models);
|
||||||
buildVertexHealthConfig(vertex.values, model.model),
|
io.stdout.write(`│ LLM ready: yes (${validation.models.default})\n`);
|
||||||
'Vertex AI',
|
return { status: 'ready', projectDir: args.projectDir };
|
||||||
model.model,
|
|
||||||
healthCheck,
|
|
||||||
deps,
|
|
||||||
);
|
|
||||||
if (health.ok) {
|
|
||||||
await persistLlmConfig(args.projectDir, { backend: 'vertex', vertex: vertex.refs }, model.model);
|
|
||||||
io.stdout.write(`│ LLM ready: yes (${model.model})\n`);
|
|
||||||
return { status: 'ready', projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
|
|
||||||
io.stderr.write(`Vertex AI Anthropic model health check failed: ${formatVertexHealthFailure(health.message, vertex.values)}\n`);
|
|
||||||
if (args.inputMode === 'disabled') {
|
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
io.stderr.write('Choose a different Vertex AI project, location, or model, or Back.\n');
|
|
||||||
attemptArgs = buildInteractiveRetryArgs(args, backendChoice.backend);
|
|
||||||
continue;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if (backendChoice.backend === 'claude-code') {
|
if (backendChoice.backend === 'claude-code') {
|
||||||
const model = await chooseClaudeCodeModel(backendArgs, deps);
|
const preset = presetForBackend('claude-code');
|
||||||
if (model.status === 'back' && backendChoice.prompted) {
|
|
||||||
attemptArgs = buildInteractiveRetryArgs(args);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (model.status === 'invalid-credential') {
|
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
if (model.status !== 'ready') {
|
|
||||||
return { status: model.status, projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
const probe = deps.claudeCodeAuthProbe ?? runClaudeCodeAuthProbe;
|
const probe = deps.claudeCodeAuthProbe ?? runClaudeCodeAuthProbe;
|
||||||
const health = await probe({ projectDir: args.projectDir, model: model.model, env: deps.env ?? process.env });
|
const validation = await validatePresetModels(
|
||||||
if (!health.ok) {
|
preset,
|
||||||
io.stderr.write(`${health.message}\n`);
|
async (model) => probe({ projectDir: args.projectDir, model, env: deps.env ?? process.env }),
|
||||||
|
io,
|
||||||
|
);
|
||||||
|
if (validation.status !== 'ready') {
|
||||||
|
io.stderr.write(`${validation.message}\n`);
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
return { status: 'failed', projectDir: args.projectDir };
|
||||||
}
|
}
|
||||||
const warning = formatClaudeCodePromptCachingWarning(
|
const warning = formatClaudeCodePromptCachingWarning(
|
||||||
ignoredClaudeCodePromptCachingFields(
|
ignoredClaudeCodePromptCachingFields(
|
||||||
buildProjectLlmConfig(project.config.llm, { backend: 'claude-code' }, model.model),
|
buildProjectLlmConfig(project.config.llm, { backend: 'claude-code' }, validation.models),
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
if (warning) {
|
if (warning) {
|
||||||
io.stderr.write(`${warning}\n`);
|
io.stderr.write(`${warning}\n`);
|
||||||
}
|
}
|
||||||
await persistLlmConfig(args.projectDir, { backend: 'claude-code' }, model.model);
|
await persistLlmConfig(args.projectDir, { backend: 'claude-code' }, validation.models);
|
||||||
io.stdout.write(`│ LLM ready: yes (${model.model})\n`);
|
io.stdout.write(`│ LLM ready: yes (${validation.models.default})\n`);
|
||||||
return { status: 'ready', projectDir: args.projectDir };
|
return { status: 'ready', projectDir: args.projectDir };
|
||||||
}
|
}
|
||||||
|
|
||||||
if (backendChoice.backend === 'codex') {
|
if (backendChoice.backend === 'codex') {
|
||||||
const model = await chooseCodexModel(backendArgs, deps);
|
const preset = presetForBackend('codex');
|
||||||
if (model.status === 'back' && backendChoice.prompted) {
|
|
||||||
attemptArgs = buildInteractiveRetryArgs(args);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (model.status === 'invalid-credential') {
|
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
if (model.status !== 'ready') {
|
|
||||||
return { status: model.status, projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
const probe = deps.codexAuthProbe ?? runCodexAuthProbe;
|
const probe = deps.codexAuthProbe ?? runCodexAuthProbe;
|
||||||
const health = await probe({ projectDir: args.projectDir, model: model.model });
|
const validation = await validatePresetModels(preset, async (model) => probe({ projectDir: args.projectDir, model }), io);
|
||||||
if (!health.ok) {
|
if (validation.status !== 'ready') {
|
||||||
io.stderr.write(`${health.message}\n`);
|
io.stderr.write(`${validation.message}\n`);
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
return { status: 'failed', projectDir: args.projectDir };
|
||||||
}
|
}
|
||||||
// Prefix the clack gutter so the warning sits inside the setup frame
|
// Prefix the clack gutter so the warning sits inside the setup frame
|
||||||
// instead of breaking out of it; kept on stderr for scripted runs.
|
// instead of breaking out of it; kept on stderr for scripted runs.
|
||||||
io.stderr.write(`│ ${formatCodexIsolationWarning()}\n`);
|
io.stderr.write(`│ ${formatCodexIsolationWarning()}\n`);
|
||||||
await persistLlmConfig(args.projectDir, { backend: 'codex' }, model.model);
|
await persistLlmConfig(args.projectDir, { backend: 'codex' }, validation.models);
|
||||||
io.stdout.write(`│ LLM ready: yes (codex, ${model.model})\n`);
|
io.stdout.write(`│ LLM ready: yes (codex, ${validation.models.default})\n`);
|
||||||
return { status: 'ready', projectDir: args.projectDir };
|
return { status: 'ready', projectDir: args.projectDir };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1138,8 +934,21 @@ export async function runKtxSetupAnthropicModelStep(
|
||||||
return { status: credential.status, projectDir: args.projectDir };
|
return { status: credential.status, projectDir: args.projectDir };
|
||||||
}
|
}
|
||||||
|
|
||||||
const model = await chooseModel(backendArgs, credential.value, io, deps);
|
const preset = presetForBackend('anthropic');
|
||||||
if (model.status === 'invalid-credential') {
|
const validation = await validatePresetModels(
|
||||||
|
preset,
|
||||||
|
async (model) =>
|
||||||
|
runLlmHealthCheckWithProgress(
|
||||||
|
buildAnthropicHealthConfig(credential.value, model),
|
||||||
|
'Anthropic API',
|
||||||
|
model,
|
||||||
|
healthCheck,
|
||||||
|
deps,
|
||||||
|
),
|
||||||
|
io,
|
||||||
|
);
|
||||||
|
if (validation.status !== 'ready') {
|
||||||
|
io.stderr.write(`Anthropic model health check failed: ${validation.message}\n`);
|
||||||
if (args.inputMode === 'disabled') {
|
if (args.inputMode === 'disabled') {
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
return { status: 'failed', projectDir: args.projectDir };
|
||||||
}
|
}
|
||||||
|
|
@ -1147,32 +956,9 @@ export async function runKtxSetupAnthropicModelStep(
|
||||||
attemptArgs = buildInteractiveRetryArgs(args, backendChoice.backend);
|
attemptArgs = buildInteractiveRetryArgs(args, backendChoice.backend);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (model.status === 'back' && !backendArgs.anthropicApiKeyEnv && !backendArgs.anthropicApiKeyFile) {
|
|
||||||
attemptArgs = buildInteractiveRetryArgs(args, backendChoice.backend);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (model.status !== 'ready') {
|
|
||||||
return { status: model.status, projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
|
|
||||||
const health = await runLlmHealthCheckWithProgress(
|
await persistLlmConfig(args.projectDir, { backend: 'anthropic', credentialRef: credential.ref }, validation.models);
|
||||||
buildAnthropicHealthConfig(credential.value, model.model),
|
io.stdout.write(`│ LLM ready: yes (${validation.models.default})\n`);
|
||||||
'Anthropic API',
|
return { status: 'ready', projectDir: args.projectDir };
|
||||||
model.model,
|
|
||||||
healthCheck,
|
|
||||||
deps,
|
|
||||||
);
|
|
||||||
if (health.ok) {
|
|
||||||
await persistLlmConfig(args.projectDir, { backend: 'anthropic', credentialRef: credential.ref }, model.model);
|
|
||||||
io.stdout.write(`│ LLM ready: yes (${model.model})\n`);
|
|
||||||
return { status: 'ready', projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
|
|
||||||
io.stderr.write(`Anthropic model health check failed: ${health.message}\n`);
|
|
||||||
if (args.inputMode === 'disabled') {
|
|
||||||
return { status: 'failed', projectDir: args.projectDir };
|
|
||||||
}
|
|
||||||
io.stderr.write('Choose a different credential source or model, or Back.\n');
|
|
||||||
attemptArgs = buildInteractiveRetryArgs(args, backendChoice.backend);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,3 @@
|
||||||
import type { Writable } from 'node:stream';
|
|
||||||
import {
|
import {
|
||||||
autocomplete,
|
autocomplete,
|
||||||
autocompleteMultiselect,
|
autocompleteMultiselect,
|
||||||
|
|
@ -9,12 +8,13 @@ import {
|
||||||
log,
|
log,
|
||||||
multiselect,
|
multiselect,
|
||||||
note,
|
note,
|
||||||
password,
|
|
||||||
select,
|
select,
|
||||||
text,
|
text,
|
||||||
} from '@clack/prompts';
|
} from '@clack/prompts';
|
||||||
import type { KtxCliIo } from './cli-runtime.js';
|
import type { KtxCliIo } from './cli-runtime.js';
|
||||||
|
import { isWritableTtyOutput } from './io/tty.js';
|
||||||
import { withMenuOptionsSpacing, withTextInputNavigation } from './prompt-navigation.js';
|
import { withMenuOptionsSpacing, withTextInputNavigation } from './prompt-navigation.js';
|
||||||
|
import { revealPassword } from './reveal-password-prompt.js';
|
||||||
import { withSetupInterruptConfirmation } from './setup-interrupt.js';
|
import { withSetupInterruptConfirmation } from './setup-interrupt.js';
|
||||||
|
|
||||||
export interface KtxSetupPromptOption<Value extends string = string> {
|
export interface KtxSetupPromptOption<Value extends string = string> {
|
||||||
|
|
@ -189,7 +189,7 @@ export function createKtxSetupPromptAdapter(options: KtxSetupPromptAdapterOption
|
||||||
},
|
},
|
||||||
async password(promptOptions) {
|
async password(promptOptions) {
|
||||||
const value = await withSetupInterruptConfirmation(() =>
|
const value = await withSetupInterruptConfirmation(() =>
|
||||||
password({ ...promptOptions, message: withTextInputNavigation(promptOptions.message) }),
|
revealPassword({ ...promptOptions, message: withTextInputNavigation(promptOptions.message) }),
|
||||||
);
|
);
|
||||||
return isCancel(value) ? undefined : String(value);
|
return isCancel(value) ? undefined : String(value);
|
||||||
},
|
},
|
||||||
|
|
@ -211,14 +211,6 @@ export interface KtxSetupUiAdapter {
|
||||||
note(message: string, title: string, io: KtxCliIo, options?: KtxSetupNoteOptions): void;
|
note(message: string, title: string, io: KtxCliIo, options?: KtxSetupNoteOptions): void;
|
||||||
}
|
}
|
||||||
|
|
||||||
function isWritableTtyOutput(output: KtxCliIo['stdout']): output is KtxCliIo['stdout'] & Writable {
|
|
||||||
return (
|
|
||||||
output.isTTY === true &&
|
|
||||||
typeof (output as { on?: unknown }).on === 'function' &&
|
|
||||||
typeof (output as { columns?: unknown }).columns !== 'undefined'
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function createKtxSetupUiAdapter(): KtxSetupUiAdapter {
|
export function createKtxSetupUiAdapter(): KtxSetupUiAdapter {
|
||||||
return {
|
return {
|
||||||
intro(title, io) {
|
intro(title, io) {
|
||||||
|
|
|
||||||
|
|
@ -119,11 +119,11 @@ export interface KtxSetupSourcesDeps {
|
||||||
|
|
||||||
const SOURCE_OPTIONS: Array<{ value: KtxSetupSourceType; label: string }> = [
|
const SOURCE_OPTIONS: Array<{ value: KtxSetupSourceType; label: string }> = [
|
||||||
{ value: 'dbt', label: 'dbt' },
|
{ value: 'dbt', label: 'dbt' },
|
||||||
{ value: 'metricflow', label: 'MetricFlow' },
|
|
||||||
{ value: 'metabase', label: 'Metabase' },
|
{ value: 'metabase', label: 'Metabase' },
|
||||||
|
{ value: 'notion', label: 'Notion' },
|
||||||
|
{ value: 'metricflow', label: 'MetricFlow' },
|
||||||
{ value: 'looker', label: 'Looker' },
|
{ value: 'looker', label: 'Looker' },
|
||||||
{ value: 'lookml', label: 'LookML' },
|
{ value: 'lookml', label: 'LookML' },
|
||||||
{ value: 'notion', label: 'Notion' },
|
|
||||||
];
|
];
|
||||||
|
|
||||||
const SOURCE_LABELS = Object.fromEntries(SOURCE_OPTIONS.map((option) => [option.value, option.label])) as Record<
|
const SOURCE_LABELS = Object.fromEntries(SOURCE_OPTIONS.map((option) => [option.value, option.label])) as Record<
|
||||||
|
|
@ -269,8 +269,8 @@ async function chooseSourceCredentialRef(input: {
|
||||||
message: `How should KTX find your ${input.label}?`,
|
message: `How should KTX find your ${input.label}?`,
|
||||||
options: [
|
options: [
|
||||||
...(input.existingRef ? [{ value: 'keep', label: 'Keep existing credential' }] : []),
|
...(input.existingRef ? [{ value: 'keep', label: 'Keep existing credential' }] : []),
|
||||||
{ value: 'env', label: `Use ${input.envName} from the environment` },
|
|
||||||
{ value: 'paste', label: 'Paste a key and save it as a local secret file' },
|
{ value: 'paste', label: 'Paste a key and save it as a local secret file' },
|
||||||
|
{ value: 'env', label: `Use ${input.envName} from the environment` },
|
||||||
{ value: 'back', label: 'Back' },
|
{ value: 'back', label: 'Back' },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
|
|
@ -307,8 +307,8 @@ async function chooseGitAuthCredentialRef(input: {
|
||||||
message: `${label} repo requires authentication.`,
|
message: `${label} repo requires authentication.`,
|
||||||
options: [
|
options: [
|
||||||
...(input.existingRef ? [{ value: 'keep', label: 'Keep existing credential' }] : []),
|
...(input.existingRef ? [{ value: 'keep', label: 'Keep existing credential' }] : []),
|
||||||
{ value: 'env', label: 'Use GITHUB_TOKEN from the environment' },
|
|
||||||
{ value: 'paste', label: 'Paste a token and save it as a local secret file' },
|
{ value: 'paste', label: 'Paste a token and save it as a local secret file' },
|
||||||
|
{ value: 'env', label: 'Use GITHUB_TOKEN from the environment' },
|
||||||
{ value: 'skip', label: 'Skip — try without authentication' },
|
{ value: 'skip', label: 'Skip — try without authentication' },
|
||||||
{ value: 'back', label: 'Back' },
|
{ value: 'back', label: 'Back' },
|
||||||
],
|
],
|
||||||
|
|
@ -1063,8 +1063,8 @@ async function promptForInteractiveSource(
|
||||||
const selectedLocation = await prompts.select({
|
const selectedLocation = await prompts.select({
|
||||||
message: `${source} source location`,
|
message: `${source} source location`,
|
||||||
options: [
|
options: [
|
||||||
{ value: 'path', label: 'Local path' },
|
|
||||||
{ value: 'git', label: 'Git URL' },
|
{ value: 'git', label: 'Git URL' },
|
||||||
|
{ value: 'path', label: 'Local path' },
|
||||||
{ value: 'back', label: 'Back' },
|
{ value: 'back', label: 'Back' },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
|
|
@ -1343,8 +1343,8 @@ async function promptForInteractiveSource(
|
||||||
const crawlMode = await prompts.select({
|
const crawlMode = await prompts.select({
|
||||||
message: 'Which Notion pages should KTX ingest?',
|
message: 'Which Notion pages should KTX ingest?',
|
||||||
options: [
|
options: [
|
||||||
{ value: 'selected_roots', label: 'Specific pages and their subpages (choose them in a picker)' },
|
|
||||||
{ value: 'all_accessible', label: 'All pages the integration can access' },
|
{ value: 'all_accessible', label: 'All pages the integration can access' },
|
||||||
|
{ value: 'selected_roots', label: 'Specific pages and their subpages (choose them in a picker)' },
|
||||||
{ value: 'back', label: 'Back' },
|
{ value: 'back', label: 'Back' },
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
|
|
@ -2064,7 +2064,7 @@ export async function runKtxSetupSourcesStep(
|
||||||
const addMore = await prompts.select({
|
const addMore = await prompts.select({
|
||||||
message: `${readyConnectionIds.length} context source${readyConnectionIds.length > 1 ? 's' : ''} configured (${readyConnectionIds.join(', ')}). Add another?`,
|
message: `${readyConnectionIds.length} context source${readyConnectionIds.length > 1 ? 's' : ''} configured (${readyConnectionIds.join(', ')}). Add another?`,
|
||||||
options: [
|
options: [
|
||||||
{ value: 'done', label: 'Done — continue to context build' },
|
{ value: 'done', label: 'Done adding context sources' },
|
||||||
{ value: 'edit', label: 'Edit an existing context source' },
|
{ value: 'edit', label: 'Edit an existing context source' },
|
||||||
{ value: 'add', label: 'Add another context source' },
|
{ value: 'add', label: 'Add another context source' },
|
||||||
],
|
],
|
||||||
|
|
|
||||||
|
|
@ -6,11 +6,13 @@ import { ktxLocalStateDbPath } from './context/project/local-state-db.js';
|
||||||
import { loadKtxProject, type KtxLocalProject } from './context/project/project.js';
|
import { loadKtxProject, type KtxLocalProject } from './context/project/project.js';
|
||||||
import { readKtxSetupState } from './context/project/setup-config.js';
|
import { readKtxSetupState } from './context/project/setup-config.js';
|
||||||
import { getKtxCliPackageInfo, type KtxCliIo } from './cli-runtime.js';
|
import { getKtxCliPackageInfo, type KtxCliIo } from './cli-runtime.js';
|
||||||
|
import { SLACK_SETUP_NOTE } from './community-cta.js';
|
||||||
import { formatNextStepLines, formatSetupNextStepLines } from './next-steps.js';
|
import { formatNextStepLines, formatSetupNextStepLines } from './next-steps.js';
|
||||||
import { runtimeInstallPolicyFromFlags } from './managed-python-command.js';
|
import { runtimeInstallPolicyFromFlags } from './managed-python-command.js';
|
||||||
import { readManagedPythonRuntimeStatus } from './managed-python-runtime.js';
|
import { readManagedPythonRuntimeStatus } from './managed-python-runtime.js';
|
||||||
import { resolveProjectRuntimeRequirements } from './runtime-requirements.js';
|
import { resolveProjectRuntimeRequirements } from './runtime-requirements.js';
|
||||||
import { isKtxSetupExitError } from './setup-interrupt.js';
|
import { isKtxSetupExitError } from './setup-interrupt.js';
|
||||||
|
import type { CommandOutcome } from './telemetry/index.js';
|
||||||
import {
|
import {
|
||||||
type KtxAgentScope,
|
type KtxAgentScope,
|
||||||
type KtxAgentTarget,
|
type KtxAgentTarget,
|
||||||
|
|
@ -80,12 +82,12 @@ export type KtxSetupArgs =
|
||||||
agentScope?: KtxAgentScope;
|
agentScope?: KtxAgentScope;
|
||||||
skipAgents?: boolean;
|
skipAgents?: boolean;
|
||||||
inputMode: 'auto' | 'disabled';
|
inputMode: 'auto' | 'disabled';
|
||||||
|
debug?: boolean;
|
||||||
yes: boolean;
|
yes: boolean;
|
||||||
cliVersion: string;
|
cliVersion: string;
|
||||||
llmBackend?: KtxSetupLlmBackend;
|
llmBackend?: KtxSetupLlmBackend;
|
||||||
anthropicApiKeyEnv?: string;
|
anthropicApiKeyEnv?: string;
|
||||||
anthropicApiKeyFile?: string;
|
anthropicApiKeyFile?: string;
|
||||||
llmModel?: string;
|
|
||||||
vertexProject?: string;
|
vertexProject?: string;
|
||||||
vertexLocation?: string;
|
vertexLocation?: string;
|
||||||
skipLlm: boolean;
|
skipLlm: boolean;
|
||||||
|
|
@ -210,6 +212,80 @@ function setupTelemetryOutcome(
|
||||||
return 'abandoned';
|
return 'abandoned';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
interface SetupCommandAnnotation {
|
||||||
|
outcome: CommandOutcome;
|
||||||
|
errorClass?: string;
|
||||||
|
errorDetail?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Classify a terminal non-ready setup status into the `command` telemetry
|
||||||
|
* outcome. The setup flow is the decision-maker and knows the difference:
|
||||||
|
* - `failed` is a genuine error; attach a step-scoped reason so the dashboard
|
||||||
|
* shows an actionable signature instead of a blank.
|
||||||
|
* - `missing-input` from a *non-interactive* run is an automation error
|
||||||
|
* (required flags absent and no prompt was possible); attach a reason too.
|
||||||
|
* - `missing-input` from an interactive prompt, or a project `cancelled`, is the
|
||||||
|
* user backing out of the wizard — an abort, not a failure. Keep it out of
|
||||||
|
* error telemetry so it stops inflating the error count.
|
||||||
|
*
|
||||||
|
* `interactive` must reflect whether a prompt could actually be shown — input
|
||||||
|
* is enabled AND a TTY is attached. `inputMode: 'auto'` alone is not enough: a
|
||||||
|
* piped/CI run without `--no-input` is still non-interactive, and steps such as
|
||||||
|
* the project step return `missing-input` ("pass --yes …") there without ever
|
||||||
|
* prompting. Treating that as an abort would make a broken automation run exit
|
||||||
|
* 0, so it must classify as an error.
|
||||||
|
*
|
||||||
|
* Reasons are synthetic, step-scoped strings (no user input), so they satisfy
|
||||||
|
* the telemetry privacy rules. The step's own `errorDetail`, when present, has
|
||||||
|
* already been vetted for the `setup_step` event and is safe to reuse.
|
||||||
|
*/
|
||||||
|
function setupCommandOutcomeAnnotation(input: {
|
||||||
|
status: 'failed' | 'missing-input' | 'cancelled';
|
||||||
|
step: TelemetrySetupStep;
|
||||||
|
interactive: boolean;
|
||||||
|
errorDetail?: string;
|
||||||
|
}): SetupCommandAnnotation {
|
||||||
|
if (input.status === 'failed') {
|
||||||
|
return {
|
||||||
|
outcome: 'error',
|
||||||
|
errorClass: 'KtxSetupStepFailed',
|
||||||
|
errorDetail: input.errorDetail ?? `${input.step} setup step failed`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (input.status === 'missing-input' && !input.interactive) {
|
||||||
|
return {
|
||||||
|
outcome: 'error',
|
||||||
|
errorClass: 'KtxSetupMissingInput',
|
||||||
|
errorDetail: `${input.step} setup step requires input not provided in a non-interactive run`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { outcome: 'aborted' };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Single source of truth for how a non-ready setup step ends: the process exit
|
||||||
|
* code and the telemetry annotation are both derived from one classification,
|
||||||
|
* so they can never disagree. A genuine failure (`error`) exits non-zero; an
|
||||||
|
* abort — the user leaving an interactive wizard — exits 0, matching the entry
|
||||||
|
* menu's "Exit", a project cancellation, and a confirmed Ctrl+C.
|
||||||
|
*/
|
||||||
|
/** @internal */
|
||||||
|
export function setupTerminalOutcome(input: {
|
||||||
|
status: 'failed' | 'missing-input' | 'cancelled';
|
||||||
|
step: TelemetrySetupStep;
|
||||||
|
interactive: boolean;
|
||||||
|
errorDetail?: string;
|
||||||
|
}): { exitCode: number; annotation: SetupCommandAnnotation } {
|
||||||
|
const annotation = setupCommandOutcomeAnnotation(input);
|
||||||
|
return { exitCode: annotation.outcome === 'error' ? 1 : 0, annotation };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function annotateSetupCommandOutcome(annotation: SetupCommandAnnotation): Promise<void> {
|
||||||
|
const { annotateCommandOutcome } = await import('./telemetry/index.js');
|
||||||
|
annotateCommandOutcome(annotation);
|
||||||
|
}
|
||||||
|
|
||||||
async function recordSetupStep(input: {
|
async function recordSetupStep(input: {
|
||||||
projectDir: string;
|
projectDir: string;
|
||||||
step: TelemetrySetupStep;
|
step: TelemetrySetupStep;
|
||||||
|
|
@ -572,6 +648,10 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
|
||||||
args.inputMode !== 'disabled' &&
|
args.inputMode !== 'disabled' &&
|
||||||
!args.agents &&
|
!args.agents &&
|
||||||
(io.stdout.isTTY === true || deps.entryMenuDeps?.prompts !== undefined);
|
(io.stdout.isTTY === true || deps.entryMenuDeps?.prompts !== undefined);
|
||||||
|
// A prompt is only possible when input is enabled AND a TTY is attached. A
|
||||||
|
// piped/CI `ktx setup` without `--no-input` is still `inputMode: 'auto'` but
|
||||||
|
// cannot prompt, so its `missing-input` is an automation error, not an abort.
|
||||||
|
const interactive = args.inputMode !== 'disabled' && io.stdout.isTTY === true;
|
||||||
|
|
||||||
setupLoop: while (true) {
|
setupLoop: while (true) {
|
||||||
entryAction = undefined;
|
entryAction = undefined;
|
||||||
|
|
@ -618,7 +698,13 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
|
||||||
}
|
}
|
||||||
|
|
||||||
if (projectResult.status !== 'ready') {
|
if (projectResult.status !== 'ready') {
|
||||||
return projectResult.status === 'cancelled' ? 0 : 1;
|
const terminal = setupTerminalOutcome({
|
||||||
|
status: projectResult.status,
|
||||||
|
step: 'project',
|
||||||
|
interactive,
|
||||||
|
});
|
||||||
|
await annotateSetupCommandOutcome(terminal.annotation);
|
||||||
|
return terminal.exitCode;
|
||||||
}
|
}
|
||||||
|
|
||||||
const agentsRequested = args.agents || entryAction === 'agents';
|
const agentsRequested = args.agents || entryAction === 'agents';
|
||||||
|
|
@ -699,7 +785,6 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
|
||||||
...(args.llmBackend ? { llmBackend: args.llmBackend } : {}),
|
...(args.llmBackend ? { llmBackend: args.llmBackend } : {}),
|
||||||
...(args.anthropicApiKeyEnv ? { anthropicApiKeyEnv: args.anthropicApiKeyEnv } : {}),
|
...(args.anthropicApiKeyEnv ? { anthropicApiKeyEnv: args.anthropicApiKeyEnv } : {}),
|
||||||
...(args.anthropicApiKeyFile ? { anthropicApiKeyFile: args.anthropicApiKeyFile } : {}),
|
...(args.anthropicApiKeyFile ? { anthropicApiKeyFile: args.anthropicApiKeyFile } : {}),
|
||||||
...(args.llmModel ? { llmModel: args.llmModel } : {}),
|
|
||||||
...(args.vertexProject ? { vertexProject: args.vertexProject } : {}),
|
...(args.vertexProject ? { vertexProject: args.vertexProject } : {}),
|
||||||
...(args.vertexLocation ? { vertexLocation: args.vertexLocation } : {}),
|
...(args.vertexLocation ? { vertexLocation: args.vertexLocation } : {}),
|
||||||
forcePrompt: forcePromptSteps.has('models') || runOnly === 'models',
|
forcePrompt: forcePromptSteps.has('models') || runOnly === 'models',
|
||||||
|
|
@ -735,6 +820,7 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
|
||||||
{
|
{
|
||||||
projectDir: projectResult.projectDir,
|
projectDir: projectResult.projectDir,
|
||||||
inputMode: args.inputMode,
|
inputMode: args.inputMode,
|
||||||
|
...(args.debug !== undefined ? { debug: args.debug } : {}),
|
||||||
yes: args.yes,
|
yes: args.yes,
|
||||||
cliVersion: args.cliVersion,
|
cliVersion: args.cliVersion,
|
||||||
runtimeInstallPolicy: setupRuntimeInstallPolicy(args),
|
runtimeInstallPolicy: setupRuntimeInstallPolicy(args),
|
||||||
|
|
@ -855,11 +941,15 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
|
||||||
...(stepResult.errorDetail ? { errorDetail: stepResult.errorDetail } : {}),
|
...(stepResult.errorDetail ? { errorDetail: stepResult.errorDetail } : {}),
|
||||||
});
|
});
|
||||||
|
|
||||||
if (stepResult.status === 'failed') {
|
if (stepResult.status === 'failed' || stepResult.status === 'missing-input') {
|
||||||
return 1;
|
const terminal = setupTerminalOutcome({
|
||||||
}
|
status: stepResult.status,
|
||||||
if (stepResult.status === 'missing-input') {
|
step,
|
||||||
return 1;
|
interactive,
|
||||||
|
...(stepResult.errorDetail ? { errorDetail: stepResult.errorDetail } : {}),
|
||||||
|
});
|
||||||
|
await annotateSetupCommandOutcome(terminal.annotation);
|
||||||
|
return terminal.exitCode;
|
||||||
}
|
}
|
||||||
if (stepResult.status === 'back') {
|
if (stepResult.status === 'back') {
|
||||||
const previousIndex = previousNavigableStepIndex(stepIndex);
|
const previousIndex = previousNavigableStepIndex(stepIndex);
|
||||||
|
|
@ -921,5 +1011,6 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
setupUi.note(SLACK_SETUP_NOTE.body, SLACK_SETUP_NOTE.title, io);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,5 @@
|
||||||
import { readFile } from 'node:fs/promises';
|
import { readFile } from 'node:fs/promises';
|
||||||
import type { KtxCliIo } from './cli-runtime.js';
|
import type { KtxCliIo } from './cli-runtime.js';
|
||||||
import { createDefaultLocalQueryExecutor } from './context/connections/local-query-executor.js';
|
|
||||||
import type { KtxSqlQueryExecutorPort } from './context/connections/query-executor.js';
|
import type { KtxSqlQueryExecutorPort } from './context/connections/query-executor.js';
|
||||||
import { KtxIngestEmbeddingPortAdapter } from './context/llm/embedding-port.js';
|
import { KtxIngestEmbeddingPortAdapter } from './context/llm/embedding-port.js';
|
||||||
import type { KtxEmbeddingPort } from './context/core/embedding.js';
|
import type { KtxEmbeddingPort } from './context/core/embedding.js';
|
||||||
|
|
@ -20,13 +19,15 @@ import {
|
||||||
resolveProjectEmbeddingProvider,
|
resolveProjectEmbeddingProvider,
|
||||||
type EmbeddingProviderResolution,
|
type EmbeddingProviderResolution,
|
||||||
} from './embedding-resolution.js';
|
} from './embedding-resolution.js';
|
||||||
|
import { createKtxCliIngestQueryExecutor } from './ingest-query-executor.js';
|
||||||
import type { PrintListColumn } from './io/print-list.js';
|
import type { PrintListColumn } from './io/print-list.js';
|
||||||
import {
|
import {
|
||||||
createManagedPythonSemanticLayerComputePort,
|
createManagedPythonSemanticLayerComputePort,
|
||||||
type KtxManagedPythonInstallPolicy,
|
type KtxManagedPythonInstallPolicy,
|
||||||
} from './managed-python-command.js';
|
} from './managed-python-command.js';
|
||||||
import { profileMark } from './startup-profile.js';
|
import { profileMark } from './startup-profile.js';
|
||||||
import { emitTelemetryEvent } from './telemetry/index.js';
|
import { emitTelemetryEvent, reportException } from './telemetry/index.js';
|
||||||
|
import { collectTelemetryRedactionSecrets } from './telemetry/redaction-secrets.js';
|
||||||
import { scrubErrorClass } from './telemetry/scrubber.js';
|
import { scrubErrorClass } from './telemetry/scrubber.js';
|
||||||
|
|
||||||
profileMark('module:sl');
|
profileMark('module:sl');
|
||||||
|
|
@ -80,7 +81,7 @@ interface KtxSlDeps {
|
||||||
io: KtxSlIo;
|
io: KtxSlIo;
|
||||||
projectDir?: string;
|
projectDir?: string;
|
||||||
}) => Promise<KtxSemanticLayerComputePort>;
|
}) => Promise<KtxSemanticLayerComputePort>;
|
||||||
createQueryExecutor?: () => KtxSqlQueryExecutorPort;
|
createQueryExecutor?: (project: KtxLocalProject) => KtxSqlQueryExecutorPort;
|
||||||
}
|
}
|
||||||
|
|
||||||
function resolutionToEmbeddingPort(resolution: EmbeddingProviderResolution): KtxEmbeddingPort | null {
|
function resolutionToEmbeddingPort(resolution: EmbeddingProviderResolution): KtxEmbeddingPort | null {
|
||||||
|
|
@ -202,8 +203,9 @@ function ambiguousSourceMessage(sourceName: string, connectionIds: readonly stri
|
||||||
export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: KtxSlDeps = {}): Promise<number> {
|
export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: KtxSlDeps = {}): Promise<number> {
|
||||||
const startedAt = performance.now();
|
const startedAt = performance.now();
|
||||||
let queryForTelemetry: SemanticLayerQueryInput | undefined;
|
let queryForTelemetry: SemanticLayerQueryInput | undefined;
|
||||||
|
let project: KtxLocalProject | undefined;
|
||||||
try {
|
try {
|
||||||
const project = await (deps.loadProject ?? loadKtxProject)({ projectDir: args.projectDir });
|
project = await (deps.loadProject ?? loadKtxProject)({ projectDir: args.projectDir });
|
||||||
if (args.command === 'list') {
|
if (args.command === 'list') {
|
||||||
const sources = await listLocalSlSources(project, { connectionId: args.connectionId });
|
const sources = await listLocalSlSources(project, { connectionId: args.connectionId });
|
||||||
await printSlSources({
|
await printSlSources({
|
||||||
|
|
@ -319,8 +321,8 @@ export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: Ktx
|
||||||
io,
|
io,
|
||||||
projectDir: args.projectDir,
|
projectDir: args.projectDir,
|
||||||
});
|
});
|
||||||
const queryExecutor = args.execute ? (deps.createQueryExecutor ?? createDefaultLocalQueryExecutor)() : undefined;
|
const queryExecutor = args.execute ? (deps.createQueryExecutor ?? createKtxCliIngestQueryExecutor)(project) : undefined;
|
||||||
const result = await compileLocalSlQuery(project as KtxLocalProject, {
|
const result = await compileLocalSlQuery(project, {
|
||||||
connectionId: args.connectionId,
|
connectionId: args.connectionId,
|
||||||
query,
|
query,
|
||||||
compute,
|
compute,
|
||||||
|
|
@ -351,6 +353,20 @@ export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: Ktx
|
||||||
const _exhaustive: never = args;
|
const _exhaustive: never = args;
|
||||||
throw new Error(`Unsupported sl command: ${JSON.stringify(_exhaustive)}`);
|
throw new Error(`Unsupported sl command: ${JSON.stringify(_exhaustive)}`);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
await reportException({
|
||||||
|
error,
|
||||||
|
context: { source: `sl ${args.command}`, handled: true, fatal: false },
|
||||||
|
projectDir: args.projectDir,
|
||||||
|
io,
|
||||||
|
redactionSecrets: await collectTelemetryRedactionSecrets({
|
||||||
|
project,
|
||||||
|
projectDir: args.projectDir,
|
||||||
|
connectionId: args.connectionId,
|
||||||
|
includeLlm: args.command === 'query',
|
||||||
|
includeEmbeddings: args.command === 'search' || args.command === 'query',
|
||||||
|
env: process.env,
|
||||||
|
}),
|
||||||
|
});
|
||||||
if (args.command === 'validate') {
|
if (args.command === 'validate') {
|
||||||
const errorClass = scrubErrorClass(error);
|
const errorClass = scrubErrorClass(error);
|
||||||
await emitTelemetryEvent({
|
await emitTelemetryEvent({
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue