apunkt/ktx
1
0
Fork 0
mirror of https://github.com/Kaelio/ktx.git synced 2026-06-07 07:55:13 +02:00
Code Issues Projects Releases Packages Wiki Activity

docs(concepts): add Wiki retrieval pillar page (#191)

Browse source 
* docs(concepts): add Wiki retrieval pillar page

Adds a dedicated concept page covering the wiki side of the context
layer: the page contract, the hybrid retrieval pipeline (lexical,
semantic, token lanes fused by RRF), the refs/sl_refs/[[wikilink]]
graph, validation that keeps edges live, and where ingest sources
pages. Wired into concepts nav and cross-linked from the-context-layer
to mirror the existing Semantic querying link.

* test: derive release versions in tests instead of hardcoding 0.1.0-rc.1

After @semantic-release/git started committing version bumps back to the
branch, the 0.4.0 release rewrote package.json, packages/cli/package.json,
and release-policy.json — but the script and CLI tests still pinned the
pre-bump strings (0.0.0-private, 0.1.0-rc.1, 0.1.0rc1), so every new
branch off main failed TypeScript checks and Coverage.

Drive the version off the existing source of truth instead: read
@ktx/cli/package.json via createRequire in the CLI tests, and reuse the
already-imported PUBLIC_NPM_PACKAGE_VERSION / RUNTIME_WHEEL_PACKAGE_VERSION
constants in the script tests. The two assertions that pinned those
constants to specific values become semver shape checks.
This commit is contained in:
Andrey Avtomonov 2026-05-21 01:26:58 +02:00 • committed by GitHub
parent 2f647d5c68
commit ed2d2f9be0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 365 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs-site/content/docs/concepts/meta.json
View file

@ -1,5 +1,5 @@
{ {
"title": "Concepts", "title": "Concepts",
"defaultOpen": true, "defaultOpen": true,
"pages": ["the-context-layer", "semantic-layer-internals", "context-as-code"] "pages": ["the-context-layer", "semantic-layer-internals", "wiki-retrieval", "context-as-code"]
} }

4
docs-site/content/docs/concepts/the-context-layer.mdx
View file

@ -195,6 +195,10 @@ wiki pages are written and prunes `sl_refs` during ingest when their target
sources are deleted or their measures are renamed - so a stale page can never sources are deleted or their measures are renamed - so a stale page can never
quietly route an agent to a definition that no longer exists. quietly route an agent to a definition that no longer exists.
For how the hybrid search pipeline ranks pages, how `[[wikilinks]]` extend
the graph, and how ingest authors pages from evidence, read
[Wiki retrieval](/docs/concepts/wiki-retrieval).
The split between the two pillars is sharp: The split between the two pillars is sharp:
| Put it in YAML | Put it in Markdown | | Put it in YAML | Put it in Markdown |

280
docs-site/content/docs/concepts/wiki-retrieval.mdx Normal file
View file

@ -0,0 +1,280 @@
---
title: Wiki retrieval
description: How ktx ranks wiki pages with hybrid search, links them into a graph, and keeps both sides anchored to evidence.
---
The wiki is the prose half of the context layer. Agents reach it two ways:
they search for a page, then follow references inside the pages they
already opened. This page covers how both work.
- The wiki page contract that retrieval and validation depend on.
- The hybrid search pipeline that turns a question into ranked pages.
- The reference graph agents traverse without rerunning search.
- How pages get authored from evidence, and how broken edges get pruned.
## The wiki page contract
A wiki page is a Markdown file with a YAML frontmatter block. Frontmatter
carries metadata; the prose below it is free-form. Keys are flat tokens
(`revenue`, `mart_account_segments`), not paths, so every page is
addressable as `[[key]]` from any other page.
```markdown
# wiki/global/revenue.md
---
summary: Paid order value after refunds
tags: [finance, orders]
sl_refs: [warehouse.orders]
refs: [segment-classification]
usage_mode: auto
---
Revenue is paid order amount after refund adjustments.
Use `orders.total_revenue` for recognized order value and
`orders.order_count` for paid order volume.
```
| Field | Purpose |
|-------|---------|
| `summary` | One-line description shown in search results and the agent's knowledge index |
| `tags` | Topic labels mixed into the search text and used for filtering |
| `refs` | Outgoing edges to other wiki pages by key |
| `sl_refs` | Outgoing edges to semantic-layer sources by `connection.source` name |
| `usage_mode` | `always`, `auto`, or `never` - whether the agent must, may, or must not surface this page |
| `source` | Where the page came from when authored by ingest (e.g. `historic-sql`, `dbt`) |
| `usage` | Stats attached to historic-SQL pattern pages: executions, distinct users, runtime percentiles, error rate |
Pages live under two scopes. `wiki/global/*.md` is the team's shared
context; `wiki/user/<user>/*.md` is per-agent scratch space that shadows
global pages with the same key.
## What retrieval does
A wiki search runs the same ordered steps every time.
1. **Normalize the query.** Lowercase, tokenize, deduplicate terms.
2. **Score in three lanes.** Lexical (SQLite FTS5 bm25), semantic
(cosine similarity over embeddings), and token (term-overlap fallback)
each rank every page independently.
3. **Fuse with Reciprocal Rank Fusion.** Each lane contributes
`weight / (60 + rank)` to a candidate's score. Lanes that fail or
skip are dropped, not zeroed.
4. **Order and trim.** Sort by fused score, then by how many lanes
matched, then by id for stable tie-breaks. Return the top `limit`
results with their summaries.
5. **Hydrate on demand.** The agent calls `wiki_read` to load full
bodies for the few pages that look relevant.
<figure
className="not-prose my-10 overflow-hidden rounded-lg border border-fd-border bg-fd-card shadow-sm"
aria-label="Three retrieval lanes fused with Reciprocal Rank Fusion"
>
<div className="border-b border-fd-border bg-fd-muted/35 px-5 py-4">
<p className="text-[11px] font-semibold uppercase tracking-[0.08em] text-fd-primary">
{"Hybrid retrieval"}
</p>
<h3
className="mt-1 text-base font-semibold tracking-normal text-fd-foreground sm:text-lg"
style={{ fontFamily: "var(--font-display)" }}
>
{"Three lanes, one ranking"}
</h3>
</div>
<div className="p-6">
<div className="grid gap-3 md:grid-cols-3">
<div className="rounded-md border border-fd-border bg-fd-background p-4">
<p className="text-sm font-semibold text-fd-foreground">{"lexical"}</p>
<p className="mt-1 font-mono text-[11px] text-fd-muted-foreground">{"sqlite fts5 / bm25"}</p>
<p className="mt-3 text-xs leading-5 text-fd-muted-foreground">
{"Matches stems and phrases. Strong on the exact terms the team already uses."}
</p>
<p className="mt-3 text-[11px] uppercase tracking-[0.08em] text-fd-muted-foreground">
<span className="text-fd-foreground">{"weight "}</span>{"1.5"}
</p>
</div>
<div className="rounded-md border border-fd-border bg-fd-background p-4">
<p className="text-sm font-semibold text-fd-foreground">{"semantic"}</p>
<p className="mt-1 font-mono text-[11px] text-fd-muted-foreground">{"cosine over embeddings"}</p>
<p className="mt-3 text-xs leading-5 text-fd-muted-foreground">
{"Catches synonyms and paraphrases the lexical lane misses."}
</p>
<p className="mt-3 text-[11px] uppercase tracking-[0.08em] text-fd-muted-foreground">
<span className="text-fd-foreground">{"weight "}</span>{"2"}
</p>
</div>
<div className="rounded-md border border-fd-border bg-fd-background p-4">
<p className="text-sm font-semibold text-fd-foreground">{"token"}</p>
<p className="mt-1 font-mono text-[11px] text-fd-muted-foreground">{"term-overlap fallback"}</p>
<p className="mt-3 text-xs leading-5 text-fd-muted-foreground">
{"Always available, so short queries still produce candidates."}
</p>
<p className="mt-3 text-[11px] uppercase tracking-[0.08em] text-fd-muted-foreground">
<span className="text-fd-foreground">{"weight "}</span>{"0.75"}
</p>
</div>
</div>
<div className="mt-5 rounded-md border border-fd-primary/40 bg-fd-background p-4">
<p className="text-[11px] font-semibold uppercase tracking-[0.08em] text-fd-primary">
{"Reciprocal Rank Fusion"}
</p>
<p className="mt-1 font-mono text-[12px] text-fd-foreground">
{"score = Σ weight / (60 + rank)"}
</p>
<p className="mt-2 text-xs leading-5 text-fd-muted-foreground">
{"Pages that rank well in multiple lanes outscore pages that rank well in only one."}
</p>
</div>
</div>
<figcaption className="border-t border-fd-border bg-fd-muted/25 px-5 py-3 text-[11.5px] leading-5 text-fd-muted-foreground">
<span className="font-medium text-fd-foreground">{"Defaults are tunable. "}</span>
{"Lane weights and the RRF constant K are configuration, not assumptions."}
</figcaption>
</figure>
The text each lane scores is built deterministically: page key, summary,
body, and tags concatenated in that order. A precise summary and the
right tags make a page reachable before its body matches anything.
## The page graph
Two frontmatter fields and one inline syntax turn the wiki into a graph
the agent traverses without re-running search.
| Edge | Source | Target |
|------|--------|--------|
| `sl_refs: [warehouse.orders]` | Frontmatter | Semantic source by name |
| `refs: [segment-classification]` | Frontmatter | Another wiki page by key |
| `[[segment-classification]]` | Inline in body | Another wiki page by key |
`refs` stays in the prose layer; `sl_refs` crosses into the executable
half of the context layer. Inline `[[wikilinks]]` are extracted from
page bodies at validation time and treated as declared `refs`.
<figure
className="not-prose my-10 overflow-hidden rounded-lg border border-fd-border bg-fd-card shadow-sm"
aria-label="Example wiki cross-reference graph"
>
<div className="border-b border-fd-border bg-fd-muted/35 px-5 py-4">
<p className="text-[11px] font-semibold uppercase tracking-[0.08em] text-fd-primary">
{"Anatomy of a traversal"}
</p>
<h3
className="mt-1 text-base font-semibold tracking-normal text-fd-foreground sm:text-lg"
style={{ fontFamily: "var(--font-display)" }}
>
{"Edges to prose, edges to SQL"}
</h3>
</div>
<div className="p-6">
<div className="grid gap-3 md:grid-cols-[1fr_1fr]">
<div className="rounded-md border-2 bg-fd-background p-4" style={{ borderColor: "#10b981" }}>
<p className="font-mono text-[11px] font-semibold" style={{ color: "#10b981" }}>
{"wiki/global/revenue.md"}
</p>
<p className="mt-2 text-sm font-semibold text-fd-foreground">{"revenue"}</p>
<p className="mt-3 text-[11px] uppercase tracking-[0.08em] text-fd-muted-foreground">
{"declares"}
</p>
<ul className="mt-1 space-y-1 text-xs text-fd-muted-foreground">
<li><span className="font-mono text-fd-foreground">{"sl_refs"}</span>: warehouse.orders</li>
<li><span className="font-mono text-fd-foreground">{"refs"}</span>: segment-classification</li>
</ul>
</div>
<div className="rounded-md border bg-fd-background p-4" style={{ borderColor: "#10b981", borderStyle: "dashed" }}>
<p className="font-mono text-[11px] font-semibold" style={{ color: "#10b981" }}>
{"wiki/global/segment-classification.md"}
</p>
<p className="mt-2 text-sm font-semibold text-fd-foreground">{"segment-classification"}</p>
<p className="mt-3 text-[11px] uppercase tracking-[0.08em] text-fd-muted-foreground">
{"declares"}
</p>
<ul className="mt-1 space-y-1 text-xs text-fd-muted-foreground">
<li><span className="font-mono text-fd-foreground">{"sl_refs"}</span>: warehouse.customers</li>
</ul>
</div>
</div>
<div className="my-4 grid gap-2 text-center text-xs font-medium text-fd-muted-foreground md:grid-cols-[1fr_1fr]">
<div>{"revenue → warehouse.orders · sl_refs"}</div>
<div>{"revenue → segment-classification · refs"}</div>
</div>
<div className="grid gap-3 md:grid-cols-[1fr_1fr]">
<div className="rounded-md border-2 bg-fd-background p-4" style={{ borderColor: "#3b82f6" }}>
<p className="font-mono text-[11px] font-semibold" style={{ color: "#3b82f6" }}>
{"semantic-layer/warehouse/orders.yaml"}
</p>
<p className="mt-2 text-sm font-semibold text-fd-foreground">{"warehouse.orders"}</p>
<p className="mt-1 text-xs text-fd-muted-foreground">{"grain: order_id · measure: total_revenue"}</p>
</div>
<div className="rounded-md border-2 bg-fd-background p-4" style={{ borderColor: "#3b82f6" }}>
<p className="font-mono text-[11px] font-semibold" style={{ color: "#3b82f6" }}>
{"semantic-layer/warehouse/customers.yaml"}
</p>
<p className="mt-2 text-sm font-semibold text-fd-foreground">{"warehouse.customers"}</p>
<p className="mt-1 text-xs text-fd-muted-foreground">{"grain: customer_id · dim: segment"}</p>
</div>
</div>
</div>
<figcaption className="border-t border-fd-border bg-fd-muted/25 px-5 py-3 text-[11.5px] leading-5 text-fd-muted-foreground">
{"Green nodes are wiki pages; blue nodes are semantic sources."}
</figcaption>
</figure>
## Keeping the graph live
A page that references a deleted source is worse than no reference at
all - it sends the agent confidently to a definition that no longer
exists. **ktx** prevents that with three layered checks:
- **At write time.** Every `refs` entry and `[[wikilink]]` is validated
against the pages visible in the current scope. A write that targets
a missing page is rejected before any file changes.
- **At ingest time.** Adapters prune `sl_refs` when the target source
is deleted, mark stale pattern pages with `stale_since`, and set
`archived_since` on retired pages instead of removing them silently.
- **At session end.** Every page touched by an ingest run is re-scanned
for references that resolved at write time but no longer point at
a live target. Dangling pairs are reported so the next iteration can
fix them.
## Where the pages come from
**ktx** writes wiki pages from evidence, not free invention. Each input
contributes a different kind of page, and accepted edits feed the next
ingest as input.
| Evidence | What it produces |
|----------|------------------|
| Schema scans | One page per material table, with grain, columns, and known constraints |
| Query history | Pattern pages with `usage` frontmatter for executions, distinct users, runtime percentiles, and error rate |
| dbt manifests | Pages per model, exposure, and test, with `sl_refs` to the matching semantic source |
| MetricFlow, Looker, Metabase | Pages per metric, explore, or saved question, linked back to the source artifact |
| Notion, docs, analyst notes | Pages preserving business definitions, policies, and incident write-ups |
| Agent and analyst edits | First-class input to the next ingest, not a fork |
Provenance stays with the page. Ingested pages keep HTML comments like
`<!-- from: raw-sources/.../cards/69.json -->` inline, so a reviewer can
walk from the prose back to the artifact that produced it.
## Agent usage notes
Point an agent at this page when it needs to explain why a wiki search
returned the pages it did, why a write was rejected, or how the wiki
stays in step with the semantic layer.
| Agent task | Relevant section | Next page |
|------------|------------------|-----------|
| Explain why two searches return different pages for the same query | What retrieval does | [ktx wiki](/docs/cli-reference/ktx-wiki) |
| Decide whether to add a `refs` or `sl_refs` entry | The page graph | [Writing Context](/docs/guides/writing-context) |
| Repair a wiki write rejected for missing references | Keeping the graph live | [Writing Context](/docs/guides/writing-context) |
| Describe how historic SQL becomes a wiki page | Where the pages come from | [Building Context](/docs/guides/building-context) |
| Explain raw-source provenance comments | Where the pages come from | [Context as Code](/docs/concepts/context-as-code) |

7
packages/cli/src/admin-reindex.test.ts
View file

@ -1,8 +1,13 @@
import { createRequire } from 'node:module';
import type { ReindexSummary } from '@ktx/context/index-sync'; import type { ReindexSummary } from '@ktx/context/index-sync';
import { describe, expect, it, vi } from 'vitest'; import { describe, expect, it, vi } from 'vitest';
import { renderReindexJson, renderReindexPlain, reindexHasErrors } from './admin-reindex.js'; import { renderReindexJson, renderReindexPlain, reindexHasErrors } from './admin-reindex.js';
import { runKtxCli } from './index.js'; import { runKtxCli } from './index.js';
const cliVersion = (createRequire(import.meta.url)('@ktx/cli/package.json') as { version: string })
.version;
function makeIo(options: { stdoutIsTTY?: boolean } = {}) { function makeIo(options: { stdoutIsTTY?: boolean } = {}) {
let stdout = ''; let stdout = '';
let stderr = ''; let stderr = '';
@ -137,7 +142,7 @@ describe('admin reindex Commander routing', () => {
force: true, force: true,
json: true, json: true,
output: 'plain', output: 'plain',
cliVersion: '0.0.0-private', cliVersion,
}, },
io.io, io.io,
); );

46
packages/cli/src/index.test.ts
View file

@ -19,6 +19,9 @@ import {
const require = createRequire(import.meta.url); const require = createRequire(import.meta.url);
const cliPackageJson = require('@ktx/cli/package.json') as { name: string; version: string };
const cliVersion = cliPackageJson.version;
function makeIo(options: { stdoutIsTty?: boolean } = {}) { function makeIo(options: { stdoutIsTty?: boolean } = {}) {
let stdout = ''; let stdout = '';
let stderr = ''; let stderr = '';
@ -45,7 +48,7 @@ describe('getKtxCliPackageInfo', () => {
it('identifies the CLI package and its context dependency', () => { it('identifies the CLI package and its context dependency', () => {
expect(getKtxCliPackageInfo()).toEqual({ expect(getKtxCliPackageInfo()).toEqual({
name: '@ktx/cli', name: '@ktx/cli',
version: '0.0.0-private', version: cliVersion,
contextPackageName: '@ktx/context', contextPackageName: '@ktx/context',
}); });
}); });
@ -55,8 +58,9 @@ describe('getKtxCliPackageInfo', () => {
expect(packageJson).toMatchObject({ expect(packageJson).toMatchObject({
name: '@ktx/cli', name: '@ktx/cli',
version: '0.0.0-private', version: cliVersion,
}); });
expect(cliVersion).toMatch(/^\d+\.\d+\.\d+/);
}); });
it('normalizes public package metadata from package.json contents', () => { it('normalizes public package metadata from package.json contents', () => {
@ -114,7 +118,7 @@ describe('runKtxCli', () => {
await expect(runKtxCli(['--version'], testIo.io)).resolves.toBe(0); await expect(runKtxCli(['--version'], testIo.io)).resolves.toBe(0);
expect(testIo.stdout()).toBe('@ktx/cli 0.0.0-private\n'); expect(testIo.stdout()).toBe(`@ktx/cli ${cliVersion}\n`);
expect(testIo.stderr()).toBe(''); expect(testIo.stderr()).toBe('');
}); });
@ -304,7 +308,7 @@ describe('runKtxCli', () => {
1, 1,
{ {
command: 'install', command: 'install',
cliVersion: '0.0.0-private', cliVersion,
feature: 'local-embeddings', feature: 'local-embeddings',
force: true, force: true,
}, },
@ -314,7 +318,7 @@ describe('runKtxCli', () => {
2, 2,
{ {
command: 'start', command: 'start',
cliVersion: '0.0.0-private', cliVersion,
projectDir: expect.any(String), projectDir: expect.any(String),
feature: 'local-embeddings', feature: 'local-embeddings',
force: true, force: true,
@ -325,7 +329,7 @@ describe('runKtxCli', () => {
3, 3,
{ {
command: 'stop', command: 'stop',
cliVersion: '0.0.0-private', cliVersion,
projectDir: expect.any(String), projectDir: expect.any(String),
all: false, all: false,
}, },
@ -335,7 +339,7 @@ describe('runKtxCli', () => {
4, 4,
{ {
command: 'stop', command: 'stop',
cliVersion: '0.0.0-private', cliVersion,
projectDir: expect.any(String), projectDir: expect.any(String),
all: true, all: true,
}, },
@ -345,7 +349,7 @@ describe('runKtxCli', () => {
5, 5,
{ {
command: 'status', command: 'status',
cliVersion: '0.0.0-private', cliVersion,
json: true, json: true,
}, },
statusIo.io, statusIo.io,
@ -418,7 +422,7 @@ describe('runKtxCli', () => {
expect.objectContaining({ expect.objectContaining({
command: 'query', command: 'query',
projectDir: tempDir, projectDir: tempDir,
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'prompt', runtimeInstallPolicy: 'prompt',
query: expect.objectContaining({ measures: ['orders.order_count'], dimensions: [] }), query: expect.objectContaining({ measures: ['orders.order_count'], dimensions: [] }),
}), }),
@ -433,7 +437,7 @@ describe('runKtxCli', () => {
).resolves.toBe(0); ).resolves.toBe(0);
expect(sl).toHaveBeenLastCalledWith( expect(sl).toHaveBeenLastCalledWith(
expect.objectContaining({ expect.objectContaining({
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'auto', runtimeInstallPolicy: 'auto',
}), }),
autoIo.io, autoIo.io,
@ -449,7 +453,7 @@ describe('runKtxCli', () => {
).resolves.toBe(0); ).resolves.toBe(0);
expect(sl).toHaveBeenLastCalledWith( expect(sl).toHaveBeenLastCalledWith(
expect.objectContaining({ expect.objectContaining({
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'never', runtimeInstallPolicy: 'never',
}), }),
noInputIo.io, noInputIo.io,
@ -585,7 +589,7 @@ describe('runKtxCli', () => {
skipAgents: false, skipAgents: false,
inputMode: 'auto', inputMode: 'auto',
yes: false, yes: false,
cliVersion: '0.0.0-private', cliVersion,
skipLlm: false, skipLlm: false,
skipEmbeddings: false, skipEmbeddings: false,
databaseSchemas: [], databaseSchemas: [],
@ -715,7 +719,7 @@ describe('runKtxCli', () => {
inputMode: 'disabled', inputMode: 'disabled',
depth: 'fast', depth: 'fast',
queryHistory: 'default', queryHistory: 'default',
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'never', runtimeInstallPolicy: 'never',
}, },
testIo.io, testIo.io,
@ -742,7 +746,7 @@ describe('runKtxCli', () => {
inputMode: 'auto', inputMode: 'auto',
depth: 'deep', depth: 'deep',
queryHistory: 'default', queryHistory: 'default',
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'prompt', runtimeInstallPolicy: 'prompt',
}, },
testIo.io, testIo.io,
@ -819,7 +823,7 @@ describe('runKtxCli', () => {
json: false, json: false,
inputMode: 'disabled', inputMode: 'disabled',
queryHistory: 'default', queryHistory: 'default',
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'never', runtimeInstallPolicy: 'never',
}, },
testIo.io, testIo.io,
@ -1124,7 +1128,7 @@ describe('runKtxCli', () => {
command: 'run', command: 'run',
projectDir: tempDir, projectDir: tempDir,
inputMode: 'disabled', inputMode: 'disabled',
cliVersion: '0.0.0-private', cliVersion,
anthropicApiKeyEnv: 'ANTHROPIC_API_KEY', // pragma: allowlist secret anthropicApiKeyEnv: 'ANTHROPIC_API_KEY', // pragma: allowlist secret
llmModel: 'claude-sonnet-4-6', llmModel: 'claude-sonnet-4-6',
skipLlm: false, skipLlm: false,
@ -1163,7 +1167,7 @@ describe('runKtxCli', () => {
command: 'run', command: 'run',
projectDir: tempDir, projectDir: tempDir,
inputMode: 'disabled', inputMode: 'disabled',
cliVersion: '0.0.0-private', cliVersion,
llmBackend: 'vertex', llmBackend: 'vertex',
vertexProject: 'local-gcp-project', vertexProject: 'local-gcp-project',
vertexLocation: 'us-east5', vertexLocation: 'us-east5',
@ -1200,7 +1204,7 @@ describe('runKtxCli', () => {
command: 'run', command: 'run',
projectDir: tempDir, projectDir: tempDir,
inputMode: 'disabled', inputMode: 'disabled',
cliVersion: '0.0.0-private', cliVersion,
llmBackend: 'claude-code', llmBackend: 'claude-code',
llmModel: 'opus', llmModel: 'opus',
skipLlm: false, skipLlm: false,
@ -1308,7 +1312,7 @@ describe('runKtxCli', () => {
projectDir: '/tmp/project', projectDir: '/tmp/project',
inputMode: 'disabled', inputMode: 'disabled',
yes: true, yes: true,
cliVersion: '0.0.0-private', cliVersion,
skipLlm: true, skipLlm: true,
skipEmbeddings: true, skipEmbeddings: true,
databaseDrivers: ['postgres'], databaseDrivers: ['postgres'],
@ -1649,7 +1653,7 @@ describe('runKtxCli', () => {
queryFile: '/tmp/query.json', queryFile: '/tmp/query.json',
execute: false, execute: false,
format: 'json', format: 'json',
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'auto', runtimeInstallPolicy: 'auto',
}, },
autoIo.io, autoIo.io,
@ -1663,7 +1667,7 @@ describe('runKtxCli', () => {
queryFile: '/tmp/query.json', queryFile: '/tmp/query.json',
execute: false, execute: false,
format: 'json', format: 'json',
cliVersion: '0.0.0-private', cliVersion,
runtimeInstallPolicy: 'never', runtimeInstallPolicy: 'never',
}, },
neverIo.io, neverIo.io,

15
scripts/build-public-npm-package.test.mjs
View file

@ -139,12 +139,15 @@ async function writeWorkspaceFixture(root) {
} }
describe('publicNpmPackageLayout', () => { describe('publicNpmPackageLayout', () => {
it('uses the first public npm release version for the tarball name', () => { it('uses the public npm release version for the tarball name', () => {
const layout = publicNpmPackageLayout('/repo/ktx'); const layout = publicNpmPackageLayout('/repo/ktx');
assert.equal(PUBLIC_NPM_PACKAGE_VERSION, '0.1.0-rc.1'); assert.match(PUBLIC_NPM_PACKAGE_VERSION, /^\d+\.\d+\.\d+/);
assert.equal(publicNpmPackageTarballName(), 'kaelio-ktx-0.1.0-rc.1.tgz'); assert.equal(publicNpmPackageTarballName(), `kaelio-ktx-${PUBLIC_NPM_PACKAGE_VERSION}.tgz`);
assert.equal(layout.tarballPath, '/repo/ktx/dist/artifacts/npm/kaelio-ktx-0.1.0-rc.1.tgz'); assert.equal(
layout.tarballPath,
`/repo/ktx/dist/artifacts/npm/kaelio-ktx-${PUBLIC_NPM_PACKAGE_VERSION}.tgz`,
);
}); });
}); });
@ -211,7 +214,7 @@ describe('publicNpmPackageJson', () => {
); );
assert.equal(packageJson.name, PUBLIC_NPM_PACKAGE_NAME); assert.equal(packageJson.name, PUBLIC_NPM_PACKAGE_NAME);
assert.equal(packageJson.version, '0.1.0-rc.1'); assert.equal(packageJson.version, PUBLIC_NPM_PACKAGE_VERSION);
assert.equal(packageJson.private, false); assert.equal(packageJson.private, false);
assert.deepEqual(packageJson.bin, { ktx: './dist/bin.js' }); assert.deepEqual(packageJson.bin, { ktx: './dist/bin.js' });
assert.deepEqual(packageJson.dependencies, { commander: '14.0.3' }); assert.deepEqual(packageJson.dependencies, { commander: '14.0.3' });
@ -275,7 +278,7 @@ describe('publicNpmPackCommand', () => {
'--config.node-linker=hoisted', '--config.node-linker=hoisted',
'pack', 'pack',
'--out', '--out',
'/repo/ktx/dist/artifacts/npm/kaelio-ktx-0.1.0-rc.1.tgz', `/repo/ktx/dist/artifacts/npm/kaelio-ktx-${PUBLIC_NPM_PACKAGE_VERSION}.tgz`,
], ],
cwd: '/repo/ktx/dist/public-npm-package', cwd: '/repo/ktx/dist/public-npm-package',
}); });

4
scripts/build-python-runtime-wheel.test.mjs
View file

@ -52,7 +52,7 @@ describe('runtimeWheelPyproject', () => {
const pyproject = runtimeWheelPyproject(); const pyproject = runtimeWheelPyproject();
assert.match(pyproject, /name = "kaelio-ktx"/); assert.match(pyproject, /name = "kaelio-ktx"/);
assert.match(pyproject, /version = "0\.1\.0rc1"/); assert.match(pyproject, new RegExp(`version = "${RUNTIME_WHEEL_PACKAGE_VERSION.replace(/\./g, '\\.')}"`));
assert.match(pyproject, /ktx-daemon = "ktx_daemon\.__main__:main"/); assert.match(pyproject, /ktx-daemon = "ktx_daemon\.__main__:main"/);
assert.match(pyproject, /packages = \["semantic_layer", "ktx_daemon"\]/); assert.match(pyproject, /packages = \["semantic_layer", "ktx_daemon"\]/);
assert.match(pyproject, /\[project\.optional-dependencies\]/); assert.match(pyproject, /\[project\.optional-dependencies\]/);
@ -110,6 +110,6 @@ describe('runtimeWheelBuildCommand', () => {
cwd: '/repo/ktx', cwd: '/repo/ktx',
}); });
assert.equal(RUNTIME_WHEEL_DISTRIBUTION_NAME, 'kaelio-ktx'); assert.equal(RUNTIME_WHEEL_DISTRIBUTION_NAME, 'kaelio-ktx');
assert.equal(RUNTIME_WHEEL_PACKAGE_VERSION, '0.1.0rc1'); assert.match(RUNTIME_WHEEL_PACKAGE_VERSION, /^\d+\.\d+\.\d+/);
}); });
}); });

13
scripts/local-embeddings-runtime-smoke.test.mjs
View file

@ -2,6 +2,7 @@ import assert from 'node:assert/strict';
import { readFile } from 'node:fs/promises'; import { readFile } from 'node:fs/promises';
import { describe, it } from 'node:test'; import { describe, it } from 'node:test';
import { PUBLIC_NPM_PACKAGE_VERSION } from './build-public-npm-package.mjs';
import { import {
buildLocalEmbeddingsSmokeEnv, buildLocalEmbeddingsSmokeEnv,
expectedPublicKtxVersionPattern, expectedPublicKtxVersionPattern,
@ -12,6 +13,9 @@ import {
validateEmbeddingResponse, validateEmbeddingResponse,
} from './local-embeddings-runtime-smoke.mjs'; } from './local-embeddings-runtime-smoke.mjs';
const PUBLIC_TARBALL_NAME = `kaelio-ktx-${PUBLIC_NPM_PACKAGE_VERSION}.tgz`;
const OTHER_PUBLIC_TARBALL_NAME = 'kaelio-ktx-9.9.9.tgz';
describe('localEmbeddingsSmokeOptIn', () => { describe('localEmbeddingsSmokeOptIn', () => {
it('skips unless the smoke is explicitly enabled', () => { it('skips unless the smoke is explicitly enabled', () => {
assert.deepEqual(localEmbeddingsSmokeOptIn({}, []), { assert.deepEqual(localEmbeddingsSmokeOptIn({}, []), {
@ -35,10 +39,7 @@ describe('localEmbeddingsSmokeOptIn', () => {
describe('publicKtxTarballName', () => { describe('publicKtxTarballName', () => {
it('selects the public @kaelio/ktx tarball name', () => { it('selects the public @kaelio/ktx tarball name', () => {
assert.equal( assert.equal(publicKtxTarballName([PUBLIC_TARBALL_NAME, 'ignore-me.tgz']), PUBLIC_TARBALL_NAME);
publicKtxTarballName(['kaelio-ktx-0.1.0-rc.1.tgz', 'ignore-me.tgz']),
'kaelio-ktx-0.1.0-rc.1.tgz',
);
}); });
it('fails when the public package tarball is missing', () => { it('fails when the public package tarball is missing', () => {
@ -50,7 +51,7 @@ describe('publicKtxTarballName', () => {
it('fails when multiple public package tarballs are present', () => { it('fails when multiple public package tarballs are present', () => {
assert.throws( assert.throws(
() => publicKtxTarballName(['kaelio-ktx-0.1.0-rc.1.tgz', 'kaelio-ktx-0.2.0.tgz']), () => publicKtxTarballName([PUBLIC_TARBALL_NAME, OTHER_PUBLIC_TARBALL_NAME]),
/Expected exactly one @kaelio\/ktx tarball/, /Expected exactly one @kaelio\/ktx tarball/,
); );
}); });
@ -60,7 +61,7 @@ describe('expectedPublicKtxVersionPattern', () => {
it('matches the public package version and rejects the private workspace version', () => { it('matches the public package version and rejects the private workspace version', () => {
const pattern = expectedPublicKtxVersionPattern(); const pattern = expectedPublicKtxVersionPattern();
assert.match('@kaelio/ktx 0.1.0-rc.1\n', pattern); assert.match(`@kaelio/ktx ${PUBLIC_NPM_PACKAGE_VERSION}\n`, pattern);
assert.doesNotMatch('@kaelio/ktx 0.0.0-private\n', pattern); assert.doesNotMatch('@kaelio/ktx 0.0.0-private\n', pattern);
}); });
}); });

43
scripts/package-artifacts.test.mjs
View file

@ -72,6 +72,10 @@ async function writeReleaseMetadataInputs(root) {
} }
} }
function runtimeWheelFilename(version = RUNTIME_WHEEL_PACKAGE_VERSION) {
return `kaelio_ktx-${version}-py3-none-any.whl`;
}
async function writeUploadableArtifactFixtures(layout) { async function writeUploadableArtifactFixtures(layout) {
await mkdir(layout.npmDir, { recursive: true }); await mkdir(layout.npmDir, { recursive: true });
await mkdir(layout.pythonDir, { recursive: true }); await mkdir(layout.pythonDir, { recursive: true });
@ -82,7 +86,7 @@ async function writeUploadableArtifactFixtures(layout) {
`${packageInfo.name}-tarball`, `${packageInfo.name}-tarball`,
]), ]),
[ [
join(layout.pythonDir, 'kaelio_ktx-0.1.0rc1-py3-none-any.whl'), join(layout.pythonDir, runtimeWheelFilename()),
'kaelio-ktx-runtime-wheel', 'kaelio-ktx-runtime-wheel',
], ],
]); ]);
@ -99,7 +103,10 @@ describe('packageArtifactLayout', () => {
assert.equal(layout.artifactDir, '/repo/ktx/dist/artifacts'); assert.equal(layout.artifactDir, '/repo/ktx/dist/artifacts');
assert.equal(layout.npmDir, '/repo/ktx/dist/artifacts/npm'); assert.equal(layout.npmDir, '/repo/ktx/dist/artifacts/npm');
assert.equal(layout.pythonDir, '/repo/ktx/dist/artifacts/python'); assert.equal(layout.pythonDir, '/repo/ktx/dist/artifacts/python');
assert.equal(layout.cliTarball, '/repo/ktx/dist/artifacts/npm/kaelio-ktx-0.1.0-rc.1.tgz'); assert.equal(
layout.cliTarball,
`/repo/ktx/dist/artifacts/npm/kaelio-ktx-${PUBLIC_NPM_PACKAGE_VERSION}.tgz`,
);
assert.deepEqual(Object.keys(layout.npmTarballs), ['@kaelio/ktx']); assert.deepEqual(Object.keys(layout.npmTarballs), ['@kaelio/ktx']);
}); });
}); });
@ -131,7 +138,7 @@ describe('packageReleaseMetadata', () => {
ecosystem: 'npm', ecosystem: 'npm',
packageName: '@kaelio/ktx', packageName: '@kaelio/ktx',
packageRoot: 'packages/cli', packageRoot: 'packages/cli',
packageVersion: '0.1.0-rc.1', packageVersion: PUBLIC_NPM_PACKAGE_VERSION,
private: false, private: false,
releaseMode: 'ci-artifact-only', releaseMode: 'ci-artifact-only',
}, },
@ -139,7 +146,7 @@ describe('packageReleaseMetadata', () => {
ecosystem: 'python', ecosystem: 'python',
packageName: 'kaelio-ktx', packageName: 'kaelio-ktx',
packageRoot: 'python/runtime-wheel', packageRoot: 'python/runtime-wheel',
packageVersion: '0.1.0rc1', packageVersion: RUNTIME_WHEEL_PACKAGE_VERSION,
private: false, private: false,
releaseMode: 'ci-artifact-only', releaseMode: 'ci-artifact-only',
}, },
@ -154,10 +161,10 @@ describe('findPythonArtifacts', () => {
it('finds the bundled runtime wheel only', async () => { it('finds the bundled runtime wheel only', async () => {
const root = await mkdtemp(join(tmpdir(), 'ktx-artifacts-test-')); const root = await mkdtemp(join(tmpdir(), 'ktx-artifacts-test-'));
try { try {
await writeFile(join(root, 'kaelio_ktx-0.1.0rc1-py3-none-any.whl'), ''); await writeFile(join(root, runtimeWheelFilename()), '');
assert.deepEqual(await findPythonArtifacts(root), { assert.deepEqual(await findPythonArtifacts(root), {
runtimeWheel: join(root, 'kaelio_ktx-0.1.0rc1-py3-none-any.whl'), runtimeWheel: join(root, runtimeWheelFilename()),
}); });
} finally { } finally {
await rm(root, { recursive: true, force: true }); await rm(root, { recursive: true, force: true });
@ -197,7 +204,7 @@ describe('artifact manifest', () => {
ecosystem: 'npm', ecosystem: 'npm',
packageName: '@kaelio/ktx', packageName: '@kaelio/ktx',
packageRoot: 'packages/cli', packageRoot: 'packages/cli',
packageVersion: '0.1.0-rc.1', packageVersion: PUBLIC_NPM_PACKAGE_VERSION,
private: false, private: false,
releaseMode: 'ci-artifact-only', releaseMode: 'ci-artifact-only',
}, },
@ -210,7 +217,7 @@ describe('artifact manifest', () => {
ecosystem: 'python', ecosystem: 'python',
packageName: 'kaelio-ktx', packageName: 'kaelio-ktx',
packageRoot: 'python/runtime-wheel', packageRoot: 'python/runtime-wheel',
packageVersion: '0.1.0rc1', packageVersion: RUNTIME_WHEEL_PACKAGE_VERSION,
private: false, private: false,
releaseMode: 'ci-artifact-only', releaseMode: 'ci-artifact-only',
}, },
@ -232,8 +239,8 @@ describe('artifact manifest', () => {
artifactKind: 'tarball', artifactKind: 'tarball',
ecosystem: 'npm', ecosystem: 'npm',
packageName: '@kaelio/ktx', packageName: '@kaelio/ktx',
packageVersion: '0.1.0-rc.1', packageVersion: PUBLIC_NPM_PACKAGE_VERSION,
path: 'npm/kaelio-ktx-0.1.0-rc.1.tgz', path: `npm/kaelio-ktx-${PUBLIC_NPM_PACKAGE_VERSION}.tgz`,
}, },
], ],
); );
@ -252,13 +259,15 @@ describe('artifact manifest', () => {
artifactKind: 'wheel', artifactKind: 'wheel',
ecosystem: 'python', ecosystem: 'python',
packageName: 'kaelio-ktx', packageName: 'kaelio-ktx',
packageVersion: '0.1.0rc1', packageVersion: RUNTIME_WHEEL_PACKAGE_VERSION,
path: 'python/kaelio_ktx-0.1.0rc1-py3-none-any.whl', path: `python/${runtimeWheelFilename()}`,
}, },
], ],
); );
const npmEntry = manifest.files.find((file) => file.path === 'npm/kaelio-ktx-0.1.0-rc.1.tgz'); const npmEntry = manifest.files.find(
(file) => file.path === `npm/kaelio-ktx-${PUBLIC_NPM_PACKAGE_VERSION}.tgz`,
);
assert.ok(npmEntry); assert.ok(npmEntry);
assert.equal(npmEntry.bytes, Buffer.byteLength('@kaelio/ktx-tarball')); assert.equal(npmEntry.bytes, Buffer.byteLength('@kaelio/ktx-tarball'));
assert.equal(npmEntry.sha256, createHash('sha256').update('@kaelio/ktx-tarball').digest('hex')); assert.equal(npmEntry.sha256, createHash('sha256').update('@kaelio/ktx-tarball').digest('hex'));
@ -362,17 +371,17 @@ describe('copyRuntimeWheelAssets', () => {
try { try {
await mkdir(layout.pythonDir, { recursive: true }); await mkdir(layout.pythonDir, { recursive: true });
await writeFile( await writeFile(
join(layout.pythonDir, 'kaelio_ktx-0.1.0rc1-py3-none-any.whl'), join(layout.pythonDir, runtimeWheelFilename()),
'kaelio-ktx-runtime-wheel', 'kaelio-ktx-runtime-wheel',
); );
const assets = await copyRuntimeWheelAssets(layout, { const assets = await copyRuntimeWheelAssets(layout, {
runtimeWheel: join(layout.pythonDir, 'kaelio_ktx-0.1.0rc1-py3-none-any.whl'), runtimeWheel: join(layout.pythonDir, runtimeWheelFilename()),
}); });
assert.equal( assert.equal(
assets.wheelPath, assets.wheelPath,
join(root, 'packages', 'cli', 'assets', 'python', 'kaelio_ktx-0.1.0rc1-py3-none-any.whl'), join(root, 'packages', 'cli', 'assets', 'python', runtimeWheelFilename()),
); );
assert.equal( assert.equal(
assets.manifestPath, assets.manifestPath,
@ -385,7 +394,7 @@ describe('copyRuntimeWheelAssets', () => {
normalizedName: RUNTIME_WHEEL_NORMALIZED_NAME, normalizedName: RUNTIME_WHEEL_NORMALIZED_NAME,
version: RUNTIME_WHEEL_PACKAGE_VERSION, version: RUNTIME_WHEEL_PACKAGE_VERSION,
wheel: { wheel: {
file: 'kaelio_ktx-0.1.0rc1-py3-none-any.whl', file: runtimeWheelFilename(),
sha256: createHash('sha256') sha256: createHash('sha256')
.update('kaelio-ktx-runtime-wheel') .update('kaelio-ktx-runtime-wheel')
.digest('hex'), .digest('hex'),

6
scripts/release-readiness.test.mjs
View file

@ -11,6 +11,7 @@ import {
writeArtifactManifest, writeArtifactManifest,
} from './package-artifacts.mjs'; } from './package-artifacts.mjs';
import { PUBLIC_NPM_PACKAGE_VERSION } from './build-public-npm-package.mjs'; import { PUBLIC_NPM_PACKAGE_VERSION } from './build-public-npm-package.mjs';
import { RUNTIME_WHEEL_PACKAGE_VERSION } from './build-python-runtime-wheel.mjs';
import { readReleasePolicy, releasePolicyPath, releaseReadinessReport } from './release-readiness.mjs'; import { readReleasePolicy, releasePolicyPath, releaseReadinessReport } from './release-readiness.mjs';
async function writeJson(path, value) { async function writeJson(path, value) {
@ -37,7 +38,10 @@ async function writeUploadableArtifactFixtures(layout) {
layout.npmTarballs[packageInfo.name], layout.npmTarballs[packageInfo.name],
`${packageInfo.name}-tarball`, `${packageInfo.name}-tarball`,
]), ]),
[join(layout.pythonDir, 'kaelio_ktx-0.1.0rc1-py3-none-any.whl'), 'kaelio-ktx-runtime-wheel'], [
join(layout.pythonDir, `kaelio_ktx-${RUNTIME_WHEEL_PACKAGE_VERSION}-py3-none-any.whl`),
'kaelio-ktx-runtime-wheel',
],
]); ]);
for (const [path, contents] of fileContents) { for (const [path, contents] of fileContents) {