feat: merge ingest and scan

* docs: add CLI component reuse guidance

* docs: add unified ingest ux design

* Refine unified ingest UX design after adversarial review iteration 1

* Refine unified ingest UX design after adversarial review iteration 2

* Refine unified ingest UX design after adversarial review iteration 3

* feat(cli): route public connection ingest command

* feat(cli): hide standalone scan from public help

* feat(cli): plan public ingest depth and query history

* feat(cli): execute public database ingest facets

* feat(ingest): read connection query history config

* fix(cli): use public ingest wording

* fix(config): stop generating ingest adapter allow lists

* docs: document public ingest command

* test: align ingest surface expectations

* docs: add unified ingest public CLI surface plan

* feat(cli): preflight deep public ingest readiness

* feat(setup): store query history in connection context

* feat(setup): store database context depth

* feat(setup): verify context readiness by database depth

* fix(setup): keep context build foreground only

* fix(config): reject reserved ingest connection ids

* test: close unified ingest v1 expectations

* docs: add unified ingest v1 closure plan

* fix(ingest): bypass adapter allow-list for public source ingest

* fix(ingest): honor query history window intent

* fix(ingest): hide scan internals from public database ingest

* feat(ingest): use foreground view for interactive public ingest

* fix(setup): use schema context and query history wording

* test(cli): verify unified ingest public output

* docs: add unified ingest v1 public output closure plan

* fix(setup): forward query history flags

* fix(setup): prompt for postgres query history

* fix(status): report query history readiness

* fix(ingest): remove legacy public guidance

* fix(ingest): polish foreground retry copy

* docs(examples): use unified query history wording

* chore(ingest): finish public query history cleanup

* docs: add unified ingest v1 query history status cleanup plan

* test(docs): cover unified ingest public docs

* docs: align ingest CLI reference with unified UX

* docs: update context build guides for unified ingest

* docs: update setup and primary source ingest wording

* docs: stop advertising adapter-backed example ingest

* docs: close unified ingest public docs gaps

* docs: add unified ingest v1 docs site closure plan

* fix: render unified ingest foreground warnings

* fix: explain query history schema order

* fix: add public ingest retry guidance

* fix: align setup next steps with unified ingest

* fix: remove scan wording from demo progress

* test: verify unified ingest ux closure

* docs: add unified ingest v1 foreground and retry closure plan

* fix(cli): preserve query-history pull config in public ingest

* fix(cli): omit hidden commands from docs command tree

* test(cli): close unified ingest final public surface checks

* docs: add unified ingest v1 final public surface closure plan

* fix(cli): use public source labels in ingest reports

* fix(cli): suppress low-level public ingest output

* test(cli): verify unified ingest public plain output

* docs: add unified ingest v1 public plain output closure plan

* fix(cli): add public ingest copy sanitizers

* fix(cli): sanitize public ingest progress copy

* fix(cli): rename setup schema scope prompt

* docs(plan): add progress copy closure; test: align setup back-nav fixture

Adds the iter9 plan and updates the setup back-navigation test fixture
to pass disableQueryHistory plus listSchemas/listTables stubs that the
unified ingest setup step now requires.

* docs(plan): add final ux labels plan with narrowed label scans

* fix(cli): aggregate unsupported query-history warnings

* fix(cli): align setup database labels

* test(cli): fix setup database test type-check

* fix(cli): remove primary-source wording from setup output

* test(cli): verify unified ingest setup closure

* docs(plan): add unified ingest v1 verification copy closure plan

* fix(cli): remove top-level scan command

* fix(cli): remove legacy ingest and wiki commands

* Merge scan into ingest flow

* feat(cli): split ingest progress into per-phase rows, rename work units to tasks

Each database target in the unified ingest dashboard now renders one row per
real subprocess (Schema, then Query history when enabled) instead of a single
combined bar. Each phase has its own monotonic 0-100% bar so the progress
never snaps back to zero when historic-sql starts after scan completes.
Completed phases keep their final bar, summary, and elapsed time visible as
an inline audit trail; queued and skipped phases are shown explicitly.

Also rename user-facing "work units" / "Failed work units" to "tasks" /
"Failed tasks" in ingest output and parseIngestSummary. The parser still
accepts the legacy "Work units:" wording in captured output for backward
compat. Internal memory-flow event names and type fields are left alone.

* Fix test harness failures

* Fix CI smoke checks

---------

Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
This commit is contained in:
Andrey Avtomonov 2026-05-14 01:43:06 +02:00 committed by GitHub
parent 1a472cf3ed
commit b00c1a11a9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
118 changed files with 16890 additions and 2992 deletions

View file

@ -63,11 +63,17 @@ describe('standalone example docs', () => {
const smoke = await readText('examples/postgres-historic/scripts/smoke.sh');
assert.match(examples, /postgres-historic/);
assert.match(examples, /unified Historic SQL artifacts/);
assert.match(readme, /--enable-historic-sql/);
assert.match(readme, /--historic-sql-min-executions 2/);
assert.doesNotMatch(examples, /Historic SQL/);
assert.doesNotMatch(examples, /historic-SQL/);
assert.match(examples, /query-history ingest via `pg_stat_statements`/);
assert.doesNotMatch(readme, new RegExp(['--enable-historic', 'sql'].join('-')));
assert.doesNotMatch(readme, new RegExp(['--historic', 'sql-min-executions'].join('-')));
assert.doesNotMatch(readme, /ktx ingest run --project-dir/);
assert.doesNotMatch(readme, /--adapter historic-sql/);
assert.match(readme, /--enable-query-history/);
assert.match(readme, /--query-history-min-executions 2/);
assert.match(readme, /ktx status --project-dir/);
assert.match(readme, /Postgres Historic SQL/);
assert.match(readme, /Postgres query history/);
assert.match(readme, /manifest\.json/);
assert.match(readme, /tables\/\*\.json/);
assert.match(readme, /patterns-input\.json/);
@ -89,7 +95,7 @@ describe('standalone example docs', () => {
assert.match(smoke, /historic-sql-patterns-part-/);
assert.match(smoke, /patterns-input\/part-/);
assert.doesNotMatch(smoke, new RegExp(["unitKey === 'historic", 'sql', "patterns'"].join('-')));
assert.match(smoke, /--historic-sql-min-executions 2/);
assert.match(smoke, /--query-history-min-executions 2/);
assert.match(smoke, /KTX_RUNTIME_ROOT/);
assert.match(smoke, /managedDaemon/);
assert.match(smoke, /installPolicy: 'auto'/);
@ -129,6 +135,15 @@ describe('standalone example docs', () => {
);
});
it('checked-in example configs do not include public database adapters', async () => {
const localWarehouseConfig = await readFile('examples/local-warehouse/ktx.yaml', 'utf8');
const orbitConfig = await readFile('examples/orbit-relationship-verification/ktx.yaml', 'utf8');
const legacyPublicAdapter = new RegExp(['live', 'database'].join('-'));
assert.doesNotMatch(localWarehouseConfig, legacyPublicAdapter);
assert.doesNotMatch(orbitConfig, legacyPublicAdapter);
});
it('lists every workspace package in the contributor docs', async () => {
const contributing = await readText('docs-site/content/docs/community/contributing.mdx');
@ -222,18 +237,64 @@ describe('standalone example docs', () => {
assert.doesNotMatch(readme, /python -m ktx_daemon semantic-validate/);
});
it('documents scan workflows in the docs site', async () => {
it('documents unified public ingest workflows in the docs site', async () => {
const rootReadme = await readText('README.md');
const cliMeta = await readText('docs-site/content/docs/cli-reference/meta.json');
const ingestReference = await readText('docs-site/content/docs/cli-reference/ktx-ingest.mdx');
const devReference = await readText('docs-site/content/docs/cli-reference/ktx-dev.mdx');
const setupReference = await readText('docs-site/content/docs/cli-reference/ktx-setup.mdx');
const buildingContext = await readText('docs-site/content/docs/guides/building-context.mdx');
const scanReference = await readText('docs-site/content/docs/cli-reference/ktx-scan.mdx');
const contextSources = await readText('docs-site/content/docs/integrations/context-sources.mdx');
const contextAsCode = await readText('docs-site/content/docs/concepts/context-as-code.mdx');
const quickstart = await readText('docs-site/content/docs/getting-started/quickstart.mdx');
const primarySources = await readText('docs-site/content/docs/integrations/primary-sources.mdx');
const examplesIndex = await readText('examples/README.md');
const localWarehouseReadme = await readText('examples/local-warehouse/README.md');
assert.match(ingestReference, /ktx ingest <connectionId>/);
assert.match(ingestReference, /ktx ingest --all --deep/);
assert.match(ingestReference, /--query-history-window-days <days>/);
assert.match(buildingContext, /ktx ingest <connection-id>/);
assert.match(buildingContext, /ktx ingest --all/);
assert.match(contextSources, /ktx ingest <connectionId>/);
assert.match(contextAsCode, /ktx ingest --all --no-input/);
assert.match(quickstart, /schema context/);
assert.match(primarySources, /context:\n queryHistory:/);
assert.match(rootReadme, /Databases configured: yes \(postgres-warehouse\)/);
assert.match(quickstart, /Databases:\n postgres-warehouse: deep context complete/);
assert.match(quickstart, /Databases configured: yes \(postgres-warehouse\)/);
assert.match(setupReference, /Databases configured: yes \(postgres-warehouse\)/);
assert.doesNotMatch(rootReadme, new RegExp(['Primary sources', 'configured'].join(' ')));
assert.doesNotMatch(quickstart, new RegExp(['Primary', 'sources'].join(' ')));
assert.doesNotMatch(setupReference, new RegExp(['Primary sources', 'configured'].join(' ')));
assert.doesNotMatch(cliMeta, /ktx-scan/);
assert.doesNotMatch(ingestReference, /ktx ingest run/);
assert.doesNotMatch(ingestReference, /ktx ingest status/);
assert.doesNotMatch(ingestReference, /ktx ingest replay/);
assert.doesNotMatch(ingestReference, /--adapter/);
assert.doesNotMatch(ingestReference, /ktx ingest watch/);
assert.doesNotMatch(ingestReference, /live-database/);
assert.doesNotMatch(devReference, /ktx scan/);
assert.doesNotMatch(buildingContext, /ktx ingest watch/);
assert.doesNotMatch(buildingContext, /ktx ingest status/);
assert.doesNotMatch(buildingContext, /ktx ingest replay/);
assert.doesNotMatch(buildingContext, /historic-sql/);
assert.doesNotMatch(buildingContext, /live-database/);
assert.doesNotMatch(contextSources, /ktx ingest run --connection-id/);
assert.doesNotMatch(contextSources, /--adapter <adapter>/);
assert.doesNotMatch(contextAsCode, /ktx ingest run --connection-id/);
assert.doesNotMatch(quickstart, /Historic SQL/);
assert.doesNotMatch(quickstart, /--enable-historic-sql/);
assert.doesNotMatch(quickstart, /press <kbd>d<\/kbd> to detach/);
assert.doesNotMatch(primarySources, /historicSql/);
assert.doesNotMatch(primarySources, /Historic SQL/);
assert.doesNotMatch(examplesIndex, /ktx ingest run --project-dir/);
assert.doesNotMatch(localWarehouseReadme, /ktx ingest run --project-dir/);
assert.match(buildingContext, /ktx scan <connection-id>/);
assert.match(buildingContext, /ktx status/);
assert.doesNotMatch(buildingContext, /ktx scan status <run-id>/);
assert.doesNotMatch(buildingContext, /ktx scan report <run-id>/);
assert.match(scanReference, /ktx scan <connectionId> \[options\]/);
assert.match(rootReadme, /raw-sources\//);
assert.match(rootReadme, /live-database\//);
assert.doesNotMatch(rootReadme, new RegExp(`${['live', 'database'].join('-')}/`));
assert.doesNotMatch(rootReadme, /ktx scan/);
assert.doesNotMatch(rootReadme, /Run a local ingest smoke test/);
assert.doesNotMatch(rootReadme, /ktx ingest run --project-dir/);
assert.doesNotMatch(rootReadme, /ktx ingest status --project-dir/);

View file

@ -95,34 +95,23 @@ export function buildKtxYaml(postgresUrl) {
'storage:',
' state: sqlite',
' search: sqlite-fts5',
'ingest:',
' adapters:',
' - live-database',
'',
].join('\n');
}
export function buildLiveDatabaseIngestArgs(projectDir, databaseIntrospectionUrl) {
export function buildLiveDatabaseIngestArgs(projectDir, _databaseIntrospectionUrl, connectionId = 'warehouse') {
return [
'exec',
'ktx',
'ingest',
'run',
connectionId,
'--project-dir',
projectDir,
'--connection-id',
'warehouse',
'--adapter',
'live-database',
'--database-introspection-url',
databaseIntrospectionUrl,
'--fast',
'--no-input',
];
}
export function buildLiveDatabaseStatusArgs(projectDir, runId) {
return ['exec', 'ktx', 'ingest', 'status', '--project-dir', projectDir, runId];
}
async function run(command, args, options = {}) {
process.stdout.write(`$ ${command} ${args.join(' ')}\n`);
return new Promise((resolve) => {
@ -173,7 +162,7 @@ function requireOutput(label, result, pattern) {
function getRunId(stdout) {
const match = stdout.match(/^Run: (.+)$/m);
if (!match) {
throw new Error(`ingest run output did not include a run id\nstdout:\n${stdout}`);
throw new Error(`ingest output did not include a run id\nstdout:\n${stdout}`);
}
return match[1];
}
@ -323,24 +312,11 @@ async function main() {
env: managedRuntimeEnv(cleanInstallDir),
timeout: 120_000,
});
requireSuccess('ktx ingest run live-database', ingestRun);
requireOutput('ktx ingest run live-database', ingestRun, /Status: done/);
requireOutput('ktx ingest run live-database', ingestRun, /Adapter: live-database/);
requireOutput('ktx ingest run live-database', ingestRun, /Diff: \+4\/~0\/-0\/=0/);
requireOutput('ktx ingest run live-database', ingestRun, /Raw files: 4/);
requireOutput('ktx ingest run live-database', ingestRun, /Work units: 2/);
requireSuccess('ktx ingest warehouse --fast', ingestRun);
requireOutput('ktx ingest warehouse --fast', ingestRun, /Ingest finished/);
requireOutput('ktx ingest warehouse --fast', ingestRun, /Database schema/);
const runId = getRunId(ingestRun.stdout);
const ingestStatus = await run('pnpm', buildLiveDatabaseStatusArgs(projectDir, runId), {
cwd: cleanInstallDir,
env: managedRuntimeEnv(cleanInstallDir),
timeout: 30_000,
});
requireSuccess('ktx ingest status live-database', ingestStatus);
requireOutput('ktx ingest status live-database', ingestStatus, new RegExp(`Run: ${runId}`));
requireOutput('ktx ingest status live-database', ingestStatus, /Status: done/);
requireOutput('ktx ingest status live-database', ingestStatus, /Raw files: 4/);
requireOutput('ktx ingest status live-database', ingestStatus, /Work units: 2/);
await assertPathExists(join(projectDir, '.ktx', 'db.sqlite'), 'SQLite local ingest state');
process.stdout.write(`Installed live-database artifact smoke passed: ${runId}\n`);
} finally {

View file

@ -5,7 +5,6 @@ import {
buildDockerRunArgs,
buildKtxYaml,
buildLiveDatabaseIngestArgs,
buildLiveDatabaseStatusArgs,
buildPostgresUrl,
buildPostgresReadyArgs,
buildSeedSql,
@ -50,7 +49,7 @@ describe('installed live-database artifact smoke helpers', () => {
);
});
it('writes a live-database-only KTX project config with SQLite local state', () => {
it('writes a public database ingest KTX project config with SQLite local state', () => {
assert.equal(
buildKtxYaml('postgresql://ktx:postgres@127.0.0.1:15432/warehouse'), // pragma: allowlist secret
[
@ -62,9 +61,6 @@ describe('installed live-database artifact smoke helpers', () => {
'storage:',
' state: sqlite',
' search: sqlite-fts5',
'ingest:',
' adapters:',
' - live-database',
'',
].join('\n'),
);
@ -97,30 +93,17 @@ describe('installed live-database artifact smoke helpers', () => {
]);
});
it('builds installed CLI live-database ingest and status commands', () => {
it('builds the installed CLI public database ingest command', () => {
assert.deepEqual(buildLiveDatabaseIngestArgs('/tmp/project', 'http://127.0.0.1:8765'), [
'exec',
'ktx',
'ingest',
'run',
'warehouse',
'--project-dir',
'/tmp/project',
'--connection-id',
'warehouse',
'--adapter',
'live-database',
'--database-introspection-url',
'http://127.0.0.1:8765',
'--fast',
'--no-input',
]);
assert.deepEqual(buildLiveDatabaseStatusArgs('/tmp/project', 'local-run-1'), [
'exec',
'ktx',
'ingest',
'status',
'--project-dir',
'/tmp/project',
'local-run-1',
]);
});
});

View file

@ -518,7 +518,7 @@ function requireSuccess(label, result) {
assert.equal(result.stderr, '', label + ' wrote unexpected stderr');
}
function requireProjectStderr(label, result, projectDir) {
function requireSuccessWithProjectStderr(label, result, projectDir) {
assert.equal(
result.code,
0,
@ -527,6 +527,15 @@ function requireProjectStderr(label, result, projectDir) {
assert.equal(result.stderr, 'Project: ' + projectDir + '\\n', label + ' wrote unexpected stderr');
}
function requireExitCodeWithProjectStderr(label, result, projectDir, expectedCode) {
assert.equal(
result.code,
expectedCode,
label + ' failed with code ' + result.code + '\\nstdout:\\n' + result.stdout + '\\nstderr:\\n' + result.stderr,
);
assert.equal(result.stderr, 'Project: ' + projectDir + '\\n', label + ' wrote unexpected stderr');
}
function requireSuccessWithStderr(label, result, stderrPattern) {
assert.equal(
result.code,
@ -559,12 +568,6 @@ function requireIncludes(values, expected, label) {
assert.ok(values.includes(expected), label + ' did not include ' + expected + ': ' + values.join(', '));
}
function getRunId(stdout) {
const match = stdout.match(/^Run: (.+)$/m);
assert.ok(match, 'ingest run output did not include a run id');
return match[1];
}
async function writeSqliteWarehouse(projectDir) {
const database = new DatabaseSync(join(projectDir, 'warehouse.db'));
try {
@ -588,7 +591,6 @@ process.env.KTX_RUNTIME_ROOT = join(root, 'managed-runtime');
let daemonStarted = false;
try {
const projectDir = join(root, 'project');
const sourceDir = join(root, 'source');
const version = await run('pnpm', ['exec', 'ktx', '--version']);
requireSuccess('ktx public package version', version);
@ -619,7 +621,6 @@ try {
'--skip-agents',
]);
requireSuccess('ktx setup', init);
requireOutput('ktx setup', init, /Project: /);
const emptyProjectDir = join(root, 'empty-project');
const emptyInit = await run('pnpm', [
@ -652,10 +653,6 @@ try {
'scan:',
' enrichment:',
' mode: deterministic',
'ingest:',
' adapters:',
' - fake',
' - live-database',
'',
].join('\\n'),
'utf-8',
@ -818,52 +815,32 @@ try {
requireOutput('ktx dev runtime stop', runtimeStop, /Stopped KTX Python daemon/);
process.stdout.write('ktx dev runtime daemon lifecycle verified\\n');
const structuralScan = await run('pnpm', ['exec', 'ktx', 'scan', 'warehouse',
const structuralScan = await run('pnpm', ['exec', 'ktx', 'ingest', 'warehouse',
'--project-dir',
projectDir,
'--fast',
'--no-input',
]);
requireProjectStderr('ktx scan structural', structuralScan, projectDir);
requireOutput('ktx scan structural', structuralScan, /Status: done/);
requireOutput('ktx scan structural', structuralScan, /Mode: structural/);
requireOutput('ktx scan structural', structuralScan, /Needs attention\\s+None/);
const structuralScanRunId = getRunId(structuralScan.stdout);
requireSuccessWithProjectStderr('ktx ingest fast', structuralScan, projectDir);
requireOutput('ktx ingest fast', structuralScan, /Ingest finished/);
requireOutput('ktx ingest fast', structuralScan, /Database schema/);
requireOutput('ktx ingest fast', structuralScan, /warehouse\\s+done/);
await access(join(projectDir, 'semantic-layer', 'warehouse', '_schema', 'public.yaml'));
process.stdout.write('ktx scan structural verified: ' + structuralScanRunId + '\\n');
process.stdout.write('ktx ingest fast verified\\n');
const enrichedScan = await run('pnpm', ['exec', 'ktx', 'scan', 'warehouse',
const enrichedScan = await run('pnpm', ['exec', 'ktx', 'ingest', 'warehouse',
'--project-dir',
projectDir,
'--mode',
'enriched',
'--deep',
'--no-input',
]);
requireProjectStderr('ktx scan enriched', enrichedScan, projectDir);
requireOutput('ktx scan enriched', enrichedScan, /Status: done/);
requireOutput('ktx scan enriched', enrichedScan, /Mode: enriched/);
requireOutput('ktx scan enriched', enrichedScan, /Enrichment artifacts:/);
const enrichedScanRunId = getRunId(enrichedScan.stdout);
process.stdout.write('ktx scan enriched verified: ' + enrichedScanRunId + '\\n');
await mkdir(join(sourceDir, 'orders'), { recursive: true });
await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\\n', 'utf-8');
const ingestRun = await run('pnpm', ['exec', 'ktx', 'ingest', 'run',
'--project-dir',
projectDir,
'--connection-id',
'warehouse',
'--adapter',
'fake',
'--source-dir',
sourceDir,
]);
assert.equal(ingestRun.code, 1, 'ktx ingest run without an LLM provider must fail');
assert.match(
ingestRun.stderr,
/ktx ingest run requires llm\\.provider\\.backend: anthropic, vertex, or gateway, or an injected agentRunner/,
);
requireExitCodeWithProjectStderr('ktx ingest deep readiness guard', enrichedScan, projectDir, 1);
requireOutput('ktx ingest deep readiness guard', enrichedScan, /Ingest finished with partial failures/);
requireOutput('ktx ingest deep readiness guard', enrichedScan, /requires deep ingest readiness/);
process.stdout.write('ktx ingest deep readiness guard verified\\n');
await access(join(projectDir, '.ktx', 'db.sqlite'));
process.stdout.write('ktx ingest provider guard verified\\n');
process.stdout.write('ktx ingest state verified\\n');
} finally {
if (daemonStarted) {
await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'stop']);
@ -939,7 +916,7 @@ try {
assert.ok([0, 1].includes(doctor.code), 'ktx status setup exit code must be 0 or 1');
requireStdout('ktx status setup', doctor, /KTX status/);
requireStdout('ktx status setup', doctor, /No project here yet\\./);
requireStdout('ktx status setup', doctor, /Before you can run ktx setup/);
requireStdout('ktx status setup', doctor, /ktx setup/);
requireStdout('ktx status setup', doctor, /Node 22\\+/);
assert.equal(doctor.stderr, '', 'ktx status setup wrote unexpected stderr');
} finally {

View file

@ -464,7 +464,7 @@ describe('verification snippets', () => {
assert.match(source, /node:sqlite/);
assert.match(source, /driver: sqlite/);
assert.match(source, /path: warehouse\.db/);
assert.match(source, /live-database/);
assert.doesNotMatch(source, /live-database/);
assert.match(source, /'--execute'/);
assert.match(source, /"mode": "compile_only"/);
assert.match(source, /"mode": "executed"/);
@ -488,18 +488,18 @@ describe('verification snippets', () => {
assert.match(source, /ktx dev runtime stop/);
assert.doesNotMatch(source, /ktx dev runtime prune/);
assert.doesNotMatch(source, /staleRuntimeDir/);
assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'scan',\s*'warehouse'/);
assert.match(source, /'--mode',\s*'enriched'/);
assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'ingest',\s*'warehouse'/);
assert.match(source, /'--deep'/);
assert.doesNotMatch(source, /'--enrich'/);
assert.match(source, /ktx scan structural verified/);
assert.match(source, /ktx scan enriched verified/);
assert.match(source, /ktx ingest fast verified/);
assert.match(source, /ktx ingest deep readiness guard verified/);
assert.match(source, /enrichment:/);
assert.match(source, /mode: deterministic/);
assert.match(source, /run\('pnpm', \['exec', 'ktx', 'ingest', 'run'/);
assert.doesNotMatch(source, /run\('pnpm', \['exec', 'ktx', 'ingest', 'run'/);
assert.match(source, /access\(join\(projectDir, '\.ktx', 'db\.sqlite'\)\)/);
assert.match(source, /SQLite wiki index/);
assert.match(source, /ktx ingest run requires llm\\.provider\\.backend: anthropic, vertex, or gateway/);
assert.match(source, /ktx ingest provider guard verified/);
assert.doesNotMatch(source, /ktx ingest run requires llm\\.provider\\.backend: anthropic, vertex, or gateway/);
assert.match(source, /ktx ingest state verified/);
});
describe('npmCliSmokeSource', () => {
@ -511,6 +511,8 @@ describe('verification snippets', () => {
assert.match(source, /Usage: ktx setup/);
assert.doesNotMatch(source, new RegExp(["'demo'", "'--mode'", "'deterministic'"].join(', ')));
assert.match(source, /'status', '--verbose', '--no-input'/);
assert.match(source, /KTX status/);
assert.match(source, /No project here yet/);
assert.doesNotMatch(source, /function requireProjectStderr/);
assert.match(source, /Object\.keys\(packageJson\.dependencies\)/);
assert.match(source, /'@kaelio\/ktx'/);

View file

@ -41,8 +41,8 @@ export function defaultOrbitVerificationProjectDir() {
return defaultProjectDir;
}
function shellCommand(argv) {
return ['pnpm', 'run', 'ktx', '--', ...argv].join(' ');
function internalScanCommand(input) {
return `internal runKtxScan connection=${input.connectionId} mode=relationships projectDir=${input.projectDir}`;
}
function firstNonEmptyLine(...values) {
@ -55,7 +55,7 @@ function firstNonEmptyLine(...values) {
return line;
}
}
return 'Orbit scan command failed before producing diagnostic output';
return 'Orbit relationship scan failed before producing diagnostic output';
}
function parseArgs(argv) {
@ -88,8 +88,15 @@ function parseArgs(argv) {
return options;
}
export function buildOrbitScanArgv(input) {
return ['scan', input.connectionId, '--mode', 'relationships', '--project-dir', input.projectDir];
export function buildOrbitScanArgs(input) {
return {
command: 'run',
projectDir: input.projectDir,
connectionId: input.connectionId,
mode: 'relationships',
detectRelationships: true,
dryRun: false,
};
}
export function extractRunId(stdout) {
@ -171,7 +178,7 @@ function formatBlocked(result) {
'',
'## Evidence',
'',
'- Orbit verification was not executed because the current local Orbit scan command failed.',
'- Orbit verification was not executed because the current local Orbit relationship scan failed.',
'- Re-run with `--report-path` to write verification evidence to a custom location.',
'',
'Scan stdout:',
@ -228,6 +235,36 @@ async function runBufferedWorkspaceKtx(runner, argv, rootDir, execFile) {
};
}
function cliScanModulePath(rootDir) {
return resolve(rootDir, 'packages/cli/dist/scan.js');
}
async function loadRunKtxScan(rootDir) {
const module = await import(pathToFileURL(cliScanModulePath(rootDir)).href);
return module.runKtxScan;
}
async function runBufferedInternalScan(input) {
const stdout = new BufferWriter();
const stderr = new BufferWriter();
let runKtxScan = input.runKtxScan;
if (!runKtxScan) {
const build = await runBufferedWorkspaceKtx(input.runner, ['--version'], input.rootDir, input.execFile);
if (build.exitCode !== 0) {
return build;
}
runKtxScan = await loadRunKtxScan(input.rootDir);
}
const exitCode = await runKtxScan(input.scanArgs, { stdout, stderr });
return {
exitCode,
stdout: stdout.text(),
stderr: stderr.text(),
};
}
function orbitVerificationEnv(projectDir) {
if (projectDir !== defaultProjectDir) {
return process.env;
@ -253,8 +290,15 @@ export async function runOrbitVerification(options = {}) {
const env = options.env ?? orbitVerificationEnv(projectDir);
const runWithEnv = (argv, runnerOptions) => runner(argv, { ...runnerOptions, env });
const scanArgv = buildOrbitScanArgv({ connectionId, projectDir });
const scan = await runBufferedWorkspaceKtx(runWithEnv, scanArgv, rootDir, execFile);
const scanArgs = buildOrbitScanArgs({ connectionId, projectDir });
const scanCommand = internalScanCommand({ connectionId, projectDir });
const scan = await runBufferedInternalScan({
scanArgs,
rootDir,
execFile,
runner: runWithEnv,
runKtxScan: options.runKtxScan,
});
let result;
if (scan.exitCode !== 0) {
@ -263,7 +307,7 @@ export async function runOrbitVerification(options = {}) {
date,
connectionId,
projectDir,
scanCommand: shellCommand(scanArgv),
scanCommand,
scanExitCode: scan.exitCode,
blocker: firstNonEmptyLine(scan.stderr, scan.stdout),
scanStdout: scan.stdout,
@ -277,7 +321,7 @@ export async function runOrbitVerification(options = {}) {
date,
connectionId,
projectDir,
scanCommand: shellCommand(scanArgv),
scanCommand,
scanExitCode: scan.exitCode,
blocker: 'KTX scan completed without printing a Run id',
scanStdout: scan.stdout,
@ -291,7 +335,7 @@ export async function runOrbitVerification(options = {}) {
date,
connectionId,
projectDir,
scanCommand: shellCommand(scanArgv),
scanCommand,
scanExitCode: scan.exitCode,
blocker: 'KTX scan completed without printing a report artifact path',
scanStdout: scan.stdout,
@ -304,7 +348,7 @@ export async function runOrbitVerification(options = {}) {
date,
connectionId,
projectDir,
scanCommand: shellCommand(scanArgv),
scanCommand,
reportPath: fullScanReportPath,
scanExitCode: scan.exitCode,
scanStdout: scan.stdout,

View file

@ -1,9 +1,8 @@
import assert from 'node:assert/strict';
import { readFile } from 'node:fs/promises';
import { dirname } from 'node:path';
import { describe, it } from 'node:test';
import {
buildOrbitScanArgv,
buildOrbitScanArgs,
defaultOrbitVerificationProjectDir,
extractReportPath,
extractRunId,
@ -49,6 +48,14 @@ function successReportJson() {
});
}
function successfulRunKtxScan(calls = []) {
return async (args, io) => {
calls.push(args);
io.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n Report: reports/scan-report.json\n');
return 0;
};
}
describe('relationship Orbit verification helper', () => {
it('exposes the Orbit verification command from the KTX workspace package', async () => {
const packageJson = JSON.parse(await readFile(new URL('../package.json', import.meta.url), 'utf8'));
@ -59,20 +66,19 @@ describe('relationship Orbit verification helper', () => {
);
});
it('builds the current KTX launcher arguments for scan commands', () => {
assert.deepEqual(buildOrbitScanArgv({ connectionId: 'orbit', projectDir: '/tmp/orbit-project' }), [
'scan',
'orbit',
'--mode',
'relationships',
'--project-dir',
'/tmp/orbit-project',
]);
it('builds the internal relationship scan arguments', () => {
assert.deepEqual(buildOrbitScanArgs({ connectionId: 'orbit', projectDir: '/tmp/orbit-project' }), {
command: 'run',
projectDir: '/tmp/orbit-project',
connectionId: 'orbit',
mode: 'relationships',
detectRelationships: true,
dryRun: false,
});
});
it('uses the checked-in Orbit verification project by default', async () => {
const calls = [];
const envs = [];
const scanCalls = [];
const writes = [];
const defaultProjectDir = defaultOrbitVerificationProjectDir();
@ -83,27 +89,28 @@ describe('relationship Orbit verification helper', () => {
writeFile: async (path, content) => {
writes.push({ path, content });
},
runWorkspaceKtx: async (argv, options) => {
calls.push(argv);
envs.push(options.env);
options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n Report: reports/scan-report.json\n');
return 0;
},
runKtxScan: successfulRunKtxScan(scanCalls),
readFile: async () => successReportJson(),
});
assert.equal(result.status, 'success');
assert.deepEqual(calls, [
['scan', 'orbit', '--mode', 'relationships', '--project-dir', defaultProjectDir],
assert.deepEqual(scanCalls, [
{
command: 'run',
projectDir: defaultProjectDir,
connectionId: 'orbit',
mode: 'relationships',
detectRelationships: true,
dryRun: false,
},
]);
assert.equal(envs[0].GIT_CEILING_DIRECTORIES, dirname(defaultProjectDir));
assert.equal(writes.length, 1);
assert.match(writes[0].content, new RegExp(defaultProjectDir.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')));
});
it('uses KTX_PROJECT_DIR for the Orbit verification project override', async () => {
const previousProjectDir = process.env.KTX_PROJECT_DIR;
const calls = [];
const scanCalls = [];
try {
process.env.KTX_PROJECT_DIR = '/tmp/orbit-project-from-env';
@ -113,17 +120,20 @@ describe('relationship Orbit verification helper', () => {
now: () => new Date('2026-05-07T10:00:00.000Z'),
mkdir: async () => {},
writeFile: async () => {},
runWorkspaceKtx: async (argv, options) => {
calls.push(argv);
options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n Report: reports/scan-report.json\n');
return 0;
},
runKtxScan: successfulRunKtxScan(scanCalls),
readFile: async () => successReportJson(),
});
assert.equal(result.projectDir, '/tmp/orbit-project-from-env');
assert.deepEqual(calls, [
['scan', 'orbit', '--mode', 'relationships', '--project-dir', '/tmp/orbit-project-from-env'],
assert.deepEqual(scanCalls, [
{
command: 'run',
projectDir: '/tmp/orbit-project-from-env',
connectionId: 'orbit',
mode: 'relationships',
detectRelationships: true,
dryRun: false,
},
]);
} finally {
if (previousProjectDir === undefined) {
@ -146,7 +156,7 @@ describe('relationship Orbit verification helper', () => {
date: '2026-05-07',
connectionId: 'orbit',
projectDir: '/tmp/orbit-project',
scanCommand: 'pnpm run ktx -- scan orbit --mode relationships --project-dir /tmp/orbit-project',
scanCommand: 'internal runKtxScan connection=orbit mode=relationships projectDir=/tmp/orbit-project',
reportPath: '/tmp/orbit-project/reports/scan-report.json',
scanExitCode: 0,
scanStdout: 'KTX scan completed\nRun: scan-orbit-1\n',
@ -171,7 +181,7 @@ describe('relationship Orbit verification helper', () => {
date: '2026-05-07',
connectionId: 'orbit',
projectDir: '/tmp/orbit-project',
scanCommand: 'pnpm run ktx -- scan orbit --mode relationships --project-dir /tmp/orbit-project',
scanCommand: 'internal runKtxScan connection=orbit mode=relationships projectDir=/tmp/orbit-project',
scanExitCode: 1,
blocker: 'Connection "orbit" was not found',
scanStdout: '',
@ -180,12 +190,12 @@ describe('relationship Orbit verification helper', () => {
assert.match(markdown, /Exit code: 1/);
assert.match(markdown, /Connection "orbit" was not found/);
assert.match(markdown, /Orbit verification was not executed because the current local Orbit scan command failed/);
assert.match(markdown, /Orbit verification was not executed because the current local Orbit relationship scan failed/);
assert.doesNotMatch(markdown, /scan\.enrichment\.mode is required/);
});
it('runs scan then reads the report artifact and writes success Markdown', async () => {
const calls = [];
const scanCalls = [];
const writes = [];
const result = await runOrbitVerification({
connectionId: 'orbit',
@ -196,24 +206,27 @@ describe('relationship Orbit verification helper', () => {
writeFile: async (path, content) => {
writes.push({ path, content });
},
runWorkspaceKtx: async (argv, options) => {
calls.push(argv);
options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n Report: reports/scan-report.json\n');
return 0;
},
runKtxScan: successfulRunKtxScan(scanCalls),
readFile: async () => successReportJson(),
});
assert.equal(result.status, 'success');
assert.deepEqual(calls, [
['scan', 'orbit', '--mode', 'relationships', '--project-dir', '/tmp/orbit-project'],
assert.deepEqual(scanCalls, [
{
command: 'run',
projectDir: '/tmp/orbit-project',
connectionId: 'orbit',
mode: 'relationships',
detectRelationships: true,
dryRun: false,
},
]);
assert.equal(writes.length, 1);
assert.equal(writes[0].path, '/tmp/orbit-report.md');
assert.match(writes[0].content, /Accepted: 14/);
});
it('writes blocked Markdown when the scan command fails before a run id exists', async () => {
it('writes blocked Markdown when the internal scan fails before a run id exists', async () => {
const writes = [];
const result = await runOrbitVerification({
connectionId: 'orbit',
@ -224,8 +237,8 @@ describe('relationship Orbit verification helper', () => {
writeFile: async (path, content) => {
writes.push({ path, content });
},
runWorkspaceKtx: async (_argv, options) => {
options.stderr.write('Connection "orbit" was not found\n');
runKtxScan: async (_args, io) => {
io.stderr.write('Connection "orbit" was not found\n');
return 1;
},
});
@ -236,7 +249,7 @@ describe('relationship Orbit verification helper', () => {
assert.match(writes[0].content, /Connection "orbit" was not found/);
});
it('runs the workspace launcher in buffered mode so real scan errors are captured', async () => {
it('runs the workspace launcher in buffered mode when preparing the internal scan module', async () => {
let sawExecFile = false;
const result = await runOrbitVerification({
connectionId: 'orbit',
@ -246,7 +259,8 @@ describe('relationship Orbit verification helper', () => {
mkdir: async () => {},
writeFile: async () => {},
execFile: async () => ({ stdout: '', stderr: '' }),
runWorkspaceKtx: async (_argv, options) => {
runWorkspaceKtx: async (argv, options) => {
assert.deepEqual(argv, ['--version']);
sawExecFile = typeof options.execFile === 'function';
options.stderr.write('ENOENT: no such file or directory, open \'/tmp/orbit-project/ktx.yaml\'\n');
return 1;

View file

@ -152,7 +152,7 @@ test('runWorkspaceKtx rebuilds before running when workspace sources are newer t
const logs = [];
let sourceMtimeMs = 3000;
const exitCode = await runWorkspaceKtx(['scan', 'orbit', '--mode', 'relationships'], {
const exitCode = await runWorkspaceKtx(['status', '--json', '--no-input'], {
rootDir: '/workspace/ktx',
access: async () => undefined,
stat: async (path) => ({
@ -174,7 +174,7 @@ test('runWorkspaceKtx rebuilds before running when workspace sources are newer t
sourceMtimeMs = 1000;
return { stdout: 'build ok\n', stderr: '' };
}
return { stdout: 'scan ok\n', stderr: '' };
return { stdout: '{"status":"ready"}\n', stderr: '' };
},
stdout: { write: (chunk) => logs.push(['stdout', chunk]) },
stderr: { write: (chunk) => logs.push(['stderr', chunk]) },
@ -185,12 +185,12 @@ test('runWorkspaceKtx rebuilds before running when workspace sources are newer t
calls.map((call) => [call.command, call.args]),
[
['pnpm', ['run', 'build']],
[process.execPath, ['/workspace/ktx/packages/cli/dist/bin.js', 'scan', 'orbit', '--mode', 'relationships']],
[process.execPath, ['/workspace/ktx/packages/cli/dist/bin.js', 'status', '--json', '--no-input']],
],
);
assert.deepEqual(logs, [
['stderr', 'KTX CLI build output is stale. Rebuilding it now with `pnpm run build`...\n'],
['stdout', 'build ok\n'],
['stdout', 'scan ok\n'],
['stdout', '{"status":"ready"}\n'],
]);
});