ktx/packages/cli/src/admin-reindex.ts
Andrey Avtomonov 2366b00301
chore(workspace): gate dead-code with knip production mode (#196)
* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm

* refactor(workspace): rewrite @ktx/llm imports to relative paths

* refactor(workspace): fold internal packages into cli

* chore(workspace): gate dead-code with knip production mode

Turn on production-mode knip plus an autofix run in pre-commit and the
`pnpm dead-code` script, document the `/** @internal */` convention for
test-only exports in AGENTS.md, annotate test-only exports across the
CLI with that JSDoc, and drop dead exports/wrappers the new gate
surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`,
`createLocalScanEnrichmentProvidersFromConfig`,
`PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports).
Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit
production entries so cross-package barrel leaks are caught.

* refactor(cli): delete internal barrel index.ts files

The 34 `index.ts` re-export barrels inside `packages/cli/src/` were
holdovers from the pre-fold multi-workspace structure. Post-fold-in they
served no production purpose: external consumers go through the single
package main entry, and in-repo callers mostly imported through them
only because the path was short. Internally, knip flagged most barrel
re-exports as production-dead (only reached via tests).

This change:
- Deletes every internal barrel except `packages/cli/src/index.ts`
  (the published package entry).
- Rewrites ~270 source/test files to import each name directly from
  the file that defines it.
- Moves `tools/warehouse-verification/index.ts` to
  `create-warehouse-verification-tools.ts` (the function it defined
  locally) and updates its single consumer.
- Renames `search/backend-conformance.ts` → `.test-utils.ts` to match
  the existing test-helper file convention.
- Deletes 13 dead test-only chains (dbt-descriptions/*,
  live-database/extracted-schema, live-database/structural-sync,
  relationship-* feedback/review chain) plus their tests and a
  cascading orphan integration test.
- Updates test mocks that pointed at deleted barrel paths
  (notion-client, connector barrels in scan/local-scan-connectors
  tests) to mock the source files instead.
- Points the maintainer benchmark script
  (`scripts/relationship-benchmark-report.mjs`) at source files
  instead of `dist/context/scan/index.js`.
- Drops the barrel `!` entries from `knip.json`; adds explicit
  production entries only for the benchmark code reached via dist by
  the maintainer script.

Net: 413 files changed, ~1.2k insertions, ~9.4k deletions.

`pnpm run dead-code` (Biome + knip default + knip production) and
`pnpm run type-check` are clean; 2277 tests pass.

* refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly

Promote the CLI workspace package to the public name `@kaelio/ktx` and
drop the separate `scripts/build-public-npm-package.mjs` wrapper. The
CLI package is now publishable in place (`publishConfig.access: public`,
`provenance: true`), so artifact packing uses `pnpm pack` against
`packages/cli/` instead of assembling a parallel package tree.

Updates all workspace filter invocations, docs, tests, and release
readiness checks to reference the new package name, and folds the
tarball-name helper into `scripts/public-npm-release-metadata.mjs`.

* docs: align "agent clients" and "data agents" terminology

Replace "client agents" with "agent clients" and "database agents" with
"data agents" across AGENTS.md, README.md, the docs-site copy, and the
matching setup-agents test description, matching the canonical
vocabulary in docs/terminology.md.

Also moves packages/cli/tsconfig.json's tsBuildInfoFile from
node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive
node_modules reinstalls.

* refactor(release): single source of truth for package version

Make packages/cli/package.json the single source of truth for the
@kaelio/ktx version. publicNpmPackageVersion() now reads it directly,
so artifact filenames, release-readiness checks, and the Python wheel
version all derive from one field. The duplicate
release-policy.json.publicNpmPackageVersion is removed.

Previously the two fields could drift: tarballs were named
kaelio-ktx-0.4.1.tgz while internally containing
@kaelio/ktx@0.0.0-private.

- update-public-release-version.mjs rewrites both Python pyproject.toml
  files (ktx-daemon, ktx-sl) alongside the npm package.jsons,
  normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2).
- semantic-release-config.cjs adds the two pyproject.toml files to
  @semantic-release/git assets so the release commit back to main
  carries every version source in lockstep.
- The six "?? '0.0.0-private'" fallback literals across the CLI are
  replaced with "?? getKtxCliPackageInfo().version", and
  createDefaultKtxMcpServer makes its version arg required.
- docs/release.md describes the actual commit-back model: the dev tree
  always reflects the most recent release; no sentinel pin to
  maintain.

Verified: pnpm run artifacts:build now produces
kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with
@kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and
2287 vitests + 173 script tests pass.

* refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime

Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and
scan command entrypoints so tests can stub them, and teach
resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime
feature when ktx.yaml selects sentence-transformers.

* chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal

Both symbols are consumed only by status-project.test.ts. Annotating with
/** @internal */ keeps knip's production-mode check clean without changing
runtime behavior.

* fix(cli): use real package metadata in print-command-tree

The stubbed package name embedded a forbidden product identifier that
tripped the boundary check in CI. Read the metadata from package.json
instead — keeps the rendered tree unchanged and removes a duplicate
source of truth.

* feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts

Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer
source counts, computed with `SUM(embedding_json IS NOT NULL)` over
`knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to
"Wiki" (canonical per `docs/terminology.md`) and rename the matching
`localStats.knowledgePages` field to `localStats.wikiPages`.

Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those
duplicated the per-surface rows above. Disk now reports only actual byte
usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` /
`semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry`
helpers, and the `filter` arg on `summarizeDir` are removed.
2026-05-21 15:28:58 +02:00

190 lines
7.3 KiB
TypeScript

import { KtxIngestEmbeddingPortAdapter } from './context/llm/embedding-port.js';
import type { KtxEmbeddingPort } from './context/core/embedding.js';
import { reindexLocalIndexes } from './context/index-sync/reindex.js';
import type { ReindexScopeResult, ReindexSummary } from './context/index-sync/types.js';
import { loadKtxProject } from './context/project/project.js';
import { Option, type Command } from '@commander-js/extra-typings';
import { cancel, intro, log, note, outro } from '@clack/prompts';
import type { KtxCliCommandContext } from './cli-program.js';
import type { KtxCliIo } from './cli-runtime.js';
import { resolveProjectEmbeddingProvider } from './embedding-resolution.js';
import { resolveOutputMode } from './io/mode.js';
import { green, red, SYMBOLS } from './io/symbols.js';
export interface KtxAdminReindexArgs {
projectDir: string;
force: boolean;
output?: 'pretty' | 'plain' | 'json';
json?: boolean;
cliVersion: string;
}
export function registerAdminReindexCommand(admin: Command, context: KtxCliCommandContext): void {
admin
.command('reindex')
.description('Sync local wiki and semantic-layer search indexes from disk')
.option('--force', 'Clear each discovered scope before rebuilding it', false)
.option('--json', 'Shortcut for --output=json (overrides --output)', false)
.addOption(
new Option('--output <mode>', 'Output mode: pretty, plain, or json').choices(['pretty', 'plain', 'json']),
)
.action(async (options: { force?: boolean; json?: boolean; output?: 'pretty' | 'plain' | 'json' }, command) => {
const runner = context.deps.adminReindex ?? runKtxAdminReindex;
const { resolveCommandProjectDir } = await import('./cli-program.js');
context.setExitCode(
await runner(
{
projectDir: resolveCommandProjectDir(command),
force: options.force === true,
json: options.json === true,
output: options.output,
cliVersion: context.packageInfo.version,
},
context.io,
),
);
});
}
function scopeKey(scope: ReindexScopeResult): string {
if (scope.kind === 'wiki') {
return scope.scope === 'user' ? `wiki/user/${scope.scopeId ?? 'local'}` : 'wiki/global';
}
return `sl/${scope.connectionId ?? scope.label}`;
}
function quotePlainValue(value: string): string {
return `"${value.replaceAll('\\', '\\\\').replaceAll('"', '\\"')}"`;
}
/** @internal */
export function reindexHasErrors(summary: ReindexSummary): boolean {
return summary.scopes.some((scope) => scope.error);
}
/** @internal */
export function renderReindexPlain(summary: ReindexSummary, io: KtxCliIo): void {
const updateKey = summary.force ? 'rebuilt' : 'updated';
for (const scope of summary.scopes) {
const cells = [
scopeKey(scope),
`scanned=${scope.scanned}`,
`${updateKey}=${scope.updated}`,
`deleted=${scope.deleted}`,
`embeddings=${summary.embeddingsAvailable ? String(scope.embeddingsRecomputed) : '-'}`,
`duration_ms=${scope.durationMs}`,
...(scope.error ? [`error=${quotePlainValue(scope.error)}`] : []),
];
io.stderr.write(`${cells.join('\t')}\n`);
}
const failed = summary.scopes.filter((scope) => scope.error).length;
io.stdout.write(
[
'reindex',
`scopes=${summary.scopes.length}`,
`scanned=${summary.totals.scanned}`,
`${updateKey}=${summary.totals.updated}`,
`deleted=${summary.totals.deleted}`,
`embeddings=${summary.embeddingsAvailable ? String(summary.totals.embeddingsRecomputed) : '-'}`,
`duration_ms=${summary.durationMs}`,
...(failed > 0 ? [`failed=${failed}`] : []),
].join('\t') + '\n',
);
}
/** @internal */
export function renderReindexJson(summary: ReindexSummary, io: KtxCliIo): void {
io.stdout.write(`${JSON.stringify({ kind: 'reindex', data: summary, meta: { command: 'admin reindex' } }, null, 2)}\n`);
}
function noun(scope: ReindexScopeResult): string {
return scope.kind === 'wiki' ? 'pages' : 'sources';
}
function formatScopeLine(scope: ReindexScopeResult, force: boolean, embeddingsAvailable: boolean): string {
if (scope.error) {
return `${scope.kind === 'wiki' ? 'Wiki' : 'SL'}: ${scope.label} ${SYMBOLS.emDash} failed: ${scope.error}`;
}
const changedLabel = force ? 'rebuilt' : 'updated';
const parts = [`${scope.scanned} ${noun(scope)}`];
if (scope.updated > 0) {
parts.push(`${scope.updated} ${changedLabel}`);
} else {
parts.push('unchanged');
}
if (!force && scope.deleted > 0) {
parts.push(`${scope.deleted} deleted`);
}
if (embeddingsAvailable) {
parts.push(`${scope.embeddingsRecomputed} embeddings recomputed`);
}
parts.push(`${scope.durationMs}ms`);
return `${scope.kind === 'wiki' ? 'Wiki' : 'SL'}: ${scope.label} ${SYMBOLS.emDash} ${parts.join(` ${SYMBOLS.middot} `)}`;
}
function renderReindexPretty(summary: ReindexSummary, io: KtxCliIo): void {
intro(summary.force ? 'ktx admin reindex --force' : 'ktx admin reindex');
if (!summary.embeddingsAvailable) {
log.warn(`Embeddings: not configured ${SYMBOLS.emDash} indexing lexical only`);
}
for (const scope of summary.scopes) {
const line = formatScopeLine(scope, summary.force, summary.embeddingsAvailable);
if (scope.error) {
log.error(red(line));
} else {
log.success(green(line));
}
}
const failed = summary.scopes.filter((scope) => scope.error).length;
note(
[
`scopes ${summary.scopes.length}`,
`scanned ${summary.totals.scanned}`,
`${summary.force ? 'rebuilt' : 'updated'} ${summary.totals.updated}`,
`deleted ${summary.totals.deleted}`,
`embeddings ${summary.embeddingsAvailable ? summary.totals.embeddingsRecomputed : SYMBOLS.emDash}`,
`index ${summary.dbPath}`,
...(failed > 0 ? [`failed ${failed}`] : []),
].join('\n'),
'Summary',
);
if (failed > 0) {
cancel(`reindex completed with ${failed} error${failed === 1 ? '' : 's'}`);
} else {
outro(`Done in ${(summary.durationMs / 1000).toFixed(1)}s`);
}
void io;
}
async function runKtxAdminReindex(args: KtxAdminReindexArgs, io: KtxCliIo = process): Promise<number> {
try {
const project = await loadKtxProject({ projectDir: args.projectDir });
const resolution = await resolveProjectEmbeddingProvider(project, {
mode: 'use-if-running',
cliVersion: args.cliVersion,
io,
});
const embeddingService: KtxEmbeddingPort | null =
resolution.kind === 'configured' || resolution.kind === 'managed-running' || resolution.kind === 'managed-started'
? new KtxIngestEmbeddingPortAdapter(resolution.provider)
: null;
const summary = await reindexLocalIndexes(project, { force: args.force, embeddingService });
const mode = resolveOutputMode({ explicit: args.output, json: args.json, io });
if (!summary.embeddingsAvailable && mode === 'plain') {
io.stderr.write(`Embeddings: not configured ${SYMBOLS.emDash} indexing lexical only\n`);
}
if (mode === 'json') {
renderReindexJson(summary, io);
} else if (mode === 'plain') {
renderReindexPlain(summary, io);
} else {
renderReindexPretty(summary, io);
}
return reindexHasErrors(summary) ? 1 : 0;
} catch (error) {
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
return 1;
}
}