chore(workspace): gate dead-code with knip production mode (#196)

* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm

* refactor(workspace): rewrite @ktx/llm imports to relative paths

* refactor(workspace): fold internal packages into cli

* chore(workspace): gate dead-code with knip production mode

Turn on production-mode knip plus an autofix run in pre-commit and the
`pnpm dead-code` script, document the `/** @internal */` convention for
test-only exports in AGENTS.md, annotate test-only exports across the
CLI with that JSDoc, and drop dead exports/wrappers the new gate
surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`,
`createLocalScanEnrichmentProvidersFromConfig`,
`PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports).
Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit
production entries so cross-package barrel leaks are caught.

* refactor(cli): delete internal barrel index.ts files

The 34 `index.ts` re-export barrels inside `packages/cli/src/` were
holdovers from the pre-fold multi-workspace structure. Post-fold-in they
served no production purpose: external consumers go through the single
package main entry, and in-repo callers mostly imported through them
only because the path was short. Internally, knip flagged most barrel
re-exports as production-dead (only reached via tests).

This change:
- Deletes every internal barrel except `packages/cli/src/index.ts`
  (the published package entry).
- Rewrites ~270 source/test files to import each name directly from
  the file that defines it.
- Moves `tools/warehouse-verification/index.ts` to
  `create-warehouse-verification-tools.ts` (the function it defined
  locally) and updates its single consumer.
- Renames `search/backend-conformance.ts` → `.test-utils.ts` to match
  the existing test-helper file convention.
- Deletes 13 dead test-only chains (dbt-descriptions/*,
  live-database/extracted-schema, live-database/structural-sync,
  relationship-* feedback/review chain) plus their tests and a
  cascading orphan integration test.
- Updates test mocks that pointed at deleted barrel paths
  (notion-client, connector barrels in scan/local-scan-connectors
  tests) to mock the source files instead.
- Points the maintainer benchmark script
  (`scripts/relationship-benchmark-report.mjs`) at source files
  instead of `dist/context/scan/index.js`.
- Drops the barrel `!` entries from `knip.json`; adds explicit
  production entries only for the benchmark code reached via dist by
  the maintainer script.

Net: 413 files changed, ~1.2k insertions, ~9.4k deletions.

`pnpm run dead-code` (Biome + knip default + knip production) and
`pnpm run type-check` are clean; 2277 tests pass.

* refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly

Promote the CLI workspace package to the public name `@kaelio/ktx` and
drop the separate `scripts/build-public-npm-package.mjs` wrapper. The
CLI package is now publishable in place (`publishConfig.access: public`,
`provenance: true`), so artifact packing uses `pnpm pack` against
`packages/cli/` instead of assembling a parallel package tree.

Updates all workspace filter invocations, docs, tests, and release
readiness checks to reference the new package name, and folds the
tarball-name helper into `scripts/public-npm-release-metadata.mjs`.

* docs: align "agent clients" and "data agents" terminology

Replace "client agents" with "agent clients" and "database agents" with
"data agents" across AGENTS.md, README.md, the docs-site copy, and the
matching setup-agents test description, matching the canonical
vocabulary in docs/terminology.md.

Also moves packages/cli/tsconfig.json's tsBuildInfoFile from
node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive
node_modules reinstalls.

* refactor(release): single source of truth for package version

Make packages/cli/package.json the single source of truth for the
@kaelio/ktx version. publicNpmPackageVersion() now reads it directly,
so artifact filenames, release-readiness checks, and the Python wheel
version all derive from one field. The duplicate
release-policy.json.publicNpmPackageVersion is removed.

Previously the two fields could drift: tarballs were named
kaelio-ktx-0.4.1.tgz while internally containing
@kaelio/ktx@0.0.0-private.

- update-public-release-version.mjs rewrites both Python pyproject.toml
  files (ktx-daemon, ktx-sl) alongside the npm package.jsons,
  normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2).
- semantic-release-config.cjs adds the two pyproject.toml files to
  @semantic-release/git assets so the release commit back to main
  carries every version source in lockstep.
- The six "?? '0.0.0-private'" fallback literals across the CLI are
  replaced with "?? getKtxCliPackageInfo().version", and
  createDefaultKtxMcpServer makes its version arg required.
- docs/release.md describes the actual commit-back model: the dev tree
  always reflects the most recent release; no sentinel pin to
  maintain.

Verified: pnpm run artifacts:build now produces
kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with
@kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and
2287 vitests + 173 script tests pass.

* refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime

Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and
scan command entrypoints so tests can stub them, and teach
resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime
feature when ktx.yaml selects sentence-transformers.

* chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal

Both symbols are consumed only by status-project.test.ts. Annotating with
/** @internal */ keeps knip's production-mode check clean without changing
runtime behavior.

* fix(cli): use real package metadata in print-command-tree

The stubbed package name embedded a forbidden product identifier that
tripped the boundary check in CI. Read the metadata from package.json
instead — keeps the rendered tree unchanged and removes a duplicate
source of truth.

* feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts

Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer
source counts, computed with `SUM(embedding_json IS NOT NULL)` over
`knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to
"Wiki" (canonical per `docs/terminology.md`) and rename the matching
`localStats.knowledgePages` field to `localStats.wikiPages`.

Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those
duplicated the per-surface rows above. Disk now reports only actual byte
usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` /
`semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry`
helpers, and the `filter` arg on `summarizeDir` are removed.

packages/cli/src/connectors/sqlite/connector.ts

@@ -0,0 +1,354 @@
+import Database from 'better-sqlite3';
+import { existsSync, readFileSync, statSync } from 'node:fs';
+import { homedir } from 'node:os';
+import { isAbsolute, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
+import { normalizeQueryRows } from '../../context/connections/query-executor.js';
+import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
+import { KtxSqliteDialect } from './dialect.js';
+
+export interface KtxSqliteConnectionConfig {
+  driver?: string;
+  path?: string;
+  url?: string;
+  [key: string]: unknown;
+}
+
+/** @internal */
+export interface SqliteDatabasePathInput {
+  connectionId: string;
+  projectDir?: string;
+  connection: KtxSqliteConnectionConfig | undefined;
+}
+
+export interface KtxSqliteScanConnectorOptions extends SqliteDatabasePathInput {
+  now?: () => Date;
+}
+
+export interface KtxSqliteReadOnlyQueryInput extends KtxReadOnlyQueryInput {
+  params?: Record<string, unknown> | unknown[];
+}
+
+export interface KtxSqliteColumnDistinctValuesOptions {
+  maxCardinality: number;
+  limit: number;
+  sampleSize?: number;
+}
+
+export interface KtxSqliteColumnDistinctValuesResult {
+  values: string[] | null;
+  cardinality: number;
+}
+
+interface SqliteMasterRow {
+  name: string;
+  type: 'table' | 'view';
+}
+
+interface SqliteTableInfoRow {
+  cid: number;
+  name: string;
+  type: string;
+  notnull: number;
+  dflt_value: unknown;
+  pk: number;
+}
+
+interface SqliteForeignKeyRow {
+  id: number;
+  seq: number;
+  table: string;
+  from: string;
+  to: string;
+}
+
+function stringConfigValue(
+  connection: KtxSqliteConnectionConfig | undefined,
+  key: keyof KtxSqliteConnectionConfig,
+): string | undefined {
+  const value = connection?.[key];
+  return typeof value === 'string' && value.trim().length > 0 ? resolveStringReference(key, value.trim()) : undefined;
+}
+
+function resolveStringReference(key: keyof KtxSqliteConnectionConfig, value: string): string {
+  if (value.startsWith('env:')) {
+    return process.env[value.slice('env:'.length)] ?? '';
+  }
+  // `file:` on the `url` key is SQLite's native URI form (e.g. `file:///db.sqlite`), not a
+  // file-contents reference — skip the read so the URI passes through verbatim.
+  if (key !== 'url' && value.startsWith('file:')) {
+    const rawPath = value.slice('file:'.length);
+    const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
+    return readFileSync(path, 'utf-8').trim();
+  }
+  return value;
+}
+
+function sqlitePathFromUrl(url: string): string {
+  if (url.startsWith('file:')) {
+    return fileURLToPath(url);
+  }
+  if (url.startsWith('sqlite:')) {
+    const parsed = new URL(url);
+    return decodeURIComponent(parsed.pathname);
+  }
+  return url;
+}
+
+function stripLeadingSqlComments(sql: string): string {
+  let index = 0;
+  while (index < sql.length) {
+    while (/\s/.test(sql[index] ?? '')) {
+      index += 1;
+    }
+    if (sql.startsWith('--', index)) {
+      const end = sql.indexOf('\n', index + 2);
+      index = end === -1 ? sql.length : end + 1;
+      continue;
+    }
+    if (sql.startsWith('/*', index)) {
+      const end = sql.indexOf('*/', index + 2);
+      if (end === -1) {
+        return sql.slice(index);
+      }
+      index = end + 2;
+      continue;
+    }
+    break;
+  }
+  return sql.slice(index);
+}
+
+export function isKtxSqliteConnectionConfig(
+  connection: KtxSqliteConnectionConfig | undefined,
+): connection is KtxSqliteConnectionConfig {
+  const driver = String(connection?.driver ?? '').toLowerCase();
+  return driver === 'sqlite' || driver === 'sqlite3';
+}
+
+/** @internal */
+export function sqliteDatabasePathFromConfig(input: SqliteDatabasePathInput): string {
+  const inputDriver = input.connection?.driver ?? 'unknown';
+  if (!isKtxSqliteConnectionConfig(input.connection)) {
+    throw new Error(`Native SQLite connector cannot run driver "${inputDriver}"`);
+  }
+  const configuredPath = stringConfigValue(input.connection, 'path') ?? sqlitePathFromUrl(stringConfigValue(input.connection, 'url') ?? '');
+  if (!configuredPath) {
+    throw new Error(`Native SQLite connector requires connections.${input.connectionId}.path or url`);
+  }
+  return isAbsolute(configuredPath) ? configuredPath : resolve(input.projectDir ?? process.cwd(), configuredPath);
+}
+
+export class KtxSqliteScanConnector implements KtxScanConnector {
+  readonly id: string;
+  readonly driver = 'sqlite' as const;
+  readonly capabilities = createKtxConnectorCapabilities({
+    tableSampling: true,
+    columnSampling: true,
+    columnStats: false,
+    readOnlySql: true,
+    nestedAnalysis: false,
+    formalForeignKeys: true,
+    estimatedRowCounts: true,
+  });
+
+  private readonly connectionId: string;
+  private readonly dbPath: string;
+  private readonly now: () => Date;
+  private readonly dialect = new KtxSqliteDialect();
+  private db: Database.Database | null = null;
+
+  constructor(options: KtxSqliteScanConnectorOptions) {
+    this.connectionId = options.connectionId;
+    this.dbPath = sqliteDatabasePathFromConfig(options);
+    this.now = options.now ?? (() => new Date());
+    this.id = `sqlite:${options.connectionId}`;
+  }
+
+  async testConnection(): Promise<{ success: boolean; error?: string }> {
+    try {
+      if (!existsSync(this.dbPath) || !statSync(this.dbPath).isFile()) {
+        return { success: false, error: `File not found: ${this.dbPath}` };
+      }
+      this.database().prepare('SELECT 1').get();
+      return { success: true };
+    } catch (error) {
+      return { success: false, error: error instanceof Error ? error.message : String(error) };
+    }
+  }
+
+  async introspect(input: KtxScanInput, _ctx: KtxScanContext): Promise<KtxSchemaSnapshot> {
+    this.assertConnection(input.connectionId);
+    const database = this.database();
+    const rawTables = database
+      .prepare(
+        `SELECT name, type FROM sqlite_master WHERE type IN ('table', 'view') AND name NOT LIKE 'sqlite_%' ORDER BY name`,
+      )
+      .all() as SqliteMasterRow[];
+    const tables = rawTables.map((table) => this.readTable(database, table));
+    const fileStats = existsSync(this.dbPath) ? statSync(this.dbPath) : null;
+    return {
+      connectionId: this.connectionId,
+      driver: 'sqlite',
+      extractedAt: this.now().toISOString(),
+      scope: {},
+      metadata: {
+        file_path: this.dbPath,
+        file_size: fileStats ? fileStats.size : 0,
+        table_count: tables.length,
+        total_columns: tables.reduce((sum, table) => sum + table.columns.length, 0),
+      },
+      tables,
+    };
+  }
+
+  async sampleTable(input: KtxTableSampleInput, _ctx: KtxScanContext): Promise<KtxTableSampleResult> {
+    this.assertConnection(input.connectionId);
+    const result = this.query(this.dialect.generateSampleQuery(this.qTableName(input.table), input.limit, input.columns));
+    return { headers: result.headers, rows: result.rows, totalRows: result.totalRows };
+  }
+
+  async sampleColumn(input: KtxColumnSampleInput, _ctx: KtxScanContext): Promise<KtxColumnSampleResult> {
+    this.assertConnection(input.connectionId);
+    const result = this.query(
+      this.dialect.generateColumnSampleQuery(this.qTableName(input.table), input.column, input.limit),
+    );
+    const values = result.rows.filter((row) => row.length > 0 && row[0] !== null).map((row) => row[0]);
+    return { values, nullCount: null, distinctCount: null };
+  }
+
+  async columnStats(_input: KtxColumnStatsInput, _ctx: KtxScanContext): Promise<KtxColumnStatsResult | null> {
+    return null;
+  }
+
+  async executeReadOnly(input: KtxSqliteReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {
+    this.assertConnection(input.connectionId);
+    const result = this.query(limitSqlForExecution(stripLeadingSqlComments(input.sql), input.maxRows), input.params);
+    return { ...result, rowCount: result.rows.length };
+  }
+
+  async getColumnDistinctValues(
+    table: KtxTableRef,
+    columnName: string,
+    options: KtxSqliteColumnDistinctValuesOptions,
+  ): Promise<KtxSqliteColumnDistinctValuesResult | null> {
+    const sampleSize = options.sampleSize ?? 10000;
+    const tableName = this.qTableName(table);
+    const quotedColumn = this.dialect.quoteIdentifier(columnName);
+    const cardinalityResult = this.query(
+      this.dialect.generateCardinalitySampleQuery(tableName, quotedColumn, sampleSize),
+    );
+    if (cardinalityResult.rows.length === 0) {
+      return null;
+    }
+    const cardinality = Number(cardinalityResult.rows[0][0]);
+    if (Number.isNaN(cardinality)) {
+      return null;
+    }
+    if (cardinality === 0) {
+      return { values: [], cardinality: 0 };
+    }
+    if (cardinality > options.maxCardinality) {
+      return { values: null, cardinality };
+    }
+    const valuesResult = this.query(this.dialect.generateDistinctValuesQuery(tableName, quotedColumn, options.limit));
+    return {
+      values: valuesResult.rows.filter((row) => row.length > 0 && row[0] !== null).map((row) => String(row[0])),
+      cardinality,
+    };
+  }
+
+  async getTableRowCount(tableName: string): Promise<number> {
+    const result = this.query(`SELECT COUNT(*) AS count FROM ${this.dialect.quoteIdentifier(tableName)}`);
+    return Number(result.rows[0]?.[0] ?? 0);
+  }
+
+  qTableName(table: Pick<KtxTableRef, 'name'>): string {
+    return this.dialect.formatTableName(table);
+  }
+
+  quoteIdentifier(identifier: string): string {
+    return this.dialect.quoteIdentifier(identifier);
+  }
+
+  async cleanup(): Promise<void> {
+    if (this.db) {
+      this.db.close();
+      this.db = null;
+    }
+  }
+
+  private database(): Database.Database {
+    if (!this.db) {
+      this.db = new Database(this.dbPath, { readonly: true, fileMustExist: true });
+    }
+    return this.db;
+  }
+
+  private query(sql: string, params?: Record<string, unknown> | unknown[]): Omit<KtxQueryResult, 'rowCount'> {
+    const statement = this.database().prepare(assertReadOnlySql(sql));
+    const rows = (params ? statement.all(params) : statement.all()) as unknown[];
+    return {
+      headers: statement.columns().map((column) => column.name),
+      rows: normalizeQueryRows(rows),
+      totalRows: rows.length,
+    };
+  }
+
+  private readTable(database: Database.Database, table: SqliteMasterRow): KtxSchemaTable {
+    const columns = database
+      .prepare(`PRAGMA table_info(${this.dialect.quoteIdentifier(table.name)})`)
+      .all() as SqliteTableInfoRow[];
+    const foreignKeys = database
+      .prepare(`PRAGMA foreign_key_list(${this.dialect.quoteIdentifier(table.name)})`)
+      .all() as SqliteForeignKeyRow[];
+    const estimatedRows =
+      table.type === 'table'
+        ? Number(
+            (
+              database
+                .prepare(`SELECT COUNT(*) AS count FROM ${this.dialect.quoteIdentifier(table.name)}`)
+                .get() as { count: unknown }
+            ).count,
+          )
+        : null;
+    return {
+      catalog: null,
+      db: null,
+      name: table.name,
+      kind: table.type,
+      comment: null,
+      estimatedRows,
+      columns: columns.map((column) => ({
+        name: column.name,
+        nativeType: column.type,
+        normalizedType: this.dialect.mapDataType(column.type),
+        dimensionType: this.dialect.mapToDimensionType(column.type),
+        nullable: column.notnull === 0 && column.pk === 0,
+        primaryKey: column.pk > 0,
+        comment: null,
+      })),
+      foreignKeys: this.mapForeignKeys(foreignKeys),
+    };
+  }
+
+  private mapForeignKeys(rows: SqliteForeignKeyRow[]): KtxSchemaForeignKey[] {
+    return rows
+      .sort((a, b) => a.id - b.id || a.seq - b.seq)
+      .map((row) => ({
+        fromColumn: row.from,
+        toCatalog: null,
+        toDb: null,
+        toTable: row.table,
+        toColumn: row.to,
+        constraintName: null,
+      }));
+  }
+
+  private assertConnection(connectionId: string): void {
+    if (connectionId !== this.connectionId) {
+      throw new Error(`KTX SQLite connector ${this.id} cannot serve connection ${connectionId}`);
+    }
+  }
+}