chore(workspace): gate dead-code with knip production mode (#196)

* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
2026-07-19 11:41:02 +02:00 · 2026-05-21 15:28:58 +02:00 · 2026-05-21 15:28:58 +02:00 · 2366b00301
commit 2366b00301
parent a1cfb03d73
1002 changed files with 2286 additions and 12051 deletions
--- a/packages/cli/src/context/skills/skills-registry.service.test.ts
+++ b/packages/cli/src/context/skills/skills-registry.service.test.ts
@ -0,0 +1,212 @@
+import { mkdtemp, mkdir, rm, writeFile } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import { SkillsRegistryService } from './skills-registry.service.js';
+
+describe('SkillsRegistryService', () => {
+  let service: SkillsRegistryService;
+  let tempDir: string;
+
+  beforeEach(async () => {
+    tempDir = await mkdtemp(join(tmpdir(), 'skills-registry-'));
+    service = new SkillsRegistryService({ skillsDir: tempDir });
+  });
+
+  afterEach(async () => {
+    await rm(tempDir, { recursive: true, force: true });
+  });
+
+  const writeSkill = async (dirName: string, body: string) => {
+    const dir = join(tempDir, dirName);
+    await mkdir(dir, { recursive: true });
+    await writeFile(join(dir, 'SKILL.md'), body, 'utf-8');
+  };
+
+  describe('parseFrontmatter', () => {
+    it('parses name and description', () => {
+      const frontmatter = service.parseFrontmatter('---\nname: foo\ndescription: Bar baz\n---\n\n# body');
+      expect(frontmatter).toEqual({ name: 'foo', description: 'Bar baz' });
+    });
+
+    it('supports wrapped description continuation lines', () => {
+      const frontmatter = service.parseFrontmatter(
+        ['---', 'name: sl', 'description: Line one', '  continuation of the description.', '---', '', '# body'].join(
+          '\n',
+        ),
+      );
+      expect(frontmatter.name).toBe('sl');
+      expect(frontmatter.description).toContain('Line one');
+      expect(frontmatter.description).toContain('continuation');
+    });
+
+    it('returns empty fields when no frontmatter block', () => {
+      expect(service.parseFrontmatter('# just a heading')).toEqual({});
+    });
+  });
+
+  describe('stripFrontmatter', () => {
+    it('removes the frontmatter block and leading blank line', () => {
+      const body = '---\nname: x\ndescription: y\n---\n\n# Hello\n\nparagraph';
+      expect(service.stripFrontmatter(body)).toBe('# Hello\n\nparagraph');
+    });
+
+    it('is a no-op when no frontmatter exists', () => {
+      expect(service.stripFrontmatter('# hello')).toBe('# hello');
+    });
+  });
+
+  describe('discoverSkills', () => {
+    it('returns an empty map when the directory does not exist', async () => {
+      const catalog = await service.discoverSkills(join(tempDir, 'missing'));
+      expect(catalog.size).toBe(0);
+    });
+
+    it('discovers valid skills and skips invalid ones', async () => {
+      await writeSkill('sl', '---\nname: sl\ndescription: Semantic layer.\n---\n\n# SL');
+      await writeSkill('wiki_capture', '---\nname: wiki_capture\ndescription: Wiki capture.\n---\n\n# KC');
+      await writeSkill('broken', '# no frontmatter at all');
+      await mkdir(join(tempDir, 'not_a_skill'), { recursive: true });
+
+      const catalog = await service.discoverSkills(tempDir);
+      expect(catalog.size).toBe(2);
+      expect(catalog.get('sl')?.name).toBe('sl');
+      expect(catalog.get('wiki_capture')?.description).toContain('Wiki capture');
+      expect(catalog.has('broken')).toBe(false);
+    });
+  });
+
+  describe('buildSkillsPrompt', () => {
+    it('formats bullet list with name and description', () => {
+      const output = service.buildSkillsPrompt([
+        { name: 'sl', description: 'Semantic layer.', path: '/tmp/sl' },
+        { name: 'wiki_capture', description: 'Wiki capture.', path: '/tmp/kc' },
+      ]);
+      expect(output).toContain('- sl: Semantic layer.');
+      expect(output).toContain('- wiki_capture: Wiki capture.');
+      expect(output).toContain('Use the `load_skill` tool');
+    });
+
+    it('returns empty string when no skills are available', () => {
+      expect(service.buildSkillsPrompt([])).toBe('');
+    });
+
+    it('appends the async capture note for the research caller', () => {
+      const output = service.buildSkillsPrompt(
+        [{ name: 'sl', description: 'Semantic layer.', path: '/tmp/sl' }],
+        'research',
+      );
+      expect(output).toContain('captured automatically by a post-turn memory agent');
+      expect(output).toContain('Focus on answering, not on saving');
+    });
+
+    it('does not append the note for memory_agent caller', () => {
+      const output = service.buildSkillsPrompt(
+        [{ name: 'sl_capture', description: 'Capture skill.', path: '/tmp/cap' }],
+        'memory_agent',
+      );
+      expect(output).not.toContain('captured automatically by a post-turn memory agent');
+    });
+  });
+
+  describe('parseFrontmatter callers field', () => {
+    it('parses inline-array form', () => {
+      const frontmatter = service.parseFrontmatter('---\nname: x\ndescription: y\ncallers: [memory_agent]\n---\n');
+      expect(frontmatter.callers).toEqual(['memory_agent']);
+    });
+
+    it('parses comma-separated form', () => {
+      const frontmatter = service.parseFrontmatter('---\nname: x\ndescription: y\ncallers: research, memory_agent\n---\n');
+      expect(frontmatter.callers).toEqual(['research', 'memory_agent']);
+    });
+
+    it('returns undefined when callers is absent', () => {
+      const frontmatter = service.parseFrontmatter('---\nname: x\ndescription: y\n---\n');
+      expect(frontmatter.callers).toBeUndefined();
+    });
+
+    it('drops unknown caller names with a warning', () => {
+      const frontmatter = service.parseFrontmatter('---\nname: x\ndescription: y\ncallers: [bogus, memory_agent]\n---\n');
+      expect(frontmatter.callers).toEqual(['memory_agent']);
+    });
+
+    it('returns undefined when the value is empty', () => {
+      const frontmatter = service.parseFrontmatter('---\nname: x\ndescription: y\ncallers:\n---\n');
+      expect(frontmatter.callers).toBeUndefined();
+    });
+  });
+
+  describe('listSkills and getSkill caller filter', () => {
+    beforeEach(async () => {
+      await writeSkill('sl', '---\nname: sl\ndescription: Open to all.\n---\n\n# SL');
+      await writeSkill(
+        'sl_capture',
+        '---\nname: sl_capture\ndescription: Memory-only capture skill.\ncallers: [memory_agent]\n---\n\n# Capture',
+      );
+      await writeSkill(
+        'wiki_capture',
+        '---\nname: wiki_capture\ndescription: Wiki capture.\ncallers: [memory_agent]\n---\n\n# KC',
+      );
+      service = new SkillsRegistryService({ skillsDir: tempDir });
+    });
+
+    it('research caller sees only open skills', async () => {
+      const skills = await service.listSkills('research');
+      expect(skills.map((skill) => skill.name).sort()).toEqual(['sl']);
+    });
+
+    it('memory_agent caller sees memory-only and open skills', async () => {
+      const skills = await service.listSkills('memory_agent');
+      expect(skills.map((skill) => skill.name).sort()).toEqual(['sl', 'sl_capture', 'wiki_capture']);
+    });
+
+    it('listSkills with names and caller intersects both filters', async () => {
+      const skills = await service.listSkills(['sl', 'sl_capture'], 'research');
+      expect(skills.map((skill) => skill.name)).toEqual(['sl']);
+    });
+
+    it('getSkill returns null for memory-only skill when caller is research', async () => {
+      const skill = await service.getSkill('sl_capture', 'research');
+      expect(skill).toBeNull();
+    });
+
+    it('getSkill returns the skill when caller has access', async () => {
+      const skill = await service.getSkill('sl_capture', 'memory_agent');
+      expect(skill?.name).toBe('sl_capture');
+    });
+
+    it('getSkill without caller returns the skill regardless of callers field', async () => {
+      const skill = await service.getSkill('sl_capture');
+      expect(skill?.name).toBe('sl_capture');
+    });
+
+  });
+
+  it('discovers skills from additional directories when the primary directory misses', async () => {
+    const extraDir = await mkdtemp(join(tmpdir(), 'skills-registry-extra-'));
+    try {
+      await mkdir(join(extraDir, 'wiki_capture'), { recursive: true });
+      await writeFile(
+        join(extraDir, 'wiki_capture', 'SKILL.md'),
+        [
+          '---',
+          'name: wiki_capture',
+          'description: Packaged knowledge capture skill.',
+          'callers: [memory_agent]',
+          '---',
+          '',
+          '# Wiki Capture',
+        ].join('\n'),
+        'utf-8',
+      );
+      service = new SkillsRegistryService({ skillsDir: tempDir, additionalSkillDirs: [extraDir] });
+
+      const skills = await service.listSkills(['wiki_capture'], 'memory_agent');
+
+      expect(skills.map((skill) => skill.name)).toEqual(['wiki_capture']);
+      expect(skills[0]?.path).toBe(join(extraDir, 'wiki_capture'));
+    } finally {
+      await rm(extraDir, { recursive: true, force: true });
+    }
+  });
+});
--- a/packages/cli/src/context/skills/skills-registry.service.ts
+++ b/packages/cli/src/context/skills/skills-registry.service.ts
@ -0,0 +1,255 @@
+import { readFile, readdir, stat } from 'node:fs/promises';
+import { join } from 'node:path';
+import { noopLogger, type KtxLogger } from '../../context/core/config.js';
+
+export type SkillCaller = 'research' | 'memory_agent';
+
+export interface SkillMetadata {
+  name: string;
+  description: string;
+  path: string;
+  callers?: SkillCaller[];
+}
+
+export interface FrontmatterFields {
+  name?: string;
+  description?: string;
+  callers?: SkillCaller[];
+}
+
+export interface SkillsRegistryServiceOptions {
+  skillsDir: string;
+  additionalSkillDirs?: string[];
+  logger?: KtxLogger;
+}
+
+const SKILL_FILENAME = 'SKILL.md';
+const VALID_CALLERS: ReadonlySet<SkillCaller> = new Set(['research', 'memory_agent']);
+
+export class SkillsRegistryService {
+  private readonly logger: KtxLogger;
+  private readonly skillsDir: string;
+  private readonly additionalSkillDirs: string[];
+  private catalogPromise: Promise<Map<string, SkillMetadata>> | null = null;
+
+  constructor(options: SkillsRegistryServiceOptions) {
+    this.logger = options.logger ?? noopLogger;
+    this.skillsDir = options.skillsDir;
+    this.additionalSkillDirs = options.additionalSkillDirs ?? [];
+  }
+
+  private async loadCatalog(): Promise<Map<string, SkillMetadata>> {
+    if (!this.catalogPromise) {
+      this.catalogPromise = this.discoverAllSkills();
+    }
+    return this.catalogPromise;
+  }
+
+  async discoverSkills(rootDir: string): Promise<Map<string, SkillMetadata>> {
+    const catalog = new Map<string, SkillMetadata>();
+
+    let entries: string[];
+    try {
+      entries = await readdir(rootDir);
+    } catch (error) {
+      const message = error instanceof Error ? error.message : String(error);
+      this.logger.warn(`Skills directory not found or unreadable at ${rootDir}: ${message}`);
+      return catalog;
+    }
+
+    for (const entry of entries.sort()) {
+      const dir = join(rootDir, entry);
+      const skillFile = join(dir, SKILL_FILENAME);
+      let isDir = false;
+      try {
+        isDir = (await stat(dir)).isDirectory();
+      } catch {
+        continue;
+      }
+      if (!isDir) {
+        continue;
+      }
+
+      let content: string;
+      try {
+        content = await readFile(skillFile, 'utf-8');
+      } catch {
+        this.logger.warn(`Skipping skill directory '${entry}': missing ${SKILL_FILENAME}`);
+        continue;
+      }
+
+      const frontmatter = this.parseFrontmatter(content);
+      if (!frontmatter.name || !frontmatter.description) {
+        this.logger.warn(`Skipping skill '${entry}': frontmatter missing name or description`);
+        continue;
+      }
+
+      const key = frontmatter.name.toLowerCase();
+      if (catalog.has(key)) {
+        this.logger.warn(`Duplicate skill name '${frontmatter.name}' in '${entry}'; first found wins`);
+        continue;
+      }
+      catalog.set(key, {
+        name: frontmatter.name,
+        description: frontmatter.description,
+        path: dir,
+        callers: frontmatter.callers,
+      });
+    }
+
+    this.logger.log(`Discovered ${catalog.size} skill(s): ${[...catalog.values()].map((skill) => skill.name).join(', ')}`);
+    return catalog;
+  }
+
+  private async discoverAllSkills(): Promise<Map<string, SkillMetadata>> {
+    const catalog = new Map<string, SkillMetadata>();
+    for (const rootDir of [this.skillsDir, ...this.additionalSkillDirs]) {
+      const discovered = await this.discoverSkills(rootDir);
+      for (const [key, skill] of discovered) {
+        if (!catalog.has(key)) {
+          catalog.set(key, skill);
+        }
+      }
+    }
+    return catalog;
+  }
+
+  parseFrontmatter(content: string): FrontmatterFields {
+    if (!content.startsWith('---')) {
+      return {};
+    }
+    const end = content.indexOf('\n---', 3);
+    if (end === -1) {
+      return {};
+    }
+
+    const block = content.slice(3, end).trim();
+    const fields: FrontmatterFields = {};
+    let index = 0;
+    const lines = block.split(/\r?\n/);
+    while (index < lines.length) {
+      const line = lines[index];
+      const match = /^([A-Za-z_][\w-]*):\s*(.*)$/.exec(line);
+      if (!match) {
+        index += 1;
+        continue;
+      }
+
+      const [, key, rest] = match;
+      let value = rest.trim();
+      const continuation: string[] = [];
+      let nextIndex = index + 1;
+      while (nextIndex < lines.length) {
+        const next = lines[nextIndex];
+        if (!next.trim() || /^[A-Za-z_][\w-]*:/.test(next) || !/^\s/.test(next)) {
+          break;
+        }
+        continuation.push(next.trim());
+        nextIndex += 1;
+      }
+      if (continuation.length > 0) {
+        value = [value, ...continuation].filter(Boolean).join(' ');
+      }
+      if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) {
+        value = value.slice(1, -1);
+      }
+
+      if (key === 'name' || key === 'description') {
+        fields[key] = value;
+      } else if (key === 'callers') {
+        fields.callers = this.parseCallersValue(value);
+      }
+      index = nextIndex;
+    }
+    return fields;
+  }
+
+  stripFrontmatter(content: string): string {
+    if (!content.startsWith('---')) {
+      return content;
+    }
+    const end = content.indexOf('\n---', 3);
+    if (end === -1) {
+      return content;
+    }
+    return content.slice(end + 4).replace(/^(?:\r?\n)+/, '');
+  }
+
+  async listSkills(namesOrCaller?: string[] | SkillCaller, caller?: SkillCaller): Promise<SkillMetadata[]> {
+    let names: string[] | undefined;
+    let resolvedCaller: SkillCaller | undefined;
+    if (Array.isArray(namesOrCaller)) {
+      names = namesOrCaller;
+      resolvedCaller = caller;
+    } else if (typeof namesOrCaller === 'string') {
+      resolvedCaller = namesOrCaller;
+    }
+
+    const catalog = await this.loadCatalog();
+    let skills = [...catalog.values()].sort((left, right) => left.name.localeCompare(right.name));
+    if (resolvedCaller) {
+      skills = skills.filter((skill) => this.isAllowedFor(skill, resolvedCaller));
+    }
+    if (!names || names.length === 0) {
+      return skills;
+    }
+    const requested = new Set(names.map((name) => name.toLowerCase()));
+    return skills.filter((skill) => requested.has(skill.name.toLowerCase()));
+  }
+
+  async getSkill(name: string, caller?: SkillCaller): Promise<SkillMetadata | null> {
+    const catalog = await this.loadCatalog();
+    const skill = catalog.get(name.toLowerCase()) ?? null;
+    if (!skill) {
+      return null;
+    }
+    if (caller && !this.isAllowedFor(skill, caller)) {
+      return null;
+    }
+    return skill;
+  }
+
+  isAllowedFor(skill: SkillMetadata, caller: SkillCaller): boolean {
+    if (!skill.callers || skill.callers.length === 0) {
+      return true;
+    }
+    return skill.callers.includes(caller);
+  }
+
+  buildSkillsPrompt(skills: SkillMetadata[], caller?: SkillCaller): string {
+    if (skills.length === 0) {
+      return '';
+    }
+    const list = skills.map((skill) => `- ${skill.name}: ${skill.description}`).join('\n');
+    const captureNote =
+      caller === 'research'
+        ? '\n\nWiki pages and semantic-layer sources are captured automatically by a post-turn memory agent. Focus on answering, not on saving. Use `wiki_read`/`wiki_search` and `sl_read_source` to consult what already exists; the memory agent will write any new conventions or measures the turn surfaces.'
+        : '';
+    return `\n## Skills\n\nUse the \`load_skill\` tool to load a skill when the task benefits from specialized instructions.${captureNote}\n\nAvailable skills:\n${list}\n`;
+  }
+
+  private parseCallersValue(raw: string): SkillCaller[] | undefined {
+    const trimmed = raw.trim();
+    if (!trimmed) {
+      return undefined;
+    }
+    const inner = trimmed.startsWith('[') && trimmed.endsWith(']') ? trimmed.slice(1, -1) : trimmed;
+    const parts = inner
+      .split(',')
+      .map((part) => part.trim().replace(/^['"]|['"]$/g, ''))
+      .filter(Boolean);
+    if (parts.length === 0) {
+      return undefined;
+    }
+
+    const valid: SkillCaller[] = [];
+    for (const part of parts) {
+      if (VALID_CALLERS.has(part as SkillCaller)) {
+        valid.push(part as SkillCaller);
+      } else {
+        this.logger.warn(`Unknown caller '${part}' in skill frontmatter; ignoring`);
+      }
+    }
+    return valid.length > 0 ? valid : undefined;
+  }
+}