ktx/packages/cli/test/demo-assets.test.ts

170 lines
6.9 KiB
TypeScript
Raw Normal View History

import { access, readFile, rm, stat } from 'node:fs/promises';
2026-05-10 23:12:26 +02:00
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205) * feat: add telemetry phase 1 * feat: add node telemetry event catalog * feat: add telemetry event helpers * feat: emit setup and connection telemetry * feat: emit connection and stack telemetry * feat: emit ingest and scan telemetry * feat: emit query telemetry * feat: emit sampled mcp telemetry * docs: expand telemetry event catalog * feat: add telemetry schema sync artifact * feat: pass telemetry project id to semantic daemon * feat: add daemon telemetry foundation * feat: emit semantic daemon telemetry * feat: emit daemon lifecycle telemetry * docs: document full telemetry event catalog * feat(telemetry): dim first-run notice * feat(telemetry): show first-run notice before command output * feat(telemetry): wire ktx PostHog project for live ingestion * docs(telemetry): drop posthog project name and host from storage section * docs(telemetry): trim to general overview and disclaimer * docs(agents): add short telemetry guidelines * feat(telemetry): enable posthog geoip enrichment * docs(telemetry): drop ip-geoip note from public overview * refactor(telemetry): drop no-op groupIdentify, rely on capture groups field * fix(telemetry): respect CI kill switch in python daemon identity * fix(sql): route table-count analysis to existing analyze-batch endpoint * fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander * fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check * fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests * fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
import { afterEach, describe, expect, it, vi } from 'vitest';
2026-05-10 23:12:26 +02:00
import {
DEMO_ADAPTER,
DEMO_CONNECTION_ID,
DEMO_REPLAY_FILE,
defaultDemoProjectDir,
ensureDemoProject,
ensureSeededDemoProject,
test: split cli tests from source tree (#216) * feat(cli): define full warehouse dialect contract * test(cli): keep dialect edge tests focused * fix(cli): stabilize dialect contract foundation * refactor(connectors): own read-only query preparation * refactor(connectors): resolve dialects through registry * refactor(connectors): keep concrete dialect classes internal * chore(workspace): enforce dialect import boundary * refactor(cli): resolve relationship dialect at scan boundary * refactor(cli): use dialect display parsing for entity details * refactor(cli): use dialect display parsing for warehouse catalog * refactor(cli): use dialect SQL in relationship workflows * test(cli): verify solid dialect scan workflow closure * test: split cli tests from source tree * refactor(cli): standardize BigQuery scope listing * feat(sqlite): implement connector scope listing * test(connectors): cover required table listing * feat(cli): add warehouse driver registry * refactor(setup): route scope discovery through driver registry * refactor(cli): route local query execution through driver registry * refactor(historic-sql): route dialect support through driver registry * refactor(cli): test warehouse connections through driver registry * fix(cli): close driver registry type export gaps * Improve setup daemon diagnostics * refactor(setup): centralize rail-prefixed diagnostics + query-history fallback Extract errorMessage, writePrefixedLines, and flushPrefixedBufferedCommandOutput into clack.ts so the setup wizard, managed daemons, and embedding/agent steps share one rail-formatted writer. setup-databases.ts also adds a "disable query history and retry" option when the schema-context build fails and query history is the likely culprit, surfaced via a new failed-query-history-unavailable status. * fix(cli): carry catalog through the picker so BigQuery/Snowflake/SQL Server scope filters match The setup picker's KtxTableListEntry was a 2-level { schema, name }, so qualifiedTableId always wrote db.name into enabled_tables. When BigQuery, Snowflake, or SQL Server later ran fast ingest, their introspect step filtered the scope set with scopedTableNames(scope, { catalog: projectId|database, db }) — catalog was non-null on the introspect side but null in the scope refs, so every entry was rejected, the live-database adapter staged zero table files, and detect() failed with 'Adapter "live-database" did not recognize fetched source output'. Align the picker boundary with the canonical 3-level KtxTableRef: - Add catalog: string | null to KtxTableListEntry. - BigQuery/Snowflake/SQL Server listTables populate catalog from the resolved projectId / database; Postgres/MySQL/ClickHouse/SQLite set null. - qualifiedTableId emits catalog.schema.name when catalog is non-null (resolveEnabledTables already accepts the 3-part shape) and schemasFromEnabledTables now goes through parseDottedTableEntry so it recovers the schema correctly from both 2-part and 3-part entries. - Export parseDottedTableEntry from enabled-tables.ts (@internal) for picker reuse. Update listTables expectations in all seven connector tests and the setup / picker test fixtures. Add a picker regression test that covers the catalog-bearing round-trip (save + refine). * fix(cli): allow debug telemetry under opt-out env
2026-05-26 08:49:05 +02:00
} from '../src/demo-assets.js';
2026-05-10 23:12:26 +02:00
const packagedDemoSource = 'packaged-orbit-demo';
fix(cli): isolate ktx-owned project repositories (#283) * fix(cli): isolate ktx project git repos * fix(cli): remove inert auto commit config * test(cli): drop stale auto commit fixtures * docs: document isolated ktx project repos * test(cli): keep stale config grep clean * fix(cli): guide setup away from foreign repos at the project dir ktx owns the git repo rooted at the project dir and refuses to adopt one it did not create (the Finding 3 isolation invariant). But setup steered users straight into that failure: the interactive menu offers "Current directory" first, and `--no-input --yes --project-dir <repo-root>` created directly in place — both then threw a generic "Failed to initialize git repository:" wrapper from deep in GitService.initialize(). Extract the ownership rule into a shared `classifyKtxRepoOwnership(dir)` used by both GitService.initialize() (the invariant) and the setup wizard (pre-flight guidance), so the decision derives from one rule. Setup now detects a foreign repo before constructing GitService and: interactively re-prompts (the user picks the existing `ktx-project` subfolder), or non-interactively returns a clean missing-input with the actionable message. The typed foreign-repo error is also surfaced verbatim instead of being buried under the generic wrapper. Empty/non-repo current directories still work — only foreign repos are blocked. * fix(cli): keep classifyKtxRepoOwnership total for non-directory paths The setup ownership guard runs before the existing not-a-directory check, so pointing a custom/--project-dir path at a file made classifyKtxRepoOwnership lstat `<file>/.git`, hit ENOTDIR, and throw — crashing the setup step instead of returning the friendly "path exists and is not a directory" result. A path that is a file (or missing) holds no git repo for ktx to avoid, so treat ENOTDIR like ENOENT and return 'unowned'. The downstream existingFolderState check still rejects a non-directory with its friendly message, and the classifier no longer throws raw errno for any caller.
2026-06-10 14:12:25 +02:00
const removedAutoCommitKey = ['auto', 'commit'].join('_');
2026-05-10 23:12:26 +02:00
function packagedDemoAssetPath(relativePath: string): string {
return fileURLToPath(new URL(`../assets/demo/orbit/${relativePath}`, import.meta.url));
}
async function readPackagedJson<T>(relativePath: string): Promise<T> {
return JSON.parse(await readFile(packagedDemoAssetPath(relativePath), 'utf-8')) as T;
}
feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205) * feat: add telemetry phase 1 * feat: add node telemetry event catalog * feat: add telemetry event helpers * feat: emit setup and connection telemetry * feat: emit connection and stack telemetry * feat: emit ingest and scan telemetry * feat: emit query telemetry * feat: emit sampled mcp telemetry * docs: expand telemetry event catalog * feat: add telemetry schema sync artifact * feat: pass telemetry project id to semantic daemon * feat: add daemon telemetry foundation * feat: emit semantic daemon telemetry * feat: emit daemon lifecycle telemetry * docs: document full telemetry event catalog * feat(telemetry): dim first-run notice * feat(telemetry): show first-run notice before command output * feat(telemetry): wire ktx PostHog project for live ingestion * docs(telemetry): drop posthog project name and host from storage section * docs(telemetry): trim to general overview and disclaimer * docs(agents): add short telemetry guidelines * feat(telemetry): enable posthog geoip enrichment * docs(telemetry): drop ip-geoip note from public overview * refactor(telemetry): drop no-op groupIdentify, rely on capture groups field * fix(telemetry): respect CI kill switch in python daemon identity * fix(sql): route table-count analysis to existing analyze-batch endpoint * fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander * fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check * fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests * fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
function makeIo() {
let stderr = '';
return {
stdout: {
isTTY: true,
write() {},
},
stderr: {
write(chunk: string) {
stderr += chunk;
},
},
stderrText: () => stderr,
};
}
2026-05-10 23:12:26 +02:00
describe('demo assets', () => {
2026-05-10 23:51:24 +02:00
const projectDir = join(tmpdir(), `ktx-demo-assets-${process.pid}`);
2026-05-10 23:12:26 +02:00
afterEach(async () => {
feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205) * feat: add telemetry phase 1 * feat: add node telemetry event catalog * feat: add telemetry event helpers * feat: emit setup and connection telemetry * feat: emit connection and stack telemetry * feat: emit ingest and scan telemetry * feat: emit query telemetry * feat: emit sampled mcp telemetry * docs: expand telemetry event catalog * feat: add telemetry schema sync artifact * feat: pass telemetry project id to semantic daemon * feat: add daemon telemetry foundation * feat: emit semantic daemon telemetry * feat: emit daemon lifecycle telemetry * docs: document full telemetry event catalog * feat(telemetry): dim first-run notice * feat(telemetry): show first-run notice before command output * feat(telemetry): wire ktx PostHog project for live ingestion * docs(telemetry): drop posthog project name and host from storage section * docs(telemetry): trim to general overview and disclaimer * docs(agents): add short telemetry guidelines * feat(telemetry): enable posthog geoip enrichment * docs(telemetry): drop ip-geoip note from public overview * refactor(telemetry): drop no-op groupIdentify, rely on capture groups field * fix(telemetry): respect CI kill switch in python daemon identity * fix(sql): route table-count analysis to existing analyze-batch endpoint * fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander * fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check * fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests * fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
vi.unstubAllEnvs();
2026-05-10 23:12:26 +02:00
await rm(projectDir, { recursive: true, force: true });
});
it('resolves the default demo root under the OS temp directory', () => {
const dir = defaultDemoProjectDir();
2026-05-10 23:51:24 +02:00
expect(dir.startsWith(join(tmpdir(), 'ktx-demo-'))).toBe(true);
expect(dir).toMatch(/ktx-demo-[a-f0-9]{8}$/);
2026-05-10 23:12:26 +02:00
});
it('exports the packaged Orbit demo identity', () => {
expect(DEMO_CONNECTION_ID).toBe('orbit_demo');
expect(DEMO_ADAPTER).toBe('live-database');
expect(DEMO_REPLAY_FILE).toBe('replay.memory-flow.v1.json');
});
it('ships the seeded demo bundle required by the May 6 PRD', async () => {
const manifest = await readPackagedJson<{
demoAssetSchemaVersion: number;
mode: string;
source: string;
sources: {
warehouse: { tables: number; rowCounts: Record<string, number> };
dbt: { models: number; sourceTables: number };
bi: { explores: number; dashboards: number };
notion: { pages: number };
};
name: string;
displayName: string;
generated: {
semanticLayer: { path: string; sourceCount: number };
knowledge: { pageCount: number };
links: { linkCount: number };
};
}>('manifest.json');
expect(manifest).toMatchObject({
demoAssetSchemaVersion: 2,
name: 'orbit',
displayName: 'Orbit Demo',
mode: 'seeded',
source: packagedDemoSource,
});
expect(manifest.sources.warehouse.tables).toBeGreaterThanOrEqual(5);
expect(manifest.sources.warehouse.tables).toBeLessThanOrEqual(10);
expect(Object.keys(manifest.sources.warehouse.rowCounts).sort()).toEqual([
'accounts',
'arr_movements',
'contracts',
'invoices',
'plans',
'purchase_requests',
'support_tickets',
'users',
]);
expect(manifest.sources.dbt.models).toBeGreaterThanOrEqual(3);
expect(manifest.sources.dbt.models).toBeLessThanOrEqual(6);
expect(manifest.sources.bi.explores).toBeGreaterThanOrEqual(2);
expect(manifest.sources.bi.dashboards).toBeGreaterThanOrEqual(2);
expect(manifest.sources.notion.pages).toBeGreaterThanOrEqual(5);
expect(manifest.generated.semanticLayer.sourceCount).toBeGreaterThanOrEqual(40);
expect(manifest.generated.knowledge.pageCount).toBeGreaterThanOrEqual(20);
2026-05-10 23:12:26 +02:00
expect(manifest.generated.links.linkCount).toBeGreaterThanOrEqual(10);
const dbStat = await stat(packagedDemoAssetPath('demo.db'));
expect(dbStat.size).toBeGreaterThan(0);
expect(dbStat.size).toBeLessThan(10 * 1024 * 1024);
await expect(access(packagedDemoAssetPath('semantic-layer/dbt-main/mart_arr_daily.yaml'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('semantic-layer/postgres-warehouse/mart_account_activity.yaml'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('wiki/global/orbit-company-overview.md'))).resolves.toBeUndefined();
2026-05-10 23:12:26 +02:00
await expect(access(packagedDemoAssetPath('links/provenance.json'))).resolves.toBeUndefined();
await expect(access(packagedDemoAssetPath('reports/seeded-demo-report.json'))).resolves.toBeUndefined();
});
it('initializes a flat demo project without writing literal credentials', async () => {
const result = await ensureDemoProject({ projectDir, force: false });
expect(result.projectDir).toBe(projectDir);
await expect(access(join(projectDir, 'demo.db'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'state.sqlite'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'reports'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'semantic-layer'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'wiki'))).resolves.toBeUndefined();
2026-05-10 23:12:26 +02:00
await expect(access(join(projectDir, 'replays', 'replay.memory-flow.v1.json'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'raw-sources'))).resolves.toBeUndefined();
await expect(access(join(projectDir, '_schema'))).rejects.toMatchObject({ code: 'ENOENT' });
2026-05-10 23:51:24 +02:00
const config = await readFile(join(projectDir, 'ktx.yaml'), 'utf-8');
2026-05-10 23:12:26 +02:00
expect(config).toContain('backend: anthropic');
expect(config).toContain('api_key: env:ANTHROPIC_API_KEY');
fix(cli): isolate ktx-owned project repositories (#283) * fix(cli): isolate ktx project git repos * fix(cli): remove inert auto commit config * test(cli): drop stale auto commit fixtures * docs: document isolated ktx project repos * test(cli): keep stale config grep clean * fix(cli): guide setup away from foreign repos at the project dir ktx owns the git repo rooted at the project dir and refuses to adopt one it did not create (the Finding 3 isolation invariant). But setup steered users straight into that failure: the interactive menu offers "Current directory" first, and `--no-input --yes --project-dir <repo-root>` created directly in place — both then threw a generic "Failed to initialize git repository:" wrapper from deep in GitService.initialize(). Extract the ownership rule into a shared `classifyKtxRepoOwnership(dir)` used by both GitService.initialize() (the invariant) and the setup wizard (pre-flight guidance), so the decision derives from one rule. Setup now detects a foreign repo before constructing GitService and: interactively re-prompts (the user picks the existing `ktx-project` subfolder), or non-interactively returns a clean missing-input with the actionable message. The typed foreign-repo error is also surfaced verbatim instead of being buried under the generic wrapper. Empty/non-repo current directories still work — only foreign repos are blocked. * fix(cli): keep classifyKtxRepoOwnership total for non-directory paths The setup ownership guard runs before the existing not-a-directory check, so pointing a custom/--project-dir path at a file made classifyKtxRepoOwnership lstat `<file>/.git`, hit ENOTDIR, and throw — crashing the setup step instead of returning the friendly "path exists and is not a directory" result. A path that is a file (or missing) holds no git repo for ktx to avoid, so treat ENOTDIR like ENOENT and return 'unowned'. The downstream existingFolderState check still rejects a non-directory with its friendly message, and the classifier no longer throws raw errno for any caller.
2026-06-10 14:12:25 +02:00
expect(config).not.toContain(removedAutoCommitKey);
expect(config).not.toContain('memory:');
2026-05-10 23:12:26 +02:00
expect(config).not.toContain('sk-ant-');
});
it('rejects an existing demo project unless force is set', async () => {
await ensureDemoProject({ projectDir, force: false });
await expect(ensureDemoProject({ projectDir, force: false })).rejects.toThrow('Demo project already exists');
await expect(ensureDemoProject({ projectDir, force: true })).resolves.toMatchObject({ projectDir });
});
feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205) * feat: add telemetry phase 1 * feat: add node telemetry event catalog * feat: add telemetry event helpers * feat: emit setup and connection telemetry * feat: emit connection and stack telemetry * feat: emit ingest and scan telemetry * feat: emit query telemetry * feat: emit sampled mcp telemetry * docs: expand telemetry event catalog * feat: add telemetry schema sync artifact * feat: pass telemetry project id to semantic daemon * feat: add daemon telemetry foundation * feat: emit semantic daemon telemetry * feat: emit daemon lifecycle telemetry * docs: document full telemetry event catalog * feat(telemetry): dim first-run notice * feat(telemetry): show first-run notice before command output * feat(telemetry): wire ktx PostHog project for live ingestion * docs(telemetry): drop posthog project name and host from storage section * docs(telemetry): trim to general overview and disclaimer * docs(agents): add short telemetry guidelines * feat(telemetry): enable posthog geoip enrichment * docs(telemetry): drop ip-geoip note from public overview * refactor(telemetry): drop no-op groupIdentify, rely on capture groups field * fix(telemetry): respect CI kill switch in python daemon identity * fix(sql): route table-count analysis to existing analyze-batch endpoint * fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander * fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check * fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests * fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
it('emits debug telemetry when the demo connection is created', async () => {
vi.stubEnv('KTX_TELEMETRY_DEBUG', '1');
vi.stubEnv('CI', '');
const io = makeIo();
await ensureDemoProject({ projectDir, force: false, io, cliVersion: '0.2.0' });
expect(io.stderrText()).toContain('"event":"connection_added"');
expect(io.stderrText()).toContain('"driver":"sqlite"');
expect(io.stderrText()).toContain('"isDemoConnection":true');
expect(io.stderrText()).not.toContain(projectDir);
});
it('copies the seeded project assets used by the setup wizard tour', async () => {
await ensureSeededDemoProject({ projectDir, force: false });
2026-05-10 23:12:26 +02:00
await expect(access(join(projectDir, 'semantic-layer', 'dbt-main', 'mart_arr_daily.yaml'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'wiki', 'global', 'orbit-company-overview.md'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'links', 'provenance.json'))).resolves.toBeUndefined();
await expect(access(join(projectDir, 'reports', 'seeded-demo-report.json'))).resolves.toBeUndefined();
2026-05-10 23:12:26 +02:00
});
});