ktx/packages/cli/test/telemetry/command-hook.test.ts

133 lines
4 KiB
TypeScript
Raw Permalink Normal View History

feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205) * feat: add telemetry phase 1 * feat: add node telemetry event catalog * feat: add telemetry event helpers * feat: emit setup and connection telemetry * feat: emit connection and stack telemetry * feat: emit ingest and scan telemetry * feat: emit query telemetry * feat: emit sampled mcp telemetry * docs: expand telemetry event catalog * feat: add telemetry schema sync artifact * feat: pass telemetry project id to semantic daemon * feat: add daemon telemetry foundation * feat: emit semantic daemon telemetry * feat: emit daemon lifecycle telemetry * docs: document full telemetry event catalog * feat(telemetry): dim first-run notice * feat(telemetry): show first-run notice before command output * feat(telemetry): wire ktx PostHog project for live ingestion * docs(telemetry): drop posthog project name and host from storage section * docs(telemetry): trim to general overview and disclaimer * docs(agents): add short telemetry guidelines * feat(telemetry): enable posthog geoip enrichment * docs(telemetry): drop ip-geoip note from public overview * refactor(telemetry): drop no-op groupIdentify, rely on capture groups field * fix(telemetry): respect CI kill switch in python daemon identity * fix(sql): route table-count analysis to existing analyze-batch endpoint * fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander * fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check * fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests * fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
import { describe, expect, it } from 'vitest';
fix(cli): classify ktx setup abandonment as aborted, not a blank error (#278) * fix(cli): classify ktx setup abandonment as aborted, not a blank error ktx setup returned a non-zero exit code without throwing when a user abandoned the interactive wizard, so the command telemetry recorded outcome=error with no errorClass/errorDetail — an unactionable blank in the errors dashboard, where most ktx setup "errors" were really people backing out of the wizard. Add annotateCommandOutcome() to the command span so the setup flow (the decision-maker) records the true outcome: genuine step failures and --no-input missing input become outcome=error with a self-diagnosing reason, while interactive abandonment and project cancellation become outcome=aborted and drop out of the error view. Unify the exit code and telemetry through setupTerminalOutcome() so they can never diverge: aborts now exit 0 (matching the entry-menu Exit, project cancel, and a confirmed Ctrl+C), while failures and automation errors still exit 1. * fix(cli): treat non-TTY setup missing-input as an error, not an abort setupTerminalOutcome classified `missing-input` by `args.inputMode`, but `auto` only means "interactive if a TTY is attached". A piped/CI `ktx setup` without `--no-input` and without `--yes` is still `auto`, yet the project and agents steps return `missing-input` there without ever prompting (e.g. "pass --yes to create a project outside an interactive terminal"). Classifying that as `aborted` made a broken automation run exit 0 — a silent failure. Key the classification off actual interactivity instead: input enabled AND `io.stdout.isTTY === true`. Non-interactive missing-input now exits 1 with a `KtxSetupMissingInput` reason; only a genuine interactive abort exits 0. Adds a non-TTY regression test and fixes the abandonment test to use a real TTY.
2026-06-09 12:53:15 +02:00
import {
annotateCommandOutcome,
beginCommandSpan,
completeCommandSpan,
resetCommandSpan,
} from '../../src/telemetry/command-hook.js';
feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205) * feat: add telemetry phase 1 * feat: add node telemetry event catalog * feat: add telemetry event helpers * feat: emit setup and connection telemetry * feat: emit connection and stack telemetry * feat: emit ingest and scan telemetry * feat: emit query telemetry * feat: emit sampled mcp telemetry * docs: expand telemetry event catalog * feat: add telemetry schema sync artifact * feat: pass telemetry project id to semantic daemon * feat: add daemon telemetry foundation * feat: emit semantic daemon telemetry * feat: emit daemon lifecycle telemetry * docs: document full telemetry event catalog * feat(telemetry): dim first-run notice * feat(telemetry): show first-run notice before command output * feat(telemetry): wire ktx PostHog project for live ingestion * docs(telemetry): drop posthog project name and host from storage section * docs(telemetry): trim to general overview and disclaimer * docs(agents): add short telemetry guidelines * feat(telemetry): enable posthog geoip enrichment * docs(telemetry): drop ip-geoip note from public overview * refactor(telemetry): drop no-op groupIdentify, rely on capture groups field * fix(telemetry): respect CI kill switch in python daemon identity * fix(sql): route table-count analysis to existing analyze-batch endpoint * fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander * fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check * fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests * fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
describe('telemetry command hook', () => {
it('builds a completed command event from a span', () => {
resetCommandSpan();
beginCommandSpan({
commandPath: ['ktx', 'status'],
flagsPresent: { projectDir: true, json: true },
projectDir: '/tmp/private',
hasProject: true,
attachProjectGroup: true,
startedAt: 100,
});
expect(
completeCommandSpan({
completedAt: 125,
outcome: 'ok',
}),
).toEqual({
commandPath: ['ktx', 'status'],
durationMs: 25,
outcome: 'ok',
flagsPresent: { projectDir: true, json: true },
hasProject: true,
projectDir: '/tmp/private',
projectGroupAttached: true,
});
});
it('returns undefined when no preAction span exists', () => {
resetCommandSpan();
expect(completeCommandSpan({ completedAt: 200, outcome: 'ok' })).toBeUndefined();
});
it('captures errorClass and raw errorDetail on a failed command', () => {
resetCommandSpan();
beginCommandSpan({
commandPath: ['ktx', 'ingest'],
flagsPresent: {},
hasProject: true,
attachProjectGroup: false,
startedAt: 0,
});
class KtxConnectionError extends Error {}
const error = new KtxConnectionError('connect ECONNREFUSED 127.0.0.1:5432');
const completed = completeCommandSpan({ completedAt: 10, outcome: 'error', error });
expect(completed?.outcome).toBe('error');
expect(completed?.errorClass).toBe('KtxConnectionError');
expect(completed?.errorDetail).toBe('connect ECONNREFUSED 127.0.0.1:5432');
});
fix(cli): classify ktx setup abandonment as aborted, not a blank error (#278) * fix(cli): classify ktx setup abandonment as aborted, not a blank error ktx setup returned a non-zero exit code without throwing when a user abandoned the interactive wizard, so the command telemetry recorded outcome=error with no errorClass/errorDetail — an unactionable blank in the errors dashboard, where most ktx setup "errors" were really people backing out of the wizard. Add annotateCommandOutcome() to the command span so the setup flow (the decision-maker) records the true outcome: genuine step failures and --no-input missing input become outcome=error with a self-diagnosing reason, while interactive abandonment and project cancellation become outcome=aborted and drop out of the error view. Unify the exit code and telemetry through setupTerminalOutcome() so they can never diverge: aborts now exit 0 (matching the entry-menu Exit, project cancel, and a confirmed Ctrl+C), while failures and automation errors still exit 1. * fix(cli): treat non-TTY setup missing-input as an error, not an abort setupTerminalOutcome classified `missing-input` by `args.inputMode`, but `auto` only means "interactive if a TTY is attached". A piped/CI `ktx setup` without `--no-input` and without `--yes` is still `auto`, yet the project and agents steps return `missing-input` there without ever prompting (e.g. "pass --yes to create a project outside an interactive terminal"). Classifying that as `aborted` made a broken automation run exit 0 — a silent failure. Key the classification off actual interactivity instead: input enabled AND `io.stdout.isTTY === true`. Non-interactive missing-input now exits 1 with a `KtxSetupMissingInput` reason; only a genuine interactive abort exits 0. Adds a non-TTY regression test and fixes the abandonment test to use a real TTY.
2026-06-09 12:53:15 +02:00
it('applies an annotated outcome when no error is thrown', () => {
resetCommandSpan();
beginCommandSpan({
commandPath: ['ktx', 'setup'],
flagsPresent: {},
hasProject: false,
attachProjectGroup: true,
startedAt: 0,
});
annotateCommandOutcome({ outcome: 'aborted' });
// The wrapper derives 'error' from a non-zero exit code, but the action
// knows the user aborted — the annotation must win on the non-throwing path.
const completed = completeCommandSpan({ completedAt: 5, outcome: 'error' });
expect(completed?.outcome).toBe('aborted');
expect(completed?.errorClass).toBeUndefined();
expect(completed?.errorDetail).toBeUndefined();
});
it('applies an annotated reason so a non-throwing failure is self-diagnosing', () => {
resetCommandSpan();
beginCommandSpan({
commandPath: ['ktx', 'setup'],
flagsPresent: {},
hasProject: false,
attachProjectGroup: true,
startedAt: 0,
});
annotateCommandOutcome({
outcome: 'error',
errorClass: 'KtxSetupStepFailed',
errorDetail: 'runtime setup step failed',
});
const completed = completeCommandSpan({ completedAt: 5, outcome: 'error' });
expect(completed?.outcome).toBe('error');
expect(completed?.errorClass).toBe('KtxSetupStepFailed');
expect(completed?.errorDetail).toBe('runtime setup step failed');
});
it('lets a thrown error take precedence over an annotation', () => {
resetCommandSpan();
beginCommandSpan({
commandPath: ['ktx', 'setup'],
flagsPresent: {},
hasProject: false,
attachProjectGroup: true,
startedAt: 0,
});
annotateCommandOutcome({ outcome: 'aborted', errorDetail: 'user aborted' });
class KtxSetupBoomError extends Error {}
const completed = completeCommandSpan({
completedAt: 5,
outcome: 'error',
error: new KtxSetupBoomError('boom'),
});
expect(completed?.outcome).toBe('error');
expect(completed?.errorClass).toBe('KtxSetupBoomError');
expect(completed?.errorDetail).toBe('boom');
});
it('ignores annotation when no span is active', () => {
resetCommandSpan();
expect(() => annotateCommandOutcome({ outcome: 'aborted' })).not.toThrow();
expect(completeCommandSpan({ completedAt: 1, outcome: 'ok' })).toBeUndefined();
});
feat(telemetry): anonymous posthog usage telemetry across node cli and python daemon (#205) * feat: add telemetry phase 1 * feat: add node telemetry event catalog * feat: add telemetry event helpers * feat: emit setup and connection telemetry * feat: emit connection and stack telemetry * feat: emit ingest and scan telemetry * feat: emit query telemetry * feat: emit sampled mcp telemetry * docs: expand telemetry event catalog * feat: add telemetry schema sync artifact * feat: pass telemetry project id to semantic daemon * feat: add daemon telemetry foundation * feat: emit semantic daemon telemetry * feat: emit daemon lifecycle telemetry * docs: document full telemetry event catalog * feat(telemetry): dim first-run notice * feat(telemetry): show first-run notice before command output * feat(telemetry): wire ktx PostHog project for live ingestion * docs(telemetry): drop posthog project name and host from storage section * docs(telemetry): trim to general overview and disclaimer * docs(agents): add short telemetry guidelines * feat(telemetry): enable posthog geoip enrichment * docs(telemetry): drop ip-geoip note from public overview * refactor(telemetry): drop no-op groupIdentify, rely on capture groups field * fix(telemetry): respect CI kill switch in python daemon identity * fix(sql): route table-count analysis to existing analyze-batch endpoint * fix(telemetry): emit install_first_run from notice path and derive flagsPresent from commander * fix(telemetry): read package info via getKtxCliPackageInfo to satisfy boundary check * fix(telemetry): make python identity env={} bypass os.environ and unset CI in tests * fix(telemetry): unset CI kill switch in cli-program-telemetry tests
2026-05-22 18:18:47 +02:00
});