mirror of
https://github.com/Kaelio/ktx.git
synced 2026-07-04 10:52:13 +02:00
feat: merge ingest and scan
* docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
This commit is contained in:
Andrey Avtomonov
GitHub
parent
1a472cf3ed
commit
b00c1a11a9
118 changed files with 16890 additions and 2992 deletions
|
|
@ -35,7 +35,7 @@ import { profileMark } from './startup-profile.js';
|
|||
|
||||
profileMark('module:ingest');
|
||||
|
||||
export type KtxIngestOutputMode = 'plain' | 'json' | 'viz';
|
||||
type KtxIngestOutputMode = 'plain' | 'json' | 'viz';
|
||||
type KtxIngestInputMode = 'auto' | 'disabled';
|
||||
|
||||
export type KtxIngestArgs =
|
||||
|
|
@ -49,6 +49,8 @@ export type KtxIngestArgs =
|
|||
cliVersion?: string;
|
||||
runtimeInstallPolicy?: KtxManagedPythonInstallPolicy;
|
||||
debugLlmRequestFile?: string;
|
||||
allowImplicitAdapter?: boolean;
|
||||
historicSqlPullConfigOverride?: Record<string, unknown>;
|
||||
outputMode: KtxIngestOutputMode;
|
||||
inputMode?: KtxIngestInputMode;
|
||||
}
|
||||
|
|
@ -101,19 +103,42 @@ function reportStatus(report: IngestReportSnapshot): 'done' | 'error' {
|
|||
return report.body.failedWorkUnits.length > 0 ? 'error' : 'done';
|
||||
}
|
||||
|
||||
const REPORT_SOURCE_LABELS = new Map<string, string>([
|
||||
['live-database', 'Database schema'],
|
||||
['historic-sql', 'Query history'],
|
||||
['dbt', 'dbt'],
|
||||
['metricflow', 'MetricFlow'],
|
||||
['lookml', 'LookML'],
|
||||
['looker', 'Looker'],
|
||||
['metabase', 'Metabase'],
|
||||
['notion', 'Notion'],
|
||||
]);
|
||||
|
||||
function reportSourceLabel(sourceKey: string): string {
|
||||
const label = REPORT_SOURCE_LABELS.get(sourceKey);
|
||||
if (label) {
|
||||
return label;
|
||||
}
|
||||
return sourceKey
|
||||
.split(/[-_]+/)
|
||||
.filter((part) => part.length > 0)
|
||||
.map((part) => `${part[0]?.toUpperCase() ?? ''}${part.slice(1)}`)
|
||||
.join(' ');
|
||||
}
|
||||
|
||||
function writeReportStatus(report: IngestReportSnapshot, io: KtxIngestIo): void {
|
||||
const counts = savedMemoryCountsForReport(report);
|
||||
io.stdout.write(`Report: ${report.id}\n`);
|
||||
io.stdout.write(`Run: ${report.runId}\n`);
|
||||
io.stdout.write(`Job: ${report.jobId}\n`);
|
||||
io.stdout.write(`Status: ${reportStatus(report)}\n`);
|
||||
io.stdout.write(`Adapter: ${report.sourceKey}\n`);
|
||||
io.stdout.write(`Source: ${reportSourceLabel(report.sourceKey)}\n`);
|
||||
io.stdout.write(`Connection: ${report.connectionId}\n`);
|
||||
io.stdout.write(`Sync: ${report.body.syncId}\n`);
|
||||
io.stdout.write(
|
||||
`Diff: +${report.body.diffSummary.added}/~${report.body.diffSummary.modified}/-${report.body.diffSummary.deleted}/=${report.body.diffSummary.unchanged}\n`,
|
||||
);
|
||||
io.stdout.write(`Work units: ${report.body.workUnits.length}\n`);
|
||||
io.stdout.write(`Tasks: ${report.body.workUnits.length}\n`);
|
||||
io.stdout.write(`Saved memory: ${counts.wikiCount} wiki, ${counts.slCount} SL\n`);
|
||||
io.stdout.write(`Provenance rows: ${report.body.provenanceRows.length}\n`);
|
||||
}
|
||||
|
|
@ -133,8 +158,8 @@ function writeMetabaseFanoutStatus(result: LocalMetabaseFanoutResult, io: KtxIng
|
|||
io.stdout.write(`Source: ${result.metabaseConnectionId}\n`);
|
||||
io.stdout.write(`Children: ${result.children.length}\n`);
|
||||
if (result.totals) {
|
||||
io.stdout.write(`Work units: ${result.totals.workUnits}\n`);
|
||||
io.stdout.write(`Failed work units: ${result.totals.failedWorkUnits}\n`);
|
||||
io.stdout.write(`Tasks: ${result.totals.workUnits}\n`);
|
||||
io.stdout.write(`Failed tasks: ${result.totals.failedWorkUnits}\n`);
|
||||
}
|
||||
io.stdout.write(`Saved memory: ${counts.wikiCount} wiki, ${counts.slCount} SL\n`);
|
||||
for (const child of result.children) {
|
||||
|
|
@ -255,19 +280,19 @@ function plainIngestEventProgress(
|
|||
if (event.workUnitCount === 0) {
|
||||
return {
|
||||
percent: 80,
|
||||
message: 'No work units to process; finalizing ingest',
|
||||
message: 'No tasks to process; finalizing ingest',
|
||||
};
|
||||
}
|
||||
return {
|
||||
percent: 45,
|
||||
message: `Planned ${pluralize(event.workUnitCount, 'work unit')}`,
|
||||
message: `Planned ${pluralize(event.workUnitCount, 'task')}`,
|
||||
};
|
||||
case 'stage_skipped':
|
||||
return { percent: 45, message: `Skipped ${event.stage}: ${event.reason}` };
|
||||
case 'work_unit_started': {
|
||||
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
||||
const ordinal = workUnitOrdinalThrough(snapshot, eventIndex, event.unitKey);
|
||||
const progress = total > 0 ? `${ordinal}/${total} work units: ` : '';
|
||||
const progress = total > 0 ? `${ordinal}/${total} tasks: ` : '';
|
||||
return { percent: 55, message: `Processing ${progress}${event.unitKey}` };
|
||||
}
|
||||
case 'work_unit_step': {
|
||||
|
|
@ -279,7 +304,7 @@ function plainIngestEventProgress(
|
|||
const latest = `${event.unitKey} step ${event.stepIndex}/${event.stepBudget}`;
|
||||
return {
|
||||
percent,
|
||||
message: `Processing work units: ${completed}/${total} complete, ${active} active; latest ${latest}`,
|
||||
message: `Processing tasks: ${completed}/${total} complete, ${active} active; latest ${latest}`,
|
||||
transient: true,
|
||||
};
|
||||
}
|
||||
|
|
@ -289,7 +314,7 @@ function plainIngestEventProgress(
|
|||
const percent = total > 0 ? 55 + Math.round((completed / total) * 25) : 80;
|
||||
return {
|
||||
percent,
|
||||
message: `Processed ${completed}/${total} work units`,
|
||||
message: `Processed ${completed}/${total} tasks`,
|
||||
};
|
||||
}
|
||||
case 'reconciliation_finished':
|
||||
|
|
@ -571,6 +596,19 @@ export async function runKtxIngest(
|
|||
const project = await loadKtxProject({ projectDir: args.projectDir });
|
||||
const env = deps.env ?? process.env;
|
||||
if (args.command === 'run') {
|
||||
const ingestProject =
|
||||
args.allowImplicitAdapter && !project.config.ingest.adapters.includes(args.adapter)
|
||||
? {
|
||||
...project,
|
||||
config: {
|
||||
...project.config,
|
||||
ingest: {
|
||||
...project.config.ingest,
|
||||
adapters: [...project.config.ingest.adapters, args.adapter],
|
||||
},
|
||||
},
|
||||
}
|
||||
: project;
|
||||
const createAdapters =
|
||||
deps.createAdapters ??
|
||||
(deps.runLocalIngest || deps.runLocalMetabaseIngest ? () => [] : createKtxCliLocalIngestAdapters);
|
||||
|
|
@ -583,11 +621,14 @@ export async function runKtxIngest(
|
|||
...(args.databaseIntrospectionUrl ? { databaseIntrospectionUrl: args.databaseIntrospectionUrl } : {}),
|
||||
...(managedDaemon ? { managedDaemon } : {}),
|
||||
...(args.adapter === 'historic-sql' ? { historicSqlConnectionId: args.connectionId } : {}),
|
||||
...(args.historicSqlPullConfigOverride
|
||||
? { historicSqlPullConfigOverride: args.historicSqlPullConfigOverride }
|
||||
: {}),
|
||||
logger: operationalLogger,
|
||||
};
|
||||
const queryExecutor =
|
||||
localIngestOptions.queryExecutor ??
|
||||
(deps.createQueryExecutor ?? createKtxCliIngestQueryExecutor)(project);
|
||||
(deps.createQueryExecutor ?? createKtxCliIngestQueryExecutor)(ingestProject);
|
||||
if (args.adapter === 'metabase' && args.sourceDir) {
|
||||
throw new Error('source-dir uploads are not supported for the Metabase fan-out adapter');
|
||||
}
|
||||
|
|
@ -604,8 +645,8 @@ export async function runKtxIngest(
|
|||
deps.progress,
|
||||
);
|
||||
const result = await executeMetabaseFanout({
|
||||
project,
|
||||
adapters: createAdapters(project, adapterOptions),
|
||||
project: ingestProject,
|
||||
adapters: createAdapters(ingestProject, adapterOptions),
|
||||
metabaseConnectionId: args.connectionId,
|
||||
...localIngestOptions,
|
||||
queryExecutor,
|
||||
|
|
@ -668,8 +709,8 @@ export async function runKtxIngest(
|
|||
|
||||
try {
|
||||
const result = await executeLocalIngest({
|
||||
project,
|
||||
adapters: createAdapters(project, adapterOptions),
|
||||
project: ingestProject,
|
||||
adapters: createAdapters(ingestProject, adapterOptions),
|
||||
adapter: args.adapter,
|
||||
connectionId: args.connectionId,
|
||||
sourceDir: args.sourceDir,
|
||||
|
|
@ -720,7 +761,7 @@ export async function runKtxIngest(
|
|||
throw new Error(
|
||||
args.runId
|
||||
? `Local ingest run or report "${args.runId}" was not found`
|
||||
: 'No local ingest reports were found. Run `ktx ingest run --connection-id <id> --adapter <adapter>` first.',
|
||||
: 'No local ingest reports were found. Run `ktx ingest <connectionId>` first.',
|
||||
);
|
||||
}
|
||||
await writeReportRecord(report, args.outputMode, io, {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue