mirror of
https://github.com/Kaelio/ktx.git
synced 2026-07-22 11:51:01 +02:00
feat: merge ingest and scan
* docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
This commit is contained in:
Andrey Avtomonov
GitHub
parent
1a472cf3ed
commit
b00c1a11a9
118 changed files with 16890 additions and 2992 deletions
|
|
@ -109,12 +109,12 @@ function shouldShowSetupEntryMenu(
|
|||
newDatabaseConnectionId?: string;
|
||||
databaseUrl?: string;
|
||||
databaseSchema?: string[];
|
||||
enableHistoricSql?: boolean;
|
||||
disableHistoricSql?: boolean;
|
||||
historicSqlWindowDays?: number;
|
||||
historicSqlMinExecutions?: number;
|
||||
historicSqlServiceAccountPattern?: string[];
|
||||
historicSqlRedactionPattern?: string[];
|
||||
enableQueryHistory?: boolean;
|
||||
disableQueryHistory?: boolean;
|
||||
queryHistoryWindowDays?: number;
|
||||
queryHistoryMinExecutions?: number;
|
||||
queryHistoryServiceAccountPattern?: string[];
|
||||
queryHistoryRedactionPattern?: string[];
|
||||
skipDatabases?: boolean;
|
||||
source?: KtxSetupSourceType;
|
||||
sourceConnectionId?: string;
|
||||
|
|
@ -147,10 +147,10 @@ function shouldShowSetupEntryMenu(
|
|||
if (options.databaseSchema && options.databaseSchema.length > 0) {
|
||||
return false;
|
||||
}
|
||||
if (options.historicSqlServiceAccountPattern && options.historicSqlServiceAccountPattern.length > 0) {
|
||||
if (options.queryHistoryServiceAccountPattern && options.queryHistoryServiceAccountPattern.length > 0) {
|
||||
return false;
|
||||
}
|
||||
if (options.historicSqlRedactionPattern && options.historicSqlRedactionPattern.length > 0) {
|
||||
if (options.queryHistoryRedactionPattern && options.queryHistoryRedactionPattern.length > 0) {
|
||||
return false;
|
||||
}
|
||||
if (options.notionRootPageId && options.notionRootPageId.length > 0) {
|
||||
|
|
@ -179,10 +179,10 @@ function shouldShowSetupEntryMenu(
|
|||
'skipEmbeddings',
|
||||
'newDatabaseConnectionId',
|
||||
'databaseUrl',
|
||||
'enableHistoricSql',
|
||||
'disableHistoricSql',
|
||||
'historicSqlWindowDays',
|
||||
'historicSqlMinExecutions',
|
||||
'enableQueryHistory',
|
||||
'disableQueryHistory',
|
||||
'queryHistoryWindowDays',
|
||||
'queryHistoryMinExecutions',
|
||||
'skipDatabases',
|
||||
'source',
|
||||
'sourceConnectionId',
|
||||
|
|
@ -282,33 +282,37 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
|
|||
.hideHelp(),
|
||||
)
|
||||
.addOption(
|
||||
new Option('--enable-historic-sql', 'Enable Historic SQL when the selected database supports it')
|
||||
new Option('--enable-query-history', 'Enable query history when the selected database supports it')
|
||||
.hideHelp()
|
||||
.default(false),
|
||||
)
|
||||
.addOption(
|
||||
new Option('--disable-historic-sql', 'Disable Historic SQL for the selected database').hideHelp().default(false),
|
||||
new Option('--disable-query-history', 'Disable query history for the selected database').hideHelp().default(false),
|
||||
)
|
||||
.addOption(new Option('--historic-sql-window-days <number>', 'Historic SQL query-history window').argParser(positiveInteger).hideHelp())
|
||||
.addOption(
|
||||
new Option('--historic-sql-min-executions <number>', 'Minimum Historic SQL executions for a template')
|
||||
new Option('--query-history-window-days <number>', 'Query-history lookback window')
|
||||
.argParser(positiveInteger)
|
||||
.hideHelp(),
|
||||
)
|
||||
.addOption(
|
||||
new Option('--historic-sql-service-account-pattern <pattern>', 'Historic SQL service-account regex; repeatable')
|
||||
new Option('--query-history-min-executions <number>', 'Minimum executions for a query-history template')
|
||||
.argParser(positiveInteger)
|
||||
.hideHelp(),
|
||||
)
|
||||
.addOption(
|
||||
new Option('--query-history-service-account-pattern <pattern>', 'Query-history service-account regex; repeatable')
|
||||
.argParser((value, previous: string[]) => [...previous, value])
|
||||
.default([] as string[])
|
||||
.hideHelp(),
|
||||
)
|
||||
.addOption(
|
||||
new Option('--historic-sql-redaction-pattern <pattern>', 'Historic SQL SQL-literal redaction regex; repeatable')
|
||||
new Option('--query-history-redaction-pattern <pattern>', 'Query-history SQL-literal redaction regex; repeatable')
|
||||
.argParser((value, previous: string[]) => [...previous, value])
|
||||
.default([] as string[])
|
||||
.hideHelp(),
|
||||
)
|
||||
.addOption(
|
||||
new Option('--skip-databases', 'Leave database setup incomplete; KTX cannot work until a primary source is added')
|
||||
new Option('--skip-databases', 'Leave database setup incomplete; KTX cannot work until a database is added')
|
||||
.hideHelp()
|
||||
.default(false),
|
||||
)
|
||||
|
|
@ -371,9 +375,9 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
|
|||
context.setExitCode(1);
|
||||
return;
|
||||
}
|
||||
if (options.enableHistoricSql && options.disableHistoricSql) {
|
||||
if (options.enableQueryHistory && options.disableQueryHistory) {
|
||||
context.io.stderr.write(
|
||||
'Choose only one Historic SQL action: --enable-historic-sql or --disable-historic-sql.\n',
|
||||
'Choose only one query-history action: --enable-query-history or --disable-query-history.\n',
|
||||
);
|
||||
context.setExitCode(1);
|
||||
return;
|
||||
|
|
@ -418,17 +422,17 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
|
|||
...(options.newDatabaseConnectionId ? { databaseConnectionId: options.newDatabaseConnectionId } : {}),
|
||||
...(options.databaseUrl ? { databaseUrl: options.databaseUrl } : {}),
|
||||
databaseSchemas: options.databaseSchema,
|
||||
...(options.enableHistoricSql ? { enableHistoricSql: true } : {}),
|
||||
...(options.disableHistoricSql ? { disableHistoricSql: true } : {}),
|
||||
...(options.historicSqlWindowDays !== undefined ? { historicSqlWindowDays: options.historicSqlWindowDays } : {}),
|
||||
...(options.historicSqlMinExecutions !== undefined
|
||||
? { historicSqlMinExecutions: options.historicSqlMinExecutions }
|
||||
...(options.enableQueryHistory ? { enableQueryHistory: true } : {}),
|
||||
...(options.disableQueryHistory ? { disableQueryHistory: true } : {}),
|
||||
...(options.queryHistoryWindowDays !== undefined ? { queryHistoryWindowDays: options.queryHistoryWindowDays } : {}),
|
||||
...(options.queryHistoryMinExecutions !== undefined
|
||||
? { queryHistoryMinExecutions: options.queryHistoryMinExecutions }
|
||||
: {}),
|
||||
...(options.historicSqlServiceAccountPattern.length > 0
|
||||
? { historicSqlServiceAccountPatterns: options.historicSqlServiceAccountPattern }
|
||||
...(options.queryHistoryServiceAccountPattern.length > 0
|
||||
? { queryHistoryServiceAccountPatterns: options.queryHistoryServiceAccountPattern }
|
||||
: {}),
|
||||
...(options.historicSqlRedactionPattern.length > 0
|
||||
? { historicSqlRedactionPatterns: options.historicSqlRedactionPattern }
|
||||
...(options.queryHistoryRedactionPattern.length > 0
|
||||
? { queryHistoryRedactionPatterns: options.queryHistoryRedactionPattern }
|
||||
: {}),
|
||||
skipDatabases: options.skipDatabases === true,
|
||||
...(options.source ? { source: options.source } : {}),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue