mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-19 08:28:06 +02:00
feat: merge ingest and scan
* docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
This commit is contained in:
Andrey Avtomonov
GitHub
parent
1a472cf3ed
commit
b00c1a11a9
118 changed files with 16890 additions and 2992 deletions
|
|
@ -52,14 +52,14 @@ describe('dev Commander tree', () => {
|
|||
expect(testIo.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('keeps dev callable while hiding it from root command rows', async () => {
|
||||
it('lists dev in root command rows', async () => {
|
||||
const testIo = makeIo();
|
||||
|
||||
await expect(runKtxCli(['--help'], testIo.io)).resolves.toBe(0);
|
||||
|
||||
expect(testIo.stdout()).toContain('Advanced:');
|
||||
expect(testIo.stdout()).toContain('ktx dev');
|
||||
expect(testIo.stdout()).not.toContain('dev Low-level diagnostics');
|
||||
expect(testIo.stdout()).not.toContain('Advanced:');
|
||||
expect(testIo.stdout()).toContain('dev');
|
||||
expect(testIo.stdout()).toMatch(/Low-level project initialization and runtime\s+management/);
|
||||
expect(testIo.stderr()).toBe('');
|
||||
});
|
||||
|
||||
|
|
@ -129,21 +129,11 @@ describe('dev Commander tree', () => {
|
|||
argv: ['dev', 'runtime', '--help'],
|
||||
expected: ['Usage: ktx dev runtime', 'install', 'start', 'stop', 'status'],
|
||||
},
|
||||
{
|
||||
argv: ['scan', '--help'],
|
||||
expected: ['Usage: ktx scan [options] <connectionId>', '--mode <mode>', 'structural', 'relationships', '--dry-run'],
|
||||
},
|
||||
{
|
||||
argv: ['ingest', 'run', '--help'],
|
||||
expected: ['Usage: ktx ingest run [options]', '--connection-id <connectionId>', '--adapter <adapter>'],
|
||||
},
|
||||
])('prints generated nested help for $argv', async ({ argv, expected }) => {
|
||||
const io = makeIo();
|
||||
const doctor = vi.fn(async () => 0);
|
||||
const ingest = vi.fn(async () => 0);
|
||||
const scan = vi.fn(async () => 0);
|
||||
|
||||
await expect(runKtxCli(argv, io.io, { doctor, ingest, scan })).resolves.toBe(0);
|
||||
await expect(runKtxCli(argv, io.io, { doctor })).resolves.toBe(0);
|
||||
|
||||
for (const text of expected) {
|
||||
expect(io.stdout()).toContain(text);
|
||||
|
|
@ -154,109 +144,46 @@ describe('dev Commander tree', () => {
|
|||
}
|
||||
expect(io.stderr()).toBe('');
|
||||
expect(doctor).not.toHaveBeenCalled();
|
||||
expect(ingest).not.toHaveBeenCalled();
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('dispatches top-level scan through Commander with injected dependencies', async () => {
|
||||
const scanIo = makeIo();
|
||||
const scan = vi.fn(async () => 0);
|
||||
it('rejects old adapter-backed ingest flags through public option parsing and keeps run out of ingest help', async () => {
|
||||
const helpIo = makeIo();
|
||||
const runIo = makeIo();
|
||||
const publicIngest = vi.fn(async () => 0);
|
||||
|
||||
await expect(runKtxCli(['ingest', '--help'], helpIo.io, { publicIngest })).resolves.toBe(0);
|
||||
await expect(
|
||||
runKtxCli(['scan', 'warehouse', '--project-dir', '/tmp/project', '--dry-run'], scanIo.io, { scan }),
|
||||
).resolves.toBe(0);
|
||||
runKtxCli(
|
||||
['ingest', 'run', '--connection-id', 'warehouse', '--adapter', 'metabase', '--project-dir', '/tmp/project'],
|
||||
runIo.io,
|
||||
{ publicIngest },
|
||||
),
|
||||
).resolves.toBe(1);
|
||||
|
||||
expect(scan).toHaveBeenCalledWith(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir: '/tmp/project',
|
||||
connectionId: 'warehouse',
|
||||
mode: 'structural',
|
||||
detectRelationships: false,
|
||||
dryRun: true,
|
||||
databaseIntrospectionUrl: undefined,
|
||||
cliVersion: '0.0.0-private',
|
||||
runtimeInstallPolicy: 'prompt',
|
||||
},
|
||||
scanIo.io,
|
||||
);
|
||||
expect(scanIo.stderr()).toBe('Project: /tmp/project\n');
|
||||
});
|
||||
|
||||
it('dispatches top-level scan --mode relationships through Commander', async () => {
|
||||
const io = makeIo();
|
||||
const scan = vi.fn(async () => 0);
|
||||
|
||||
await expect(
|
||||
runKtxCli(['scan', 'warehouse', '--project-dir', '/tmp/project', '--mode', 'relationships'], io.io, {
|
||||
scan,
|
||||
}),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(scan).toHaveBeenCalledWith(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir: '/tmp/project',
|
||||
connectionId: 'warehouse',
|
||||
mode: 'relationships',
|
||||
detectRelationships: true,
|
||||
dryRun: false,
|
||||
databaseIntrospectionUrl: undefined,
|
||||
cliVersion: '0.0.0-private',
|
||||
runtimeInstallPolicy: 'prompt',
|
||||
},
|
||||
io.io,
|
||||
);
|
||||
expect(io.stderr()).toBe('Project: /tmp/project\n');
|
||||
});
|
||||
|
||||
it.each(['--enrich', '--detect-relationships'])('rejects removed scan shorthand option %s', async (option) => {
|
||||
const io = makeIo();
|
||||
const scan = vi.fn(async () => 0);
|
||||
|
||||
await expect(runKtxCli(['scan', 'warehouse', option], io.io, { scan })).resolves.toBe(1);
|
||||
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
expect(io.stderr()).toContain(`unknown option '${option}'`);
|
||||
});
|
||||
|
||||
it('rejects scan without a connection id', async () => {
|
||||
const io = makeIo();
|
||||
const scan = vi.fn(async () => 0);
|
||||
|
||||
await expect(runKtxCli(['scan', '--dry-run'], io.io, { scan })).resolves.toBe(1);
|
||||
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
expect(io.stderr()).toMatch(/missing required argument/i);
|
||||
});
|
||||
|
||||
it('rejects invalid scan modes before dispatch', async () => {
|
||||
const io = makeIo();
|
||||
const scan = vi.fn(async () => 0);
|
||||
|
||||
await expect(runKtxCli(['scan', 'warehouse', '--mode', 'deep'], io.io, { scan })).resolves.toBe(1);
|
||||
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
expect(io.stderr()).toContain("argument 'deep' is invalid");
|
||||
expect(io.stderr()).toContain('Allowed choices are structural, enriched, relationships');
|
||||
expect(helpIo.stdout()).not.toMatch(/^ run\s/m);
|
||||
expect(runIo.stderr()).toMatch(/unknown option '--connection-id'|error:/);
|
||||
expect(publicIngest).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
['scan', 'report', 'scan-run-1'],
|
||||
['scan', 'relationships', 'scan-run-1'],
|
||||
])('rejects removed scan subcommand %s %s', async (command, subcommand, runId) => {
|
||||
{ argv: ['scan'] },
|
||||
{ argv: ['scan', '--help'] },
|
||||
{ argv: ['scan', 'warehouse'] },
|
||||
{ argv: ['scan', 'warehouse', '--project-dir', '/tmp/project', '--dry-run'] },
|
||||
{ argv: ['scan', 'warehouse', '--project-dir', '/tmp/project', '--mode', 'relationships'] },
|
||||
])('rejects removed top-level scan command $argv', async ({ argv }) => {
|
||||
const io = makeIo();
|
||||
const scan = vi.fn(async () => 0);
|
||||
const publicIngest = vi.fn(async () => 0);
|
||||
|
||||
await expect(runKtxCli([command, subcommand, runId], io.io, { scan })).resolves.toBe(1);
|
||||
await expect(runKtxCli(argv, io.io, { publicIngest })).resolves.toBe(1);
|
||||
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
expect(io.stderr()).toMatch(/too many arguments|unknown command|error:/);
|
||||
expect(publicIngest).not.toHaveBeenCalled();
|
||||
expect(io.stderr()).toMatch(/unknown command|error:/);
|
||||
});
|
||||
|
||||
it('dispatches top-level ingest run through the low-level ingest Commander registration', async () => {
|
||||
it('rejects old adapter-backed top-level ingest flags without low-level ingest registration', async () => {
|
||||
const io = makeIo();
|
||||
const ingest = vi.fn(async () => 0);
|
||||
const publicIngest = vi.fn(async () => 0);
|
||||
|
||||
await expect(
|
||||
runKtxCli(
|
||||
|
|
@ -272,24 +199,11 @@ describe('dev Commander tree', () => {
|
|||
'--json',
|
||||
],
|
||||
io.io,
|
||||
{ ingest },
|
||||
{ publicIngest },
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
).resolves.toBe(1);
|
||||
|
||||
expect(ingest).toHaveBeenCalledWith(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir: '/tmp/project',
|
||||
connectionId: 'warehouse',
|
||||
adapter: 'metabase',
|
||||
sourceDir: undefined,
|
||||
databaseIntrospectionUrl: undefined,
|
||||
cliVersion: '0.0.0-private',
|
||||
runtimeInstallPolicy: 'prompt',
|
||||
outputMode: 'json',
|
||||
},
|
||||
io.io,
|
||||
);
|
||||
expect(io.stderr()).toBe('');
|
||||
expect(publicIngest).not.toHaveBeenCalled();
|
||||
expect(io.stderr()).toMatch(/unknown option '--connection-id'|error:/);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue