mirror of
https://github.com/Kaelio/ktx.git
synced 2026-07-10 11:12:10 +02:00
fix(snowflake): unblock multi-schema ingest and relationship discovery (#204)
* feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure Snowflake setup previously asked for a single schema as free text, then ran a multiselect against the discovered schemas — two schema questions back-to-back, with the first being only a session bootstrap. The SDK's `schema` is optional, so the bootstrap step is unnecessary. - Remove the free-text Snowflake schema prompt; only pass `schema` to snowflake-sdk when one is configured. - When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the user for a comma-separated list, persist it as `schema_names`, and use it as both the table-list filter and the multiselect default. Applies to every driver with a scope-discovery spec, not just Snowflake. - Update docs to lead with `schema_names`; keep `schema_name` as a documented single-schema shorthand. * fix(snowflake): keep introspecting when primary-key discovery is denied The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the connection role may not have. Previously a 'SQL compilation error: Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist or not authorized' aborted the entire introspect — schemas, columns, and row counts were all discarded over a missing nice-to-have. Wrap the constraint query in try/catch, log a one-line warning per schema, and return an empty PK map. Columns end up with primaryKey=false; relationship inference still has FK and profiling to fall back on. * fix(scan): unblock relationship discovery on Snowflake Two adjacent bugs prevented the scan's relationship pipeline from producing any joins on a Snowflake warehouse: - relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table profile query failed with "Unknown function GROUP_CONCAT". Add an explicit Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected). - description-generation.ts destructured `connector.sampleTable` and `connector.sampleColumn` into bare locals, losing the `this` binding when the class-method connectors (Snowflake, Postgres, MySQL) were invoked. Every sample call threw "Cannot read properties of undefined (reading 'assertConnection')" and degraded LLM descriptions to metadata-only prompts. Call the methods through the connector instead. Without these, even after the primary-key probe is allowed to fail softly, the scan ends up with 0 validated relationships and an empty `joins:` block in every shard YAML. * test(scan): cover table-ref helpers * feat(scan): plumb tableScope through live-database introspection port * feat(scan): apply tableScope during metadata fetch * feat(scan): enforce table scope at fetch boundary * feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206) * feat(cli): add RSA key-pair auth option to Snowflake setup wizard Extends the interactive Snowflake setup flow with an authentication-method prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key path (env/file/absolute) and an optional passphrase; the resulting connection config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead of `password`. * feat(scan): pool Snowflake sessions * fix(scan): reuse structural snapshots and cleanup connectors * feat(scan): parallelize relationship profiling * feat(scan): batch table description generation * docs: document Snowflake ingest concurrency knobs * fix(scan): close Snowflake ingest perf verification gaps * fix(scan): keep batched description failure bounded * feat(scan): dispatch query-history probes by connection driver Extract historic-sql dialect resolution into a shared helper so the status-project readiness check and the local ingest factory agree on which connections enable query history and which probe to run. The status command now picks the postgres/snowflake/bigquery probe based on the connection's driver instead of always reporting against postgres, which previously caused snowflake connections with queryHistory.enabled to surface a misleading "driver is snowflake" failure. Also drops a noisy console.warn from Snowflake primary-key discovery — INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only roles and the FK + profiling paths handle the empty PK map already. * fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject The Claude Code agent SDK announces an internal pseudo-tool named StructuredOutput in the system/init message whenever outputFormat is set to { type: 'json_schema' }. The runtime's isolation check built its allowedToolIds set only from MCP tool ids and treated StructuredOutput as an unexpected host-injected tool, so every generateObject call threw "Claude Code runtime isolation failed: tools=StructuredOutput ..." and the table-descriptions and relationship-LLM-proposal enrichment stages recorded null output across the board. Whitelist StructuredOutput specifically in generateObject's allowedToolIds — the check also enforces missing_tools symmetry, so generateText and runAgentLoop, which do not see StructuredOutput, must not require it. generateObject also ran with maxTurns: 1, which the model intermittently breached when it emitted thinking text before the structured response. Raised to 5 to give the schema-bound call enough headroom without allowing unbounded loops. The existing tests now exercise the path with an init message that announces StructuredOutput so the regression cannot slip back in. * chore(scripts): add ktx-reset.sh project-cleanup helper Convenience script for repeatable ingest testing: takes a project directory and prunes everything except ktx.yaml and .ktx/secrets/, so the next ktx setup or ktx ingest run starts from a known-clean state.
This commit is contained in:
Andrey Avtomonov
GitHub
parent
b0dd13ce7c
commit
394a985d2a
72 changed files with 3508 additions and 655 deletions
|
|
@ -601,10 +601,47 @@ function markTargetResult(
|
|||
};
|
||||
}
|
||||
|
||||
function markTargetWithSkippedQueryHistory(
|
||||
target: KtxPublicIngestPlanTarget,
|
||||
args: Extract<KtxPublicIngestArgs, { command: 'run' }>,
|
||||
detail: string,
|
||||
): KtxPublicIngestTargetResult {
|
||||
const baseline = markTargetResult(target, args, 'done');
|
||||
return {
|
||||
...baseline,
|
||||
steps: baseline.steps.map((step) =>
|
||||
step.operation === 'query-history' ? { ...step, status: 'skipped', detail } : step,
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
function queryHistoryFailureDetail(input: {
|
||||
target: KtxPublicIngestPlanTarget;
|
||||
args: Extract<KtxPublicIngestArgs, { command: 'run' }>;
|
||||
capturedOutput?: string;
|
||||
}): string {
|
||||
const captured = capturedFailureMessage(input.capturedOutput ?? '');
|
||||
return failureDetailWithRetry({
|
||||
target: input.target,
|
||||
args: input.args,
|
||||
failedOperation: 'query-history',
|
||||
failureDetail: captured,
|
||||
});
|
||||
}
|
||||
|
||||
function resultFailed(result: KtxPublicIngestTargetResult): boolean {
|
||||
return result.steps.some((step) => step.status === 'failed');
|
||||
}
|
||||
|
||||
function resultSkippedQueryHistory(
|
||||
result: KtxPublicIngestTargetResult,
|
||||
): { connectionId: string; detail: string } | null {
|
||||
const skipped = result.steps.find(
|
||||
(step) => step.operation === 'query-history' && step.status === 'skipped' && step.detail !== undefined,
|
||||
);
|
||||
return skipped?.detail ? { connectionId: result.connectionId, detail: skipped.detail } : null;
|
||||
}
|
||||
|
||||
function rowsBucket(): '<10k' | '<100k' | '<1M' | '<10M' | '>=10M' {
|
||||
return '<10k';
|
||||
}
|
||||
|
|
@ -644,7 +681,17 @@ function stepStatus(result: KtxPublicIngestTargetResult, operation: KtxPublicIng
|
|||
|
||||
function renderPlainResults(results: KtxPublicIngestTargetResult[], io: KtxCliIo): void {
|
||||
const failures = results.filter(resultFailed);
|
||||
io.stdout.write(failures.length > 0 ? 'Ingest finished with partial failures\n' : 'Ingest finished\n');
|
||||
const skippedQueryHistory = results.map(resultSkippedQueryHistory).filter((entry) => entry !== null) as Array<{
|
||||
connectionId: string;
|
||||
detail: string;
|
||||
}>;
|
||||
const headerSuffix =
|
||||
failures.length > 0
|
||||
? ' with partial failures'
|
||||
: skippedQueryHistory.length > 0
|
||||
? ' with skipped query history'
|
||||
: '';
|
||||
io.stdout.write(`Ingest finished${headerSuffix}\n`);
|
||||
io.stdout.write('\n');
|
||||
io.stdout.write('Source Database schema Query history Source ingest Memory update\n');
|
||||
for (const result of results) {
|
||||
|
|
@ -659,17 +706,22 @@ function renderPlainResults(results: KtxPublicIngestTargetResult[], io: KtxCliIo
|
|||
);
|
||||
}
|
||||
|
||||
if (failures.length === 0) {
|
||||
return;
|
||||
if (failures.length > 0) {
|
||||
io.stdout.write('\nFailed sources:\n');
|
||||
for (const result of failures) {
|
||||
const failedStep = result.steps.find((step) => step.status === 'failed');
|
||||
if (!failedStep) {
|
||||
continue;
|
||||
}
|
||||
io.stdout.write(` ${failedStep.detail ?? `${result.connectionId} failed.`}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
io.stdout.write('\nFailed sources:\n');
|
||||
for (const result of failures) {
|
||||
const failedStep = result.steps.find((step) => step.status === 'failed');
|
||||
if (!failedStep) {
|
||||
continue;
|
||||
if (skippedQueryHistory.length > 0) {
|
||||
io.stdout.write('\nSkipped query history:\n');
|
||||
for (const { detail } of skippedQueryHistory) {
|
||||
io.stdout.write(` ${detail}\n`);
|
||||
}
|
||||
io.stdout.write(` ${failedStep.detail ?? `${result.connectionId} failed.`}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -849,14 +901,13 @@ export async function executePublicIngestTarget(
|
|||
? await runIngest(ingestArgs, ingestIo, ingestDeps)
|
||||
: await runIngest(ingestArgs, ingestIo);
|
||||
if (qhExitCode !== 0) {
|
||||
deps.onPhaseEnd?.('query-history', 'failed');
|
||||
return markTargetResult(
|
||||
const detail = queryHistoryFailureDetail({
|
||||
target,
|
||||
args,
|
||||
'failed',
|
||||
'query-history',
|
||||
capturedIngestIo ? capturedFailureMessage(capturedIngestIo.capturedOutput()) : undefined,
|
||||
);
|
||||
capturedOutput: capturedIngestIo ? capturedIngestIo.capturedOutput() : undefined,
|
||||
});
|
||||
deps.onPhaseEnd?.('query-history', 'failed', detail);
|
||||
return markTargetWithSkippedQueryHistory(target, args, detail);
|
||||
}
|
||||
deps.onPhaseEnd?.('query-history', 'done');
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue