mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-07 07:55:13 +02:00
test: split cli tests from source tree (#216)
* feat(cli): define full warehouse dialect contract
* test(cli): keep dialect edge tests focused
* fix(cli): stabilize dialect contract foundation
* refactor(connectors): own read-only query preparation
* refactor(connectors): resolve dialects through registry
* refactor(connectors): keep concrete dialect classes internal
* chore(workspace): enforce dialect import boundary
* refactor(cli): resolve relationship dialect at scan boundary
* refactor(cli): use dialect display parsing for entity details
* refactor(cli): use dialect display parsing for warehouse catalog
* refactor(cli): use dialect SQL in relationship workflows
* test(cli): verify solid dialect scan workflow closure
* test: split cli tests from source tree
* refactor(cli): standardize BigQuery scope listing
* feat(sqlite): implement connector scope listing
* test(connectors): cover required table listing
* feat(cli): add warehouse driver registry
* refactor(setup): route scope discovery through driver registry
* refactor(cli): route local query execution through driver registry
* refactor(historic-sql): route dialect support through driver registry
* refactor(cli): test warehouse connections through driver registry
* fix(cli): close driver registry type export gaps
* Improve setup daemon diagnostics
* refactor(setup): centralize rail-prefixed diagnostics + query-history fallback
Extract errorMessage, writePrefixedLines, and flushPrefixedBufferedCommandOutput
into clack.ts so the setup wizard, managed daemons, and embedding/agent steps
share one rail-formatted writer. setup-databases.ts also adds a
"disable query history and retry" option when the schema-context build fails
and query history is the likely culprit, surfaced via a new
failed-query-history-unavailable status.
* fix(cli): carry catalog through the picker so BigQuery/Snowflake/SQL Server scope filters match
The setup picker's KtxTableListEntry was a 2-level { schema, name }, so
qualifiedTableId always wrote db.name into enabled_tables. When BigQuery,
Snowflake, or SQL Server later ran fast ingest, their introspect step filtered
the scope set with scopedTableNames(scope, { catalog: projectId|database, db })
— catalog was non-null on the introspect side but null in the scope refs, so
every entry was rejected, the live-database adapter staged zero table files,
and detect() failed with 'Adapter "live-database" did not recognize fetched
source output'.
Align the picker boundary with the canonical 3-level KtxTableRef:
- Add catalog: string | null to KtxTableListEntry.
- BigQuery/Snowflake/SQL Server listTables populate catalog from the
resolved projectId / database; Postgres/MySQL/ClickHouse/SQLite set null.
- qualifiedTableId emits catalog.schema.name when catalog is non-null
(resolveEnabledTables already accepts the 3-part shape) and
schemasFromEnabledTables now goes through parseDottedTableEntry so it
recovers the schema correctly from both 2-part and 3-part entries.
- Export parseDottedTableEntry from enabled-tables.ts (@internal) for picker
reuse.
Update listTables expectations in all seven connector tests and the setup /
picker test fixtures. Add a picker regression test that covers the
catalog-bearing round-trip (save + refine).
* fix(cli): allow debug telemetry under opt-out env
This commit is contained in:
parent
924868841d
commit
56985b7e09
548 changed files with 5048 additions and 2228 deletions
|
|
@ -137,8 +137,10 @@ Enabling query history makes deep ingest readiness matter for later
|
|||
When query history is enabled for PostgreSQL, Snowflake, or BigQuery,
|
||||
`ktx setup` runs a non-blocking readiness probe after the connection test
|
||||
passes. A failed probe still writes setup changes, prints the warehouse-specific
|
||||
grant or extension remediation, and leaves query-history ingest disabled until
|
||||
you fix the prerequisite.
|
||||
grant or extension remediation, and skips query-history processing until you
|
||||
fix the prerequisite. If the later schema-context build also fails, interactive
|
||||
setup offers **Disable query history and retry** so you can finish database
|
||||
setup with `connections.<id>.context.queryHistory.enabled: false`.
|
||||
|
||||
For BigQuery, the remediation tells you to grant `roles/bigquery.resourceViewer`
|
||||
on the BigQuery project, or grant a custom role that contains
|
||||
|
|
|
|||
|
|
@ -89,3 +89,41 @@ enough reason to fix it even when the local code "works."
|
|||
(`loadX` vs `loadHigherX`, `createY` vs `createDefaultY`, `xClient`
|
||||
vs `xService`), assume callers will pick the wrong one. Unify, or
|
||||
document inline why both must exist.
|
||||
|
||||
## Dispatch and contract leaks across per-variant layers
|
||||
|
||||
Layers with multiple per-variant implementations (warehouse drivers,
|
||||
dialects, LLM providers, ingest adapters, historic-SQL probes) drift
|
||||
toward parallel switches and informal contracts. The patterns below
|
||||
look locally reasonable per file but multiply with the number of
|
||||
variants times the number of consumers — every fix has to be applied
|
||||
N times, and silent drift between variants is invisible until a user
|
||||
hits it.
|
||||
|
||||
- **MUST NOT**: Maintain two or more files that switch on the same
|
||||
enum or string union to dispatch to per-variant behavior. Promote
|
||||
the dispatch to a single registry table keyed by the union, exposed
|
||||
through one resolution function. If you find yourself writing the
|
||||
third such switch, the second one was already a bug.
|
||||
- **MUST**: When every variant of an abstraction implements the same
|
||||
method, the method belongs on the shared interface. An informal
|
||||
contract that every implementation happens to satisfy is a leak
|
||||
waiting to happen — callers will reach for the concrete class
|
||||
instead of the contract, and the next variant added will silently
|
||||
forget to implement it.
|
||||
- **MUST**: When a layer has both a thin shared interface and rich
|
||||
per-variant concrete classes, they must agree. Either widen the
|
||||
interface so callers never need the concrete class, or make the
|
||||
concrete class private (test-only `/** @internal */` JSDoc plus a
|
||||
boundary check in `scripts/check-boundaries.mjs`). A class that is
|
||||
public AND has methods the interface does not expose is the exact
|
||||
configuration that produces leaks.
|
||||
|
||||
The warehouse driver / dialect layer in
|
||||
`packages/cli/src/connectors/<driver>/` plus
|
||||
`packages/cli/src/context/connections/{dialects,drivers}.ts` is the
|
||||
canonical worked example: per-driver dialect classes carry
|
||||
`/** @internal */`, `scripts/check-boundaries.mjs` enforces the import
|
||||
boundary, and dispatch lives in the two registry files. Apply the
|
||||
same shape to any other per-variant layer that grows beyond two
|
||||
implementations.
|
||||
|
|
|
|||
|
|
@ -14,8 +14,8 @@
|
|||
"src/telemetry/schema-writer.ts!",
|
||||
"src/telemetry/index.ts!",
|
||||
"scripts/**/*.mjs",
|
||||
"src/**/*.test-utils.ts",
|
||||
"src/**/acceptance-fixtures.ts",
|
||||
"test/**/*.test-utils.ts",
|
||||
"test/**/acceptance-fixtures.ts",
|
||||
"src/context/scan/relationship-benchmarks.ts!",
|
||||
"src/context/scan/relationship-benchmark-report.ts!"
|
||||
]
|
||||
|
|
|
|||
|
|
@ -32,12 +32,12 @@
|
|||
"build": "tsc -p tsconfig.json && node dist/telemetry/schema-writer.js src/telemetry/events.schema.json ../../python/ktx-daemon/src/ktx_daemon/telemetry/events.schema.json && node scripts/copy-runtime-assets.mjs && node ../../scripts/prepare-cli-bin.mjs",
|
||||
"clean": "node -e \"fs.rmSync('dist', { recursive: true, force: true })\"",
|
||||
"docs:commands": "pnpm run build && node dist/print-command-tree.js",
|
||||
"smoke": "vitest run src/standalone-smoke.test.ts src/example-smoke.test.ts --testTimeout 30000",
|
||||
"test": "vitest run --exclude src/standalone-smoke.test.ts --exclude src/example-smoke.test.ts --exclude src/setup-databases.test.ts --exclude src/scan.test.ts --exclude src/commands/connection-metabase-setup.test.ts --exclude src/setup-models.test.ts --exclude src/setup-sources.test.ts --exclude src/setup.test.ts --exclude src/connection.test.ts --exclude src/setup-embeddings.test.ts --exclude src/ingest.test.ts --exclude src/commands/connection-mapping.test.ts --exclude src/ingest-viz.test.ts --exclude src/demo.test.ts --exclude src/setup-project.test.ts --exclude src/sl.test.ts --exclude src/local-scan-connectors.test.ts --exclude src/commands/connection-notion.test.ts --exclude src/context/scan/local-scan.test.ts --exclude src/context/mcp/local-project-ports.test.ts --exclude src/context/ingest/local-stage-ingest.test.ts --exclude src/context/sl/pglite-sl-search-prototype.test.ts --exclude src/context/core/git.service.test.ts --exclude src/context/ingest/local-adapters.test.ts --exclude src/context/ingest/local-bundle-ingest.test.ts --exclude src/context/ingest/local-metabase-ingest.test.ts --exclude src/context/sl/local-sl.test.ts --exclude src/context/search/pglite-owner-process.test.ts --exclude src/context/scan/local-enrichment-artifacts.test.ts --exclude src/context/search/pglite-spike.test.ts --exclude src/context/wiki/local-knowledge.test.ts --exclude src/context/sl/local-query.test.ts --exclude src/context/scan/relationship-review-decisions.test.ts --exclude src/context/scan/relationship-profiling.test.ts",
|
||||
"test:slow": "vitest run src/setup-databases.test.ts src/scan.test.ts src/commands/connection-metabase-setup.test.ts src/setup-models.test.ts src/setup-sources.test.ts src/setup.test.ts src/connection.test.ts src/setup-embeddings.test.ts src/ingest.test.ts src/commands/connection-mapping.test.ts src/ingest-viz.test.ts src/demo.test.ts src/setup-project.test.ts src/sl.test.ts src/local-scan-connectors.test.ts src/commands/connection-notion.test.ts src/context/scan/local-scan.test.ts src/context/mcp/local-project-ports.test.ts src/context/ingest/local-stage-ingest.test.ts src/context/sl/pglite-sl-search-prototype.test.ts src/context/core/git.service.test.ts src/context/ingest/local-adapters.test.ts src/context/ingest/local-bundle-ingest.test.ts src/context/ingest/local-metabase-ingest.test.ts src/context/sl/local-sl.test.ts src/context/search/pglite-owner-process.test.ts src/context/scan/local-enrichment-artifacts.test.ts src/context/search/pglite-spike.test.ts src/context/wiki/local-knowledge.test.ts src/context/sl/local-query.test.ts src/context/scan/relationship-review-decisions.test.ts src/context/scan/relationship-profiling.test.ts --testTimeout 30000",
|
||||
"type-check": "tsc -p tsconfig.json --noEmit",
|
||||
"smoke": "vitest run test/standalone-smoke.test.ts test/example-smoke.test.ts --testTimeout 30000",
|
||||
"test": "vitest run --exclude test/standalone-smoke.test.ts --exclude test/example-smoke.test.ts --exclude test/setup-databases.test.ts --exclude test/scan.test.ts --exclude test/commands/connection-metabase-setup.test.ts --exclude test/setup-models.test.ts --exclude test/setup-sources.test.ts --exclude test/setup.test.ts --exclude test/connection.test.ts --exclude test/setup-embeddings.test.ts --exclude test/ingest.test.ts --exclude test/commands/connection-mapping.test.ts --exclude test/ingest-viz.test.ts --exclude test/demo.test.ts --exclude test/setup-project.test.ts --exclude test/sl.test.ts --exclude test/local-scan-connectors.test.ts --exclude test/commands/connection-notion.test.ts --exclude test/context/scan/local-scan.test.ts --exclude test/context/mcp/local-project-ports.test.ts --exclude test/context/ingest/local-stage-ingest.test.ts --exclude test/context/sl/pglite-sl-search-prototype.test.ts --exclude test/context/core/git.service.test.ts --exclude test/context/ingest/local-adapters.test.ts --exclude test/context/ingest/local-bundle-ingest.test.ts --exclude test/context/ingest/local-metabase-ingest.test.ts --exclude test/context/sl/local-sl.test.ts --exclude test/context/search/pglite-owner-process.test.ts --exclude test/context/scan/local-enrichment-artifacts.test.ts --exclude test/context/search/pglite-spike.test.ts --exclude test/context/wiki/local-knowledge.test.ts --exclude test/context/sl/local-query.test.ts --exclude test/context/scan/relationship-review-decisions.test.ts --exclude test/context/scan/relationship-profiling.test.ts",
|
||||
"test:slow": "vitest run test/setup-databases.test.ts test/scan.test.ts test/commands/connection-metabase-setup.test.ts test/setup-models.test.ts test/setup-sources.test.ts test/setup.test.ts test/connection.test.ts test/setup-embeddings.test.ts test/ingest.test.ts test/commands/connection-mapping.test.ts test/ingest-viz.test.ts test/demo.test.ts test/setup-project.test.ts test/sl.test.ts test/local-scan-connectors.test.ts test/commands/connection-notion.test.ts test/context/scan/local-scan.test.ts test/context/mcp/local-project-ports.test.ts test/context/ingest/local-stage-ingest.test.ts test/context/sl/pglite-sl-search-prototype.test.ts test/context/core/git.service.test.ts test/context/ingest/local-adapters.test.ts test/context/ingest/local-bundle-ingest.test.ts test/context/ingest/local-metabase-ingest.test.ts test/context/sl/local-sl.test.ts test/context/search/pglite-owner-process.test.ts test/context/scan/local-enrichment-artifacts.test.ts test/context/search/pglite-spike.test.ts test/context/wiki/local-knowledge.test.ts test/context/sl/local-query.test.ts test/context/scan/relationship-review-decisions.test.ts test/context/scan/relationship-profiling.test.ts --testTimeout 30000",
|
||||
"type-check": "tsc -p tsconfig.json --noEmit && tsc -p tsconfig.test.json --noEmit",
|
||||
"relationships:benchmarks": "pnpm --silent run build && node ../../scripts/relationship-benchmark-report.mjs",
|
||||
"relationships:benchmarks:test": "KTX_RUN_RELATIONSHIP_BENCHMARKS=1 vitest run src/context/scan/relationship-benchmarks.test.ts",
|
||||
"relationships:benchmarks:test": "KTX_RUN_RELATIONSHIP_BENCHMARKS=1 vitest run test/context/scan/relationship-benchmarks.test.ts",
|
||||
"search:pglite-spike": "node ../../scripts/pglite-hybrid-search-spike.mjs",
|
||||
"search:pglite-owner-prototype": "node ../../scripts/pglite-owner-process-prototype.mjs",
|
||||
"search:pglite-sl-prototype": "node ../../scripts/pglite-sl-search-prototype.mjs"
|
||||
|
|
|
|||
|
|
@ -1,7 +1,30 @@
|
|||
import { cancel, confirm, isCancel, log, spinner } from '@clack/prompts';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
|
||||
const ESC = String.fromCharCode(0x1b);
|
||||
|
||||
export interface RailBufferedSource {
|
||||
stdoutText(): string;
|
||||
stderrText(): string;
|
||||
}
|
||||
|
||||
export function errorMessage(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
export function writePrefixedLines(write: (chunk: string) => void, output: string): void {
|
||||
for (const line of output.split(/\r?\n/)) {
|
||||
if (line.length > 0) {
|
||||
write(`│ ${line}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function flushPrefixedBufferedCommandOutput(io: KtxCliIo, buffered: RailBufferedSource): void {
|
||||
writePrefixedLines((chunk) => io.stdout.write(chunk), buffered.stdoutText());
|
||||
writePrefixedLines((chunk) => io.stderr.write(chunk), buffered.stderrText());
|
||||
}
|
||||
|
||||
export interface KtxCliSpinner {
|
||||
start(message: string): void;
|
||||
message(message: string): void;
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import { type NotionBotInfo, NotionClient } from './context/ingest/adapters/noti
|
|||
import { createLocalLookerCredentialResolver } from './context/ingest/adapters/looker/local-looker.adapter.js';
|
||||
import { metabaseRuntimeConfigFromLocalConnection } from './context/ingest/adapters/metabase/local-metabase.adapter.js';
|
||||
import { testRepoConnection } from './context/ingest/repo-fetch.js';
|
||||
import { getDriverRegistration } from './context/connections/drivers.js';
|
||||
import { parseNotionConnectionConfig, resolveNotionConnectionAuthToken } from './context/connections/notion-config.js';
|
||||
import { resolveKtxConfigReference } from './context/core/config-reference.js';
|
||||
import { type KtxLocalProject, loadKtxProject } from './context/project/project.js';
|
||||
|
|
@ -272,15 +273,7 @@ async function testConnectionByDriver(
|
|||
return { driver, detailKey: 'Repo', detailValue: result.repoUrl };
|
||||
}
|
||||
|
||||
if (
|
||||
driver === 'sqlite' ||
|
||||
driver === 'postgres' ||
|
||||
driver === 'mysql' ||
|
||||
driver === 'clickhouse' ||
|
||||
driver === 'sqlserver' ||
|
||||
driver === 'bigquery' ||
|
||||
driver === 'snowflake'
|
||||
) {
|
||||
if (getDriverRegistration(driver)) {
|
||||
const result = await testNativeConnection(
|
||||
project,
|
||||
connectionId,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import { BigQuery, type TableField } from '@google-cloud/bigquery';
|
||||
import { normalizeBigQueryProjectId, normalizeBigQueryRegion } from '../../context/connections/bigquery-identifiers.js';
|
||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||
|
|
@ -26,7 +27,6 @@ import {
|
|||
import { readFileSync } from 'node:fs';
|
||||
import { homedir } from 'node:os';
|
||||
import { resolve } from 'node:path';
|
||||
import { KtxBigQueryDialect } from './dialect.js';
|
||||
|
||||
export interface KtxBigQueryConnectionConfig {
|
||||
driver?: string;
|
||||
|
|
@ -235,6 +235,23 @@ function normalizeValue(value: unknown): unknown {
|
|||
return value;
|
||||
}
|
||||
|
||||
/** @internal */
|
||||
export function prepareBigQueryReadOnlyQuery(
|
||||
sql: string,
|
||||
params?: Record<string, unknown>,
|
||||
): { sql: string; params?: Record<string, unknown> } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
let processedSql = sql;
|
||||
const processedParams: Record<string, unknown> = {};
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
processedSql = processedSql.replace(new RegExp(`:${key}\\b`, 'g'), `@${key}`);
|
||||
processedParams[key] = value;
|
||||
}
|
||||
return { sql: processedSql, params: Object.keys(processedParams).length > 0 ? processedParams : undefined };
|
||||
}
|
||||
|
||||
export function isKtxBigQueryConnectionConfig(
|
||||
connection: KtxBigQueryConnectionConfig | undefined,
|
||||
): connection is KtxBigQueryConnectionConfig {
|
||||
|
|
@ -286,7 +303,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
|
|||
private readonly now: () => Date;
|
||||
private readonly maxBytesBilled?: number | string;
|
||||
private readonly queryTimeoutMs?: number;
|
||||
private readonly dialect = new KtxBigQueryDialect();
|
||||
private readonly dialect = getDialectForDriver('bigquery');
|
||||
private client: KtxBigQueryClient | null = null;
|
||||
|
||||
constructor(options: KtxBigQueryScanConnectorOptions) {
|
||||
|
|
@ -364,7 +381,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
|
|||
async executeReadOnly(input: KtxBigQueryReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {
|
||||
this.assertConnection(input.connectionId);
|
||||
const limitedSql = limitSqlForExecution(assertReadOnlySql(input.sql), input.maxRows);
|
||||
const prepared = this.dialect.prepareQuery(limitedSql, input.params);
|
||||
const prepared = prepareBigQueryReadOnlyQuery(limitedSql, input.params);
|
||||
const result = await this.query(prepared.sql, prepared.params);
|
||||
return { ...result, rowCount: result.rows.length };
|
||||
}
|
||||
|
|
@ -411,7 +428,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
|
|||
return this.dialect.quoteIdentifier(identifier);
|
||||
}
|
||||
|
||||
async listDatasets(): Promise<string[]> {
|
||||
async listSchemas(): Promise<string[]> {
|
||||
const [datasets] = await this.getClient().getDatasets();
|
||||
return datasets.map((dataset) => dataset.id).filter((id): id is string => Boolean(id));
|
||||
}
|
||||
|
|
@ -437,6 +454,7 @@ export class KtxBigQueryScanConnector implements KtxScanConnector {
|
|||
params,
|
||||
);
|
||||
return rows.map((row) => ({
|
||||
catalog: this.resolved.projectId,
|
||||
schema: row.table_schema,
|
||||
name: row.table_name,
|
||||
kind:
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
import type { KtxDialect } from '../../context/connections/dialects.js';
|
||||
import {
|
||||
columnDisplayPartCount,
|
||||
formatDialectDisplayRef,
|
||||
formatDialectTableName,
|
||||
limitOffsetClause,
|
||||
parseDialectDisplayRef,
|
||||
} from '../../context/connections/dialect-helpers.js';
|
||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||
|
||||
type BigQueryTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export class KtxBigQueryDialect {
|
||||
readonly type = 'bigquery';
|
||||
/** @internal */
|
||||
export class KtxBigQueryDialect implements KtxDialect {
|
||||
readonly type = 'bigquery' as const;
|
||||
|
||||
private readonly typeMappings: Record<string, KtxSchemaDimensionType> = {
|
||||
TIMESTAMP: 'time',
|
||||
|
|
@ -27,13 +36,19 @@ export class KtxBigQueryDialect {
|
|||
}
|
||||
|
||||
formatTableName(table: BigQueryTableNameRef): string {
|
||||
if (table.catalog && table.db) {
|
||||
return `${this.quoteIdentifier(table.catalog)}.${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`;
|
||||
}
|
||||
if (table.db) {
|
||||
return `${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`;
|
||||
}
|
||||
return this.quoteIdentifier(table.name);
|
||||
return formatDialectTableName(table, this.quoteIdentifier.bind(this), 'three-part');
|
||||
}
|
||||
|
||||
formatDisplayRef(table: BigQueryTableNameRef): string {
|
||||
return formatDialectDisplayRef(table, 'three-part');
|
||||
}
|
||||
|
||||
parseDisplayRef(display: string): KtxTableRef | null {
|
||||
return parseDialectDisplayRef(display, 'three-part');
|
||||
}
|
||||
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3 {
|
||||
return columnDisplayPartCount('three-part');
|
||||
}
|
||||
|
||||
mapDataType(nativeType: string): string {
|
||||
|
|
@ -93,19 +108,6 @@ export class KtxBigQueryDialect {
|
|||
return `SELECT ${quotedColumn} FROM ${tableName} WHERE ${quotedColumn} IS NOT NULL AND TRIM(CAST(${quotedColumn} AS STRING)) != '' ORDER BY RAND() LIMIT ${limit}`;
|
||||
}
|
||||
|
||||
prepareQuery(sql: string, params?: Record<string, unknown>): { sql: string; params?: Record<string, unknown> } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
let processedSql = sql;
|
||||
const processedParams: Record<string, unknown> = {};
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
processedSql = processedSql.replace(new RegExp(`:${key}\\b`, 'g'), `@${key}`);
|
||||
processedParams[key] = value;
|
||||
}
|
||||
return { sql: processedSql, params: Object.keys(processedParams).length > 0 ? processedParams : undefined };
|
||||
}
|
||||
|
||||
getRandomSampleFilter(samplePct: number): string {
|
||||
if (samplePct <= 0 || samplePct >= 1) {
|
||||
return '';
|
||||
|
|
@ -121,7 +123,11 @@ export class KtxBigQueryDialect {
|
|||
}
|
||||
|
||||
getLimitOffsetClause(limit: number, offset?: number): string {
|
||||
return offset !== undefined && offset > 0 ? `LIMIT ${limit} OFFSET ${offset}` : `LIMIT ${limit}`;
|
||||
return limitOffsetClause(limit, offset);
|
||||
}
|
||||
|
||||
getTopClause(_limit: number): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
getNullCountExpression(column: string): string {
|
||||
|
|
@ -132,6 +138,18 @@ export class KtxBigQueryDialect {
|
|||
return `APPROX_COUNT_DISTINCT(${column})`;
|
||||
}
|
||||
|
||||
textLengthExpression(columnSql: string): string {
|
||||
return `LENGTH(CAST(${columnSql} AS STRING))`;
|
||||
}
|
||||
|
||||
castToText(columnSql: string): string {
|
||||
return `CAST(${columnSql} AS STRING)`;
|
||||
}
|
||||
|
||||
getSampleValueAggregation(innerSql: string): string {
|
||||
return `(SELECT STRING_AGG(CAST(value AS STRING), '\\u001F') FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||
return `
|
||||
WITH sampled AS (
|
||||
|
|
@ -172,36 +190,4 @@ export class KtxBigQueryDialect {
|
|||
FROM sampled
|
||||
`;
|
||||
}
|
||||
|
||||
getTimeTruncExpression(
|
||||
column: string,
|
||||
granularity: 'day' | 'week' | 'month' | 'quarter' | 'year',
|
||||
timezone?: string,
|
||||
): string {
|
||||
const bigQueryGranularity = granularity.toUpperCase();
|
||||
if (timezone) {
|
||||
return `DATE_TRUNC(DATETIME(${column}, '${timezone}'), ${bigQueryGranularity})`;
|
||||
}
|
||||
return `DATE_TRUNC(${column}, ${bigQueryGranularity})`;
|
||||
}
|
||||
|
||||
getCustomTimeTruncExpression(column: string, interval: string, origin?: string, timezone?: string): string {
|
||||
const col = timezone ? `DATETIME(${column}, '${timezone}')` : column;
|
||||
const [rawAmount, rawUnit] = interval.split(' ');
|
||||
let diffUnit = rawUnit!.toUpperCase();
|
||||
let amount = Number(rawAmount);
|
||||
let addUnit = diffUnit;
|
||||
if (diffUnit === 'WEEK') {
|
||||
diffUnit = 'DAY';
|
||||
amount = amount * 7;
|
||||
addUnit = 'DAY';
|
||||
}
|
||||
const originExpr = origin ? `TIMESTAMP '${origin}'` : `TIMESTAMP '1970-01-01'`;
|
||||
return `TIMESTAMP_ADD(${originExpr}, INTERVAL CAST(FLOOR(TIMESTAMP_DIFF(${col}, ${originExpr}, ${diffUnit}) / ${amount}) * ${amount} AS INT64) ${addUnit})`;
|
||||
}
|
||||
|
||||
parseIntervalToSql(interval: string): string {
|
||||
const [amount, unit] = interval.split(' ');
|
||||
return `INTERVAL ${amount} ${unit!.toUpperCase()}`;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { createClient } from '@clickhouse/client';
|
||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||
|
|
@ -6,7 +7,6 @@ import { readFileSync } from 'node:fs';
|
|||
import { Agent as HttpsAgent } from 'node:https';
|
||||
import { homedir } from 'node:os';
|
||||
import { resolve } from 'node:path';
|
||||
import { KtxClickHouseDialect } from './dialect.js';
|
||||
|
||||
export interface KtxClickHouseConnectionConfig {
|
||||
driver?: string;
|
||||
|
|
@ -198,6 +198,49 @@ function clickHouseTableKey(database: string, table: string): string {
|
|||
return `${database}.${table}`;
|
||||
}
|
||||
|
||||
function inferClickHouseQueryParamType(value: unknown): string {
|
||||
if (value === null || value === undefined) {
|
||||
return 'String';
|
||||
}
|
||||
if (typeof value === 'boolean') {
|
||||
return 'Bool';
|
||||
}
|
||||
if (typeof value === 'number') {
|
||||
return Number.isInteger(value) ? 'Int64' : 'Float64';
|
||||
}
|
||||
if (value instanceof Date) {
|
||||
return 'DateTime';
|
||||
}
|
||||
return 'String';
|
||||
}
|
||||
|
||||
/** @internal */
|
||||
export function prepareClickHouseReadOnlyQuery(
|
||||
sql: string,
|
||||
params?: Record<string, unknown>,
|
||||
): { sql: string; params?: Record<string, unknown> } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
|
||||
let parameterizedQuery = sql;
|
||||
const queryParams: Record<string, unknown> = {};
|
||||
const sortedKeys = Object.keys(params).sort((a, b) => b.length - a.length);
|
||||
|
||||
for (const key of sortedKeys) {
|
||||
const placeholder = `:${key}`;
|
||||
if (parameterizedQuery.includes(placeholder)) {
|
||||
parameterizedQuery = parameterizedQuery.replace(
|
||||
new RegExp(`:${key}\\b`, 'g'),
|
||||
`{${key}:${inferClickHouseQueryParamType(params[key])}}`,
|
||||
);
|
||||
queryParams[key] = params[key];
|
||||
}
|
||||
}
|
||||
|
||||
return { sql: parameterizedQuery, params: Object.keys(queryParams).length > 0 ? queryParams : undefined };
|
||||
}
|
||||
|
||||
export function isKtxClickHouseConnectionConfig(
|
||||
connection: KtxClickHouseConnectionConfig | undefined,
|
||||
): connection is KtxClickHouseConnectionConfig {
|
||||
|
|
@ -256,7 +299,7 @@ export class KtxClickHouseScanConnector implements KtxScanConnector {
|
|||
private readonly clientFactory: KtxClickHouseClientFactory;
|
||||
private readonly endpointResolver?: KtxClickHouseEndpointResolver;
|
||||
private readonly now: () => Date;
|
||||
private readonly dialect = new KtxClickHouseDialect();
|
||||
private readonly dialect = getDialectForDriver('clickhouse');
|
||||
private client: KtxClickHouseClient | null = null;
|
||||
private resolvedEndpoint: KtxClickHouseResolvedEndpoint | null = null;
|
||||
|
||||
|
|
@ -408,7 +451,7 @@ export class KtxClickHouseScanConnector implements KtxScanConnector {
|
|||
async executeReadOnly(input: KtxClickHouseReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {
|
||||
this.assertConnection(input.connectionId);
|
||||
const limitedSql = limitSqlForExecution(assertReadOnlySql(input.sql), input.maxRows);
|
||||
const prepared = this.dialect.prepareQuery(limitedSql, input.params);
|
||||
const prepared = prepareClickHouseReadOnlyQuery(limitedSql, input.params);
|
||||
const result = await this.query(prepared.sql, prepared.params);
|
||||
return { ...result, rowCount: result.rows.length };
|
||||
}
|
||||
|
|
@ -488,6 +531,7 @@ export class KtxClickHouseScanConnector implements KtxScanConnector {
|
|||
{ schemas: filterSchemas },
|
||||
);
|
||||
return rows.map((row) => ({
|
||||
catalog: null,
|
||||
schema: row.database,
|
||||
name: row.name,
|
||||
kind: row.engine === 'View' || row.engine === 'MaterializedView' ? ('view' as const) : ('table' as const),
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
import type { KtxDialect } from '../../context/connections/dialects.js';
|
||||
import {
|
||||
columnDisplayPartCount,
|
||||
formatDialectDisplayRef,
|
||||
formatDialectTableName,
|
||||
limitOffsetClause,
|
||||
parseDialectDisplayRef,
|
||||
} from '../../context/connections/dialect-helpers.js';
|
||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||
|
||||
type ClickHouseTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export class KtxClickHouseDialect {
|
||||
readonly type = 'clickhouse';
|
||||
/** @internal */
|
||||
export class KtxClickHouseDialect implements KtxDialect {
|
||||
readonly type = 'clickhouse' as const;
|
||||
|
||||
private readonly typeMappings: Record<string, KtxSchemaDimensionType> = {
|
||||
date: 'time',
|
||||
|
|
@ -45,9 +54,19 @@ export class KtxClickHouseDialect {
|
|||
}
|
||||
|
||||
formatTableName(table: ClickHouseTableNameRef): string {
|
||||
return table.db
|
||||
? `${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`
|
||||
: this.quoteIdentifier(table.name);
|
||||
return formatDialectTableName(table, this.quoteIdentifier.bind(this), 'ansi');
|
||||
}
|
||||
|
||||
formatDisplayRef(table: ClickHouseTableNameRef): string {
|
||||
return formatDialectDisplayRef(table, 'ansi');
|
||||
}
|
||||
|
||||
parseDisplayRef(display: string): KtxTableRef | null {
|
||||
return parseDialectDisplayRef(display, 'ansi');
|
||||
}
|
||||
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3 {
|
||||
return columnDisplayPartCount('ansi');
|
||||
}
|
||||
|
||||
mapDataType(nativeType: string): string {
|
||||
|
|
@ -97,29 +116,6 @@ export class KtxClickHouseDialect {
|
|||
return `SELECT ${quotedColumn} FROM ${tableName} WHERE ${quotedColumn} IS NOT NULL AND trim(toString(${quotedColumn})) != '' LIMIT ${limit}`;
|
||||
}
|
||||
|
||||
prepareQuery(sql: string, params?: Record<string, unknown>): { sql: string; params?: Record<string, unknown> } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
|
||||
let parameterizedQuery = sql;
|
||||
const queryParams: Record<string, unknown> = {};
|
||||
const sortedKeys = Object.keys(params).sort((a, b) => b.length - a.length);
|
||||
|
||||
for (const key of sortedKeys) {
|
||||
const placeholder = `:${key}`;
|
||||
if (parameterizedQuery.includes(placeholder)) {
|
||||
parameterizedQuery = parameterizedQuery.replace(
|
||||
new RegExp(`:${key}\\b`, 'g'),
|
||||
`{${key}:${this.inferClickHouseType(params[key])}}`,
|
||||
);
|
||||
queryParams[key] = params[key];
|
||||
}
|
||||
}
|
||||
|
||||
return { sql: parameterizedQuery, params: queryParams };
|
||||
}
|
||||
|
||||
getRandomSampleFilter(samplePct: number): string {
|
||||
if (samplePct <= 0 || samplePct >= 1) {
|
||||
return '';
|
||||
|
|
@ -132,7 +128,11 @@ export class KtxClickHouseDialect {
|
|||
}
|
||||
|
||||
getLimitOffsetClause(limit: number, offset?: number): string {
|
||||
return offset !== undefined && offset > 0 ? `LIMIT ${limit} OFFSET ${offset}` : `LIMIT ${limit}`;
|
||||
return limitOffsetClause(limit, offset);
|
||||
}
|
||||
|
||||
getTopClause(_limit: number): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
getNullCountExpression(column: string): string {
|
||||
|
|
@ -143,6 +143,18 @@ export class KtxClickHouseDialect {
|
|||
return `COUNT(DISTINCT ${column})`;
|
||||
}
|
||||
|
||||
textLengthExpression(columnSql: string): string {
|
||||
return `length(toString(${columnSql}))`;
|
||||
}
|
||||
|
||||
castToText(columnSql: string): string {
|
||||
return `toString(${columnSql})`;
|
||||
}
|
||||
|
||||
getSampleValueAggregation(innerSql: string): string {
|
||||
return `(SELECT arrayStringConcat(groupArray(toString(value)), '\\x1F') FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||
return `
|
||||
SELECT COUNT(DISTINCT val) AS cardinality
|
||||
|
|
@ -181,99 +193,9 @@ export class KtxClickHouseDialect {
|
|||
)
|
||||
`;
|
||||
}
|
||||
|
||||
getTimeTruncExpression(
|
||||
column: string,
|
||||
granularity: 'day' | 'week' | 'month' | 'quarter' | 'year',
|
||||
timezone?: string,
|
||||
): string {
|
||||
const tz = timezone ? `, '${timezone}'` : '';
|
||||
switch (granularity) {
|
||||
case 'day':
|
||||
return `toStartOfDay(${column}${tz})`;
|
||||
case 'week':
|
||||
return `toStartOfWeek(${column}, 1${tz})`;
|
||||
case 'month':
|
||||
return `toStartOfMonth(${column}${tz})`;
|
||||
case 'quarter':
|
||||
return `toStartOfQuarter(${column}${tz})`;
|
||||
case 'year':
|
||||
return `toStartOfYear(${column}${tz})`;
|
||||
}
|
||||
}
|
||||
|
||||
getCustomTimeTruncExpression(column: string, interval: string, origin?: string, timezone?: string): string {
|
||||
const col = timezone ? `toTimezone(${column}, '${timezone}')` : column;
|
||||
const [rawAmount, rawUnit] = interval.split(' ');
|
||||
const amount = Number(rawAmount);
|
||||
const unit = rawUnit!.toLowerCase();
|
||||
const originExpr = origin ? `toDateTime('${origin}')` : "toDateTime('1970-01-01')";
|
||||
const calendarUnit = this.toClickHouseDateDiffUnit(unit);
|
||||
if (calendarUnit) {
|
||||
return `dateAdd(${calendarUnit}, intDiv(dateDiff(${calendarUnit}, ${originExpr}, ${col}), ${amount}) * ${amount}, ${originExpr})`;
|
||||
}
|
||||
const seconds = this.intervalToSeconds(amount, unit);
|
||||
return `addSeconds(${originExpr}, intDiv(toUInt64(dateDiff('second', ${originExpr}, ${col})), ${seconds}) * ${seconds})`;
|
||||
}
|
||||
|
||||
parseIntervalToSql(interval: string): string {
|
||||
const [amount, unit] = interval.split(' ');
|
||||
return `INTERVAL ${amount} ${unit!.toUpperCase()}`;
|
||||
}
|
||||
|
||||
private unwrapClickHouseType(value: string, wrapper: string): string {
|
||||
const prefix = `${wrapper}(`;
|
||||
return value.startsWith(prefix) && value.endsWith(')') ? value.slice(prefix.length, -1) : value;
|
||||
}
|
||||
|
||||
private inferClickHouseType(value: unknown): string {
|
||||
if (value === null || value === undefined) {
|
||||
return 'String';
|
||||
}
|
||||
if (typeof value === 'boolean') {
|
||||
return 'Bool';
|
||||
}
|
||||
if (typeof value === 'number') {
|
||||
return Number.isInteger(value) ? 'Int64' : 'Float64';
|
||||
}
|
||||
if (value instanceof Date) {
|
||||
return 'DateTime';
|
||||
}
|
||||
return 'String';
|
||||
}
|
||||
|
||||
private toClickHouseDateDiffUnit(unit: string): string | null {
|
||||
if (unit === 'month' || unit === 'months') {
|
||||
return "'month'";
|
||||
}
|
||||
if (unit === 'quarter' || unit === 'quarters') {
|
||||
return "'quarter'";
|
||||
}
|
||||
if (unit === 'year' || unit === 'years') {
|
||||
return "'year'";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private intervalToSeconds(amount: number, unit: string): number {
|
||||
switch (unit) {
|
||||
case 'second':
|
||||
case 'seconds':
|
||||
return amount;
|
||||
case 'minute':
|
||||
case 'minutes':
|
||||
return amount * 60;
|
||||
case 'hour':
|
||||
case 'hours':
|
||||
return amount * 3600;
|
||||
case 'day':
|
||||
case 'days':
|
||||
return amount * 86400;
|
||||
case 'week':
|
||||
case 'weeks':
|
||||
return amount * 604800;
|
||||
default:
|
||||
return amount * 86400;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import mysql, { type FieldPacket, type Pool, type RowDataPacket } from 'mysql2/p
|
|||
import { readFileSync } from 'node:fs';
|
||||
import { homedir } from 'node:os';
|
||||
import { resolve } from 'node:path';
|
||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||
import {
|
||||
constraintDiscoveryWarning,
|
||||
|
|
@ -30,7 +31,6 @@ import {
|
|||
type KtxTableSampleInput,
|
||||
type KtxTableSampleResult,
|
||||
} from '../../context/scan/types.js';
|
||||
import { KtxMysqlDialect } from './dialect.js';
|
||||
|
||||
export interface KtxMysqlConnectionConfig {
|
||||
driver?: string;
|
||||
|
|
@ -303,6 +303,25 @@ function queryParams(params: Record<string, unknown> | unknown[] | undefined): u
|
|||
return Array.isArray(params) ? params : Object.values(params);
|
||||
}
|
||||
|
||||
/** @internal */
|
||||
export function prepareMysqlReadOnlyQuery(
|
||||
sql: string,
|
||||
params?: Record<string, unknown>,
|
||||
): { sql: string; params?: unknown[] } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
const values: unknown[] = [];
|
||||
const parameterizedQuery = sql.replace(/:([A-Za-z_][A-Za-z0-9_]*)\b/g, (placeholder, key: string) => {
|
||||
if (!(key in params)) {
|
||||
return placeholder;
|
||||
}
|
||||
values.push(params[key]);
|
||||
return '?';
|
||||
});
|
||||
return { sql: parameterizedQuery, params: values };
|
||||
}
|
||||
|
||||
export function isKtxMysqlConnectionConfig(
|
||||
connection: KtxMysqlConnectionConfig | undefined,
|
||||
): connection is KtxMysqlConnectionConfig {
|
||||
|
|
@ -376,7 +395,7 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
|
|||
private readonly poolFactory: KtxMysqlPoolFactory;
|
||||
private readonly endpointResolver?: KtxMysqlEndpointResolver;
|
||||
private readonly now: () => Date;
|
||||
private readonly dialect = new KtxMysqlDialect();
|
||||
private readonly dialect = getDialectForDriver('mysql');
|
||||
private pool: KtxMysqlPool | null = null;
|
||||
private resolvedEndpoint: KtxMysqlResolvedEndpoint | null = null;
|
||||
|
||||
|
|
@ -550,7 +569,7 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
|
|||
const limitedSql = limitSqlForExecution(assertReadOnlySql(input.sql), input.maxRows);
|
||||
const prepared = Array.isArray(input.params)
|
||||
? { sql: limitedSql, params: input.params }
|
||||
: this.dialect.prepareQuery(limitedSql, input.params);
|
||||
: prepareMysqlReadOnlyQuery(limitedSql, input.params);
|
||||
const result = await this.query(prepared.sql, prepared.params);
|
||||
return { ...result, rowCount: result.rows.length };
|
||||
}
|
||||
|
|
@ -625,6 +644,7 @@ export class KtxMysqlScanConnector implements KtxScanConnector {
|
|||
filterSchemas,
|
||||
);
|
||||
return rows.map((row) => ({
|
||||
catalog: null,
|
||||
schema: row.TABLE_SCHEMA,
|
||||
name: row.TABLE_NAME,
|
||||
kind: row.TABLE_TYPE === 'VIEW' ? ('view' as const) : ('table' as const),
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
import type { KtxDialect } from '../../context/connections/dialects.js';
|
||||
import {
|
||||
columnDisplayPartCount,
|
||||
formatDialectDisplayRef,
|
||||
formatDialectTableName,
|
||||
limitOffsetClause,
|
||||
parseDialectDisplayRef,
|
||||
} from '../../context/connections/dialect-helpers.js';
|
||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||
|
||||
type MysqlTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export class KtxMysqlDialect {
|
||||
readonly type = 'mysql';
|
||||
/** @internal */
|
||||
export class KtxMysqlDialect implements KtxDialect {
|
||||
readonly type = 'mysql' as const;
|
||||
|
||||
private readonly typeMappings: Record<string, KtxSchemaDimensionType> = {
|
||||
datetime: 'time',
|
||||
|
|
@ -41,9 +50,19 @@ export class KtxMysqlDialect {
|
|||
}
|
||||
|
||||
formatTableName(table: MysqlTableNameRef): string {
|
||||
return table.db
|
||||
? `${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`
|
||||
: this.quoteIdentifier(table.name);
|
||||
return formatDialectTableName(table, this.quoteIdentifier.bind(this), 'ansi');
|
||||
}
|
||||
|
||||
formatDisplayRef(table: MysqlTableNameRef): string {
|
||||
return formatDialectDisplayRef(table, 'ansi');
|
||||
}
|
||||
|
||||
parseDisplayRef(display: string): KtxTableRef | null {
|
||||
return parseDialectDisplayRef(display, 'ansi');
|
||||
}
|
||||
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3 {
|
||||
return columnDisplayPartCount('ansi');
|
||||
}
|
||||
|
||||
mapDataType(nativeType: string): string {
|
||||
|
|
@ -91,21 +110,6 @@ export class KtxMysqlDialect {
|
|||
return `SELECT ${quotedColumn} FROM ${tableName} WHERE ${quotedColumn} IS NOT NULL AND TRIM(CAST(${quotedColumn} AS CHAR)) != '' LIMIT ${limit}`;
|
||||
}
|
||||
|
||||
prepareQuery(sql: string, params?: Record<string, unknown>): { sql: string; params?: unknown[] } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
const values: unknown[] = [];
|
||||
const parameterizedQuery = sql.replace(/:([A-Za-z_][A-Za-z0-9_]*)\b/g, (placeholder, key: string) => {
|
||||
if (!(key in params)) {
|
||||
return placeholder;
|
||||
}
|
||||
values.push(params[key]);
|
||||
return '?';
|
||||
});
|
||||
return { sql: parameterizedQuery, params: values };
|
||||
}
|
||||
|
||||
getRandomSampleFilter(samplePct: number): string {
|
||||
if (samplePct <= 0 || samplePct >= 1) {
|
||||
return '';
|
||||
|
|
@ -118,7 +122,11 @@ export class KtxMysqlDialect {
|
|||
}
|
||||
|
||||
getLimitOffsetClause(limit: number, offset?: number): string {
|
||||
return offset !== undefined && offset > 0 ? `LIMIT ${limit} OFFSET ${offset}` : `LIMIT ${limit}`;
|
||||
return limitOffsetClause(limit, offset);
|
||||
}
|
||||
|
||||
getTopClause(_limit: number): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
getNullCountExpression(column: string): string {
|
||||
|
|
@ -129,6 +137,18 @@ export class KtxMysqlDialect {
|
|||
return `COUNT(DISTINCT ${column})`;
|
||||
}
|
||||
|
||||
textLengthExpression(columnSql: string): string {
|
||||
return `CHAR_LENGTH(CAST(${columnSql} AS CHAR))`;
|
||||
}
|
||||
|
||||
castToText(columnSql: string): string {
|
||||
return `CAST(${columnSql} AS CHAR)`;
|
||||
}
|
||||
|
||||
getSampleValueAggregation(innerSql: string): string {
|
||||
return `(SELECT GROUP_CONCAT(CAST(value AS CHAR) SEPARATOR CHAR(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||
return `
|
||||
SELECT COUNT(DISTINCT val) AS cardinality
|
||||
|
|
@ -167,36 +187,4 @@ export class KtxMysqlDialect {
|
|||
) AS sampled
|
||||
`;
|
||||
}
|
||||
|
||||
getTimeTruncExpression(
|
||||
column: string,
|
||||
granularity: 'day' | 'week' | 'month' | 'quarter' | 'year',
|
||||
timezone?: string,
|
||||
): string {
|
||||
const col = timezone ? `CONVERT_TZ(${column}, '+00:00', '${timezone}')` : column;
|
||||
switch (granularity) {
|
||||
case 'day':
|
||||
return `DATE(${col})`;
|
||||
case 'week':
|
||||
return `DATE(${col} - INTERVAL WEEKDAY(${col}) DAY)`;
|
||||
case 'month':
|
||||
return `DATE_FORMAT(${col}, '%Y-%m-01')`;
|
||||
case 'quarter':
|
||||
return `MAKEDATE(YEAR(${col}), 1) + INTERVAL (QUARTER(${col}) - 1) QUARTER`;
|
||||
case 'year':
|
||||
return `DATE_FORMAT(${col}, '%Y-01-01')`;
|
||||
}
|
||||
}
|
||||
|
||||
getCustomTimeTruncExpression(column: string, interval: string, origin?: string, timezone?: string): string {
|
||||
const col = timezone ? `CONVERT_TZ(${column}, '+00:00', '${timezone}')` : column;
|
||||
const [amount, unit] = interval.split(' ');
|
||||
const originExpr = origin ? `'${origin}'` : `'1970-01-01'`;
|
||||
return `DATE_ADD(${originExpr}, INTERVAL FLOOR(TIMESTAMPDIFF(${unit!.toUpperCase()}, ${originExpr}, ${col}) / ${amount}) * ${amount} ${unit!.toUpperCase()})`;
|
||||
}
|
||||
|
||||
parseIntervalToSql(interval: string): string {
|
||||
const [amount, unit] = interval.split(' ');
|
||||
return `INTERVAL ${amount} ${unit!.toUpperCase()}`;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import { readFileSync } from 'node:fs';
|
||||
import { homedir } from 'node:os';
|
||||
import { resolve } from 'node:path';
|
||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||
|
|
@ -26,7 +27,6 @@ import {
|
|||
type KtxTableSampleResult,
|
||||
} from '../../context/scan/types.js';
|
||||
import { Pool } from 'pg';
|
||||
import { KtxPostgresDialect } from './dialect.js';
|
||||
|
||||
const PG_OID_TYPE_MAP: Record<number, string> = {
|
||||
16: 'boolean',
|
||||
|
|
@ -219,6 +219,29 @@ function groupByTable<T extends { table_name: string }>(rows: T[]): Map<string,
|
|||
return grouped;
|
||||
}
|
||||
|
||||
/** @internal */
|
||||
export function preparePostgresReadOnlyQuery(
|
||||
sql: string,
|
||||
params?: Record<string, unknown>,
|
||||
): { sql: string; params?: unknown[] } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
const paramNames = Object.keys(params);
|
||||
const values: unknown[] = new Array(paramNames.length);
|
||||
const paramIndexMap = new Map<string, number>();
|
||||
paramNames.forEach((name, index) => {
|
||||
paramIndexMap.set(name, index + 1);
|
||||
values[index] = params[name];
|
||||
});
|
||||
const sortedKeys = [...paramNames].sort((a, b) => b.length - a.length);
|
||||
let parameterizedQuery = sql;
|
||||
for (const name of sortedKeys) {
|
||||
parameterizedQuery = parameterizedQuery.replace(new RegExp(`:${name}\\b`, 'g'), `$${paramIndexMap.get(name)}`);
|
||||
}
|
||||
return { sql: parameterizedQuery, params: values };
|
||||
}
|
||||
|
||||
function primaryKeyMap(rows: PostgresPrimaryKeyRow[]): Map<string, Set<string>> {
|
||||
const grouped = new Map<string, Set<string>>();
|
||||
for (const row of rows) {
|
||||
|
|
@ -400,7 +423,7 @@ export class KtxPostgresScanConnector implements KtxScanConnector {
|
|||
private readonly poolFactory: KtxPostgresPoolFactory;
|
||||
private readonly endpointResolver?: KtxPostgresEndpointResolver;
|
||||
private readonly now: () => Date;
|
||||
private readonly dialect = new KtxPostgresDialect();
|
||||
private readonly dialect = getDialectForDriver('postgres');
|
||||
private pool: KtxPostgresPool | null = null;
|
||||
private lastIdlePoolError: Error | null = null;
|
||||
private resolvedEndpoint: KtxPostgresResolvedEndpoint | null = null;
|
||||
|
|
@ -489,7 +512,7 @@ export class KtxPostgresScanConnector implements KtxScanConnector {
|
|||
const limitedSql = limitSqlForExecution(assertReadOnlySql(input.sql), input.maxRows);
|
||||
const prepared = Array.isArray(input.params)
|
||||
? { sql: limitedSql, params: input.params }
|
||||
: this.dialect.prepareQuery(limitedSql, input.params);
|
||||
: preparePostgresReadOnlyQuery(limitedSql, input.params);
|
||||
const result = await this.query(prepared.sql, prepared.params);
|
||||
return { ...result, rowCount: result.rows.length };
|
||||
}
|
||||
|
|
@ -584,6 +607,7 @@ export class KtxPostgresScanConnector implements KtxScanConnector {
|
|||
[filterSchemas],
|
||||
);
|
||||
return rows.map((row) => ({
|
||||
catalog: null,
|
||||
schema: row.schema_name,
|
||||
name: row.table_name,
|
||||
kind: row.table_kind === 'v' ? ('view' as const) : ('table' as const),
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
import type { KtxDialect } from '../../context/connections/dialects.js';
|
||||
import {
|
||||
columnDisplayPartCount,
|
||||
formatDialectDisplayRef,
|
||||
formatDialectTableName,
|
||||
limitOffsetClause,
|
||||
parseDialectDisplayRef,
|
||||
} from '../../context/connections/dialect-helpers.js';
|
||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||
|
||||
type PostgresTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export class KtxPostgresDialect {
|
||||
readonly type = 'postgresql';
|
||||
/** @internal */
|
||||
export class KtxPostgresDialect implements KtxDialect {
|
||||
readonly type = 'postgres' as const;
|
||||
|
||||
private readonly typeMappings: Record<string, KtxSchemaDimensionType> = {
|
||||
timestamp: 'time',
|
||||
|
|
@ -45,9 +54,19 @@ export class KtxPostgresDialect {
|
|||
}
|
||||
|
||||
formatTableName(table: PostgresTableNameRef): string {
|
||||
return table.db
|
||||
? `${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`
|
||||
: this.quoteIdentifier(table.name);
|
||||
return formatDialectTableName(table, this.quoteIdentifier.bind(this), 'ansi');
|
||||
}
|
||||
|
||||
formatDisplayRef(table: PostgresTableNameRef): string {
|
||||
return formatDialectDisplayRef(table, 'ansi');
|
||||
}
|
||||
|
||||
parseDisplayRef(display: string): KtxTableRef | null {
|
||||
return parseDialectDisplayRef(display, 'ansi');
|
||||
}
|
||||
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3 {
|
||||
return columnDisplayPartCount('ansi');
|
||||
}
|
||||
|
||||
mapDataType(nativeType: string): string {
|
||||
|
|
@ -92,25 +111,6 @@ export class KtxPostgresDialect {
|
|||
return `SELECT ${quotedColumn} FROM ${tableName} WHERE ${quotedColumn} IS NOT NULL AND TRIM(CAST(${quotedColumn} AS TEXT)) != '' LIMIT ${limit}`;
|
||||
}
|
||||
|
||||
prepareQuery(sql: string, params?: Record<string, unknown>): { sql: string; params?: unknown[] } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
const paramNames = Object.keys(params);
|
||||
const values: unknown[] = new Array(paramNames.length);
|
||||
const paramIndexMap = new Map<string, number>();
|
||||
paramNames.forEach((name, index) => {
|
||||
paramIndexMap.set(name, index + 1);
|
||||
values[index] = params[name];
|
||||
});
|
||||
const sortedKeys = [...paramNames].sort((a, b) => b.length - a.length);
|
||||
let parameterizedQuery = sql;
|
||||
for (const name of sortedKeys) {
|
||||
parameterizedQuery = parameterizedQuery.replace(new RegExp(`:${name}\\b`, 'g'), `$${paramIndexMap.get(name)}`);
|
||||
}
|
||||
return { sql: parameterizedQuery, params: values };
|
||||
}
|
||||
|
||||
getRandomSampleFilter(samplePct: number): string {
|
||||
if (samplePct <= 0 || samplePct >= 1) {
|
||||
return '';
|
||||
|
|
@ -126,7 +126,11 @@ export class KtxPostgresDialect {
|
|||
}
|
||||
|
||||
getLimitOffsetClause(limit: number, offset?: number): string {
|
||||
return offset !== undefined && offset > 0 ? `LIMIT ${limit} OFFSET ${offset}` : `LIMIT ${limit}`;
|
||||
return limitOffsetClause(limit, offset);
|
||||
}
|
||||
|
||||
getTopClause(_limit: number): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
getNullCountExpression(column: string): string {
|
||||
|
|
@ -137,6 +141,18 @@ export class KtxPostgresDialect {
|
|||
return `COUNT(DISTINCT ${column})`;
|
||||
}
|
||||
|
||||
textLengthExpression(columnSql: string): string {
|
||||
return `LENGTH(CAST(${columnSql} AS TEXT))`;
|
||||
}
|
||||
|
||||
castToText(columnSql: string): string {
|
||||
return `CAST(${columnSql} AS TEXT)`;
|
||||
}
|
||||
|
||||
getSampleValueAggregation(innerSql: string): string {
|
||||
return `(SELECT STRING_AGG(CAST(value AS TEXT), CHR(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||
return `
|
||||
WITH sampled AS (
|
||||
|
|
@ -191,23 +207,4 @@ export class KtxPostgresDialect {
|
|||
FROM sampled
|
||||
`;
|
||||
}
|
||||
|
||||
getTimeTruncExpression(
|
||||
column: string,
|
||||
granularity: 'day' | 'week' | 'month' | 'quarter' | 'year',
|
||||
timezone?: string,
|
||||
): string {
|
||||
const col = timezone ? `(${column} AT TIME ZONE '${timezone.replace(/'/g, "''")}')` : column;
|
||||
return `DATE_TRUNC('${granularity}', ${col})`;
|
||||
}
|
||||
|
||||
getCustomTimeTruncExpression(column: string, interval: string, origin?: string, timezone?: string): string {
|
||||
const col = timezone ? `(${column} AT TIME ZONE '${timezone.replace(/'/g, "''")}')` : column;
|
||||
const originExpr = origin ? `TIMESTAMP '${origin.replace(/'/g, "''")}'` : "TIMESTAMP '1970-01-01'";
|
||||
return `${originExpr} + FLOOR(EXTRACT(EPOCH FROM (${col} - ${originExpr})) / EXTRACT(EPOCH FROM INTERVAL '${interval.replace(/'/g, "''")}')) * INTERVAL '${interval.replace(/'/g, "''")}'`;
|
||||
}
|
||||
|
||||
parseIntervalToSql(interval: string): string {
|
||||
return `INTERVAL '${interval.replace(/'/g, "''")}'`;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import { createPrivateKey } from 'node:crypto';
|
|||
import { readFileSync } from 'node:fs';
|
||||
import { homedir } from 'node:os';
|
||||
import { resolve } from 'node:path';
|
||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||
|
|
@ -27,7 +28,6 @@ import {
|
|||
} from '../../context/scan/types.js';
|
||||
import snowflake from 'snowflake-sdk';
|
||||
import type { Bind, Binds, Connection, ConnectionOptions } from 'snowflake-sdk';
|
||||
import { KtxSnowflakeDialect } from './dialect.js';
|
||||
import { assertSafeSnowflakeIdentifier, quoteSnowflakeIdentifier } from './identifiers.js';
|
||||
import { configureSnowflakeSdkLogger } from './sdk-logger.js';
|
||||
|
||||
|
|
@ -229,6 +229,14 @@ function toSnowflakeBinds(params: unknown[] | undefined): Binds | undefined {
|
|||
return params?.map((value) => toSnowflakeBind(value));
|
||||
}
|
||||
|
||||
/** @internal */
|
||||
export function prepareSnowflakeReadOnlyQuery(
|
||||
sql: string,
|
||||
params?: Record<string, unknown>,
|
||||
): { sql: string; params?: unknown[] } {
|
||||
return { sql, params: params ? Object.values(params) : undefined };
|
||||
}
|
||||
|
||||
export function isKtxSnowflakeConnectionConfig(
|
||||
connection: KtxSnowflakeConnectionConfig | undefined,
|
||||
): connection is KtxSnowflakeConnectionConfig {
|
||||
|
|
@ -430,6 +438,7 @@ class SnowflakeSdkDriver implements KtxSnowflakeDriver {
|
|||
[this.resolved.database, ...(schemas ?? [])],
|
||||
);
|
||||
return result.rows.map((row) => ({
|
||||
catalog: this.resolved.database,
|
||||
schema: String(row[0]),
|
||||
name: String(row[1]),
|
||||
kind: String(row[2]) === 'VIEW' ? ('view' as const) : ('table' as const),
|
||||
|
|
@ -550,7 +559,7 @@ export class KtxSnowflakeScanConnector implements KtxScanConnector {
|
|||
|
||||
private readonly resolved: KtxSnowflakeResolvedConnectionConfig;
|
||||
private readonly driverFactory: KtxSnowflakeDriverFactory;
|
||||
private readonly dialect = new KtxSnowflakeDialect();
|
||||
private readonly dialect = getDialectForDriver('snowflake');
|
||||
private readonly now: () => Date;
|
||||
private driverInstance: KtxSnowflakeDriver | null = null;
|
||||
|
||||
|
|
@ -635,7 +644,7 @@ export class KtxSnowflakeScanConnector implements KtxScanConnector {
|
|||
async executeReadOnly(input: KtxSnowflakeReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {
|
||||
this.assertConnection(input.connectionId);
|
||||
const limitedSql = limitSqlForExecution(assertReadOnlySql(input.sql), input.maxRows);
|
||||
const prepared = this.dialect.prepareQuery(limitedSql, input.params);
|
||||
const prepared = prepareSnowflakeReadOnlyQuery(limitedSql, input.params);
|
||||
return this.getDriver().query(prepared.sql, prepared.params);
|
||||
}
|
||||
|
||||
|
|
@ -696,6 +705,7 @@ export class KtxSnowflakeScanConnector implements KtxScanConnector {
|
|||
[this.resolved.database, ...(schemas ?? [])],
|
||||
);
|
||||
return result.rows.map((row) => ({
|
||||
catalog: this.resolved.database,
|
||||
schema: String(row[0]),
|
||||
name: String(row[1]),
|
||||
kind: String(row[2]) === 'VIEW' ? ('view' as const) : ('table' as const),
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
import type { KtxDialect } from '../../context/connections/dialects.js';
|
||||
import {
|
||||
columnDisplayPartCount,
|
||||
formatDialectDisplayRef,
|
||||
formatDialectTableName,
|
||||
limitOffsetClause,
|
||||
parseDialectDisplayRef,
|
||||
} from '../../context/connections/dialect-helpers.js';
|
||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||
|
||||
type SnowflakeTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export class KtxSnowflakeDialect {
|
||||
readonly type = 'snowflake';
|
||||
/** @internal */
|
||||
export class KtxSnowflakeDialect implements KtxDialect {
|
||||
readonly type = 'snowflake' as const;
|
||||
|
||||
private readonly typeMappings: Record<string, KtxSchemaDimensionType> = {
|
||||
TIMESTAMP_NTZ: 'time',
|
||||
|
|
@ -45,13 +54,19 @@ export class KtxSnowflakeDialect {
|
|||
}
|
||||
|
||||
formatTableName(table: SnowflakeTableNameRef): string {
|
||||
if (table.catalog && table.db) {
|
||||
return `${this.quoteIdentifier(table.catalog)}.${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`;
|
||||
}
|
||||
if (table.db) {
|
||||
return `${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`;
|
||||
}
|
||||
return this.quoteIdentifier(table.name);
|
||||
return formatDialectTableName(table, this.quoteIdentifier.bind(this), 'three-part');
|
||||
}
|
||||
|
||||
formatDisplayRef(table: SnowflakeTableNameRef): string {
|
||||
return formatDialectDisplayRef(table, 'three-part');
|
||||
}
|
||||
|
||||
parseDisplayRef(display: string): KtxTableRef | null {
|
||||
return parseDialectDisplayRef(display, 'three-part');
|
||||
}
|
||||
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3 {
|
||||
return columnDisplayPartCount('three-part');
|
||||
}
|
||||
|
||||
mapDataType(nativeType: string): string {
|
||||
|
|
@ -96,10 +111,6 @@ export class KtxSnowflakeDialect {
|
|||
return `SELECT ${quotedColumn} FROM ${tableName} WHERE ${quotedColumn} IS NOT NULL AND TRIM(CAST(${quotedColumn} AS STRING)) != '' LIMIT ${limit}`;
|
||||
}
|
||||
|
||||
prepareQuery(sql: string, params?: Record<string, unknown>): { sql: string; params?: unknown[] } {
|
||||
return { sql, params: params ? Object.values(params) : undefined };
|
||||
}
|
||||
|
||||
getRandomSampleFilter(samplePct: number): string {
|
||||
if (samplePct <= 0 || samplePct >= 1) {
|
||||
return '';
|
||||
|
|
@ -115,7 +126,11 @@ export class KtxSnowflakeDialect {
|
|||
}
|
||||
|
||||
getLimitOffsetClause(limit: number, offset?: number): string {
|
||||
return offset !== undefined && offset > 0 ? `LIMIT ${limit} OFFSET ${offset}` : `LIMIT ${limit}`;
|
||||
return limitOffsetClause(limit, offset);
|
||||
}
|
||||
|
||||
getTopClause(_limit: number): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
getNullCountExpression(column: string): string {
|
||||
|
|
@ -126,6 +141,18 @@ export class KtxSnowflakeDialect {
|
|||
return `APPROX_COUNT_DISTINCT(${column})`;
|
||||
}
|
||||
|
||||
textLengthExpression(columnSql: string): string {
|
||||
return `LENGTH(CAST(${columnSql} AS TEXT))`;
|
||||
}
|
||||
|
||||
castToText(columnSql: string): string {
|
||||
return `CAST(${columnSql} AS VARCHAR)`;
|
||||
}
|
||||
|
||||
getSampleValueAggregation(innerSql: string): string {
|
||||
return `(SELECT LISTAGG(CAST(value AS VARCHAR), '\\x1f') FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||
return `
|
||||
WITH sampled AS (
|
||||
|
|
@ -164,24 +191,4 @@ export class KtxSnowflakeDialect {
|
|||
FROM sampled
|
||||
`;
|
||||
}
|
||||
|
||||
getTimeTruncExpression(
|
||||
column: string,
|
||||
granularity: 'day' | 'week' | 'month' | 'quarter' | 'year',
|
||||
timezone?: string,
|
||||
): string {
|
||||
const target = timezone ? `CONVERT_TIMEZONE('UTC', '${timezone}', ${column})` : column;
|
||||
return `DATE_TRUNC('${granularity}', ${target})`;
|
||||
}
|
||||
|
||||
getCustomTimeTruncExpression(column: string, interval: string, origin?: string, timezone?: string): string {
|
||||
const target = timezone ? `CONVERT_TIMEZONE('UTC', '${timezone}', ${column})` : column;
|
||||
const [amount, unit] = interval.split(' ');
|
||||
const originExpr = origin ? `'${origin}'::TIMESTAMP` : `'1970-01-01'::TIMESTAMP`;
|
||||
return `DATEADD(${unit}, FLOOR(DATEDIFF(${unit}, ${originExpr}, ${target}) / ${amount}) * ${amount}, ${originExpr})`;
|
||||
}
|
||||
|
||||
parseIntervalToSql(interval: string): string {
|
||||
return `INTERVAL '${interval}'`;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,11 +3,11 @@ import { existsSync, readFileSync, statSync } from 'node:fs';
|
|||
import { homedir } from 'node:os';
|
||||
import { isAbsolute, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||
import { normalizeQueryRows } from '../../context/connections/query-executor.js';
|
||||
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||
import { KtxSqliteDialect } from './dialect.js';
|
||||
|
||||
export interface KtxSqliteConnectionConfig {
|
||||
driver?: string;
|
||||
|
|
@ -157,7 +157,7 @@ export class KtxSqliteScanConnector implements KtxScanConnector {
|
|||
private readonly connectionId: string;
|
||||
private readonly dbPath: string;
|
||||
private readonly now: () => Date;
|
||||
private readonly dialect = new KtxSqliteDialect();
|
||||
private readonly dialect = getDialectForDriver('sqlite');
|
||||
private db: Database.Database | null = null;
|
||||
|
||||
constructor(options: KtxSqliteScanConnectorOptions) {
|
||||
|
|
@ -209,6 +209,31 @@ export class KtxSqliteScanConnector implements KtxScanConnector {
|
|||
};
|
||||
}
|
||||
|
||||
async listSchemas(): Promise<string[]> {
|
||||
return [];
|
||||
}
|
||||
|
||||
async listTables(_schemas?: string[]): Promise<KtxTableListEntry[]> {
|
||||
const rows = this.database()
|
||||
.prepare(
|
||||
`
|
||||
SELECT name, type
|
||||
FROM sqlite_master
|
||||
WHERE type IN ('table', 'view')
|
||||
AND name NOT LIKE 'sqlite_%'
|
||||
ORDER BY name
|
||||
`,
|
||||
)
|
||||
.all() as SqliteMasterRow[];
|
||||
|
||||
return rows.map((row) => ({
|
||||
catalog: null,
|
||||
schema: '',
|
||||
name: row.name,
|
||||
kind: row.type === 'view' ? ('view' as const) : ('table' as const),
|
||||
}));
|
||||
}
|
||||
|
||||
async sampleTable(input: KtxTableSampleInput, _ctx: KtxScanContext): Promise<KtxTableSampleResult> {
|
||||
this.assertConnection(input.connectionId);
|
||||
const result = this.query(this.dialect.generateSampleQuery(this.qTableName(input.table), input.limit, input.columns));
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
import type { KtxDialect } from '../../context/connections/dialects.js';
|
||||
import {
|
||||
columnDisplayPartCount,
|
||||
formatDialectDisplayRef,
|
||||
formatDialectTableName,
|
||||
limitOffsetClause,
|
||||
parseDialectDisplayRef,
|
||||
} from '../../context/connections/dialect-helpers.js';
|
||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||
|
||||
type SqliteTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export class KtxSqliteDialect {
|
||||
readonly type = 'sqlite';
|
||||
/** @internal */
|
||||
export class KtxSqliteDialect implements KtxDialect {
|
||||
readonly type = 'sqlite' as const;
|
||||
|
||||
private readonly typeMappings: Record<string, KtxSchemaDimensionType> = {
|
||||
DATETIME: 'time',
|
||||
|
|
@ -29,7 +38,19 @@ export class KtxSqliteDialect {
|
|||
}
|
||||
|
||||
formatTableName(table: SqliteTableNameRef): string {
|
||||
return this.quoteIdentifier(table.name);
|
||||
return formatDialectTableName(table, this.quoteIdentifier.bind(this), 'sqlite');
|
||||
}
|
||||
|
||||
formatDisplayRef(table: SqliteTableNameRef): string {
|
||||
return formatDialectDisplayRef(table, 'sqlite');
|
||||
}
|
||||
|
||||
parseDisplayRef(display: string): KtxTableRef | null {
|
||||
return parseDialectDisplayRef(display, 'sqlite');
|
||||
}
|
||||
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3 {
|
||||
return columnDisplayPartCount('sqlite');
|
||||
}
|
||||
|
||||
mapDataType(nativeType: string): string {
|
||||
|
|
@ -76,10 +97,6 @@ export class KtxSqliteDialect {
|
|||
return `SELECT ${quoted} FROM ${tableName} WHERE ${quoted} IS NOT NULL AND TRIM(CAST(${quoted} AS TEXT)) != '' LIMIT ${limit}`;
|
||||
}
|
||||
|
||||
prepareQuery(sql: string, params?: Record<string, unknown>): { sql: string; params?: unknown } {
|
||||
return params ? { sql, params } : { sql };
|
||||
}
|
||||
|
||||
getRandomSampleFilter(samplePct: number): string {
|
||||
if (samplePct <= 0 || samplePct >= 1) {
|
||||
return '';
|
||||
|
|
@ -92,7 +109,11 @@ export class KtxSqliteDialect {
|
|||
}
|
||||
|
||||
getLimitOffsetClause(limit: number, offset?: number): string {
|
||||
return offset !== undefined && offset > 0 ? `LIMIT ${limit} OFFSET ${offset}` : `LIMIT ${limit}`;
|
||||
return limitOffsetClause(limit, offset);
|
||||
}
|
||||
|
||||
getTopClause(_limit: number): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
getNullCountExpression(column: string): string {
|
||||
|
|
@ -103,6 +124,18 @@ export class KtxSqliteDialect {
|
|||
return `COUNT(DISTINCT ${column})`;
|
||||
}
|
||||
|
||||
textLengthExpression(columnSql: string): string {
|
||||
return `LENGTH(CAST(${columnSql} AS TEXT))`;
|
||||
}
|
||||
|
||||
castToText(columnSql: string): string {
|
||||
return `CAST(${columnSql} AS TEXT)`;
|
||||
}
|
||||
|
||||
getSampleValueAggregation(innerSql: string): string {
|
||||
return `(SELECT GROUP_CONCAT(CAST(value AS TEXT), char(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||
return `
|
||||
WITH sampled AS (
|
||||
|
|
@ -143,35 +176,4 @@ export class KtxSqliteDialect {
|
|||
FROM sampled
|
||||
`;
|
||||
}
|
||||
|
||||
getTimeTruncExpression(
|
||||
column: string,
|
||||
granularity: 'day' | 'week' | 'month' | 'quarter' | 'year',
|
||||
_timezone?: string,
|
||||
): string {
|
||||
switch (granularity) {
|
||||
case 'day':
|
||||
return `DATE(${column})`;
|
||||
case 'week':
|
||||
return `DATE(${column}, 'weekday 0', '-6 days')`;
|
||||
case 'month':
|
||||
return `DATE(${column}, 'start of month')`;
|
||||
case 'quarter':
|
||||
return `DATE(${column}, 'start of month', '-' || ((CAST(STRFTIME('%m', ${column}) AS INTEGER) - 1) % 3) || ' months')`;
|
||||
case 'year':
|
||||
return `DATE(${column}, 'start of year')`;
|
||||
}
|
||||
}
|
||||
|
||||
getCustomTimeTruncExpression(column: string, interval: string, origin?: string, _timezone?: string): string {
|
||||
const [amount, unit] = interval.split(' ');
|
||||
const originExpr = origin ? `julianday('${origin}')` : `julianday('1970-01-01')`;
|
||||
const unitDays = unit === 'day' ? 1 : unit === 'week' ? 7 : 30;
|
||||
const intervalDays = Number(amount) * unitDays;
|
||||
return `DATE(julianday('1970-01-01') + (CAST((julianday(${column}) - ${originExpr}) / ${intervalDays} AS INTEGER) * ${intervalDays}))`;
|
||||
}
|
||||
|
||||
parseIntervalToSql(interval: string): string {
|
||||
return `'${interval}'`;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { assertReadOnlySql } from '../../context/connections/read-only-sql.js';
|
||||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
|
||||
import { scopedTableNames } from '../../context/scan/table-ref.js';
|
||||
import {
|
||||
|
|
@ -26,7 +27,6 @@ import { readFileSync } from 'node:fs';
|
|||
import { homedir } from 'node:os';
|
||||
import { resolve } from 'node:path';
|
||||
import sql from 'mssql';
|
||||
import { KtxSqlServerDialect } from './dialect.js';
|
||||
|
||||
export interface KtxSqlServerConnectionConfig {
|
||||
driver?: string;
|
||||
|
|
@ -158,6 +158,21 @@ function tableScopeSql(
|
|||
return { clause: `AND ${columnExpression} IN (${placeholders.join(', ')})`, params };
|
||||
}
|
||||
|
||||
/** @internal */
|
||||
export function prepareSqlServerReadOnlyQuery(
|
||||
sql: string,
|
||||
params?: Record<string, unknown>,
|
||||
): { sql: string; params?: Record<string, unknown> } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
let parameterizedQuery = sql;
|
||||
for (const key of Object.keys(params)) {
|
||||
parameterizedQuery = parameterizedQuery.replace(new RegExp(`:${key}\\b`, 'g'), `@${key}`);
|
||||
}
|
||||
return { sql: parameterizedQuery, params };
|
||||
}
|
||||
|
||||
class DefaultSqlServerPoolFactory implements KtxSqlServerPoolFactory {
|
||||
async createPool(config: KtxSqlServerPoolConfig): Promise<KtxSqlServerPool> {
|
||||
const pool = await new sql.ConnectionPool(config as sql.config).connect();
|
||||
|
|
@ -349,7 +364,7 @@ export class KtxSqlServerScanConnector implements KtxScanConnector {
|
|||
private readonly poolFactory: KtxSqlServerPoolFactory;
|
||||
private readonly endpointResolver?: KtxSqlServerEndpointResolver;
|
||||
private readonly now: () => Date;
|
||||
private readonly dialect = new KtxSqlServerDialect();
|
||||
private readonly dialect = getDialectForDriver('sqlserver');
|
||||
private pool: KtxSqlServerPool | null = null;
|
||||
private resolvedEndpoint: KtxSqlServerResolvedEndpoint | null = null;
|
||||
|
||||
|
|
@ -427,7 +442,7 @@ export class KtxSqlServerScanConnector implements KtxScanConnector {
|
|||
async executeReadOnly(input: KtxSqlServerReadOnlyQueryInput, _ctx: KtxScanContext): Promise<KtxQueryResult> {
|
||||
this.assertConnection(input.connectionId);
|
||||
const limitedSql = limitSqlForSqlServerExecution(input.sql, input.maxRows);
|
||||
const prepared = this.dialect.prepareQuery(limitedSql, input.params);
|
||||
const prepared = prepareSqlServerReadOnlyQuery(limitedSql, input.params);
|
||||
const result = await this.query(prepared.sql, prepared.params);
|
||||
return { ...result, rowCount: result.rows.length };
|
||||
}
|
||||
|
|
@ -517,6 +532,7 @@ export class KtxSqlServerScanConnector implements KtxScanConnector {
|
|||
params,
|
||||
);
|
||||
return rows.map((row) => ({
|
||||
catalog: this.poolConfig.database,
|
||||
schema: row.schema_name,
|
||||
name: row.table_name,
|
||||
kind: row.table_type === 'VIEW' ? ('view' as const) : ('table' as const),
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
import type { KtxDialect } from '../../context/connections/dialects.js';
|
||||
import {
|
||||
columnDisplayPartCount,
|
||||
formatDialectDisplayRef,
|
||||
formatDialectTableName,
|
||||
parseDialectDisplayRef,
|
||||
safeSqlLimit,
|
||||
} from '../../context/connections/dialect-helpers.js';
|
||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||
|
||||
type SqlServerTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export class KtxSqlServerDialect {
|
||||
readonly type = 'sqlserver';
|
||||
/** @internal */
|
||||
export class KtxSqlServerDialect implements KtxDialect {
|
||||
readonly type = 'sqlserver' as const;
|
||||
|
||||
private readonly typeMappings: Record<string, KtxSchemaDimensionType> = {
|
||||
datetime: 'time',
|
||||
|
|
@ -39,9 +48,19 @@ export class KtxSqlServerDialect {
|
|||
}
|
||||
|
||||
formatTableName(table: SqlServerTableNameRef): string {
|
||||
return table.db
|
||||
? `${this.quoteIdentifier(table.db)}.${this.quoteIdentifier(table.name)}`
|
||||
: this.quoteIdentifier(table.name);
|
||||
return formatDialectTableName(table, this.quoteIdentifier.bind(this), 'three-part');
|
||||
}
|
||||
|
||||
formatDisplayRef(table: SqlServerTableNameRef): string {
|
||||
return formatDialectDisplayRef(table, 'three-part');
|
||||
}
|
||||
|
||||
parseDisplayRef(display: string): KtxTableRef | null {
|
||||
return parseDialectDisplayRef(display, 'three-part');
|
||||
}
|
||||
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3 {
|
||||
return columnDisplayPartCount('three-part');
|
||||
}
|
||||
|
||||
mapDataType(nativeType: string): string {
|
||||
|
|
@ -86,17 +105,6 @@ export class KtxSqlServerDialect {
|
|||
return `SELECT TOP ${limit} ${quotedColumn} FROM ${tableName} WHERE ${quotedColumn} IS NOT NULL AND LTRIM(RTRIM(CAST(${quotedColumn} AS NVARCHAR(MAX)))) != ''`;
|
||||
}
|
||||
|
||||
prepareQuery(sql: string, params?: Record<string, unknown>): { sql: string; params?: Record<string, unknown> } {
|
||||
if (!params) {
|
||||
return { sql, params: undefined };
|
||||
}
|
||||
let parameterizedQuery = sql;
|
||||
for (const key of Object.keys(params)) {
|
||||
parameterizedQuery = parameterizedQuery.replace(new RegExp(`:${key}\\b`, 'g'), `@${key}`);
|
||||
}
|
||||
return { sql: parameterizedQuery, params };
|
||||
}
|
||||
|
||||
getRandomSampleFilter(samplePct: number): string {
|
||||
if (samplePct <= 0 || samplePct >= 1) {
|
||||
return '';
|
||||
|
|
@ -111,12 +119,12 @@ export class KtxSqlServerDialect {
|
|||
return `TABLESAMPLE (${samplePct * 100} PERCENT)`;
|
||||
}
|
||||
|
||||
getLimitOffsetClause(limit: number, offset?: number): string {
|
||||
return offset !== undefined && offset > 0 ? `OFFSET ${offset} ROWS FETCH NEXT ${limit} ROWS ONLY` : '';
|
||||
getLimitOffsetClause(_limit: number, _offset?: number): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
getTopClause(limit: number): string {
|
||||
return `TOP ${limit}`;
|
||||
return `TOP (${safeSqlLimit(limit)})`;
|
||||
}
|
||||
|
||||
getNullCountExpression(column: string): string {
|
||||
|
|
@ -127,6 +135,18 @@ export class KtxSqlServerDialect {
|
|||
return `COUNT(DISTINCT ${column})`;
|
||||
}
|
||||
|
||||
textLengthExpression(columnSql: string): string {
|
||||
return `LEN(CAST(${columnSql} AS NVARCHAR(MAX)))`;
|
||||
}
|
||||
|
||||
castToText(columnSql: string): string {
|
||||
return `CAST(${columnSql} AS NVARCHAR(MAX))`;
|
||||
}
|
||||
|
||||
getSampleValueAggregation(innerSql: string): string {
|
||||
return `(SELECT STRING_AGG(CAST(value AS NVARCHAR(MAX)), CHAR(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string {
|
||||
return `
|
||||
WITH sampled AS (
|
||||
|
|
@ -167,35 +187,4 @@ export class KtxSqlServerDialect {
|
|||
FROM sampled
|
||||
`;
|
||||
}
|
||||
|
||||
getTimeTruncExpression(
|
||||
column: string,
|
||||
granularity: 'day' | 'week' | 'month' | 'quarter' | 'year',
|
||||
timezone?: string,
|
||||
): string {
|
||||
const col = timezone ? `${column} AT TIME ZONE 'UTC' AT TIME ZONE '${timezone}'` : column;
|
||||
switch (granularity) {
|
||||
case 'day':
|
||||
return `CAST(${col} AS DATE)`;
|
||||
case 'week':
|
||||
return `DATEADD(WEEK, DATEDIFF(WEEK, 0, ${col}), 0)`;
|
||||
case 'month':
|
||||
return `DATEFROMPARTS(YEAR(${col}), MONTH(${col}), 1)`;
|
||||
case 'quarter':
|
||||
return `DATEFROMPARTS(YEAR(${col}), (DATEPART(QUARTER, ${col}) - 1) * 3 + 1, 1)`;
|
||||
case 'year':
|
||||
return `DATEFROMPARTS(YEAR(${col}), 1, 1)`;
|
||||
}
|
||||
}
|
||||
|
||||
getCustomTimeTruncExpression(column: string, interval: string, origin?: string, timezone?: string): string {
|
||||
const col = timezone ? `${column} AT TIME ZONE 'UTC' AT TIME ZONE '${timezone}'` : column;
|
||||
const [amount, unit] = interval.split(' ');
|
||||
const originExpr = origin ? `'${origin}'` : `'1970-01-01'`;
|
||||
return `DATEADD(${unit}, (DATEDIFF(${unit}, ${originExpr}, ${col}) / ${amount}) * ${amount}, ${originExpr})`;
|
||||
}
|
||||
|
||||
parseIntervalToSql(interval: string): string {
|
||||
return `'${interval}'`;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
87
packages/cli/src/context/connections/dialect-helpers.ts
Normal file
87
packages/cli/src/context/connections/dialect-helpers.ts
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
import type { KtxTableRef } from '../scan/types.js';
|
||||
|
||||
export type KtxDialectIdentifierShape = 'ansi' | 'sqlite' | 'three-part';
|
||||
|
||||
export type KtxDialectTableRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||
|
||||
export function safeSqlLimit(limit: number): number {
|
||||
return Math.max(1, Math.floor(limit));
|
||||
}
|
||||
|
||||
function safeSqlOffset(offset: number | undefined): number | null {
|
||||
if (offset === undefined) {
|
||||
return null;
|
||||
}
|
||||
const normalized = Math.floor(offset);
|
||||
return normalized > 0 ? normalized : null;
|
||||
}
|
||||
|
||||
function cleanIdentifierPart(part: string): string {
|
||||
return part.trim().replace(/^["'`\[]|["'`\]]$/g, '');
|
||||
}
|
||||
|
||||
function splitDisplay(display: string): string[] {
|
||||
return display.trim().split('.').map(cleanIdentifierPart).filter(Boolean);
|
||||
}
|
||||
|
||||
function tableParts(table: KtxDialectTableRef, shape: KtxDialectIdentifierShape): string[] {
|
||||
if (shape === 'sqlite') {
|
||||
return [table.name];
|
||||
}
|
||||
return [table.catalog ?? null, table.db ?? null, table.name].filter((part): part is string => Boolean(part));
|
||||
}
|
||||
|
||||
function acceptedDisplayPartCounts(shape: KtxDialectIdentifierShape): readonly number[] {
|
||||
if (shape === 'sqlite') {
|
||||
return [1];
|
||||
}
|
||||
if (shape === 'three-part') {
|
||||
return [3];
|
||||
}
|
||||
return [2, 3];
|
||||
}
|
||||
|
||||
export function formatDialectTableName(
|
||||
table: KtxDialectTableRef,
|
||||
quoteIdentifier: (identifier: string) => string,
|
||||
shape: KtxDialectIdentifierShape,
|
||||
): string {
|
||||
return tableParts(table, shape).map(quoteIdentifier).join('.');
|
||||
}
|
||||
|
||||
export function formatDialectDisplayRef(table: KtxDialectTableRef, shape: KtxDialectIdentifierShape): string {
|
||||
return tableParts(table, shape).join('.');
|
||||
}
|
||||
|
||||
export function parseDialectDisplayRef(display: string, shape: KtxDialectIdentifierShape): KtxTableRef | null {
|
||||
const parts = splitDisplay(display);
|
||||
if (!acceptedDisplayPartCounts(shape).includes(parts.length)) {
|
||||
return null;
|
||||
}
|
||||
if (parts.length === 1) {
|
||||
return { catalog: null, db: null, name: parts[0]! };
|
||||
}
|
||||
if (parts.length === 2) {
|
||||
return { catalog: null, db: parts[0]!, name: parts[1]! };
|
||||
}
|
||||
if (parts.length === 3) {
|
||||
return { catalog: parts[0]!, db: parts[1]!, name: parts[2]! };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function columnDisplayPartCount(shape: KtxDialectIdentifierShape): 1 | 2 | 3 {
|
||||
if (shape === 'sqlite') {
|
||||
return 1;
|
||||
}
|
||||
if (shape === 'three-part') {
|
||||
return 3;
|
||||
}
|
||||
return 2;
|
||||
}
|
||||
|
||||
export function limitOffsetClause(limit: number, offset?: number): string {
|
||||
const safeLimit = safeSqlLimit(limit);
|
||||
const safeOffset = safeSqlOffset(offset);
|
||||
return safeOffset === null ? `LIMIT ${safeLimit}` : `LIMIT ${safeLimit} OFFSET ${safeOffset}`;
|
||||
}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { getDialectForDriver } from './dialects.js';
|
||||
|
||||
describe('getDialectForDriver', () => {
|
||||
it.each([
|
||||
['postgres', '"public"."orders"'],
|
||||
['mysql', '`public`.`orders`'],
|
||||
['clickhouse', '`public`.`orders`'],
|
||||
['sqlite', '"orders"'],
|
||||
['snowflake', '"analytics"."public"."orders"'],
|
||||
['bigquery', '`analytics`.`public`.`orders`'],
|
||||
['sqlserver', '[analytics].[public].[orders]'],
|
||||
] as const)('formats table names for %s', (driver, expected) => {
|
||||
const dialect = getDialectForDriver(driver);
|
||||
expect(
|
||||
dialect.formatTableName({
|
||||
catalog: driver === 'snowflake' || driver === 'bigquery' || driver === 'sqlserver' ? 'analytics' : null,
|
||||
db: driver === 'sqlite' ? null : 'public',
|
||||
name: 'orders',
|
||||
}),
|
||||
).toBe(expected);
|
||||
});
|
||||
|
||||
it('throws with a supported-driver list for unknown drivers', () => {
|
||||
expect(() => getDialectForDriver('oracle')).toThrow(
|
||||
'Unsupported warehouse driver "oracle". Supported drivers: bigquery, clickhouse, mysql, postgres, sqlite, snowflake, sqlserver',
|
||||
);
|
||||
});
|
||||
|
||||
it('rejects legacy driver aliases', () => {
|
||||
expect(() => getDialectForDriver('postgresql')).toThrow('Unsupported warehouse driver "postgresql"');
|
||||
expect(() => getDialectForDriver('sqlite3')).toThrow('Unsupported warehouse driver "sqlite3"');
|
||||
});
|
||||
});
|
||||
|
|
@ -1,22 +1,40 @@
|
|||
import type { KtxSchemaDimensionType, KtxTableRef } from '../scan/types.js';
|
||||
|
||||
type SupportedDriver =
|
||||
| 'postgres'
|
||||
| 'mysql'
|
||||
| 'sqlserver'
|
||||
| 'snowflake'
|
||||
| 'bigquery'
|
||||
| 'clickhouse'
|
||||
| 'sqlite';
|
||||
import { KtxBigQueryDialect } from '../../connectors/bigquery/dialect.js';
|
||||
import { KtxClickHouseDialect } from '../../connectors/clickhouse/dialect.js';
|
||||
import { KtxMysqlDialect } from '../../connectors/mysql/dialect.js';
|
||||
import { KtxPostgresDialect } from '../../connectors/postgres/dialect.js';
|
||||
import { KtxSqliteDialect } from '../../connectors/sqlite/dialect.js';
|
||||
import { KtxSnowflakeDialect } from '../../connectors/snowflake/dialect.js';
|
||||
import { KtxSqlServerDialect } from '../../connectors/sqlserver/dialect.js';
|
||||
import type { KtxConnectionDriver, KtxSchemaDimensionType, KtxTableRef } from '../scan/types.js';
|
||||
import type { KtxDialectTableRef } from './dialect-helpers.js';
|
||||
|
||||
export interface KtxDialect {
|
||||
readonly type: SupportedDriver;
|
||||
readonly type: KtxConnectionDriver;
|
||||
quoteIdentifier(identifier: string): string;
|
||||
formatTableName(table: KtxTableRef): string;
|
||||
formatTableName(table: KtxDialectTableRef): string;
|
||||
formatDisplayRef(table: KtxDialectTableRef): string;
|
||||
parseDisplayRef(display: string): KtxTableRef | null;
|
||||
columnDisplayTablePartCount(): 1 | 2 | 3;
|
||||
getLimitOffsetClause(limit: number, offset?: number): string;
|
||||
getTopClause(limit: number): string;
|
||||
getRandomSampleFilter(samplePct: number): string;
|
||||
getTableSampleClause(samplePct: number): string;
|
||||
generateSampleQuery(tableName: string, limit: number, columns?: string[]): string;
|
||||
generateColumnSampleQuery(tableName: string, columnName: string, limit: number): string;
|
||||
getSampleValueAggregation(innerSql: string): string;
|
||||
generateCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string;
|
||||
generateRandomizedCardinalitySampleQuery(tableName: string, columnName: string, sampleSize: number): string;
|
||||
generateDistinctValuesQuery(tableName: string, columnName: string, limit: number): string;
|
||||
generateColumnStatisticsQuery(schemaName: string, tableName: string): string | null;
|
||||
getNullCountExpression(column: string): string;
|
||||
getDistinctCountExpression(column: string): string;
|
||||
textLengthExpression(columnSql: string): string;
|
||||
castToText(columnSql: string): string;
|
||||
mapToDimensionType(nativeType: string): KtxSchemaDimensionType;
|
||||
mapDataType(nativeType: string): string;
|
||||
}
|
||||
|
||||
const supportedDrivers: SupportedDriver[] = [
|
||||
const supportedDrivers: KtxConnectionDriver[] = [
|
||||
'bigquery',
|
||||
'clickhouse',
|
||||
'mysql',
|
||||
|
|
@ -26,71 +44,21 @@ const supportedDrivers: SupportedDriver[] = [
|
|||
'sqlserver',
|
||||
];
|
||||
|
||||
function doubleQuoted(identifier: string): string {
|
||||
return `"${identifier.replace(/"/g, '""')}"`;
|
||||
}
|
||||
|
||||
function backtickQuoted(identifier: string): string {
|
||||
return `\`${identifier.replace(/`/g, '``')}\``;
|
||||
}
|
||||
|
||||
function bigQueryQuoted(identifier: string): string {
|
||||
return `\`${identifier.replace(/`/g, '\\`')}\``;
|
||||
}
|
||||
|
||||
function bracketQuoted(identifier: string): string {
|
||||
return `[${identifier.replace(/\]/g, ']]')}]`;
|
||||
}
|
||||
|
||||
function inferDimensionType(nativeType: string): KtxSchemaDimensionType {
|
||||
const normalized = nativeType.toLowerCase().trim();
|
||||
if (normalized.includes('date') || normalized.includes('time')) {
|
||||
return 'time';
|
||||
}
|
||||
if (
|
||||
normalized.includes('int') ||
|
||||
normalized.includes('num') ||
|
||||
normalized.includes('dec') ||
|
||||
normalized.includes('float') ||
|
||||
normalized.includes('double') ||
|
||||
normalized.includes('real')
|
||||
) {
|
||||
return 'number';
|
||||
}
|
||||
if (normalized.includes('bool') || normalized === 'bit') {
|
||||
return 'boolean';
|
||||
}
|
||||
return 'string';
|
||||
}
|
||||
|
||||
function formatWithParts(table: KtxTableRef, quote: (identifier: string) => string, sqlite = false): string {
|
||||
const parts = sqlite ? [table.name] : [table.catalog, table.db, table.name].filter((part): part is string => !!part);
|
||||
return parts.map(quote).join('.');
|
||||
}
|
||||
|
||||
function createDialect(type: SupportedDriver, quote: (identifier: string) => string, sqlite = false): KtxDialect {
|
||||
return {
|
||||
type,
|
||||
quoteIdentifier: quote,
|
||||
formatTableName: (table) => formatWithParts(table, quote, sqlite),
|
||||
mapToDimensionType: inferDimensionType,
|
||||
};
|
||||
}
|
||||
|
||||
const dialects: Record<SupportedDriver, KtxDialect> = {
|
||||
postgres: createDialect('postgres', doubleQuoted),
|
||||
mysql: createDialect('mysql', backtickQuoted),
|
||||
clickhouse: createDialect('clickhouse', backtickQuoted),
|
||||
sqlite: createDialect('sqlite', doubleQuoted, true),
|
||||
snowflake: createDialect('snowflake', doubleQuoted),
|
||||
bigquery: createDialect('bigquery', bigQueryQuoted),
|
||||
sqlserver: createDialect('sqlserver', bracketQuoted),
|
||||
const dialectFactories: Record<KtxConnectionDriver, () => KtxDialect> = {
|
||||
bigquery: () => new KtxBigQueryDialect(),
|
||||
clickhouse: () => new KtxClickHouseDialect(),
|
||||
mysql: () => new KtxMysqlDialect(),
|
||||
postgres: () => new KtxPostgresDialect(),
|
||||
sqlite: () => new KtxSqliteDialect(),
|
||||
snowflake: () => new KtxSnowflakeDialect(),
|
||||
sqlserver: () => new KtxSqlServerDialect(),
|
||||
};
|
||||
|
||||
export function getDialectForDriver(driver: string): KtxDialect {
|
||||
const normalized = driver.toLowerCase().trim();
|
||||
if (normalized in dialects) {
|
||||
return dialects[normalized as SupportedDriver];
|
||||
const factory = dialectFactories[normalized as KtxConnectionDriver];
|
||||
if (factory) {
|
||||
return factory();
|
||||
}
|
||||
throw new Error(`Unsupported warehouse driver "${driver}". Supported drivers: ${supportedDrivers.join(', ')}`);
|
||||
}
|
||||
|
|
|
|||
199
packages/cli/src/context/connections/drivers.ts
Normal file
199
packages/cli/src/context/connections/drivers.ts
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
import type { KtxConnectionDriver, KtxScanConnector } from '../scan/types.js';
|
||||
|
||||
/** @internal */
|
||||
export type KtxScopeConfigKey = 'dataset_ids' | 'databases' | 'schemas' | 'schema_names';
|
||||
|
||||
/** @internal */
|
||||
export interface KtxDriverConnectorModule {
|
||||
isConnectionConfig(connection: unknown): boolean;
|
||||
createScanConnector(args: {
|
||||
connectionId: string;
|
||||
connection: unknown;
|
||||
projectDir: string;
|
||||
}): KtxScanConnector;
|
||||
}
|
||||
|
||||
export interface KtxDriverRegistration {
|
||||
readonly driver: KtxConnectionDriver;
|
||||
readonly scopeConfigKey: KtxScopeConfigKey | null;
|
||||
readonly hasHistoricSqlReader: boolean;
|
||||
readonly hasLocalQueryExecutor: boolean;
|
||||
load(): Promise<KtxDriverConnectorModule>;
|
||||
}
|
||||
|
||||
function invalidConnectionConfig(driver: KtxConnectionDriver): Error {
|
||||
return new Error(`Connection config does not match warehouse driver "${driver}".`);
|
||||
}
|
||||
|
||||
/** @internal */
|
||||
export const driverRegistrations: Record<KtxConnectionDriver, KtxDriverRegistration> = {
|
||||
bigquery: {
|
||||
driver: 'bigquery',
|
||||
scopeConfigKey: 'dataset_ids',
|
||||
hasHistoricSqlReader: true,
|
||||
hasLocalQueryExecutor: false,
|
||||
load: async () => {
|
||||
const m = await import('../../connectors/bigquery/connector.js');
|
||||
return {
|
||||
isConnectionConfig: (connection) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxBigQueryConnectionConfig>[0];
|
||||
return m.isKtxBigQueryConnectionConfig(typedConnection);
|
||||
},
|
||||
createScanConnector: ({ connectionId, connection }) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxBigQueryConnectionConfig>[0];
|
||||
if (!m.isKtxBigQueryConnectionConfig(typedConnection)) {
|
||||
throw invalidConnectionConfig('bigquery');
|
||||
}
|
||||
return new m.KtxBigQueryScanConnector({ connectionId, connection: typedConnection });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
clickhouse: {
|
||||
driver: 'clickhouse',
|
||||
scopeConfigKey: 'databases',
|
||||
hasHistoricSqlReader: false,
|
||||
hasLocalQueryExecutor: false,
|
||||
load: async () => {
|
||||
const m = await import('../../connectors/clickhouse/connector.js');
|
||||
return {
|
||||
isConnectionConfig: (connection) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxClickHouseConnectionConfig>[0];
|
||||
return m.isKtxClickHouseConnectionConfig(typedConnection);
|
||||
},
|
||||
createScanConnector: ({ connectionId, connection }) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxClickHouseConnectionConfig>[0];
|
||||
if (!m.isKtxClickHouseConnectionConfig(typedConnection)) {
|
||||
throw invalidConnectionConfig('clickhouse');
|
||||
}
|
||||
return new m.KtxClickHouseScanConnector({ connectionId, connection: typedConnection });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
mysql: {
|
||||
driver: 'mysql',
|
||||
scopeConfigKey: 'schemas',
|
||||
hasHistoricSqlReader: false,
|
||||
hasLocalQueryExecutor: false,
|
||||
load: async () => {
|
||||
const m = await import('../../connectors/mysql/connector.js');
|
||||
return {
|
||||
isConnectionConfig: (connection) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxMysqlConnectionConfig>[0];
|
||||
return m.isKtxMysqlConnectionConfig(typedConnection);
|
||||
},
|
||||
createScanConnector: ({ connectionId, connection }) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxMysqlConnectionConfig>[0];
|
||||
if (!m.isKtxMysqlConnectionConfig(typedConnection)) {
|
||||
throw invalidConnectionConfig('mysql');
|
||||
}
|
||||
return new m.KtxMysqlScanConnector({ connectionId, connection: typedConnection });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
postgres: {
|
||||
driver: 'postgres',
|
||||
scopeConfigKey: 'schemas',
|
||||
hasHistoricSqlReader: true,
|
||||
hasLocalQueryExecutor: true,
|
||||
load: async () => {
|
||||
const m = await import('../../connectors/postgres/connector.js');
|
||||
return {
|
||||
isConnectionConfig: (connection) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxPostgresConnectionConfig>[0];
|
||||
return m.isKtxPostgresConnectionConfig(typedConnection);
|
||||
},
|
||||
createScanConnector: ({ connectionId, connection }) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxPostgresConnectionConfig>[0];
|
||||
if (!m.isKtxPostgresConnectionConfig(typedConnection)) {
|
||||
throw invalidConnectionConfig('postgres');
|
||||
}
|
||||
return new m.KtxPostgresScanConnector({ connectionId, connection: typedConnection });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
sqlite: {
|
||||
driver: 'sqlite',
|
||||
scopeConfigKey: null,
|
||||
hasHistoricSqlReader: false,
|
||||
hasLocalQueryExecutor: true,
|
||||
load: async () => {
|
||||
const m = await import('../../connectors/sqlite/connector.js');
|
||||
return {
|
||||
isConnectionConfig: (connection) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxSqliteConnectionConfig>[0];
|
||||
return m.isKtxSqliteConnectionConfig(typedConnection);
|
||||
},
|
||||
createScanConnector: ({ connectionId, connection, projectDir }) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxSqliteConnectionConfig>[0];
|
||||
if (!m.isKtxSqliteConnectionConfig(typedConnection)) {
|
||||
throw invalidConnectionConfig('sqlite');
|
||||
}
|
||||
return new m.KtxSqliteScanConnector({ connectionId, connection: typedConnection, projectDir });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
snowflake: {
|
||||
driver: 'snowflake',
|
||||
scopeConfigKey: 'schema_names',
|
||||
hasHistoricSqlReader: true,
|
||||
hasLocalQueryExecutor: false,
|
||||
load: async () => {
|
||||
const m = await import('../../connectors/snowflake/connector.js');
|
||||
return {
|
||||
isConnectionConfig: (connection) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxSnowflakeConnectionConfig>[0];
|
||||
return m.isKtxSnowflakeConnectionConfig(typedConnection);
|
||||
},
|
||||
createScanConnector: ({ connectionId, connection, projectDir }) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxSnowflakeConnectionConfig>[0];
|
||||
if (!m.isKtxSnowflakeConnectionConfig(typedConnection)) {
|
||||
throw invalidConnectionConfig('snowflake');
|
||||
}
|
||||
return new m.KtxSnowflakeScanConnector({ connectionId, connection: typedConnection, projectDir });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
sqlserver: {
|
||||
driver: 'sqlserver',
|
||||
scopeConfigKey: 'schemas',
|
||||
hasHistoricSqlReader: false,
|
||||
hasLocalQueryExecutor: false,
|
||||
load: async () => {
|
||||
const m = await import('../../connectors/sqlserver/connector.js');
|
||||
return {
|
||||
isConnectionConfig: (connection) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxSqlServerConnectionConfig>[0];
|
||||
return m.isKtxSqlServerConnectionConfig(typedConnection);
|
||||
},
|
||||
createScanConnector: ({ connectionId, connection }) => {
|
||||
const typedConnection = connection as Parameters<typeof m.isKtxSqlServerConnectionConfig>[0];
|
||||
if (!m.isKtxSqlServerConnectionConfig(typedConnection)) {
|
||||
throw invalidConnectionConfig('sqlserver');
|
||||
}
|
||||
return new m.KtxSqlServerScanConnector({ connectionId, connection: typedConnection });
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const supportedDrivers = Object.keys(driverRegistrations).sort() as KtxConnectionDriver[];
|
||||
|
||||
function isRegisteredDriver(driver: string): driver is KtxConnectionDriver {
|
||||
return Object.prototype.hasOwnProperty.call(driverRegistrations, driver);
|
||||
}
|
||||
|
||||
export function getDriverRegistration(driver: string): KtxDriverRegistration | undefined {
|
||||
const normalized = driver.toLowerCase().trim();
|
||||
return isRegisteredDriver(normalized) ? driverRegistrations[normalized] : undefined;
|
||||
}
|
||||
|
||||
export function listSupportedDrivers(): KtxConnectionDriver[] {
|
||||
return [...supportedDrivers];
|
||||
}
|
||||
|
|
@ -1,3 +1,4 @@
|
|||
import { driverRegistrations, getDriverRegistration } from './drivers.js';
|
||||
import { createPostgresQueryExecutor } from './postgres-query-executor.js';
|
||||
import type {
|
||||
KtxSqlQueryExecutionInput,
|
||||
|
|
@ -5,6 +6,7 @@ import type {
|
|||
KtxSqlQueryExecutorPort,
|
||||
} from './query-executor.js';
|
||||
import { createSqliteQueryExecutor } from './sqlite-query-executor.js';
|
||||
import type { KtxConnectionDriver } from '../scan/types.js';
|
||||
|
||||
export interface DefaultLocalQueryExecutorOptions {
|
||||
postgres?: KtxSqlQueryExecutorPort;
|
||||
|
|
@ -15,20 +17,43 @@ function driverFor(input: KtxSqlQueryExecutionInput): string {
|
|||
return String(input.connection?.driver ?? '').toLowerCase();
|
||||
}
|
||||
|
||||
function localExecutorMap(
|
||||
options: DefaultLocalQueryExecutorOptions,
|
||||
): Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> {
|
||||
const wiredExecutors: Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> = {
|
||||
postgres: options.postgres ?? createPostgresQueryExecutor(),
|
||||
sqlite: options.sqlite ?? createSqliteQueryExecutor(),
|
||||
};
|
||||
|
||||
const executors: Partial<Record<KtxConnectionDriver, KtxSqlQueryExecutorPort>> = {};
|
||||
for (const registration of Object.values(driverRegistrations)) {
|
||||
if (!registration.hasLocalQueryExecutor) continue;
|
||||
const executor = wiredExecutors[registration.driver];
|
||||
if (executor) {
|
||||
executors[registration.driver] = executor;
|
||||
}
|
||||
}
|
||||
return executors;
|
||||
}
|
||||
|
||||
export function createDefaultLocalQueryExecutor(options: DefaultLocalQueryExecutorOptions = {}): KtxSqlQueryExecutorPort {
|
||||
const postgres = options.postgres ?? createPostgresQueryExecutor();
|
||||
const sqlite = options.sqlite ?? createSqliteQueryExecutor();
|
||||
const executors = localExecutorMap(options);
|
||||
|
||||
return {
|
||||
async execute(input: KtxSqlQueryExecutionInput): Promise<KtxSqlQueryExecutionResult> {
|
||||
const driver = driverFor(input);
|
||||
if (driver === 'postgres') {
|
||||
return postgres.execute(input);
|
||||
const registration = getDriverRegistration(driver);
|
||||
if (!registration?.hasLocalQueryExecutor) {
|
||||
throw new Error(`No local query executor is configured for driver "${input.connection?.driver ?? 'unknown'}".`);
|
||||
}
|
||||
if (driver === 'sqlite') {
|
||||
return sqlite.execute(input);
|
||||
|
||||
const executor = executors[registration.driver];
|
||||
if (!executor) {
|
||||
throw new Error(
|
||||
`Local query executor flag is enabled for driver "${registration.driver}", but no executor factory is wired.`,
|
||||
);
|
||||
}
|
||||
throw new Error(`No local query executor is configured for driver "${input.connection?.driver ?? 'unknown'}".`);
|
||||
return executor.execute(input);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,9 @@
|
|||
import { getDriverRegistration } from '../../../connections/drivers.js';
|
||||
import type { KtxConnectionDriver } from '../../../scan/types.js';
|
||||
import type { HistoricSqlDialect } from './types.js';
|
||||
|
||||
const historicSqlDialects: readonly HistoricSqlDialect[] = ['postgres', 'bigquery', 'snowflake'];
|
||||
|
||||
function recordOrNull(value: unknown): Record<string, unknown> | null {
|
||||
return value && typeof value === 'object' && !Array.isArray(value) ? (value as Record<string, unknown>) : null;
|
||||
}
|
||||
|
|
@ -10,6 +14,14 @@ function queryHistoryRecord(connection: unknown): Record<string, unknown> | null
|
|||
return context ? recordOrNull(context.queryHistory) : null;
|
||||
}
|
||||
|
||||
function historicSqlDialectForDriver(driver: KtxConnectionDriver): HistoricSqlDialect {
|
||||
const dialect = historicSqlDialects.find((candidate) => candidate === driver);
|
||||
if (!dialect) {
|
||||
throw new Error(`Driver "${driver}" is marked as historic-SQL capable but has no HistoricSqlDialect mapping.`);
|
||||
}
|
||||
return dialect;
|
||||
}
|
||||
|
||||
export function isQueryHistoryEnabled(connection: unknown): boolean {
|
||||
return queryHistoryRecord(connection)?.enabled === true;
|
||||
}
|
||||
|
|
@ -25,8 +37,6 @@ export function queryHistoryDialectForConnection(connection: unknown): HistoricS
|
|||
}
|
||||
const conn = recordOrNull(connection);
|
||||
const driver = String(conn?.driver ?? '').toLowerCase();
|
||||
if (driver === 'postgres') return 'postgres';
|
||||
if (driver === 'bigquery') return 'bigquery';
|
||||
if (driver === 'snowflake') return 'snowflake';
|
||||
return null;
|
||||
const registration = getDriverRegistration(driver);
|
||||
return registration?.hasHistoricSqlReader ? historicSqlDialectForDriver(registration.driver) : null;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -27,12 +27,13 @@ export function resolveEnabledTables(
|
|||
|
||||
function parseEnabledTableEntry(value: unknown): KtxTableRef | null {
|
||||
if (typeof value === 'string') {
|
||||
return parseDottedEntry(value);
|
||||
return parseDottedTableEntry(value);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function parseDottedEntry(value: string): KtxTableRef | null {
|
||||
/** @internal */
|
||||
export function parseDottedTableEntry(value: string): KtxTableRef | null {
|
||||
const trimmed = value.trim();
|
||||
if (trimmed.length === 0) return null;
|
||||
const parts = trimmed.split('.');
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import type { KtxLocalProject } from '../../context/project/project.js';
|
||||
import { getDialectForDriver, type KtxDialect } from '../connections/dialects.js';
|
||||
import { readLocalScanStructuralSnapshot } from './local-structural-artifacts.js';
|
||||
import type {
|
||||
KtxConnectionDriver,
|
||||
KtxScanReport,
|
||||
KtxSchemaColumn,
|
||||
KtxSchemaSnapshot,
|
||||
|
|
@ -88,59 +88,23 @@ function refsEqual(left: KtxTableRef, right: KtxTableRef): boolean {
|
|||
);
|
||||
}
|
||||
|
||||
function cleanIdentifierPart(part: string): string {
|
||||
return part.trim().replace(/^["'`\[]|["'`\]]$/g, '');
|
||||
}
|
||||
|
||||
function splitDisplay(display: string): string[] {
|
||||
return display
|
||||
.trim()
|
||||
.split('.')
|
||||
.map(cleanIdentifierPart)
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
function displayForTable(driver: KtxConnectionDriver, table: KtxTableRef): string {
|
||||
if (driver === 'sqlite') {
|
||||
return table.name;
|
||||
}
|
||||
return [table.catalog, table.db, table.name].filter((part): part is string => Boolean(part)).join('.');
|
||||
}
|
||||
|
||||
function tableRef(table: KtxSchemaTable): KtxTableRef {
|
||||
return { catalog: table.catalog, db: table.db, name: table.name };
|
||||
}
|
||||
|
||||
function candidateList(
|
||||
driver: KtxConnectionDriver,
|
||||
dialect: KtxDialect,
|
||||
tables: KtxSchemaTable[],
|
||||
): Array<{ tableRef: KtxTableRef; display: string }> {
|
||||
return tables
|
||||
.map((table) => ({
|
||||
tableRef: tableRef(table),
|
||||
display: displayForTable(driver, table),
|
||||
display: dialect.formatDisplayRef(table),
|
||||
}))
|
||||
.sort((left, right) => left.display.localeCompare(right.display));
|
||||
}
|
||||
|
||||
function parseDisplayRef(driver: KtxConnectionDriver, display: string): KtxTableRef | null {
|
||||
const parts = splitDisplay(display);
|
||||
if (driver === 'sqlite') {
|
||||
return parts.length === 1 ? { catalog: null, db: null, name: parts[0]! } : null;
|
||||
}
|
||||
if (driver === 'bigquery' || driver === 'snowflake' || driver === 'sqlserver') {
|
||||
return parts.length === 3 ? { catalog: parts[0]!, db: parts[1]!, name: parts[2]! } : null;
|
||||
}
|
||||
if (parts.length === 2) {
|
||||
return { catalog: null, db: parts[0]!, name: parts[1]! };
|
||||
}
|
||||
if (parts.length === 3) {
|
||||
return { catalog: parts[0]!, db: parts[1]!, name: parts[2]! };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function resolveTable(snapshot: KtxSchemaSnapshot, input: KtxEntityDetailsTableInput): ResolveResult {
|
||||
function resolveTable(snapshot: KtxSchemaSnapshot, input: KtxEntityDetailsTableInput, dialect: KtxDialect): ResolveResult {
|
||||
if (typeof input !== 'string') {
|
||||
const table = snapshot.tables.find((candidate) => refsEqual(candidate, input)) ?? null;
|
||||
return table
|
||||
|
|
@ -149,13 +113,13 @@ function resolveTable(snapshot: KtxSchemaSnapshot, input: KtxEntityDetailsTableI
|
|||
table: null,
|
||||
error: {
|
||||
code: 'table_not_found',
|
||||
message: `Table not found in latest scan: ${displayForTable(snapshot.driver, input)}`,
|
||||
candidates: candidateList(snapshot.driver, snapshot.tables),
|
||||
message: `Table not found in latest scan: ${dialect.formatDisplayRef(input)}`,
|
||||
candidates: candidateList(dialect, snapshot.tables),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const parsed = parseDisplayRef(snapshot.driver, input);
|
||||
const parsed = dialect.parseDisplayRef(input);
|
||||
if (parsed) {
|
||||
const table = snapshot.tables.find((candidate) => refsEqual(candidate, parsed)) ?? null;
|
||||
return table
|
||||
|
|
@ -165,7 +129,7 @@ function resolveTable(snapshot: KtxSchemaSnapshot, input: KtxEntityDetailsTableI
|
|||
error: {
|
||||
code: 'table_not_found',
|
||||
message: `Table not found in latest scan: ${input}`,
|
||||
candidates: candidateList(snapshot.driver, snapshot.tables),
|
||||
candidates: candidateList(dialect, snapshot.tables),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -180,7 +144,7 @@ function resolveTable(snapshot: KtxSchemaSnapshot, input: KtxEntityDetailsTableI
|
|||
error: {
|
||||
code: 'ambiguous_table',
|
||||
message: `Table name "${input}" is ambiguous across schemas/catalogs; pass a structured table ref.`,
|
||||
candidates: candidateList(snapshot.driver, byName),
|
||||
candidates: candidateList(dialect, byName),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -189,7 +153,7 @@ function resolveTable(snapshot: KtxSchemaSnapshot, input: KtxEntityDetailsTableI
|
|||
error: {
|
||||
code: 'table_not_found',
|
||||
message: `Table not found in latest scan: ${input}`,
|
||||
candidates: candidateList(snapshot.driver, snapshot.tables),
|
||||
candidates: candidateList(dialect, snapshot.tables),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -261,9 +225,10 @@ export function createKtxEntityDetailsService(project: KtxLocalProject) {
|
|||
}
|
||||
|
||||
const info = snapshotInfo(scan.report, scan.snapshot);
|
||||
const dialect = getDialectForDriver(scan.snapshot.driver);
|
||||
const results: KtxEntityDetailsResponse['results'] = [];
|
||||
for (const entity of input.entities) {
|
||||
const resolved = resolveTable(scan.snapshot, entity.table);
|
||||
const resolved = resolveTable(scan.snapshot, entity.table, dialect);
|
||||
if (!resolved.table) {
|
||||
results.push({
|
||||
ok: false,
|
||||
|
|
@ -289,7 +254,7 @@ export function createKtxEntityDetailsService(project: KtxLocalProject) {
|
|||
snapshot: info,
|
||||
error: {
|
||||
code: 'column_not_found',
|
||||
message: `Column(s) not found on ${displayForTable(scan.snapshot.driver, resolved.table)}: ${missing.join(', ')}`,
|
||||
message: `Column(s) not found on ${dialect.formatDisplayRef(resolved.table)}: ${missing.join(', ')}`,
|
||||
candidates: resolved.table.columns.map((column) => column.name),
|
||||
},
|
||||
});
|
||||
|
|
@ -300,7 +265,7 @@ export function createKtxEntityDetailsService(project: KtxLocalProject) {
|
|||
ok: true,
|
||||
connectionId: input.connectionId,
|
||||
tableRef: tableRef(resolved.table),
|
||||
display: displayForTable(scan.snapshot.driver, resolved.table),
|
||||
display: dialect.formatDisplayRef(resolved.table),
|
||||
kind: resolved.table.kind,
|
||||
comment: resolved.table.comment,
|
||||
estimatedRows: resolved.table.estimatedRows,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import pLimit from 'p-limit';
|
||||
import type { KtxLlmRuntimePort } from '../../context/llm/runtime-port.js';
|
||||
import { getDialectForDriver } from '../connections/dialects.js';
|
||||
import { buildDefaultKtxProjectConfig, type KtxScanRelationshipConfig } from '../project/config.js';
|
||||
import { KtxDescriptionGenerator } from './description-generation.js';
|
||||
import { buildKtxColumnEmbeddingText } from './embedding-text.js';
|
||||
|
|
@ -118,6 +119,18 @@ function targetMatchesForeignKey(table: KtxEnrichedTable, foreignKey: KtxSchemaF
|
|||
);
|
||||
}
|
||||
|
||||
function assertConnectorDriverMatchesSnapshot(input: {
|
||||
connector: KtxScanConnector;
|
||||
snapshot: KtxSchemaSnapshot;
|
||||
connectionId: string;
|
||||
}): void {
|
||||
if (input.connector.driver !== input.snapshot.driver) {
|
||||
throw new Error(
|
||||
`ktx scan connector driver "${input.connector.driver}" does not match snapshot driver "${input.snapshot.driver}" for connection "${input.connectionId}"`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function formalRelationshipsFromSnapshot(
|
||||
snapshot: KtxSchemaSnapshot,
|
||||
tables: readonly KtxEnrichedTable[],
|
||||
|
|
@ -468,6 +481,12 @@ export async function runLocalScanEnrichment(
|
|||
));
|
||||
await progress?.update(0.05, `Loaded schema snapshot with ${snapshot.tables.length} tables`);
|
||||
|
||||
assertConnectorDriverMatchesSnapshot({
|
||||
connector: input.connector,
|
||||
snapshot,
|
||||
connectionId: input.connectionId,
|
||||
});
|
||||
const dialect = getDialectForDriver(snapshot.driver);
|
||||
const now = input.now ?? (() => new Date());
|
||||
const state = completedKtxScanEnrichmentStateSummary();
|
||||
const syncId = input.syncId ?? input.context.runId;
|
||||
|
|
@ -575,7 +594,7 @@ export async function runLocalScanEnrichment(
|
|||
await relationshipProgress?.update(0, 'Detecting relationships');
|
||||
const detection = await discoverKtxRelationships({
|
||||
connectionId: input.connectionId,
|
||||
driver: snapshot.driver,
|
||||
dialect,
|
||||
connector: input.connector,
|
||||
schema,
|
||||
context: input.context,
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import { gunzipSync } from 'node:zlib';
|
|||
import Database from 'better-sqlite3';
|
||||
import YAML from 'yaml';
|
||||
import { z } from 'zod';
|
||||
import { getDialectForDriver } from '../connections/dialects.js';
|
||||
import type { KtxLlmRuntimePort } from '../llm/runtime-port.js';
|
||||
import type { KtxEnrichedRelationship, KtxEnrichedSchema, KtxRelationshipType } from './enrichment-types.js';
|
||||
import { snapshotToKtxEnrichedSchema } from './local-enrichment.js';
|
||||
|
|
@ -536,6 +537,7 @@ export function ktxRelationshipBenchmarkDetectorWithLlm(
|
|||
const formalLinks = formalMetadata.accepted.map((relationship) => relationshipToBenchmarkLink(relationship));
|
||||
const acceptedKeys = new Set(formalLinks.map(fkKey));
|
||||
const sqliteDataAvailable = Boolean(input.dataPath && input.snapshot.driver === 'sqlite');
|
||||
const dialect = getDialectForDriver(input.snapshot.driver);
|
||||
const profilingExecutor =
|
||||
sqliteDataAvailable && input.mode !== 'profiling_disabled'
|
||||
? new KtxRelationshipBenchmarkSqliteExecutor(input.dataPath as string)
|
||||
|
|
@ -550,7 +552,7 @@ export function ktxRelationshipBenchmarkDetectorWithLlm(
|
|||
})
|
||||
: await profileKtxRelationshipSchema({
|
||||
connectionId: input.snapshot.connectionId,
|
||||
driver: input.snapshot.driver,
|
||||
dialect,
|
||||
schema: input.schema,
|
||||
executor: profilingExecutor,
|
||||
ctx: { runId: `relationship-benchmark:${input.fixtureId}:${input.mode}:profile` },
|
||||
|
|
@ -580,7 +582,7 @@ export function ktxRelationshipBenchmarkDetectorWithLlm(
|
|||
: Math.max(0, input.validationBudget - profiles.queryCount);
|
||||
const validatedBroadCandidates = await validateKtxRelationshipDiscoveryCandidates({
|
||||
connectionId: input.snapshot.connectionId,
|
||||
driver: input.snapshot.driver,
|
||||
dialect,
|
||||
candidates,
|
||||
profiles,
|
||||
executor: validationExecutor,
|
||||
|
|
@ -597,7 +599,7 @@ export function ktxRelationshipBenchmarkDetectorWithLlm(
|
|||
input.mode !== 'validation_disabled'
|
||||
? await discoverKtxCompositeRelationships({
|
||||
connectionId: input.snapshot.connectionId,
|
||||
driver: input.snapshot.driver,
|
||||
dialect,
|
||||
schema: input.schema,
|
||||
profiles,
|
||||
executor: validationExecutor,
|
||||
|
|
@ -671,6 +673,7 @@ export function currentKtxRelationshipBenchmarkDetector(): KtxRelationshipBenchm
|
|||
const formalLinks = formalMetadata.accepted.map((relationship) => relationshipToBenchmarkLink(relationship));
|
||||
const acceptedKeys = new Set(formalLinks.map(fkKey));
|
||||
const sqliteDataAvailable = Boolean(input.dataPath && input.snapshot.driver === 'sqlite');
|
||||
const dialect = getDialectForDriver(input.snapshot.driver);
|
||||
const profilingExecutor =
|
||||
sqliteDataAvailable && input.mode !== 'profiling_disabled'
|
||||
? new KtxRelationshipBenchmarkSqliteExecutor(input.dataPath as string)
|
||||
|
|
@ -685,7 +688,7 @@ export function currentKtxRelationshipBenchmarkDetector(): KtxRelationshipBenchm
|
|||
})
|
||||
: await profileKtxRelationshipSchema({
|
||||
connectionId: input.snapshot.connectionId,
|
||||
driver: input.snapshot.driver,
|
||||
dialect,
|
||||
schema: input.schema,
|
||||
executor: profilingExecutor,
|
||||
ctx: { runId: `relationship-benchmark:${input.fixtureId}:${input.mode}:profile` },
|
||||
|
|
@ -702,7 +705,7 @@ export function currentKtxRelationshipBenchmarkDetector(): KtxRelationshipBenchm
|
|||
: Math.max(0, input.validationBudget - profiles.queryCount);
|
||||
const validatedBroadCandidates = await validateKtxRelationshipDiscoveryCandidates({
|
||||
connectionId: input.snapshot.connectionId,
|
||||
driver: input.snapshot.driver,
|
||||
dialect,
|
||||
candidates: broadRelationshipCandidates,
|
||||
profiles,
|
||||
executor: validationExecutor,
|
||||
|
|
@ -719,7 +722,7 @@ export function currentKtxRelationshipBenchmarkDetector(): KtxRelationshipBenchm
|
|||
input.mode !== 'validation_disabled'
|
||||
? await discoverKtxCompositeRelationships({
|
||||
connectionId: input.snapshot.connectionId,
|
||||
driver: input.snapshot.driver,
|
||||
dialect,
|
||||
schema: input.schema,
|
||||
profiles,
|
||||
executor: validationExecutor,
|
||||
|
|
|
|||
|
|
@ -1,11 +1,10 @@
|
|||
import type { KtxDialect } from '../connections/dialects.js';
|
||||
import type { KtxEnrichedColumn, KtxEnrichedSchema, KtxEnrichedTable, KtxRelationshipType } from './enrichment-types.js';
|
||||
import {
|
||||
formatKtxRelationshipTableRef,
|
||||
quoteKtxRelationshipIdentifier,
|
||||
type KtxRelationshipProfileArtifact,
|
||||
type KtxRelationshipReadOnlyExecutor,
|
||||
} from './relationship-profiling.js';
|
||||
import type { KtxConnectionDriver, KtxQueryResult, KtxScanContext, KtxTableRef } from './types.js';
|
||||
import type { KtxQueryResult, KtxScanContext, KtxTableRef } from './types.js';
|
||||
|
||||
type KtxCompositeRelationshipStatus = 'accepted' | 'review' | 'rejected';
|
||||
|
||||
|
|
@ -57,7 +56,7 @@ export interface KtxCompositeRelationshipCandidate {
|
|||
|
||||
export interface DiscoverKtxCompositeRelationshipsInput {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
schema: KtxEnrichedSchema;
|
||||
profiles: KtxRelationshipProfileArtifact;
|
||||
executor: KtxRelationshipReadOnlyExecutor | null;
|
||||
|
|
@ -224,28 +223,16 @@ function numberAt(result: KtxQueryResult, header: string): number {
|
|||
return 0;
|
||||
}
|
||||
|
||||
function topSql(driver: KtxConnectionDriver, limit: number): string {
|
||||
if (driver === 'sqlserver') {
|
||||
return ` TOP (${Math.max(1, Math.floor(limit))})`;
|
||||
}
|
||||
return '';
|
||||
function sqlSuffix(fragment: string): string {
|
||||
return fragment ? ` ${fragment}` : '';
|
||||
}
|
||||
|
||||
function limitSql(driver: KtxConnectionDriver, limit: number): string {
|
||||
if (driver === 'sqlserver') {
|
||||
return '';
|
||||
}
|
||||
return ` LIMIT ${Math.max(1, Math.floor(limit))}`;
|
||||
function aliasedTupleSelect(dialect: KtxDialect, columns: readonly string[]): string {
|
||||
return columns.map((column, index) => `${dialect.quoteIdentifier(column)} AS c${index}`).join(', ');
|
||||
}
|
||||
|
||||
function aliasedTupleSelect(driver: KtxConnectionDriver, columns: readonly string[]): string {
|
||||
return columns
|
||||
.map((column, index) => `${quoteKtxRelationshipIdentifier(driver, column)} AS c${index}`)
|
||||
.join(', ');
|
||||
}
|
||||
|
||||
function nonNullPredicate(driver: KtxConnectionDriver, columns: readonly string[]): string {
|
||||
return columns.map((column) => `${quoteKtxRelationshipIdentifier(driver, column)} IS NOT NULL`).join(' AND ');
|
||||
function nonNullPredicate(dialect: KtxDialect, columns: readonly string[]): string {
|
||||
return columns.map((column) => `${dialect.quoteIdentifier(column)} IS NOT NULL`).join(' AND ');
|
||||
}
|
||||
|
||||
function tupleEquality(columns: number): string {
|
||||
|
|
@ -255,39 +242,39 @@ function tupleEquality(columns: number): string {
|
|||
}
|
||||
|
||||
function buildTupleDistinctSql(input: {
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
table: KtxTableRef;
|
||||
columns: readonly string[];
|
||||
}): string {
|
||||
const tableSql = formatKtxRelationshipTableRef(input.driver, input.table);
|
||||
const tableSql = input.dialect.formatTableName(input.table);
|
||||
return [
|
||||
'WITH tuple_values AS (',
|
||||
`SELECT DISTINCT ${aliasedTupleSelect(input.driver, input.columns)} FROM ${tableSql}`,
|
||||
`WHERE ${nonNullPredicate(input.driver, input.columns)}`,
|
||||
`SELECT DISTINCT ${aliasedTupleSelect(input.dialect, input.columns)} FROM ${tableSql}`,
|
||||
`WHERE ${nonNullPredicate(input.dialect, input.columns)}`,
|
||||
')',
|
||||
'SELECT COUNT(*) AS distinct_count FROM tuple_values',
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
function buildCompositeCoverageSql(input: {
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
childTable: KtxTableRef;
|
||||
childColumns: readonly string[];
|
||||
parentTable: KtxTableRef;
|
||||
parentColumns: readonly string[];
|
||||
maxDistinctSourceValues: number;
|
||||
}): string {
|
||||
const childTableSql = formatKtxRelationshipTableRef(input.driver, input.childTable);
|
||||
const parentTableSql = formatKtxRelationshipTableRef(input.driver, input.parentTable);
|
||||
const top = topSql(input.driver, input.maxDistinctSourceValues);
|
||||
const limit = limitSql(input.driver, input.maxDistinctSourceValues);
|
||||
const childTableSql = input.dialect.formatTableName(input.childTable);
|
||||
const parentTableSql = input.dialect.formatTableName(input.parentTable);
|
||||
const top = input.dialect.getTopClause(input.maxDistinctSourceValues);
|
||||
const limit = sqlSuffix(input.dialect.getLimitOffsetClause(input.maxDistinctSourceValues));
|
||||
return [
|
||||
'WITH child_values AS (',
|
||||
`SELECT DISTINCT${top} ${aliasedTupleSelect(input.driver, input.childColumns)} FROM ${childTableSql}`,
|
||||
`WHERE ${nonNullPredicate(input.driver, input.childColumns)}${limit}`,
|
||||
`SELECT DISTINCT${top ? ` ${top}` : ''} ${aliasedTupleSelect(input.dialect, input.childColumns)} FROM ${childTableSql}`,
|
||||
`WHERE ${nonNullPredicate(input.dialect, input.childColumns)}${limit}`,
|
||||
'), parent_values AS (',
|
||||
`SELECT DISTINCT ${aliasedTupleSelect(input.driver, input.parentColumns)} FROM ${parentTableSql}`,
|
||||
`WHERE ${nonNullPredicate(input.driver, input.parentColumns)}`,
|
||||
`SELECT DISTINCT ${aliasedTupleSelect(input.dialect, input.parentColumns)} FROM ${parentTableSql}`,
|
||||
`WHERE ${nonNullPredicate(input.dialect, input.parentColumns)}`,
|
||||
')',
|
||||
'SELECT',
|
||||
'(SELECT COUNT(*) FROM child_values) AS child_distinct,',
|
||||
|
|
@ -335,7 +322,7 @@ function hasAcceptedSubset(
|
|||
|
||||
async function detectCompositePrimaryKeys(input: {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
table: KtxEnrichedTable;
|
||||
profiles: KtxRelationshipProfileArtifact;
|
||||
executor: KtxRelationshipReadOnlyExecutor;
|
||||
|
|
@ -379,7 +366,7 @@ async function detectCompositePrimaryKeys(input: {
|
|||
{
|
||||
connectionId: input.connectionId,
|
||||
sql: buildTupleDistinctSql({
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
table: input.table.ref,
|
||||
columns: columnNames,
|
||||
}),
|
||||
|
|
@ -439,7 +426,7 @@ function compatibleTuple(sourceColumns: readonly KtxEnrichedColumn[], targetColu
|
|||
|
||||
async function validateCompositeRelationship(input: {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
sourceTable: KtxEnrichedTable;
|
||||
sourceColumns: readonly KtxEnrichedColumn[];
|
||||
targetKey: KtxCompositePrimaryKeyCandidate;
|
||||
|
|
@ -454,7 +441,7 @@ async function validateCompositeRelationship(input: {
|
|||
{
|
||||
connectionId: input.connectionId,
|
||||
sql: buildCompositeCoverageSql({
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
childTable: input.sourceTable.ref,
|
||||
childColumns: input.sourceColumns.map((column) => column.name),
|
||||
parentTable: input.targetTable.ref,
|
||||
|
|
@ -552,7 +539,7 @@ export async function discoverKtxCompositeRelationships(
|
|||
for (const table of tables) {
|
||||
const result = await detectCompositePrimaryKeys({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
table,
|
||||
profiles: input.profiles,
|
||||
executor: input.executor,
|
||||
|
|
@ -595,7 +582,7 @@ export async function discoverKtxCompositeRelationships(
|
|||
|
||||
const result = await validateCompositeRelationship({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
sourceTable,
|
||||
sourceColumns,
|
||||
targetKey,
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import type { KtxLlmRuntimePort } from '../../context/llm/runtime-port.js';
|
||||
import type { KtxDialect } from '../connections/dialects.js';
|
||||
import type { KtxScanRelationshipConfig } from '../project/config.js';
|
||||
import type { KtxEnrichedRelationship, KtxEnrichedSchema, KtxRelationshipUpdate } from './enrichment-types.js';
|
||||
import {
|
||||
|
|
@ -24,7 +25,6 @@ import {
|
|||
} from './relationship-profiling.js';
|
||||
import { validateKtxRelationshipDiscoveryCandidates } from './relationship-validation.js';
|
||||
import type {
|
||||
KtxConnectionDriver,
|
||||
KtxScanConnector,
|
||||
KtxScanContext,
|
||||
KtxScanEnrichmentSummary,
|
||||
|
|
@ -34,7 +34,7 @@ import type {
|
|||
|
||||
export interface DiscoverKtxRelationshipsInput {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
connector: KtxScanConnector;
|
||||
schema: KtxEnrichedSchema;
|
||||
context: KtxScanContext;
|
||||
|
|
@ -122,7 +122,7 @@ function compositeSummary(relationships: readonly KtxCompositeRelationshipCandid
|
|||
|
||||
async function detectCompositeRelationships(input: {
|
||||
connectionId: string;
|
||||
driver: DiscoverKtxRelationshipsInput['driver'];
|
||||
dialect: KtxDialect;
|
||||
schema: KtxEnrichedSchema;
|
||||
profile: KtxRelationshipProfileArtifact;
|
||||
executor: KtxRelationshipReadOnlyExecutor | null;
|
||||
|
|
@ -135,7 +135,7 @@ async function detectCompositeRelationships(input: {
|
|||
try {
|
||||
const compositeDetection = await discoverKtxCompositeRelationships({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
schema: input.schema,
|
||||
profiles: input.profile,
|
||||
executor: input.executor,
|
||||
|
|
@ -223,7 +223,7 @@ export async function discoverKtxRelationships(
|
|||
const profileCache = createKtxRelationshipProfileCache();
|
||||
const profile = await profileKtxRelationshipSchema({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
schema: input.schema,
|
||||
executor,
|
||||
ctx: input.context,
|
||||
|
|
@ -256,7 +256,7 @@ export async function discoverKtxRelationships(
|
|||
warnings.push(...llmProposalResult.warnings);
|
||||
const validated = await validateKtxRelationshipDiscoveryCandidates({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
candidates,
|
||||
profiles: profile,
|
||||
executor,
|
||||
|
|
@ -282,7 +282,7 @@ export async function discoverKtxRelationships(
|
|||
});
|
||||
const compositeRelationships = await detectCompositeRelationships({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
schema: input.schema,
|
||||
profile,
|
||||
executor,
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import type { KtxDialect } from '../connections/dialects.js';
|
||||
import type { KtxEnrichedColumn, KtxEnrichedSchema, KtxEnrichedTable } from './enrichment-types.js';
|
||||
import { mapWithConcurrency } from './relationship-validation.js';
|
||||
import type {
|
||||
|
|
@ -55,7 +56,7 @@ export interface KtxRelationshipProfileCache {
|
|||
|
||||
export interface ProfileKtxRelationshipSchemaInput {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
schema: KtxEnrichedSchema;
|
||||
executor: KtxRelationshipReadOnlyExecutor | null;
|
||||
ctx: KtxScanContext;
|
||||
|
|
@ -71,75 +72,6 @@ export function createKtxRelationshipProfileCache(): KtxRelationshipProfileCache
|
|||
|
||||
const SAMPLE_VALUE_DELIMITER = '\u001f';
|
||||
|
||||
type QuoteStyle = 'double' | 'backtick' | 'bracket';
|
||||
|
||||
function quoteStyle(driver: KtxConnectionDriver): QuoteStyle {
|
||||
if (driver === 'mysql' || driver === 'clickhouse') {
|
||||
return 'backtick';
|
||||
}
|
||||
if (driver === 'sqlserver') {
|
||||
return 'bracket';
|
||||
}
|
||||
return 'double';
|
||||
}
|
||||
|
||||
export function quoteKtxRelationshipIdentifier(driver: KtxConnectionDriver, identifier: string): string {
|
||||
switch (quoteStyle(driver)) {
|
||||
case 'backtick':
|
||||
return `\`${identifier.replace(/`/g, '``')}\``;
|
||||
case 'bracket':
|
||||
return `[${identifier.replace(/\]/g, ']]')}]`;
|
||||
case 'double':
|
||||
return `"${identifier.replace(/"/g, '""')}"`;
|
||||
}
|
||||
}
|
||||
|
||||
export function formatKtxRelationshipTableRef(driver: KtxConnectionDriver, table: KtxTableRef): string {
|
||||
const parts =
|
||||
driver === 'sqlite'
|
||||
? [table.name]
|
||||
: [table.catalog, table.db, table.name].filter((value): value is string => Boolean(value));
|
||||
return parts.map((part) => quoteKtxRelationshipIdentifier(driver, part)).join('.');
|
||||
}
|
||||
|
||||
function textLengthExpression(driver: KtxConnectionDriver, columnSql: string): string {
|
||||
if (driver === 'mysql') {
|
||||
return `CHAR_LENGTH(CAST(${columnSql} AS CHAR))`;
|
||||
}
|
||||
if (driver === 'sqlserver') {
|
||||
return `LEN(CAST(${columnSql} AS NVARCHAR(MAX)))`;
|
||||
}
|
||||
if (driver === 'bigquery') {
|
||||
return `LENGTH(CAST(${columnSql} AS STRING))`;
|
||||
}
|
||||
if (driver === 'clickhouse') {
|
||||
return `length(toString(${columnSql}))`;
|
||||
}
|
||||
return `LENGTH(CAST(${columnSql} AS TEXT))`;
|
||||
}
|
||||
|
||||
function limitSql(driver: KtxConnectionDriver, limit: number): string {
|
||||
if (driver === 'sqlserver') {
|
||||
return '';
|
||||
}
|
||||
return ` LIMIT ${Math.max(1, Math.floor(limit))}`;
|
||||
}
|
||||
|
||||
function topSql(driver: KtxConnectionDriver, limit: number): string {
|
||||
if (driver === 'sqlserver') {
|
||||
return ` TOP (${Math.max(1, Math.floor(limit))})`;
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function sampledTableSql(driver: KtxConnectionDriver, tableSql: string, limit: number): string {
|
||||
const safeLimit = Math.max(1, Math.floor(limit));
|
||||
if (driver === 'sqlserver') {
|
||||
return `(SELECT TOP (${safeLimit}) * FROM ${tableSql}) AS relationship_profile_sample`;
|
||||
}
|
||||
return `(SELECT * FROM ${tableSql}${limitSql(driver, safeLimit)}) AS relationship_profile_sample`;
|
||||
}
|
||||
|
||||
function firstRow(result: KtxQueryResult): unknown[] {
|
||||
return result.rows[0] ?? [];
|
||||
}
|
||||
|
|
@ -191,7 +123,7 @@ function columnKey(table: KtxEnrichedTable, column: KtxEnrichedColumn): string {
|
|||
|
||||
function tableProfileCacheKey(input: {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
ctx: KtxScanContext;
|
||||
table: KtxTableRef;
|
||||
sampleValuesPerColumn: number;
|
||||
|
|
@ -200,7 +132,7 @@ function tableProfileCacheKey(input: {
|
|||
return [
|
||||
input.ctx.runId,
|
||||
input.connectionId,
|
||||
input.driver,
|
||||
input.dialect.type,
|
||||
input.table.catalog ?? '',
|
||||
input.table.db ?? '',
|
||||
input.table.name,
|
||||
|
|
@ -213,57 +145,47 @@ function sqlStringLiteral(value: string): string {
|
|||
return `'${value.replace(/'/g, "''")}'`;
|
||||
}
|
||||
|
||||
function sampleAggregateSql(driver: KtxConnectionDriver, innerSql: string): string {
|
||||
if (driver === 'postgres') {
|
||||
return `(SELECT STRING_AGG(CAST(value AS TEXT), CHR(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
function sqlSuffix(fragment: string): string {
|
||||
return fragment ? ` ${fragment}` : '';
|
||||
}
|
||||
|
||||
function sampledTableSql(dialect: KtxDialect, tableSql: string, limit: number): string {
|
||||
const top = dialect.getTopClause(limit);
|
||||
if (top) {
|
||||
return `(SELECT ${top} * FROM ${tableSql}) AS relationship_profile_sample`;
|
||||
}
|
||||
if (driver === 'bigquery') {
|
||||
return `(SELECT STRING_AGG(CAST(value AS STRING), '\\u001F') FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
if (driver === 'mysql') {
|
||||
return `(SELECT GROUP_CONCAT(CAST(value AS CHAR) SEPARATOR CHAR(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
if (driver === 'sqlserver') {
|
||||
return `(SELECT STRING_AGG(CAST(value AS NVARCHAR(MAX)), CHAR(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
if (driver === 'clickhouse') {
|
||||
return `(SELECT arrayStringConcat(groupArray(toString(value)), '\\x1F') FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
if (driver === 'snowflake') {
|
||||
return `(SELECT LISTAGG(CAST(value AS VARCHAR), '\\x1f') FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
}
|
||||
return `(SELECT GROUP_CONCAT(CAST(value AS TEXT), char(31)) FROM (${innerSql}) AS relationship_profile_values)`;
|
||||
return `(SELECT * FROM ${tableSql}${sqlSuffix(dialect.getLimitOffsetClause(limit))}) AS relationship_profile_sample`;
|
||||
}
|
||||
|
||||
function sampleValuesSql(input: {
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
tableSql: string;
|
||||
columnSql: string;
|
||||
limit: number;
|
||||
}): string {
|
||||
const top = input.dialect.getTopClause(input.limit);
|
||||
return [
|
||||
`SELECT${topSql(input.driver, input.limit)} ${input.columnSql} AS value`,
|
||||
`SELECT${top ? ` ${top}` : ''} ${input.columnSql} AS value`,
|
||||
`FROM ${input.tableSql}`,
|
||||
`WHERE ${input.columnSql} IS NOT NULL`,
|
||||
`GROUP BY ${input.columnSql}`,
|
||||
`ORDER BY COUNT(*) DESC, ${input.columnSql} ASC`,
|
||||
limitSql(input.driver, input.limit),
|
||||
sqlSuffix(input.dialect.getLimitOffsetClause(input.limit)),
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
function columnProfileSelectSql(input: {
|
||||
connectionDriver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
tableSql: string;
|
||||
profileTableSql: string;
|
||||
column: KtxEnrichedColumn;
|
||||
sampleValuesPerColumn: number;
|
||||
}): string {
|
||||
const columnSql = quoteKtxRelationshipIdentifier(input.connectionDriver, input.column.name);
|
||||
const textLengthSql = textLengthExpression(input.connectionDriver, columnSql);
|
||||
const samplesSql = sampleAggregateSql(
|
||||
input.connectionDriver,
|
||||
const columnSql = input.dialect.quoteIdentifier(input.column.name);
|
||||
const textLengthSql = input.dialect.textLengthExpression(columnSql);
|
||||
const samplesSql = input.dialect.getSampleValueAggregation(
|
||||
sampleValuesSql({
|
||||
driver: input.connectionDriver,
|
||||
dialect: input.dialect,
|
||||
tableSql: input.profileTableSql,
|
||||
columnSql,
|
||||
limit: input.sampleValuesPerColumn,
|
||||
|
|
@ -296,12 +218,12 @@ function splitSampleValues(value: unknown): string[] {
|
|||
|
||||
async function queryCount(input: {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
table: KtxTableRef;
|
||||
executor: KtxRelationshipReadOnlyExecutor;
|
||||
ctx: KtxScanContext;
|
||||
}): Promise<{ rowCount: number; queryCount: number }> {
|
||||
const tableSql = formatKtxRelationshipTableRef(input.driver, input.table);
|
||||
const tableSql = input.dialect.formatTableName(input.table);
|
||||
const result = await input.executor.executeReadOnly(
|
||||
{ connectionId: input.connectionId, sql: `SELECT COUNT(*) AS row_count FROM ${tableSql}`, maxRows: 1 },
|
||||
input.ctx,
|
||||
|
|
@ -311,7 +233,7 @@ async function queryCount(input: {
|
|||
|
||||
async function queryTableProfile(input: {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
table: KtxEnrichedTable;
|
||||
executor: KtxRelationshipReadOnlyExecutor;
|
||||
ctx: KtxScanContext;
|
||||
|
|
@ -325,7 +247,7 @@ async function queryTableProfile(input: {
|
|||
if (input.table.columns.length === 0) {
|
||||
const rowCount = await queryCount({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
table: input.table.ref,
|
||||
executor: input.executor,
|
||||
ctx: input.ctx,
|
||||
|
|
@ -337,12 +259,12 @@ async function queryTableProfile(input: {
|
|||
};
|
||||
}
|
||||
|
||||
const tableSql = formatKtxRelationshipTableRef(input.driver, input.table.ref);
|
||||
const profileTableSql = sampledTableSql(input.driver, tableSql, input.profileSampleRows);
|
||||
const tableSql = input.dialect.formatTableName(input.table.ref);
|
||||
const profileTableSql = sampledTableSql(input.dialect, tableSql, input.profileSampleRows);
|
||||
const sql = input.table.columns
|
||||
.map((column) =>
|
||||
columnProfileSelectSql({
|
||||
connectionDriver: input.driver,
|
||||
dialect: input.dialect,
|
||||
tableSql,
|
||||
profileTableSql,
|
||||
column,
|
||||
|
|
@ -401,7 +323,7 @@ export async function profileKtxRelationshipSchema(
|
|||
if (!input.executor) {
|
||||
return {
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
driver: input.dialect.type,
|
||||
sqlAvailable: false,
|
||||
queryCount: 0,
|
||||
tables: [],
|
||||
|
|
@ -425,7 +347,7 @@ export async function profileKtxRelationshipSchema(
|
|||
const profileSampleRows = input.profileSampleRows ?? 10000;
|
||||
const cacheKey = tableProfileCacheKey({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
ctx: input.ctx,
|
||||
table: table.ref,
|
||||
sampleValuesPerColumn,
|
||||
|
|
@ -439,7 +361,7 @@ export async function profileKtxRelationshipSchema(
|
|||
try {
|
||||
const tableProfile = await queryTableProfile({
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
dialect: input.dialect,
|
||||
table,
|
||||
executor,
|
||||
ctx: input.ctx,
|
||||
|
|
@ -481,7 +403,7 @@ export async function profileKtxRelationshipSchema(
|
|||
|
||||
return {
|
||||
connectionId: input.connectionId,
|
||||
driver: input.driver,
|
||||
driver: input.dialect.type,
|
||||
sqlAvailable: true,
|
||||
queryCount: queryTotal,
|
||||
tables,
|
||||
|
|
|
|||
|
|
@ -1,13 +1,12 @@
|
|||
import type { KtxDialect } from '../connections/dialects.js';
|
||||
import type { KtxRelationshipEndpoint } from './enrichment-types.js';
|
||||
import { applyKtxRelationshipValidationBudget, type KtxRelationshipValidationBudget } from './relationship-budget.js';
|
||||
import type { KtxRelationshipDiscoveryCandidate } from './relationship-candidates.js';
|
||||
import {
|
||||
formatKtxRelationshipTableRef,
|
||||
type KtxRelationshipProfileArtifact,
|
||||
type KtxRelationshipReadOnlyExecutor,
|
||||
quoteKtxRelationshipIdentifier,
|
||||
} from './relationship-profiling.js';
|
||||
import type { KtxConnectionDriver, KtxQueryResult, KtxScanContext } from './types.js';
|
||||
import type { KtxQueryResult, KtxScanContext, KtxTableRef } from './types.js';
|
||||
|
||||
type KtxValidatedRelationshipStatus = 'accepted' | 'review' | 'rejected';
|
||||
|
||||
|
|
@ -45,7 +44,7 @@ export interface KtxValidatedRelationshipDiscoveryCandidate
|
|||
|
||||
export interface ValidateKtxRelationshipDiscoveryCandidatesInput {
|
||||
connectionId: string;
|
||||
driver: KtxConnectionDriver;
|
||||
dialect: KtxDialect;
|
||||
candidates: readonly KtxRelationshipDiscoveryCandidate[];
|
||||
profiles: KtxRelationshipProfileArtifact;
|
||||
executor: KtxRelationshipReadOnlyExecutor | null;
|
||||
|
|
@ -104,38 +103,28 @@ function numberAt(result: KtxQueryResult, header: string): number {
|
|||
return 0;
|
||||
}
|
||||
|
||||
function limitSql(driver: KtxConnectionDriver, limit: number): string {
|
||||
if (driver === 'sqlserver') {
|
||||
return '';
|
||||
}
|
||||
return ` LIMIT ${Math.max(1, Math.floor(limit))}`;
|
||||
}
|
||||
|
||||
function topSql(driver: KtxConnectionDriver, limit: number): string {
|
||||
if (driver === 'sqlserver') {
|
||||
return ` TOP (${Math.max(1, Math.floor(limit))})`;
|
||||
}
|
||||
return '';
|
||||
function sqlSuffix(fragment: string): string {
|
||||
return fragment ? ` ${fragment}` : '';
|
||||
}
|
||||
|
||||
function buildCoverageSql(input: {
|
||||
driver: KtxConnectionDriver;
|
||||
childTable: string;
|
||||
dialect: KtxDialect;
|
||||
childTable: KtxTableRef;
|
||||
childColumn: string;
|
||||
parentTable: string;
|
||||
parentTable: KtxTableRef;
|
||||
parentColumn: string;
|
||||
maxDistinctSourceValues: number;
|
||||
}): string {
|
||||
const childTable = formatKtxRelationshipTableRef(input.driver, { catalog: null, db: null, name: input.childTable });
|
||||
const parentTable = formatKtxRelationshipTableRef(input.driver, { catalog: null, db: null, name: input.parentTable });
|
||||
const childColumn = quoteKtxRelationshipIdentifier(input.driver, input.childColumn);
|
||||
const parentColumn = quoteKtxRelationshipIdentifier(input.driver, input.parentColumn);
|
||||
const limit = limitSql(input.driver, input.maxDistinctSourceValues);
|
||||
const top = topSql(input.driver, input.maxDistinctSourceValues);
|
||||
const childTable = input.dialect.formatTableName(input.childTable);
|
||||
const parentTable = input.dialect.formatTableName(input.parentTable);
|
||||
const childColumn = input.dialect.quoteIdentifier(input.childColumn);
|
||||
const parentColumn = input.dialect.quoteIdentifier(input.parentColumn);
|
||||
const limit = sqlSuffix(input.dialect.getLimitOffsetClause(input.maxDistinctSourceValues));
|
||||
const top = input.dialect.getTopClause(input.maxDistinctSourceValues);
|
||||
|
||||
return [
|
||||
'WITH child_values AS (',
|
||||
`SELECT DISTINCT${top} ${childColumn} AS value FROM ${childTable} WHERE ${childColumn} IS NOT NULL${limit}`,
|
||||
`SELECT DISTINCT${top ? ` ${top}` : ''} ${childColumn} AS value FROM ${childTable} WHERE ${childColumn} IS NOT NULL${limit}`,
|
||||
'), parent_values AS (',
|
||||
`SELECT DISTINCT ${parentColumn} AS value FROM ${parentTable} WHERE ${parentColumn} IS NOT NULL`,
|
||||
')',
|
||||
|
|
@ -271,10 +260,10 @@ export async function validateKtxRelationshipDiscoveryCandidates(
|
|||
{
|
||||
connectionId: input.connectionId,
|
||||
sql: buildCoverageSql({
|
||||
driver: input.driver,
|
||||
childTable: candidate.from.table.name,
|
||||
dialect: input.dialect,
|
||||
childTable: candidate.from.table,
|
||||
childColumn: sourceColumn,
|
||||
parentTable: candidate.to.table.name,
|
||||
parentTable: candidate.to.table,
|
||||
parentColumn: targetColumn,
|
||||
maxDistinctSourceValues: settings.maxDistinctSourceValues,
|
||||
}),
|
||||
|
|
|
|||
|
|
@ -297,6 +297,7 @@ export interface KtxQueryResult {
|
|||
}
|
||||
|
||||
export interface KtxTableListEntry {
|
||||
catalog: string | null;
|
||||
schema: string;
|
||||
name: string;
|
||||
kind: 'table' | 'view';
|
||||
|
|
@ -313,6 +314,8 @@ export interface KtxScanConnector {
|
|||
capabilities: KtxConnectorCapabilities;
|
||||
eventStreamDiscovery?: KtxEventStreamDiscoveryPort;
|
||||
introspect(input: KtxScanInput, ctx: KtxScanContext): Promise<KtxSchemaSnapshot>;
|
||||
listSchemas(): Promise<string[]>;
|
||||
listTables(schemas?: string[]): Promise<KtxTableListEntry[]>;
|
||||
testConnection?(): Promise<KtxConnectorTestResult>;
|
||||
sampleColumn?(input: KtxColumnSampleInput, ctx: KtxScanContext): Promise<KtxColumnSampleResult>;
|
||||
sampleTable?(input: KtxTableSampleInput, ctx: KtxScanContext): Promise<KtxTableSampleResult>;
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { getDialectForDriver } from '../../context/connections/dialects.js';
|
||||
import { getDialectForDriver, type KtxDialect } from '../connections/dialects.js';
|
||||
import type { KtxFileStorePort } from '../../context/core/file-store.js';
|
||||
import type {
|
||||
KtxConnectionDriver,
|
||||
|
|
@ -128,46 +128,22 @@ function splitDisplay(display: string): string[] {
|
|||
.filter(Boolean);
|
||||
}
|
||||
|
||||
function formatDisplay(driver: CatalogDriver, table: KtxTableRef): string {
|
||||
if (driver === 'sqlite') {
|
||||
return table.name;
|
||||
}
|
||||
return [table.catalog, table.db, table.name].filter((part): part is string => Boolean(part)).join('.');
|
||||
function formatDisplay(dialect: KtxDialect, table: KtxTableRef): string {
|
||||
return dialect.formatDisplayRef(table);
|
||||
}
|
||||
|
||||
function parseDisplay(driver: CatalogDriver, display: string): KtxTableRef | null {
|
||||
function parseDisplay(dialect: KtxDialect, display: string): KtxTableRef | null {
|
||||
const parsed = dialect.parseDisplayRef(display);
|
||||
if (parsed) {
|
||||
return parsed;
|
||||
}
|
||||
const parts = splitDisplay(display);
|
||||
if (driver === 'sqlite') {
|
||||
return parts.length === 1 ? { catalog: null, db: null, name: parts[0]! } : null;
|
||||
}
|
||||
if (driver === 'bigquery' || driver === 'snowflake' || driver === 'sqlserver') {
|
||||
if (parts.length !== 3) {
|
||||
return null;
|
||||
}
|
||||
return { catalog: parts[0]!, db: parts[1]!, name: parts[2]! };
|
||||
}
|
||||
if (parts.length === 2) {
|
||||
return { catalog: null, db: parts[0]!, name: parts[1]! };
|
||||
}
|
||||
if (parts.length === 3) {
|
||||
return { catalog: parts[0]!, db: parts[1]!, name: parts[2]! };
|
||||
}
|
||||
return parts.length === 1 ? { catalog: null, db: null, name: parts[0]! } : null;
|
||||
}
|
||||
|
||||
function expectedDisplayPartCount(driver: CatalogDriver): number {
|
||||
if (driver === 'sqlite') {
|
||||
return 1;
|
||||
}
|
||||
if (driver === 'bigquery' || driver === 'snowflake' || driver === 'sqlserver') {
|
||||
return 3;
|
||||
}
|
||||
return 2;
|
||||
}
|
||||
|
||||
function parseColumnDisplay(driver: CatalogDriver, display: string): (KtxTableRef & { column: string }) | null {
|
||||
function parseColumnDisplay(dialect: KtxDialect, display: string): (KtxTableRef & { column: string }) | null {
|
||||
const parts = splitDisplay(display);
|
||||
const tablePartCount = expectedDisplayPartCount(driver);
|
||||
const tablePartCount = dialect.columnDisplayTablePartCount();
|
||||
if (parts.length !== tablePartCount + 1) {
|
||||
return null;
|
||||
}
|
||||
|
|
@ -175,7 +151,7 @@ function parseColumnDisplay(driver: CatalogDriver, display: string): (KtxTableRe
|
|||
if (!column) {
|
||||
return null;
|
||||
}
|
||||
const table = parseDisplay(driver, parts.slice(0, -1).join('.'));
|
||||
const table = dialect.parseDisplayRef(parts.slice(0, -1).join('.'));
|
||||
return table ? { ...table, column } : null;
|
||||
}
|
||||
|
||||
|
|
@ -272,6 +248,7 @@ export class WarehouseCatalogService {
|
|||
if (!table) {
|
||||
return null;
|
||||
}
|
||||
const dialect = getDialectForDriver(catalog.driver);
|
||||
const profileTables = catalog.profile?.tables ?? [];
|
||||
const profileTable = profileTables.find((candidate) => candidate.table && refsEqual(candidate.table, table));
|
||||
const profileColumns = catalog.profile?.columns ?? {};
|
||||
|
|
@ -281,7 +258,7 @@ export class WarehouseCatalogService {
|
|||
catalog: table.catalog,
|
||||
db: table.db,
|
||||
name: table.name,
|
||||
display: formatDisplay(catalog.driver, table),
|
||||
display: formatDisplay(dialect, table),
|
||||
kind: table.kind,
|
||||
comment: table.comment,
|
||||
description: firstDescription(table.descriptions),
|
||||
|
|
@ -321,16 +298,21 @@ export class WarehouseCatalogService {
|
|||
if (!catalog) {
|
||||
return { resolved: null, candidates: [], dialect: 'unknown' };
|
||||
}
|
||||
const dialect = getDialectForDriver(catalog.driver).type;
|
||||
const parsed = parseDisplay(catalog.driver, display);
|
||||
const dialect = getDialectForDriver(catalog.driver);
|
||||
const parsed = parseDisplay(dialect, display);
|
||||
if (!parsed) {
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect };
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect: dialect.type };
|
||||
}
|
||||
const table = catalog.tables.find((candidate) => refsEqual(candidate, parsed));
|
||||
const exactTable = catalog.tables.find((candidate) => refsEqual(candidate, parsed));
|
||||
const looseNameMatches =
|
||||
parsed.catalog === null && parsed.db === null
|
||||
? catalog.tables.filter((candidate) => normalize(candidate.name) === normalize(parsed.name))
|
||||
: [];
|
||||
const table = exactTable ?? (looseNameMatches.length === 1 ? looseNameMatches[0] : undefined);
|
||||
if (!table) {
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect };
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect: dialect.type };
|
||||
}
|
||||
return { resolved: { catalog: table.catalog, db: table.db, name: table.name }, candidates: [], dialect };
|
||||
return { resolved: { catalog: table.catalog, db: table.db, name: table.name }, candidates: [], dialect: dialect.type };
|
||||
}
|
||||
|
||||
async resolveDisplayTarget(connectionId: string, display: string): Promise<DisplayTargetResolution> {
|
||||
|
|
@ -339,20 +321,20 @@ export class WarehouseCatalogService {
|
|||
return { resolved: null, candidates: [], dialect: 'unknown' };
|
||||
}
|
||||
|
||||
const dialect = getDialectForDriver(catalog.driver).type;
|
||||
const dialect = getDialectForDriver(catalog.driver);
|
||||
const tableResolution = await this.resolveDisplay(connectionId, display);
|
||||
if (tableResolution.resolved) {
|
||||
return tableResolution;
|
||||
}
|
||||
|
||||
const parsedColumn = parseColumnDisplay(catalog.driver, display);
|
||||
const parsedColumn = parseColumnDisplay(dialect, display);
|
||||
if (!parsedColumn) {
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect };
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect: dialect.type };
|
||||
}
|
||||
|
||||
const table = catalog.tables.find((candidate) => refsEqual(candidate, parsedColumn));
|
||||
if (!table) {
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect };
|
||||
return { resolved: null, candidates: bestCandidates(catalog.tables, display), dialect: dialect.type };
|
||||
}
|
||||
|
||||
return {
|
||||
|
|
@ -363,7 +345,7 @@ export class WarehouseCatalogService {
|
|||
column: parsedColumn.column,
|
||||
},
|
||||
candidates: [],
|
||||
dialect,
|
||||
dialect: dialect.type,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -372,6 +354,7 @@ export class WarehouseCatalogService {
|
|||
if (!catalog) {
|
||||
return [];
|
||||
}
|
||||
const dialect = getDialectForDriver(catalog.driver);
|
||||
const hits: RawSchemaHit[] = [];
|
||||
for (const table of catalog.tables as TableWithDescriptions[]) {
|
||||
const tableMatch = matchedOnTable(table, query);
|
||||
|
|
@ -380,7 +363,7 @@ export class WarehouseCatalogService {
|
|||
kind: 'table',
|
||||
connectionId,
|
||||
ref: { catalog: table.catalog, db: table.db, name: table.name },
|
||||
display: formatDisplay(catalog.driver, table),
|
||||
display: formatDisplay(dialect, table),
|
||||
matchedOn: tableMatch,
|
||||
});
|
||||
}
|
||||
|
|
@ -393,7 +376,7 @@ export class WarehouseCatalogService {
|
|||
kind: 'column',
|
||||
connectionId,
|
||||
ref: { catalog: table.catalog, db: table.db, name: table.name, column: column.name },
|
||||
display: `${formatDisplay(catalog.driver, table)}.${column.name}`,
|
||||
display: `${formatDisplay(dialect, table)}.${column.name}`,
|
||||
matchedOn: columnMatch,
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import { parseDottedTableEntry } from './context/scan/enabled-tables.js';
|
||||
import type { KtxTableListEntry } from './context/scan/types.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { profileMark } from './startup-profile.js';
|
||||
|
|
@ -73,7 +74,9 @@ export interface PickDatabaseScopeArgs {
|
|||
}
|
||||
|
||||
function qualifiedTableId(entry: KtxTableListEntry): string {
|
||||
return `${entry.schema}.${entry.name}`;
|
||||
return entry.catalog !== null
|
||||
? `${entry.catalog}.${entry.schema}.${entry.name}`
|
||||
: `${entry.schema}.${entry.name}`;
|
||||
}
|
||||
|
||||
function tableTitle(entry: KtxTableListEntry): string {
|
||||
|
|
@ -177,7 +180,8 @@ function schemasFromEnabledTables(enabledTables: readonly string[]): string[] {
|
|||
const seen = new Set<string>();
|
||||
const result: string[] = [];
|
||||
for (const qualified of enabledTables) {
|
||||
const schema = qualified.split('.')[0] ?? '';
|
||||
const ref = parseDottedTableEntry(qualified);
|
||||
const schema = ref?.db ?? '';
|
||||
if (schema.length === 0 || seen.has(schema)) continue;
|
||||
seen.add(schema);
|
||||
result.push(schema);
|
||||
|
|
@ -228,11 +232,14 @@ async function runStageTwoTreePicker(input: {
|
|||
? initialSelectionForExisting(args.existing.enabledTables, byId)
|
||||
: initialSelectionFromDefaults(selectedSchemas, schemaIds);
|
||||
|
||||
const initialState = buildInitialState({
|
||||
tree,
|
||||
existingSelectedIds: initialSelection,
|
||||
skipEmptyAction: 'save-empty',
|
||||
});
|
||||
const initialState = {
|
||||
...buildInitialState({
|
||||
tree,
|
||||
existingSelectedIds: initialSelection,
|
||||
skipEmptyAction: 'save-empty',
|
||||
}),
|
||||
expanded: new Set(schemaIds),
|
||||
};
|
||||
|
||||
const schemaWordPlural = schemaCount === 1 ? args.schemaNoun : args.schemaNounPlural;
|
||||
const subtitleLines = [
|
||||
|
|
|
|||
28
packages/cli/src/error-message.ts
Normal file
28
packages/cli/src/error-message.ts
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
export function describeError(error: unknown): string {
|
||||
if (!(error instanceof Error)) {
|
||||
const text = String(error);
|
||||
return text.length > 0 ? text : 'unknown error';
|
||||
}
|
||||
const parts: string[] = [];
|
||||
if (error.message.length > 0) {
|
||||
parts.push(error.message);
|
||||
}
|
||||
const seen = new Set<unknown>([error]);
|
||||
let cause: unknown = error.cause;
|
||||
while (cause && !seen.has(cause)) {
|
||||
seen.add(cause);
|
||||
if (cause instanceof Error) {
|
||||
if (cause.message.length > 0) {
|
||||
parts.push(cause.message);
|
||||
}
|
||||
cause = cause.cause;
|
||||
} else {
|
||||
const text = String(cause);
|
||||
if (text.length > 0) {
|
||||
parts.push(text);
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
return parts.length > 0 ? parts.join(': ') : 'unknown error';
|
||||
}
|
||||
|
|
@ -1,3 +1,4 @@
|
|||
import { describeError } from '../error-message.js';
|
||||
import { createKtxEmbeddingProvider, type KtxEmbeddingProviderDeps } from './embedding-provider.js';
|
||||
import type { KtxEmbeddingConfig } from './types.js';
|
||||
|
||||
|
|
@ -48,7 +49,6 @@ export async function runKtxEmbeddingHealthCheck(
|
|||
}
|
||||
return { ok: true };
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return { ok: false, message: redactHealthCheckMessage(message, config) };
|
||||
return { ok: false, message: redactHealthCheckMessage(describeError(error), config) };
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,11 @@
|
|||
import {
|
||||
getDriverRegistration,
|
||||
listSupportedDrivers,
|
||||
} from './context/connections/drivers.js';
|
||||
import type { KtxLocalProject } from './context/project/project.js';
|
||||
import type { KtxScanConnector } from './context/scan/types.js';
|
||||
|
||||
const SUPPORTED_DRIVERS = 'sqlite, postgres, mysql, clickhouse, sqlserver, bigquery, snowflake';
|
||||
const SUPPORTED_DRIVERS = listSupportedDrivers().join(', ');
|
||||
|
||||
export async function createKtxCliScanConnector(
|
||||
project: KtxLocalProject,
|
||||
|
|
@ -17,58 +21,23 @@ export async function createKtxCliScanConnector(
|
|||
`Connection "${connectionId}" has no \`driver\` field in ktx.yaml. Supported drivers: ${SUPPORTED_DRIVERS}.`,
|
||||
);
|
||||
}
|
||||
if (driver === 'sqlite') {
|
||||
const { KtxSqliteScanConnector, isKtxSqliteConnectionConfig } = await import('./connectors/sqlite/connector.js');;
|
||||
if (!isKtxSqliteConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
return new KtxSqliteScanConnector({ connectionId, connection, projectDir: project.projectDir });
|
||||
|
||||
const registration = getDriverRegistration(driver);
|
||||
if (!registration) {
|
||||
throw new Error(
|
||||
`Connection "${connectionId}" uses driver "${driver}", which has no native standalone KTX scan connector. Supported drivers: ${SUPPORTED_DRIVERS}.`,
|
||||
);
|
||||
}
|
||||
if (driver === 'postgres') {
|
||||
const { KtxPostgresScanConnector, isKtxPostgresConnectionConfig } = await import('./connectors/postgres/connector.js');;
|
||||
if (!isKtxPostgresConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
return new KtxPostgresScanConnector({ connectionId, connection });
|
||||
|
||||
const connectorModule = await registration.load();
|
||||
if (!connectorModule.isConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
if (driver === 'mysql') {
|
||||
const { KtxMysqlScanConnector, isKtxMysqlConnectionConfig } = await import('./connectors/mysql/connector.js');;
|
||||
if (!isKtxMysqlConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
return new KtxMysqlScanConnector({ connectionId, connection });
|
||||
}
|
||||
if (driver === 'clickhouse') {
|
||||
const { KtxClickHouseScanConnector, isKtxClickHouseConnectionConfig } = await import('./connectors/clickhouse/connector.js');;
|
||||
if (!isKtxClickHouseConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
return new KtxClickHouseScanConnector({ connectionId, connection });
|
||||
}
|
||||
if (driver === 'sqlserver') {
|
||||
const { KtxSqlServerScanConnector, isKtxSqlServerConnectionConfig } = await import('./connectors/sqlserver/connector.js');;
|
||||
if (!isKtxSqlServerConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
return new KtxSqlServerScanConnector({ connectionId, connection });
|
||||
}
|
||||
if (driver === 'bigquery') {
|
||||
const { KtxBigQueryScanConnector, isKtxBigQueryConnectionConfig } = await import('./connectors/bigquery/connector.js');;
|
||||
if (!isKtxBigQueryConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
return new KtxBigQueryScanConnector({ connectionId, connection });
|
||||
}
|
||||
if (driver === 'snowflake') {
|
||||
const { KtxSnowflakeScanConnector, isKtxSnowflakeConnectionConfig } = await import('./connectors/snowflake/connector.js');;
|
||||
if (!isKtxSnowflakeConnectionConfig(connection)) {
|
||||
throw invalidConnectionConfigError(connectionId, driver);
|
||||
}
|
||||
return new KtxSnowflakeScanConnector({ connectionId, connection, projectDir: project.projectDir });
|
||||
}
|
||||
throw new Error(
|
||||
`Connection "${connectionId}" uses driver "${driver}", which has no native standalone KTX scan connector. Supported drivers: ${SUPPORTED_DRIVERS}.`,
|
||||
);
|
||||
return connectorModule.createScanConnector({
|
||||
connectionId,
|
||||
connection,
|
||||
projectDir: project.projectDir,
|
||||
});
|
||||
}
|
||||
|
||||
function invalidConnectionConfigError(connectionId: string, driver: string): Error {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import type { KtxEmbeddingConfig } from './llm/types.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { writePrefixedLines } from './clack.js';
|
||||
import {
|
||||
ensureManagedPythonCommandRuntime,
|
||||
type KtxManagedPythonInstallPolicy,
|
||||
|
|
@ -73,7 +74,7 @@ export async function ensureManagedLocalEmbeddingsDaemon(
|
|||
});
|
||||
|
||||
const verb = daemon.status === 'started' ? 'Started' : 'Using';
|
||||
options.io.stderr.write(`${verb} KTX daemon: ${daemon.baseUrl}\n`);
|
||||
writePrefixedLines((chunk) => options.io.stderr.write(chunk), `${verb} KTX daemon: ${daemon.baseUrl}`);
|
||||
|
||||
return {
|
||||
baseUrl: daemon.baseUrl,
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { createServer } from 'node:net';
|
|||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { promisify } from 'node:util';
|
||||
import { z } from 'zod';
|
||||
import { describeError } from './error-message.js';
|
||||
import {
|
||||
installManagedPythonRuntime,
|
||||
managedPythonDaemonLayout,
|
||||
|
|
@ -16,6 +17,17 @@ import {
|
|||
} from './managed-python-runtime.js';
|
||||
import { sanitizeChildProxyEnv } from './proxy-env.js';
|
||||
|
||||
export class ManagedPythonDaemonStartError extends Error {
|
||||
readonly detail: string;
|
||||
readonly stderrLog: string;
|
||||
constructor(detail: string, stderrLog: string) {
|
||||
super(`KTX daemon failed to start: ${detail}. stderr: ${stderrLog}`);
|
||||
this.name = 'ManagedPythonDaemonStartError';
|
||||
this.detail = detail;
|
||||
this.stderrLog = stderrLog;
|
||||
}
|
||||
}
|
||||
|
||||
export interface ManagedPythonDaemonState {
|
||||
schemaVersion: 1;
|
||||
pid: number;
|
||||
|
|
@ -237,7 +249,7 @@ async function healthOk(input: {
|
|||
}
|
||||
return { ok: true };
|
||||
} catch (error) {
|
||||
return { ok: false, detail: error instanceof Error ? error.message : String(error) };
|
||||
return { ok: false, detail: describeError(error) };
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -328,7 +340,7 @@ async function waitForHealth(input: {
|
|||
return;
|
||||
}
|
||||
lastDetail = finalHealth.detail;
|
||||
throw new Error(`KTX daemon failed to start: ${lastDetail}. stderr: ${input.state.stderrLog}`);
|
||||
throw new ManagedPythonDaemonStartError(lastDetail, input.state.stderrLog);
|
||||
}
|
||||
|
||||
async function removeState(layout: ManagedPythonDaemonLayout): Promise<void> {
|
||||
|
|
@ -721,13 +733,21 @@ export async function startManagedPythonDaemon(
|
|||
stdoutLog: layout.daemonStdoutPath,
|
||||
stderrLog: layout.daemonStderrPath,
|
||||
};
|
||||
await waitForHealth({
|
||||
state,
|
||||
cliVersion: options.cliVersion,
|
||||
fetch: fetchImpl,
|
||||
timeoutMs: options.startupTimeoutMs ?? 10_000,
|
||||
pollIntervalMs: options.pollIntervalMs ?? 100,
|
||||
});
|
||||
try {
|
||||
await waitForHealth({
|
||||
state,
|
||||
cliVersion: options.cliVersion,
|
||||
fetch: fetchImpl,
|
||||
timeoutMs: options.startupTimeoutMs ?? 30_000,
|
||||
pollIntervalMs: options.pollIntervalMs ?? 100,
|
||||
});
|
||||
} catch (error) {
|
||||
if (processAlive(state.pid)) {
|
||||
killProcess(state.pid);
|
||||
}
|
||||
await removeState(layout);
|
||||
throw error;
|
||||
}
|
||||
await writeState(layout.daemonStatePath, state);
|
||||
return { status: 'started', layout, state, baseUrl: baseUrl(state) };
|
||||
} finally {
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import type { LookerTableIdentifierParser } from './context/ingest/adapters/look
|
|||
import { createHttpSqlAnalysisPort, type KtxSqlAnalysisHttpJsonRunner } from './context/sql-analysis/http-sql-analysis-port.js';
|
||||
import type { SqlAnalysisPort } from './context/sql-analysis/ports.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { writePrefixedLines } from './clack.js';
|
||||
import {
|
||||
ensureManagedPythonCommandRuntime,
|
||||
type KtxManagedPythonInstallPolicy,
|
||||
|
|
@ -137,7 +138,7 @@ export function createManagedPythonDaemonBaseUrlResolver(
|
|||
force: false,
|
||||
});
|
||||
const verb = daemon.status === 'started' ? 'Started' : 'Using existing';
|
||||
options.io.stderr.write(`${verb} KTX daemon: ${daemon.baseUrl}\n`);
|
||||
writePrefixedLines((chunk) => options.io.stderr.write(chunk), `${verb} KTX daemon: ${daemon.baseUrl}`);
|
||||
cachedBaseUrl = daemon.baseUrl;
|
||||
return cachedBaseUrl;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import { markKtxSetupStateStepComplete } from './context/project/setup-config.js
|
|||
import { serializeKtxProjectConfig } from './context/project/config.js';
|
||||
import { strToU8, zipSync } from 'fflate';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { errorMessage, writePrefixedLines } from './clack.js';
|
||||
import {
|
||||
createKtxSetupPromptAdapter,
|
||||
createKtxSetupUiAdapter,
|
||||
|
|
@ -1230,7 +1231,7 @@ export async function runKtxSetupAgentsStep(
|
|||
}
|
||||
return { status: 'ready', projectDir: args.projectDir, installs, nextActions };
|
||||
} catch (error) {
|
||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
writePrefixedLines((chunk) => io.stderr.write(chunk), errorMessage(error));
|
||||
return { status: 'failed', projectDir: args.projectDir };
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import { type KtxLocalProject, loadKtxProject } from './context/project/project.
|
|||
import { markKtxSetupStateStepComplete, readKtxSetupState } from './context/project/setup-config.js';
|
||||
import { serializeKtxProjectConfig } from './context/project/config.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { errorMessage, writePrefixedLines } from './clack.js';
|
||||
import { buildPublicIngestPlan } from './public-ingest.js';
|
||||
import {
|
||||
type KtxDatabaseContextDepth,
|
||||
|
|
@ -745,7 +746,7 @@ export async function runKtxSetupContextStep(
|
|||
|
||||
return await runBuild(args, io, deps, project, targets);
|
||||
} catch (error) {
|
||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
writePrefixedLines((chunk) => io.stderr.write(chunk), errorMessage(error));
|
||||
return { status: 'failed', projectDir: args.projectDir };
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import { readFile, writeFile } from 'node:fs/promises';
|
|||
import { delimiter, dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { promisify } from 'node:util';
|
||||
import { getDriverRegistration } from './context/connections/drivers.js';
|
||||
import { queryHistoryDialectForConnection } from './context/ingest/adapters/historic-sql/connection-dialect.js';
|
||||
import type { HistoricSqlDialect } from './context/ingest/adapters/historic-sql/types.js';
|
||||
import {
|
||||
|
|
@ -15,6 +16,11 @@ import { loadKtxProject } from './context/project/project.js';
|
|||
import { markKtxSetupStateStepComplete, setKtxSetupDatabaseConnectionIds } from './context/project/setup-config.js';
|
||||
import type { KtxTableListEntry } from './context/scan/types.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import {
|
||||
errorMessage,
|
||||
flushPrefixedBufferedCommandOutput,
|
||||
writePrefixedLines,
|
||||
} from './clack.js';
|
||||
import { runKtxConnection } from './connection.js';
|
||||
import {
|
||||
pickDatabaseScope as defaultPickDatabaseScope,
|
||||
|
|
@ -112,13 +118,13 @@ export interface KtxSetupDatabasesDeps {
|
|||
}
|
||||
|
||||
const DRIVER_OPTIONS: Array<{ value: KtxSetupDatabaseDriver; label: string }> = [
|
||||
{ value: 'sqlite', label: 'SQLite' },
|
||||
{ value: 'postgres', label: 'PostgreSQL' },
|
||||
{ value: 'bigquery', label: 'BigQuery' },
|
||||
{ value: 'snowflake', label: 'Snowflake' },
|
||||
{ value: 'mysql', label: 'MySQL' },
|
||||
{ value: 'clickhouse', label: 'ClickHouse' },
|
||||
{ value: 'sqlserver', label: 'SQL Server' },
|
||||
{ value: 'bigquery', label: 'BigQuery' },
|
||||
{ value: 'snowflake', label: 'Snowflake' },
|
||||
{ value: 'sqlite', label: 'SQLite' },
|
||||
];
|
||||
|
||||
const DRIVER_LABELS = Object.fromEntries(DRIVER_OPTIONS.map((option) => [option.value, option.label])) as Record<
|
||||
|
|
@ -220,7 +226,7 @@ const SCOPE_DISCOVERY_SPECS: Partial<Record<KtxSetupDatabaseDriver, ScopeDiscove
|
|||
};
|
||||
|
||||
type UrlDriverType = Extract<KtxSetupDatabaseDriver, 'postgres' | 'mysql' | 'clickhouse' | 'sqlserver'>;
|
||||
type ConnectionSetupStatus = 'ready' | 'back' | 'failed';
|
||||
type ConnectionSetupStatus = 'ready' | 'back' | 'failed' | 'failed-query-history-unavailable';
|
||||
|
||||
const DRIVER_CONNECTION_DEFAULTS: Record<UrlDriverType, { port: string }> = {
|
||||
postgres: { port: '5432' },
|
||||
|
|
@ -361,74 +367,18 @@ async function defaultListSchemas(projectDir: string, connectionId: string): Pro
|
|||
const project = await loadKtxProject({ projectDir });
|
||||
const connection = project.config.connections[connectionId];
|
||||
const driver = normalizeDriver(connection?.driver);
|
||||
const registration = driver ? getDriverRegistration(driver) : undefined;
|
||||
if (!registration) return [];
|
||||
|
||||
if (driver === 'postgres') {
|
||||
const { KtxPostgresScanConnector, isKtxPostgresConnectionConfig } = await import('./connectors/postgres/connector.js');;
|
||||
if (!isKtxPostgresConnectionConfig(connection)) return [];
|
||||
const connector = new KtxPostgresScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listSchemas();
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
const connectorModule = await registration.load();
|
||||
if (!connectorModule.isConnectionConfig(connection)) return [];
|
||||
|
||||
const connector = connectorModule.createScanConnector({ connectionId, connection, projectDir });
|
||||
try {
|
||||
return await connector.listSchemas();
|
||||
} finally {
|
||||
await connector.cleanup?.();
|
||||
}
|
||||
|
||||
if (driver === 'sqlserver') {
|
||||
const { KtxSqlServerScanConnector, isKtxSqlServerConnectionConfig } = await import('./connectors/sqlserver/connector.js');;
|
||||
if (!isKtxSqlServerConnectionConfig(connection)) return [];
|
||||
const connector = new KtxSqlServerScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listSchemas();
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'mysql') {
|
||||
const { KtxMysqlScanConnector, isKtxMysqlConnectionConfig } = await import('./connectors/mysql/connector.js');;
|
||||
if (!isKtxMysqlConnectionConfig(connection)) return [];
|
||||
const connector = new KtxMysqlScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listSchemas();
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'clickhouse') {
|
||||
const { KtxClickHouseScanConnector, isKtxClickHouseConnectionConfig } = await import('./connectors/clickhouse/connector.js');;
|
||||
if (!isKtxClickHouseConnectionConfig(connection)) return [];
|
||||
const connector = new KtxClickHouseScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listSchemas();
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'bigquery') {
|
||||
const { KtxBigQueryScanConnector, isKtxBigQueryConnectionConfig } = await import('./connectors/bigquery/connector.js');;
|
||||
if (!isKtxBigQueryConnectionConfig(connection)) return [];
|
||||
const connector = new KtxBigQueryScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listDatasets();
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'snowflake') {
|
||||
const { KtxSnowflakeScanConnector, isKtxSnowflakeConnectionConfig } = await import('./connectors/snowflake/connector.js');;
|
||||
if (!isKtxSnowflakeConnectionConfig(connection)) return [];
|
||||
const connector = new KtxSnowflakeScanConnector({ connectionId, connection, projectDir });
|
||||
try {
|
||||
return await connector.listSchemas();
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
function configuredSchemas(connection: KtxProjectConnectionConfig | undefined, driver: KtxSetupDatabaseDriver): string[] | undefined {
|
||||
|
|
@ -448,74 +398,18 @@ async function defaultListTables(
|
|||
const connection = project.config.connections[connectionId];
|
||||
const driver = normalizeDriver(connection?.driver);
|
||||
const schemas = schemasOverride ?? (driver ? configuredSchemas(connection, driver) : undefined);
|
||||
const registration = driver ? getDriverRegistration(driver) : undefined;
|
||||
if (!registration) return [];
|
||||
|
||||
if (driver === 'postgres') {
|
||||
const { KtxPostgresScanConnector, isKtxPostgresConnectionConfig } = await import('./connectors/postgres/connector.js');;
|
||||
if (!isKtxPostgresConnectionConfig(connection)) return [];
|
||||
const connector = new KtxPostgresScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listTables(schemas);
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
const connectorModule = await registration.load();
|
||||
if (!connectorModule.isConnectionConfig(connection)) return [];
|
||||
|
||||
const connector = connectorModule.createScanConnector({ connectionId, connection, projectDir });
|
||||
try {
|
||||
return await connector.listTables(schemas);
|
||||
} finally {
|
||||
await connector.cleanup?.();
|
||||
}
|
||||
|
||||
if (driver === 'mysql') {
|
||||
const { KtxMysqlScanConnector, isKtxMysqlConnectionConfig } = await import('./connectors/mysql/connector.js');;
|
||||
if (!isKtxMysqlConnectionConfig(connection)) return [];
|
||||
const connector = new KtxMysqlScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listTables(schemas);
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'sqlserver') {
|
||||
const { KtxSqlServerScanConnector, isKtxSqlServerConnectionConfig } = await import('./connectors/sqlserver/connector.js');;
|
||||
if (!isKtxSqlServerConnectionConfig(connection)) return [];
|
||||
const connector = new KtxSqlServerScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listTables(schemas);
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'bigquery') {
|
||||
const { KtxBigQueryScanConnector, isKtxBigQueryConnectionConfig } = await import('./connectors/bigquery/connector.js');;
|
||||
if (!isKtxBigQueryConnectionConfig(connection)) return [];
|
||||
const connector = new KtxBigQueryScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listTables(schemas);
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'snowflake') {
|
||||
const { KtxSnowflakeScanConnector, isKtxSnowflakeConnectionConfig } = await import('./connectors/snowflake/connector.js');;
|
||||
if (!isKtxSnowflakeConnectionConfig(connection)) return [];
|
||||
const connector = new KtxSnowflakeScanConnector({ connectionId, connection, projectDir });
|
||||
try {
|
||||
return await connector.listTables(schemas);
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
if (driver === 'clickhouse') {
|
||||
const { KtxClickHouseScanConnector, isKtxClickHouseConnectionConfig } = await import('./connectors/clickhouse/connector.js');;
|
||||
if (!isKtxClickHouseConnectionConfig(connection)) return [];
|
||||
const connector = new KtxClickHouseScanConnector({ connectionId, connection });
|
||||
try {
|
||||
return await connector.listTables(schemas);
|
||||
} finally {
|
||||
await connector.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
function existingConnectionIdsByDriver(
|
||||
|
|
@ -638,9 +532,9 @@ function scriptedScopeConfigForDriver(
|
|||
databaseSchemas: string[],
|
||||
): Record<string, unknown> {
|
||||
if (databaseSchemas.length === 0) return {};
|
||||
if (driver === 'bigquery') return { dataset_ids: databaseSchemas };
|
||||
if (driver === 'clickhouse') return { databases: databaseSchemas };
|
||||
return { schemas: databaseSchemas };
|
||||
const registration = getDriverRegistration(driver);
|
||||
if (!registration?.scopeConfigKey) return {};
|
||||
return { [registration.scopeConfigKey]: databaseSchemas };
|
||||
}
|
||||
|
||||
function databaseNameFromLiteralUrl(url: string): string | undefined {
|
||||
|
|
@ -1128,25 +1022,6 @@ function createBufferedCommandIo(): BufferedCommandIo {
|
|||
};
|
||||
}
|
||||
|
||||
function flushBufferedCommandOutput(io: KtxCliIo, bufferedIo: BufferedCommandIo): void {
|
||||
const stdout = bufferedIo.stdoutText();
|
||||
const stderr = bufferedIo.stderrText();
|
||||
if (stdout.length > 0) {
|
||||
io.stdout.write(stdout);
|
||||
}
|
||||
if (stderr.length > 0) {
|
||||
io.stderr.write(stderr);
|
||||
}
|
||||
}
|
||||
|
||||
function writePrefixedLines(write: (chunk: string) => void, output: string): void {
|
||||
for (const line of output.split(/\r?\n/)) {
|
||||
if (line.length > 0) {
|
||||
write(`│ ${line}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function envWithCurrentNodeFirst(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv {
|
||||
return {
|
||||
...env,
|
||||
|
|
@ -1222,11 +1097,6 @@ async function defaultRebuildNativeSqlite(io: KtxCliIo): Promise<number> {
|
|||
}
|
||||
}
|
||||
|
||||
function flushPrefixedBufferedCommandOutput(io: KtxCliIo, bufferedIo: BufferedCommandIo): void {
|
||||
writePrefixedLines((chunk) => io.stdout.write(chunk), bufferedIo.stdoutText());
|
||||
writePrefixedLines((chunk) => io.stderr.write(chunk), bufferedIo.stderrText());
|
||||
}
|
||||
|
||||
function nativeSqliteAbiMismatchDetail(output: string): string | null {
|
||||
const mentionsBetterSqlite = /\bbetter-sqlite3\b|better_sqlite3/i.test(output);
|
||||
const mentionsAbiMismatch = /compiled against a different Node\.js version|NODE_MODULE_VERSION/i.test(output);
|
||||
|
|
@ -1318,6 +1188,20 @@ async function writeConnectionConfig(input: {
|
|||
}
|
||||
}
|
||||
|
||||
async function disableConnectionQueryHistory(projectDir: string, connectionId: string): Promise<void> {
|
||||
const project = await loadKtxProject({ projectDir });
|
||||
const connection = project.config.connections[connectionId];
|
||||
if (!connection) {
|
||||
return;
|
||||
}
|
||||
const existing = queryHistoryConfigRecord(connection) ?? historicSqlConfigRecord(connection) ?? {};
|
||||
await writeConnectionConfig({
|
||||
projectDir,
|
||||
connectionId,
|
||||
connection: withQueryHistoryConfig(connection, { ...existing, enabled: false }),
|
||||
});
|
||||
}
|
||||
|
||||
async function createConnectionConfigRollback(projectDir: string, connectionId: string): Promise<() => Promise<void>> {
|
||||
const project = await loadKtxProject({ projectDir });
|
||||
const previousConnection = project.config.connections[connectionId];
|
||||
|
|
@ -1519,9 +1403,9 @@ async function maybeConfigureDatabaseScope(input: {
|
|||
input.connectionId,
|
||||
);
|
||||
} catch (error) {
|
||||
const detail = error instanceof Error ? error.message : String(error);
|
||||
input.io.stderr.write(
|
||||
`Could not discover ${spec.promptLabel.toLowerCase()} for ${input.connectionId}; ${detail}\n`,
|
||||
writePrefixedLines(
|
||||
(chunk) => input.io.stderr.write(chunk),
|
||||
`Could not discover ${spec.promptLabel.toLowerCase()} for ${input.connectionId}; ${errorMessage(error)}`,
|
||||
);
|
||||
const typed = await promptCommaSeparatedScope({
|
||||
prompts: input.prompts,
|
||||
|
|
@ -1573,11 +1457,12 @@ async function maybeConfigureDatabaseScope(input: {
|
|||
input.io,
|
||||
);
|
||||
} catch (error) {
|
||||
const detail = error instanceof Error ? error.message : String(error);
|
||||
input.io.stderr.write(
|
||||
const detail = errorMessage(error);
|
||||
writePrefixedLines(
|
||||
(chunk) => input.io.stderr.write(chunk),
|
||||
input.forcePrompt === true
|
||||
? `Could not discover tables for ${input.connectionId}; edit was not saved. ${detail}\n`
|
||||
: `Could not discover tables for ${input.connectionId}; continuing without table filter. ${detail}\n`,
|
||||
? `Could not discover tables for ${input.connectionId}; edit was not saved. ${detail}`
|
||||
: `Could not discover tables for ${input.connectionId}; continuing without table filter. ${detail}`,
|
||||
);
|
||||
return input.forcePrompt === true ? 'failed' : 'ready';
|
||||
}
|
||||
|
|
@ -1665,19 +1550,19 @@ async function maybeRunHistoricSqlSetupProbe(input: {
|
|||
connectionId: string;
|
||||
io: KtxCliIo;
|
||||
deps: KtxSetupDatabasesDeps;
|
||||
}): Promise<void> {
|
||||
}): Promise<boolean> {
|
||||
const project = await loadKtxProject({ projectDir: input.projectDir });
|
||||
const connection = project.config.connections[input.connectionId];
|
||||
const queryHistory = queryHistoryConfigRecord(connection) ?? historicSqlConfigRecord(connection);
|
||||
if (queryHistory?.enabled !== true) {
|
||||
return;
|
||||
return true;
|
||||
}
|
||||
if (!connection) {
|
||||
return;
|
||||
return true;
|
||||
}
|
||||
const dialect = queryHistoryDialectForConnection(connection);
|
||||
if (!dialect) {
|
||||
return;
|
||||
return true;
|
||||
}
|
||||
|
||||
input.io.stdout.write('│ Query history probe...\n');
|
||||
|
|
@ -1696,6 +1581,7 @@ async function maybeRunHistoricSqlSetupProbe(input: {
|
|||
if (!result.ok) {
|
||||
input.io.stdout.write('│ Setup written; query history will be skipped until fixed.\n');
|
||||
}
|
||||
return result.ok;
|
||||
}
|
||||
|
||||
async function applyHistoricSqlConfigToExistingConnection(input: {
|
||||
|
|
@ -1785,8 +1671,11 @@ async function validateAndScanConnection(input: {
|
|||
const testIo = createBufferedCommandIo();
|
||||
const testCode = await testConnection(input.projectDir, input.connectionId, testIo);
|
||||
if (testCode !== 0) {
|
||||
flushBufferedCommandOutput(input.io, testIo);
|
||||
input.io.stderr.write(`Connection test failed for ${input.connectionId}.\n`);
|
||||
flushPrefixedBufferedCommandOutput(input.io, testIo);
|
||||
writePrefixedLines(
|
||||
(chunk) => input.io.stderr.write(chunk),
|
||||
`Connection test failed for ${input.connectionId}.`,
|
||||
);
|
||||
return 'failed';
|
||||
}
|
||||
const testOutput = testIo.stdoutText();
|
||||
|
|
@ -1800,7 +1689,7 @@ async function validateAndScanConnection(input: {
|
|||
return scopeStatus;
|
||||
}
|
||||
|
||||
await maybeRunHistoricSqlSetupProbe({
|
||||
const queryHistoryAvailable = await maybeRunHistoricSqlSetupProbe({
|
||||
projectDir: input.projectDir,
|
||||
connectionId: input.connectionId,
|
||||
io: input.io,
|
||||
|
|
@ -1857,7 +1746,7 @@ async function validateAndScanConnection(input: {
|
|||
);
|
||||
}
|
||||
if (scanCode !== 0) {
|
||||
return 'failed';
|
||||
return queryHistoryAvailable ? 'failed' : 'failed-query-history-unavailable';
|
||||
}
|
||||
}
|
||||
const scanOutput = scanIo.stdoutText();
|
||||
|
|
@ -1999,7 +1888,10 @@ async function runPrimarySourceFullEdit(input: {
|
|||
const existing = project.config.connections[input.connectionId];
|
||||
const driver = normalizeDriver(existing?.driver);
|
||||
if (!existing || !driver) {
|
||||
input.io.stderr.write(`Connection "${input.connectionId}" is not a configured database.\n`);
|
||||
writePrefixedLines(
|
||||
(chunk) => input.io.stderr.write(chunk),
|
||||
`Connection "${input.connectionId}" is not a configured database.`,
|
||||
);
|
||||
return 'failed';
|
||||
}
|
||||
|
||||
|
|
@ -2053,7 +1945,7 @@ async function runPrimarySourceFullEdit(input: {
|
|||
});
|
||||
if (validated !== 'ready') {
|
||||
await rollback();
|
||||
return validated;
|
||||
return validated === 'failed-query-history-unavailable' ? 'failed' : validated;
|
||||
}
|
||||
return 'ready';
|
||||
}
|
||||
|
|
@ -2188,7 +2080,7 @@ export async function runKtxSetupDatabasesStep(
|
|||
prompts,
|
||||
});
|
||||
} catch (error) {
|
||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
writePrefixedLines((chunk) => io.stderr.write(chunk), errorMessage(error));
|
||||
return { status: 'failed', projectDir: args.projectDir };
|
||||
}
|
||||
if (connectionChoice === 'back') {
|
||||
|
|
@ -2332,14 +2224,18 @@ export async function runKtxSetupDatabasesStep(
|
|||
break;
|
||||
}
|
||||
if (args.inputMode === 'disabled') return { status: 'failed', projectDir: args.projectDir };
|
||||
const failureOptions = [
|
||||
{ value: 'retry', label: 'Retry connection test' },
|
||||
{ value: 're-enter', label: 'Re-enter connection details' },
|
||||
...(setupStatus === 'failed-query-history-unavailable'
|
||||
? [{ value: 'disable-query-history', label: 'Disable query history and retry' }]
|
||||
: []),
|
||||
{ value: 'skip', label: 'Skip this database' },
|
||||
{ value: 'back', label: 'Back' },
|
||||
];
|
||||
const action = await prompts.select({
|
||||
message: `Database setup failed for ${connectionChoice.connectionId}`,
|
||||
options: [
|
||||
{ value: 'retry', label: 'Retry connection test' },
|
||||
{ value: 're-enter', label: 'Re-enter connection details' },
|
||||
{ value: 'skip', label: 'Skip this database' },
|
||||
{ value: 'back', label: 'Back' },
|
||||
],
|
||||
options: failureOptions,
|
||||
});
|
||||
if (action === 'back') {
|
||||
if (!canReturnToDriverSelection) return { status: 'back', projectDir: args.projectDir };
|
||||
|
|
@ -2359,6 +2255,16 @@ export async function runKtxSetupDatabasesStep(
|
|||
args,
|
||||
prompts,
|
||||
});
|
||||
} else if (action === 'disable-query-history') {
|
||||
await disableConnectionQueryHistory(args.projectDir, connectionChoice.connectionId);
|
||||
setupStatus = await validateAndScanConnection({
|
||||
projectDir: args.projectDir,
|
||||
connectionId: connectionChoice.connectionId,
|
||||
io,
|
||||
deps,
|
||||
args,
|
||||
prompts,
|
||||
});
|
||||
} else if (action === 're-enter') {
|
||||
const connection = await buildConnectionConfig({
|
||||
driver,
|
||||
|
|
|
|||
|
|
@ -6,12 +6,13 @@ import { markKtxSetupStateStepComplete, readKtxSetupState } from './context/proj
|
|||
import type { KtxEmbeddingConfig } from './llm/types.js';
|
||||
import { type KtxEmbeddingHealthCheckResult, runKtxEmbeddingHealthCheck } from './llm/embedding-health.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { createStaticCliSpinner, type KtxCliSpinner } from './clack.js';
|
||||
import { createStaticCliSpinner, errorMessage, writePrefixedLines, type KtxCliSpinner } from './clack.js';
|
||||
import {
|
||||
ensureManagedLocalEmbeddingsDaemon,
|
||||
managedLocalEmbeddingHealthConfig,
|
||||
type ManagedLocalEmbeddingsDaemon,
|
||||
} from './managed-local-embeddings.js';
|
||||
import { ManagedPythonDaemonStartError } from './managed-python-daemon.js';
|
||||
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
|
||||
import { withTextInputNavigation } from './prompt-navigation.js';
|
||||
import { envCredentialReference, writeProjectLocalSecretReference } from './setup-secrets.js';
|
||||
|
|
@ -419,7 +420,13 @@ export async function runKtxSetupEmbeddingsStep(
|
|||
io,
|
||||
});
|
||||
} catch (error) {
|
||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
const write = (chunk: string) => io.stderr.write(chunk);
|
||||
if (error instanceof ManagedPythonDaemonStartError) {
|
||||
const tail = await readLocalEmbeddingDaemonStderrTail(error.stderrLog);
|
||||
writePrefixedLines(write, localEmbeddingSetupMessage(error.detail, tail));
|
||||
} else {
|
||||
writePrefixedLines(write, errorMessage(error));
|
||||
}
|
||||
return { status: 'failed', projectDir: args.projectDir };
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import { loadKtxProject, type KtxLocalProject } from './context/project/project.js';
|
||||
import { markKtxSetupStateStepComplete } from './context/project/setup-config.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { errorMessage, writePrefixedLines } from './clack.js';
|
||||
import {
|
||||
ensureManagedLocalEmbeddingsDaemon,
|
||||
type ManagedLocalEmbeddingsDaemon,
|
||||
|
|
@ -88,7 +89,7 @@ export async function runKtxSetupRuntimeStep(
|
|||
});
|
||||
}
|
||||
} catch (error) {
|
||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
writePrefixedLines((chunk) => io.stderr.write(chunk), errorMessage(error));
|
||||
return { status: 'failed', projectDir: args.projectDir, requirements };
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ import { type KtxProjectConfig, type KtxProjectConnectionConfig, serializeKtxPro
|
|||
import { loadKtxProject } from './context/project/project.js';
|
||||
import { markKtxSetupStateStepComplete } from './context/project/setup-config.js';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { errorMessage, writePrefixedLines } from './clack.js';
|
||||
import { pickNotionRootPages } from './notion-page-picker.js';
|
||||
import { runKtxSourceMapping } from './source-mapping.js';
|
||||
import { withMultiselectNavigation, withTextInputNavigation } from './prompt-navigation.js';
|
||||
|
|
@ -1983,7 +1984,7 @@ export async function runKtxSetupSourcesStep(
|
|||
return { status: 'ready', projectDir: args.projectDir, connectionIds: readyConnectionIds };
|
||||
}
|
||||
} catch (error) {
|
||||
io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
writePrefixedLines((chunk) => io.stderr.write(chunk), errorMessage(error));
|
||||
return { status: 'failed', projectDir: args.projectDir };
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -55,6 +55,10 @@ const emittedProjectSnapshots = new Set<string>();
|
|||
const MCP_SAMPLE_RATE = 0.1 as const;
|
||||
let mcpSampled: boolean | undefined;
|
||||
|
||||
function telemetryDebugEnabled(): boolean {
|
||||
return process.env.KTX_TELEMETRY_DEBUG === '1';
|
||||
}
|
||||
|
||||
export function shouldEmitMcpTelemetry(): boolean {
|
||||
mcpSampled ??= Math.random() < MCP_SAMPLE_RATE;
|
||||
return mcpSampled;
|
||||
|
|
@ -71,19 +75,21 @@ export async function emitTelemetryEvent<Name extends TelemetryEventName>(input:
|
|||
packageInfo?: KtxCliPackageInfo;
|
||||
projectDir?: string;
|
||||
}): Promise<void> {
|
||||
const debug = telemetryDebugEnabled();
|
||||
const identity = await loadTelemetryIdentity({
|
||||
stdoutIsTTY: input.io.stdout.isTTY === true,
|
||||
stderr: input.io.stderr,
|
||||
env: process.env,
|
||||
});
|
||||
|
||||
if (!identity.enabled || !identity.installId) {
|
||||
if ((!identity.enabled || !identity.installId) && !debug) {
|
||||
return;
|
||||
}
|
||||
|
||||
const packageInfo = input.packageInfo ?? getKtxCliPackageInfo();
|
||||
const installId = identity.installId ?? 'debug';
|
||||
|
||||
const projectId = input.projectDir ? computeTelemetryProjectId(identity.installId, input.projectDir) : undefined;
|
||||
const projectId = input.projectDir ? computeTelemetryProjectId(installId, input.projectDir) : undefined;
|
||||
await trackTelemetryEvent({
|
||||
event: buildTelemetryEvent(
|
||||
input.name,
|
||||
|
|
@ -93,7 +99,7 @@ export async function emitTelemetryEvent<Name extends TelemetryEventName>(input:
|
|||
}),
|
||||
input.fields,
|
||||
),
|
||||
distinctId: identity.installId,
|
||||
distinctId: installId,
|
||||
projectId,
|
||||
env: process.env,
|
||||
stderr: input.io.stderr,
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
import { createRequire } from 'node:module';
|
||||
|
||||
import type { ReindexSummary } from './context/index-sync/types.js';
|
||||
import type { ReindexSummary } from '../src/context/index-sync/types.js';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { renderReindexJson, renderReindexPlain, reindexHasErrors } from './admin-reindex.js';
|
||||
import { runKtxCli } from './index.js';
|
||||
import { renderReindexJson, renderReindexPlain, reindexHasErrors } from '../src/admin-reindex.js';
|
||||
import { runKtxCli } from '../src/index.js';
|
||||
|
||||
const cliVersion = (createRequire(import.meta.url)('@kaelio/ktx/package.json') as { version: string })
|
||||
.version;
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { runKtxCli } from './index.js';
|
||||
import { runKtxCli } from '../src/index.js';
|
||||
|
||||
function makeIo() {
|
||||
let stdout = '';
|
||||
|
|
@ -3,8 +3,8 @@ import { tmpdir } from 'node:os';
|
|||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { runCommanderKtxCli } from './cli-program.js';
|
||||
import type { KtxCliDeps, KtxCliIo, KtxCliPackageInfo } from './cli-runtime.js';
|
||||
import { runCommanderKtxCli } from '../src/cli-program.js';
|
||||
import type { KtxCliDeps, KtxCliIo, KtxCliPackageInfo } from '../src/cli-runtime.js';
|
||||
|
||||
function makeIo(stdoutIsTTY = true): { io: KtxCliIo; stdout: () => string; stderr: () => string } {
|
||||
let stdout = '';
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { Command, type CommandUnknownOpts } from '@commander-js/extra-typings';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { buildKtxProgram, collectCommandFlagsPresent } from './cli-program.js';
|
||||
import type { KtxCliIo, KtxCliPackageInfo } from './cli-runtime.js';
|
||||
import { buildKtxProgram, collectCommandFlagsPresent } from '../src/cli-program.js';
|
||||
import type { KtxCliIo, KtxCliPackageInfo } from '../src/cli-runtime.js';
|
||||
|
||||
function stubIo(): KtxCliIo {
|
||||
return {
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
import { Command } from '@commander-js/extra-typings';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { formatCommandTree, walkCommandTree } from './command-tree.js';
|
||||
import { formatCommandTree, walkCommandTree } from '../src/command-tree.js';
|
||||
|
||||
describe('walkCommandTree', () => {
|
||||
it('captures name, description, aliases, and nested children', () => {
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { Command } from '@commander-js/extra-typings';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import type { KtxCliCommandContext } from '../cli-program.js';
|
||||
import { registerMcpCommands } from './mcp-commands.js';
|
||||
import type { KtxCliCommandContext } from '../../src/cli-program.js';
|
||||
import { registerMcpCommands } from '../../src/commands/mcp-commands.js';
|
||||
|
||||
function makeContext(overrides: Partial<KtxCliCommandContext> = {}): KtxCliCommandContext {
|
||||
let exitCode = 0;
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { Command } from '@commander-js/extra-typings';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import type { KtxCliCommandContext } from '../cli-program.js';
|
||||
import { registerSqlCommands } from './sql-commands.js';
|
||||
import type { KtxCliCommandContext } from '../../src/cli-program.js';
|
||||
import { registerSqlCommands } from '../../src/commands/sql-commands.js';
|
||||
|
||||
function makeContext(overrides: Partial<KtxCliCommandContext> = {}): KtxCliCommandContext {
|
||||
let exitCode = 0;
|
||||
|
|
@ -1,14 +1,14 @@
|
|||
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import type { LookerClient } from './context/ingest/adapters/looker/client.js';
|
||||
import type { MetabaseRuntimeClient } from './context/ingest/adapters/metabase/client-port.js';
|
||||
import type { NotionClient } from './context/ingest/adapters/notion/notion-client.js';
|
||||
import { initKtxProject } from './context/project/project.js';
|
||||
import { parseKtxProjectConfig, serializeKtxProjectConfig } from './context/project/config.js';
|
||||
import type { KtxConnectionDriver, KtxScanConnector } from './context/scan/types.js';
|
||||
import type { LookerClient } from '../src/context/ingest/adapters/looker/client.js';
|
||||
import type { MetabaseRuntimeClient } from '../src/context/ingest/adapters/metabase/client-port.js';
|
||||
import type { NotionClient } from '../src/context/ingest/adapters/notion/notion-client.js';
|
||||
import { initKtxProject } from '../src/context/project/project.js';
|
||||
import { parseKtxProjectConfig, serializeKtxProjectConfig } from '../src/context/project/config.js';
|
||||
import type { KtxConnectionDriver, KtxScanConnector } from '../src/context/scan/types.js';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { runKtxConnection } from './connection.js';
|
||||
import { runKtxConnection } from '../src/connection.js';
|
||||
|
||||
function stripAnsi(s: string): string {
|
||||
return s.replace(/\[[0-9;]*m/g, '');
|
||||
|
|
@ -59,6 +59,8 @@ function nativeConnector(
|
|||
introspect: vi.fn(async () => {
|
||||
throw new Error('introspect should not be called from connection test');
|
||||
}),
|
||||
listSchemas: vi.fn(async () => []),
|
||||
listTables: vi.fn(async () => []),
|
||||
testConnection,
|
||||
cleanup,
|
||||
};
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { bigQueryConnectionConfigFromConfig, isKtxBigQueryConnectionConfig, type KtxBigQueryClient, KtxBigQueryScanConnector, type KtxBigQueryClientFactory, type KtxBigQueryDataset, type KtxBigQueryQueryJob, type KtxBigQueryTableRef } from '../../connectors/bigquery/connector.js';
|
||||
import { createBigQueryLiveDatabaseIntrospection } from '../../connectors/bigquery/live-database-introspection.js';
|
||||
import { tableRefSet } from '../../context/scan/table-ref.js';
|
||||
import { bigQueryConnectionConfigFromConfig, isKtxBigQueryConnectionConfig, type KtxBigQueryClient, KtxBigQueryScanConnector, type KtxBigQueryClientFactory, type KtxBigQueryDataset, type KtxBigQueryQueryJob, type KtxBigQueryTableRef, prepareBigQueryReadOnlyQuery } from '../../../src/connectors/bigquery/connector.js';
|
||||
import { createBigQueryLiveDatabaseIntrospection } from '../../../src/connectors/bigquery/live-database-introspection.js';
|
||||
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
|
||||
|
||||
function fakeClientFactory(options: { primaryKeyError?: Error } = {}): KtxBigQueryClientFactory {
|
||||
const queryResults = vi.fn(async (): ReturnType<KtxBigQueryQueryJob['getQueryResults']> => [
|
||||
|
|
@ -98,6 +98,17 @@ const connection = {
|
|||
} as const;
|
||||
|
||||
describe('KtxBigQueryScanConnector', () => {
|
||||
it('prepares read-only SQL parameters with BigQuery named placeholders', () => {
|
||||
expect(prepareBigQueryReadOnlyQuery('SELECT * FROM orders WHERE id = :id AND id_2 = :id_2', { id: 1, id_2: 2 })).toEqual({
|
||||
sql: 'SELECT * FROM orders WHERE id = @id AND id_2 = @id_2',
|
||||
params: { id: 1, id_2: 2 },
|
||||
});
|
||||
expect(prepareBigQueryReadOnlyQuery('SELECT * FROM orders')).toEqual({
|
||||
sql: 'SELECT * FROM orders',
|
||||
params: undefined,
|
||||
});
|
||||
});
|
||||
|
||||
it('resolves configuration safely', () => {
|
||||
expect(isKtxBigQueryConnectionConfig(connection)).toBe(true);
|
||||
expect(isKtxBigQueryConnectionConfig({ driver: 'mysql' })).toBe(false);
|
||||
|
|
@ -256,7 +267,7 @@ describe('KtxBigQueryScanConnector', () => {
|
|||
),
|
||||
).resolves.toEqual({ values: ['open', 'paid'], cardinality: 2 });
|
||||
await expect(connector.getTableRowCount('orders')).resolves.toBe(12);
|
||||
await expect(connector.listDatasets()).resolves.toEqual(['analytics', 'staging']);
|
||||
await expect(connector.listSchemas()).resolves.toEqual(['analytics', 'staging']);
|
||||
await expect(
|
||||
connector.columnStats(
|
||||
{ connectionId: 'warehouse', table: { catalog: 'project-1', db: 'analytics', name: 'orders' }, column: 'status' },
|
||||
|
|
@ -366,9 +377,9 @@ describe('KtxBigQueryScanConnector', () => {
|
|||
});
|
||||
|
||||
await expect(connector.listTables(['analytics', 'mart'])).resolves.toEqual([
|
||||
{ schema: 'analytics', name: 'orders', kind: 'table' },
|
||||
{ schema: 'analytics', name: 'order_clone', kind: 'table' },
|
||||
{ schema: 'mart', name: 'orders_mv', kind: 'view' },
|
||||
{ catalog: 'project-1', schema: 'analytics', name: 'orders', kind: 'table' },
|
||||
{ catalog: 'project-1', schema: 'analytics', name: 'order_clone', kind: 'table' },
|
||||
{ catalog: 'project-1', schema: 'mart', name: 'orders_mv', kind: 'view' },
|
||||
]);
|
||||
|
||||
expect(createQueryJob).toHaveBeenCalledTimes(1);
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { KtxBigQueryDialect } from './dialect.js';
|
||||
import { KtxBigQueryDialect } from '../../../src/connectors/bigquery/dialect.js';
|
||||
|
||||
describe('KtxBigQueryDialect', () => {
|
||||
const dialect = new KtxBigQueryDialect();
|
||||
|
|
@ -38,14 +38,6 @@ describe('KtxBigQueryDialect', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('rewrites colon parameters to BigQuery named parameters', () => {
|
||||
expect(dialect.prepareQuery('SELECT * FROM orders WHERE id = :id AND id_2 = :id_2', { id: 1, id_2: 2 })).toEqual({
|
||||
sql: 'SELECT * FROM orders WHERE id = @id AND id_2 = @id_2',
|
||||
params: { id: 1, id_2: 2 },
|
||||
});
|
||||
expect(dialect.prepareQuery('SELECT * FROM orders')).toEqual({ sql: 'SELECT * FROM orders', params: undefined });
|
||||
});
|
||||
|
||||
it('keeps unsupported statistics explicit', () => {
|
||||
expect(dialect.generateColumnStatisticsQuery('analytics', 'orders')).toBeNull();
|
||||
});
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { clickHouseClientConfigFromConfig, isKtxClickHouseConnectionConfig, KtxClickHouseScanConnector, type KtxClickHouseClientFactory } from '../../connectors/clickhouse/connector.js';
|
||||
import { createClickHouseLiveDatabaseIntrospection } from '../../connectors/clickhouse/live-database-introspection.js';
|
||||
import { tableRefSet } from '../../context/scan/table-ref.js';
|
||||
import { clickHouseClientConfigFromConfig, isKtxClickHouseConnectionConfig, KtxClickHouseScanConnector, prepareClickHouseReadOnlyQuery, type KtxClickHouseClientFactory } from '../../../src/connectors/clickhouse/connector.js';
|
||||
import { createClickHouseLiveDatabaseIntrospection } from '../../../src/connectors/clickhouse/live-database-introspection.js';
|
||||
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
|
||||
|
||||
function result<T>(payload: T) {
|
||||
return {
|
||||
|
|
@ -15,8 +15,8 @@ function fakeClientFactory(): KtxClickHouseClientFactory {
|
|||
const query = vi.fn(async (input: { query: string; format: string; query_params?: Record<string, unknown> }) => {
|
||||
if (input.query.includes('FROM system.tables')) {
|
||||
return result([
|
||||
{ name: 'events', engine: 'MergeTree', comment: 'Event stream' },
|
||||
{ name: 'event_summary', engine: 'View', comment: '' },
|
||||
{ database: 'analytics', name: 'event_summary', engine: 'View', comment: '' },
|
||||
{ database: 'analytics', name: 'events', engine: 'MergeTree', comment: 'Event stream' },
|
||||
]);
|
||||
}
|
||||
if (input.query.includes('FROM system.columns')) {
|
||||
|
|
@ -136,6 +136,33 @@ function multiDatabaseClickHouseClientFactory(): KtxClickHouseClientFactory {
|
|||
}
|
||||
|
||||
describe('KtxClickHouseScanConnector', () => {
|
||||
it('prepares read-only SQL parameters with ClickHouse typed placeholders', () => {
|
||||
expect(
|
||||
prepareClickHouseReadOnlyQuery('select * from events where id = :id and event_name = :name', {
|
||||
id: 10,
|
||||
name: 'signup',
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select * from events where id = {id:Int64} and event_name = {name:String}',
|
||||
params: { id: 10, name: 'signup' },
|
||||
});
|
||||
expect(
|
||||
prepareClickHouseReadOnlyQuery('select * from events where enabled = :enabled and ratio = :ratio and created_at = :created_at', {
|
||||
enabled: true,
|
||||
ratio: 1.5,
|
||||
created_at: new Date('2026-05-25T00:00:00.000Z'),
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select * from events where enabled = {enabled:Bool} and ratio = {ratio:Float64} and created_at = {created_at:DateTime}',
|
||||
params: {
|
||||
enabled: true,
|
||||
ratio: 1.5,
|
||||
created_at: new Date('2026-05-25T00:00:00.000Z'),
|
||||
},
|
||||
});
|
||||
expect(prepareClickHouseReadOnlyQuery('select 1')).toEqual({ sql: 'select 1', params: undefined });
|
||||
});
|
||||
|
||||
it('resolves ClickHouse connection configuration safely', () => {
|
||||
expect(isKtxClickHouseConnectionConfig({ driver: 'clickhouse', host: 'localhost', database: 'analytics' })).toBe(
|
||||
true,
|
||||
|
|
@ -196,8 +223,8 @@ describe('KtxClickHouseScanConnector', () => {
|
|||
},
|
||||
});
|
||||
expect(snapshot.tables.map((table) => [table.name, table.kind, table.estimatedRows, table.comment])).toEqual([
|
||||
['events', 'table', 2, 'Event stream'],
|
||||
['event_summary', 'view', null, null],
|
||||
['events', 'table', 2, 'Event stream'],
|
||||
]);
|
||||
expect(snapshot.tables.find((table) => table.name === 'events')?.columns[0]).toMatchObject({
|
||||
name: 'id',
|
||||
|
|
@ -344,6 +371,10 @@ describe('KtxClickHouseScanConnector', () => {
|
|||
|
||||
await expect(connector.getTableRowCount('events')).resolves.toBe(2);
|
||||
await expect(connector.listSchemas()).resolves.toEqual(['analytics', 'warehouse']);
|
||||
await expect(connector.listTables(['analytics'])).resolves.toEqual([
|
||||
{ catalog: null, schema: 'analytics', name: 'event_summary', kind: 'view' },
|
||||
{ catalog: null, schema: 'analytics', name: 'events', kind: 'table' },
|
||||
]);
|
||||
await expect(
|
||||
connector.columnStats(
|
||||
{ connectionId: 'warehouse', table: { catalog: null, db: 'analytics', name: 'events' }, column: 'event_name' },
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { KtxClickHouseDialect } from './dialect.js';
|
||||
import { KtxClickHouseDialect } from '../../../src/connectors/clickhouse/dialect.js';
|
||||
|
||||
describe('KtxClickHouseDialect', () => {
|
||||
const dialect = new KtxClickHouseDialect();
|
||||
|
|
@ -23,7 +23,7 @@ describe('KtxClickHouseDialect', () => {
|
|||
expect(dialect.mapToDimensionType('')).toBe('string');
|
||||
});
|
||||
|
||||
it('builds sampling, distinct-value, pagination, and time SQL', () => {
|
||||
it('builds sampling, distinct-value, and pagination SQL', () => {
|
||||
expect(dialect.generateSampleQuery('`analytics`.`events`', 25, ['id', 'event_name'])).toBe(
|
||||
'SELECT `id`, `event_name` FROM `analytics`.`events` LIMIT 25',
|
||||
);
|
||||
|
|
@ -34,16 +34,6 @@ describe('KtxClickHouseDialect', () => {
|
|||
'SELECT DISTINCT toString(`event_name`) AS val',
|
||||
);
|
||||
expect(dialect.getLimitOffsetClause(10, 20)).toBe('LIMIT 10 OFFSET 20');
|
||||
expect(dialect.getTimeTruncExpression('created_at', 'week')).toBe('toStartOfWeek(created_at, 1)');
|
||||
});
|
||||
|
||||
it('prepares named parameters using ClickHouse typed placeholders', () => {
|
||||
expect(dialect.prepareQuery('select * from events where id = :id and event_name = :name', {
|
||||
id: 10,
|
||||
name: 'signup',
|
||||
})).toEqual({
|
||||
sql: 'select * from events where id = {id:Int64} and event_name = {name:String}',
|
||||
params: { id: 10, name: 'signup' },
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import type { FieldPacket, RowDataPacket } from 'mysql2/promise';
|
||||
import { createMysqlLiveDatabaseIntrospection } from '../../connectors/mysql/live-database-introspection.js';
|
||||
import { isKtxMysqlConnectionConfig, KtxMysqlScanConnector, mysqlConnectionPoolConfigFromConfig, type KtxMysqlConnectionConfig, type KtxMysqlPoolFactory } from '../../connectors/mysql/connector.js';
|
||||
import { tableRefSet } from '../../context/scan/table-ref.js';
|
||||
import { createMysqlLiveDatabaseIntrospection } from '../../../src/connectors/mysql/live-database-introspection.js';
|
||||
import { isKtxMysqlConnectionConfig, KtxMysqlScanConnector, mysqlConnectionPoolConfigFromConfig, prepareMysqlReadOnlyQuery, type KtxMysqlConnectionConfig, type KtxMysqlPoolFactory } from '../../../src/connectors/mysql/connector.js';
|
||||
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
|
||||
|
||||
function mysqlResult(rows: Record<string, unknown>[], fields: Array<{ name: string; type?: number }>): [RowDataPacket[], FieldPacket[]] {
|
||||
return [rows as RowDataPacket[], fields as FieldPacket[]];
|
||||
|
|
@ -13,9 +13,9 @@ function fakePoolFactory(): KtxMysqlPoolFactory {
|
|||
if (sql.includes('INFORMATION_SCHEMA.TABLES')) {
|
||||
return mysqlResult(
|
||||
[
|
||||
{ TABLE_NAME: 'customers', TABLE_TYPE: 'BASE TABLE', TABLE_COMMENT: 'Customer table', TABLE_ROWS: 2 },
|
||||
{ TABLE_NAME: 'orders', TABLE_TYPE: 'BASE TABLE', TABLE_COMMENT: 'InnoDB free: 1 kB; Order table', TABLE_ROWS: 2 },
|
||||
{ TABLE_NAME: 'order_summary', TABLE_TYPE: 'VIEW', TABLE_COMMENT: '', TABLE_ROWS: null },
|
||||
{ TABLE_SCHEMA: 'analytics', TABLE_NAME: 'customers', TABLE_TYPE: 'BASE TABLE', TABLE_COMMENT: 'Customer table', TABLE_ROWS: 2 },
|
||||
{ TABLE_SCHEMA: 'analytics', TABLE_NAME: 'orders', TABLE_TYPE: 'BASE TABLE', TABLE_COMMENT: 'InnoDB free: 1 kB; Order table', TABLE_ROWS: 2 },
|
||||
{ TABLE_SCHEMA: 'analytics', TABLE_NAME: 'order_summary', TABLE_TYPE: 'VIEW', TABLE_COMMENT: '', TABLE_ROWS: null },
|
||||
],
|
||||
[{ name: 'TABLE_NAME' }, { name: 'TABLE_TYPE' }, { name: 'TABLE_COMMENT' }, { name: 'TABLE_ROWS' }],
|
||||
);
|
||||
|
|
@ -173,6 +173,19 @@ function multiSchemaMysqlPoolFactory(
|
|||
}
|
||||
|
||||
describe('KtxMysqlScanConnector', () => {
|
||||
it('prepares read-only SQL parameters with MySQL positional placeholders', () => {
|
||||
expect(
|
||||
prepareMysqlReadOnlyQuery('select * from orders where id = :id and status = :status', {
|
||||
status: 'paid',
|
||||
id: 10,
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select * from orders where id = ? and status = ?',
|
||||
params: [10, 'paid'],
|
||||
});
|
||||
expect(prepareMysqlReadOnlyQuery('select 1')).toEqual({ sql: 'select 1', params: undefined });
|
||||
});
|
||||
|
||||
it('resolves MySQL connection configuration safely', () => {
|
||||
expect(isKtxMysqlConnectionConfig({ driver: 'mysql', host: 'localhost', database: 'analytics' })).toBe(true);
|
||||
expect(isKtxMysqlConnectionConfig({ driver: 'postgres', host: 'localhost', database: 'analytics' })).toBe(false);
|
||||
|
|
@ -497,6 +510,11 @@ describe('KtxMysqlScanConnector', () => {
|
|||
|
||||
await expect(connector.getTableRowCount('orders')).resolves.toBe(2);
|
||||
await expect(connector.listSchemas()).resolves.toEqual(['analytics', 'warehouse']);
|
||||
await expect(connector.listTables(['analytics'])).resolves.toEqual([
|
||||
{ catalog: null, schema: 'analytics', name: 'customers', kind: 'table' },
|
||||
{ catalog: null, schema: 'analytics', name: 'orders', kind: 'table' },
|
||||
{ catalog: null, schema: 'analytics', name: 'order_summary', kind: 'view' },
|
||||
]);
|
||||
await expect(connector.columnStats(
|
||||
{ connectionId: 'warehouse', table: { catalog: null, db: 'analytics', name: 'orders' }, column: 'status' },
|
||||
{ runId: 'scan-run-1' },
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { KtxMysqlDialect } from './dialect.js';
|
||||
import { KtxMysqlDialect } from '../../../src/connectors/mysql/dialect.js';
|
||||
|
||||
describe('KtxMysqlDialect', () => {
|
||||
const dialect = new KtxMysqlDialect();
|
||||
|
|
@ -23,7 +23,7 @@ describe('KtxMysqlDialect', () => {
|
|||
expect(dialect.mapToDimensionType('')).toBe('string');
|
||||
});
|
||||
|
||||
it('builds sampling, distinct-value, pagination, and time SQL', () => {
|
||||
it('builds sampling, distinct-value, and pagination SQL', () => {
|
||||
expect(dialect.generateSampleQuery('`analytics`.`orders`', 25, ['id', 'status'])).toBe(
|
||||
'SELECT `id`, `status` FROM `analytics`.`orders` LIMIT 25',
|
||||
);
|
||||
|
|
@ -34,16 +34,6 @@ describe('KtxMysqlDialect', () => {
|
|||
'SELECT DISTINCT CAST(`status` AS CHAR) AS val',
|
||||
);
|
||||
expect(dialect.getLimitOffsetClause(10, 20)).toBe('LIMIT 10 OFFSET 20');
|
||||
expect(dialect.getTimeTruncExpression('created_at', 'month')).toBe("DATE_FORMAT(created_at, '%Y-%m-01')");
|
||||
});
|
||||
|
||||
it('prepares named parameters in deterministic SQL placeholder order', () => {
|
||||
expect(dialect.prepareQuery('select * from orders where id = :id and status = :status', {
|
||||
status: 'paid',
|
||||
id: 10,
|
||||
})).toEqual({
|
||||
sql: 'select * from orders where id = ? and status = ?',
|
||||
params: [10, 'paid'],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { createPostgresLiveDatabaseIntrospection } from '../../connectors/postgres/live-database-introspection.js';
|
||||
import { isKtxPostgresConnectionConfig, KtxPostgresScanConnector, postgresPoolConfigFromConfig, type KtxPostgresConnectionConfig, type KtxPostgresPoolFactory } from '../../connectors/postgres/connector.js';
|
||||
import { tableRefSet } from '../../context/scan/table-ref.js';
|
||||
import { createPostgresLiveDatabaseIntrospection } from '../../../src/connectors/postgres/live-database-introspection.js';
|
||||
import { isKtxPostgresConnectionConfig, KtxPostgresScanConnector, postgresPoolConfigFromConfig, preparePostgresReadOnlyQuery, type KtxPostgresConnectionConfig, type KtxPostgresPoolFactory } from '../../../src/connectors/postgres/connector.js';
|
||||
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
|
||||
|
||||
interface FakeQueryResult {
|
||||
rows: Record<string, unknown>[];
|
||||
|
|
@ -44,9 +44,9 @@ function metadataResults(): Map<string, FakeQueryResponse> {
|
|||
'FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n',
|
||||
{
|
||||
rows: [
|
||||
{ table_name: 'customers', table_kind: 'r', row_count: '2', table_comment: 'Customers' },
|
||||
{ table_name: 'orders', table_kind: 'r', row_count: '3', table_comment: null },
|
||||
{ table_name: 'recent_orders', table_kind: 'v', row_count: '0', table_comment: 'Recent orders' },
|
||||
{ schema_name: 'public', table_name: 'customers', table_kind: 'r', row_count: '2', table_comment: 'Customers' },
|
||||
{ schema_name: 'public', table_name: 'orders', table_kind: 'r', row_count: '3', table_comment: null },
|
||||
{ schema_name: 'public', table_name: 'recent_orders', table_kind: 'v', row_count: '0', table_comment: 'Recent orders' },
|
||||
],
|
||||
},
|
||||
],
|
||||
|
|
@ -102,6 +102,28 @@ function metadataResults(): Map<string, FakeQueryResponse> {
|
|||
}
|
||||
|
||||
describe('KtxPostgresScanConnector', () => {
|
||||
it('prepares read-only SQL parameters with PostgreSQL positional placeholders', () => {
|
||||
expect(
|
||||
preparePostgresReadOnlyQuery('select * from orders where id = :id and status = :status', {
|
||||
id: 1,
|
||||
status: 'paid',
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select * from orders where id = $1 and status = $2',
|
||||
params: [1, 'paid'],
|
||||
});
|
||||
expect(
|
||||
preparePostgresReadOnlyQuery('select :Client_Name_10, :Client_Name_1', {
|
||||
Client_Name_1: 'short',
|
||||
Client_Name_10: 'long',
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select $2, $1',
|
||||
params: ['short', 'long'],
|
||||
});
|
||||
expect(preparePostgresReadOnlyQuery('select 1')).toEqual({ sql: 'select 1', params: undefined });
|
||||
});
|
||||
|
||||
it('resolves configuration safely', () => {
|
||||
expect(isKtxPostgresConnectionConfig({ driver: 'postgres', url: 'env:DATABASE_URL' })).toBe(true);
|
||||
expect(isKtxPostgresConnectionConfig({ driver: 'postgresql', host: 'db', database: 'analytics' })).toBe(false);
|
||||
|
|
@ -367,6 +389,11 @@ describe('KtxPostgresScanConnector', () => {
|
|||
});
|
||||
await expect(connector.getTableRowCount({ db: 'public', name: 'orders' })).resolves.toBe(3);
|
||||
await expect(connector.listSchemas()).resolves.toEqual(['public']);
|
||||
await expect(connector.listTables(['public'])).resolves.toEqual([
|
||||
{ catalog: null, schema: 'public', name: 'customers', kind: 'table' },
|
||||
{ catalog: null, schema: 'public', name: 'orders', kind: 'table' },
|
||||
{ catalog: null, schema: 'public', name: 'recent_orders', kind: 'view' },
|
||||
]);
|
||||
await expect(connector.testConnection()).resolves.toEqual({ success: true });
|
||||
|
||||
await expect(
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { KtxPostgresDialect } from './dialect.js';
|
||||
import { KtxPostgresDialect } from '../../../src/connectors/postgres/dialect.js';
|
||||
|
||||
describe('KtxPostgresDialect', () => {
|
||||
const dialect = new KtxPostgresDialect();
|
||||
|
|
@ -18,7 +18,7 @@ describe('KtxPostgresDialect', () => {
|
|||
expect(dialect.mapToDimensionType('jsonb')).toBe('string');
|
||||
});
|
||||
|
||||
it('generates sample, distinct-value, statistics, and time SQL', () => {
|
||||
it('generates sample, distinct-value, and statistics SQL', () => {
|
||||
expect(dialect.generateSampleQuery('"public"."orders"', 5, ['id', 'status'])).toBe(
|
||||
'SELECT "id", "status" FROM "public"."orders" LIMIT 5',
|
||||
);
|
||||
|
|
@ -29,24 +29,6 @@ describe('KtxPostgresDialect', () => {
|
|||
'SELECT DISTINCT "status"::text AS val',
|
||||
);
|
||||
expect(dialect.generateColumnStatisticsQuery('public', 'orders')).toContain('FROM pg_stats s');
|
||||
expect(dialect.getTimeTruncExpression('"created_at"', 'month')).toBe('DATE_TRUNC(\'month\', "created_at")');
|
||||
});
|
||||
|
||||
it('prepares named parameters with PostgreSQL positional parameters', () => {
|
||||
expect(
|
||||
dialect.prepareQuery('select * from orders where id = :id and status = :status', { id: 1, status: 'paid' }),
|
||||
).toEqual({
|
||||
sql: 'select * from orders where id = $1 and status = $2',
|
||||
params: [1, 'paid'],
|
||||
});
|
||||
expect(
|
||||
dialect.prepareQuery('select :Client_Name_10, :Client_Name_1', {
|
||||
Client_Name_1: 'short',
|
||||
Client_Name_10: 'long',
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select $2, $1',
|
||||
params: ['short', 'long'],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { KtxPostgresHistoricSqlQueryClient } from './historic-sql-query-client.js';
|
||||
import type { KtxPostgresPoolConfig, KtxPostgresPoolFactory } from './connector.js';
|
||||
import { KtxPostgresHistoricSqlQueryClient } from '../../../src/connectors/postgres/historic-sql-query-client.js';
|
||||
import type { KtxPostgresPoolConfig, KtxPostgresPoolFactory } from '../../../src/connectors/postgres/connector.js';
|
||||
|
||||
describe('KtxPostgresHistoricSqlQueryClient', () => {
|
||||
it('executes parameterized read-only SQL through the native Postgres connector pool', async () => {
|
||||
|
|
@ -7,9 +7,9 @@ vi.mock('snowflake-sdk', () => ({
|
|||
createPool,
|
||||
}));
|
||||
|
||||
import { createSnowflakeLiveDatabaseIntrospection } from '../../connectors/snowflake/live-database-introspection.js';
|
||||
import { isKtxSnowflakeConnectionConfig, KtxSnowflakeScanConnector, snowflakeConnectionConfigFromConfig, type KtxSnowflakeConnectionConfig, type KtxSnowflakeDriver, type KtxSnowflakeDriverFactory } from '../../connectors/snowflake/connector.js';
|
||||
import { tableRefSet } from '../../context/scan/table-ref.js';
|
||||
import { createSnowflakeLiveDatabaseIntrospection } from '../../../src/connectors/snowflake/live-database-introspection.js';
|
||||
import { isKtxSnowflakeConnectionConfig, KtxSnowflakeScanConnector, prepareSnowflakeReadOnlyQuery, snowflakeConnectionConfigFromConfig, type KtxSnowflakeConnectionConfig, type KtxSnowflakeDriver, type KtxSnowflakeDriverFactory } from '../../../src/connectors/snowflake/connector.js';
|
||||
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
|
||||
|
||||
function fakeDriverFactory(): KtxSnowflakeDriverFactory {
|
||||
const driver: KtxSnowflakeDriver = {
|
||||
|
|
@ -64,8 +64,8 @@ function fakeDriverFactory(): KtxSnowflakeDriverFactory {
|
|||
]),
|
||||
listSchemas: vi.fn(async () => ['PUBLIC', 'MART']),
|
||||
listTables: vi.fn(async () => [
|
||||
{ schema: 'PUBLIC', name: 'ORDERS', kind: 'table' as const },
|
||||
{ schema: 'PUBLIC', name: 'ORDER_SUMMARY', kind: 'view' as const },
|
||||
{ catalog: 'ANALYTICS', schema: 'PUBLIC', name: 'ORDERS', kind: 'table' as const },
|
||||
{ catalog: 'ANALYTICS', schema: 'PUBLIC', name: 'ORDER_SUMMARY', kind: 'view' as const },
|
||||
]),
|
||||
cleanup: vi.fn(async () => undefined),
|
||||
};
|
||||
|
|
@ -105,6 +105,17 @@ function installSnowflakePoolMock() {
|
|||
}
|
||||
|
||||
describe('KtxSnowflakeScanConnector', () => {
|
||||
it('prepares read-only SQL parameters with Snowflake bind arrays', () => {
|
||||
expect(prepareSnowflakeReadOnlyQuery('SELECT * FROM ORDERS WHERE ID = ? AND STATUS = ?', { id: 1, status: 'paid' })).toEqual({
|
||||
sql: 'SELECT * FROM ORDERS WHERE ID = ? AND STATUS = ?',
|
||||
params: [1, 'paid'],
|
||||
});
|
||||
expect(prepareSnowflakeReadOnlyQuery('SELECT * FROM ORDERS')).toEqual({
|
||||
sql: 'SELECT * FROM ORDERS',
|
||||
params: undefined,
|
||||
});
|
||||
});
|
||||
|
||||
it('resolves Snowflake connection configuration safely', () => {
|
||||
expect(
|
||||
isKtxSnowflakeConnectionConfig({
|
||||
|
|
@ -561,8 +572,8 @@ describe('KtxSnowflakeScanConnector', () => {
|
|||
});
|
||||
|
||||
await expect(connector.listTables(['MART', 'PUBLIC'])).resolves.toEqual([
|
||||
{ schema: 'MART', name: 'ORDERS', kind: 'table' },
|
||||
{ schema: 'PUBLIC', name: 'ORDER_SUMMARY', kind: 'view' },
|
||||
{ catalog: 'ANALYTICS', schema: 'MART', name: 'ORDERS', kind: 'table' },
|
||||
{ catalog: 'ANALYTICS', schema: 'PUBLIC', name: 'ORDER_SUMMARY', kind: 'view' },
|
||||
]);
|
||||
|
||||
expect(queries).toHaveLength(1);
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { KtxSnowflakeDialect } from './dialect.js';
|
||||
import { KtxSnowflakeDialect } from '../../../src/connectors/snowflake/dialect.js';
|
||||
|
||||
describe('KtxSnowflakeDialect', () => {
|
||||
const dialect = new KtxSnowflakeDialect();
|
||||
|
|
@ -36,14 +36,6 @@ describe('KtxSnowflakeDialect', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('passes Snowflake positional parameters as bind arrays', () => {
|
||||
expect(dialect.prepareQuery('SELECT * FROM ORDERS WHERE ID = ? AND STATUS = ?', { id: 1, status: 'paid' })).toEqual({
|
||||
sql: 'SELECT * FROM ORDERS WHERE ID = ? AND STATUS = ?',
|
||||
params: [1, 'paid'],
|
||||
});
|
||||
expect(dialect.prepareQuery('SELECT * FROM ORDERS')).toEqual({ sql: 'SELECT * FROM ORDERS', params: undefined });
|
||||
});
|
||||
|
||||
it('keeps unsupported statistics explicit', () => {
|
||||
expect(dialect.generateColumnStatisticsQuery('PUBLIC', 'ORDERS')).toBeNull();
|
||||
});
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { assertSafeSnowflakeIdentifier, quoteSnowflakeIdentifier } from './identifiers.js';
|
||||
import { assertSafeSnowflakeIdentifier, quoteSnowflakeIdentifier } from '../../../src/connectors/snowflake/identifiers.js';
|
||||
|
||||
describe('Snowflake identifier guards', () => {
|
||||
it('quotes simple Snowflake identifiers', () => {
|
||||
|
|
@ -11,7 +11,7 @@ vi.mock('snowflake-sdk', () => ({
|
|||
import {
|
||||
configureSnowflakeSdkLogger,
|
||||
resetSnowflakeSdkLoggerConfigurationForTests,
|
||||
} from './sdk-logger.js';
|
||||
} from '../../../src/connectors/snowflake/sdk-logger.js';
|
||||
|
||||
describe('configureSnowflakeSdkLogger', () => {
|
||||
let projectDir: string;
|
||||
|
|
@ -4,9 +4,9 @@ import { mkdtemp, rm } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import { createSqliteLiveDatabaseIntrospection } from '../../connectors/sqlite/live-database-introspection.js';
|
||||
import { isKtxSqliteConnectionConfig, KtxSqliteScanConnector, sqliteDatabasePathFromConfig } from '../../connectors/sqlite/connector.js';
|
||||
import { tableRefSet } from '../../context/scan/table-ref.js';
|
||||
import { createSqliteLiveDatabaseIntrospection } from '../../../src/connectors/sqlite/live-database-introspection.js';
|
||||
import { isKtxSqliteConnectionConfig, KtxSqliteScanConnector, sqliteDatabasePathFromConfig } from '../../../src/connectors/sqlite/connector.js';
|
||||
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
|
||||
|
||||
describe('KtxSqliteScanConnector', () => {
|
||||
let tempDir: string;
|
||||
|
|
@ -150,6 +150,20 @@ describe('KtxSqliteScanConnector', () => {
|
|||
]);
|
||||
});
|
||||
|
||||
it('lists schemaless tables and views for setup discovery', async () => {
|
||||
const connector = new KtxSqliteScanConnector({
|
||||
connectionId: 'warehouse',
|
||||
connection: { driver: 'sqlite', path: dbPath },
|
||||
});
|
||||
|
||||
await expect(connector.listSchemas()).resolves.toEqual([]);
|
||||
await expect(connector.listTables(['ignored'])).resolves.toEqual([
|
||||
{ catalog: null, schema: '', name: 'customers', kind: 'table' },
|
||||
{ catalog: null, schema: '', name: 'orders', kind: 'table' },
|
||||
{ catalog: null, schema: '', name: 'recent_orders', kind: 'view' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('runs samples, distinct values, statistics, and read-only SQL', async () => {
|
||||
const connector = new KtxSqliteScanConnector({
|
||||
connectionId: 'warehouse',
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { KtxSqliteDialect } from './dialect.js';
|
||||
import { KtxSqliteDialect } from '../../../src/connectors/sqlite/dialect.js';
|
||||
|
||||
describe('KtxSqliteDialect', () => {
|
||||
const dialect = new KtxSqliteDialect();
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { createSqlServerLiveDatabaseIntrospection } from '../../connectors/sqlserver/live-database-introspection.js';
|
||||
import { isKtxSqlServerConnectionConfig, KtxSqlServerScanConnector, sqlServerConnectionPoolConfigFromConfig, type KtxSqlServerConnectionConfig, type KtxSqlServerPoolFactory, type KtxSqlServerQueryResult } from '../../connectors/sqlserver/connector.js';
|
||||
import { tableRefSet } from '../../context/scan/table-ref.js';
|
||||
import { createSqlServerLiveDatabaseIntrospection } from '../../../src/connectors/sqlserver/live-database-introspection.js';
|
||||
import { isKtxSqlServerConnectionConfig, KtxSqlServerScanConnector, prepareSqlServerReadOnlyQuery, sqlServerConnectionPoolConfigFromConfig, type KtxSqlServerConnectionConfig, type KtxSqlServerPoolFactory, type KtxSqlServerQueryResult } from '../../../src/connectors/sqlserver/connector.js';
|
||||
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
|
||||
|
||||
function recordset<T extends Record<string, unknown>>(
|
||||
rows: T[],
|
||||
|
|
@ -21,9 +21,9 @@ function fakePoolFactory(options: { primaryKeyError?: Error; foreignKeyError?: E
|
|||
if (sql.includes('INFORMATION_SCHEMA.TABLES')) {
|
||||
return result(
|
||||
[
|
||||
{ table_name: 'customers', table_type: 'BASE TABLE' },
|
||||
{ table_name: 'orders', table_type: 'BASE TABLE' },
|
||||
{ table_name: 'order_summary', table_type: 'VIEW' },
|
||||
{ schema_name: 'dbo', table_name: 'customers', table_type: 'BASE TABLE' },
|
||||
{ schema_name: 'dbo', table_name: 'orders', table_type: 'BASE TABLE' },
|
||||
{ schema_name: 'dbo', table_name: 'order_summary', table_type: 'VIEW' },
|
||||
],
|
||||
['table_name', 'table_type'],
|
||||
);
|
||||
|
|
@ -100,13 +100,13 @@ function fakePoolFactory(options: { primaryKeyError?: Error; foreignKeyError?: E
|
|||
['table_name', 'row_count'],
|
||||
);
|
||||
}
|
||||
if (sql.includes('SELECT TOP 1 [id], [status] FROM [dbo].[orders]')) {
|
||||
if (sql.includes('SELECT TOP 1 [id], [status] FROM [analytics].[dbo].[orders]')) {
|
||||
return result([{ id: 10, status: 'paid' }], ['id', 'status']);
|
||||
}
|
||||
if (sql.includes('SELECT TOP 1 * FROM (select id, status from dbo.orders) AS ktx_query_result')) {
|
||||
return result([{ id: 10, status: 'paid' }], ['id', 'status']);
|
||||
}
|
||||
if (sql.includes('SELECT TOP 5 [status] FROM [dbo].[orders]')) {
|
||||
if (sql.includes('SELECT TOP 5 [status] FROM [analytics].[dbo].[orders]')) {
|
||||
return result([{ status: 'paid' }, { status: 'open' }], ['status']);
|
||||
}
|
||||
if (sql.includes('COUNT(DISTINCT val)')) {
|
||||
|
|
@ -118,6 +118,16 @@ function fakePoolFactory(options: { primaryKeyError?: Error; foreignKeyError?: E
|
|||
if (sql.includes('SUM(p.rows) AS row_count') && sql.includes('t.name = @tableName')) {
|
||||
return result([{ row_count: 2 }], ['row_count']);
|
||||
}
|
||||
if (sql.includes('FROM sys.objects o')) {
|
||||
return result(
|
||||
[
|
||||
{ schema_name: 'dbo', table_name: 'customers', table_type: 'USER_TABLE' },
|
||||
{ schema_name: 'dbo', table_name: 'order_summary', table_type: 'VIEW' },
|
||||
{ schema_name: 'dbo', table_name: 'orders', table_type: 'USER_TABLE' },
|
||||
],
|
||||
['schema_name', 'table_name', 'table_type'],
|
||||
);
|
||||
}
|
||||
if (sql.includes('SELECT s.name AS schema_name')) {
|
||||
return result([{ schema_name: 'dbo' }, { schema_name: 'sales' }], ['schema_name']);
|
||||
}
|
||||
|
|
@ -140,6 +150,19 @@ function fakePoolFactory(options: { primaryKeyError?: Error; foreignKeyError?: E
|
|||
}
|
||||
|
||||
describe('KtxSqlServerScanConnector', () => {
|
||||
it('prepares read-only SQL parameters with SQL Server named placeholders', () => {
|
||||
expect(
|
||||
prepareSqlServerReadOnlyQuery('select * from events where id = :id and name = :name', {
|
||||
id: 10,
|
||||
name: 'signup',
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select * from events where id = @id and name = @name',
|
||||
params: { id: 10, name: 'signup' },
|
||||
});
|
||||
expect(prepareSqlServerReadOnlyQuery('select 1')).toEqual({ sql: 'select 1', params: undefined });
|
||||
});
|
||||
|
||||
it('resolves SQL Server connection configuration safely', () => {
|
||||
expect(
|
||||
isKtxSqlServerConnectionConfig({
|
||||
|
|
@ -366,6 +389,11 @@ describe('KtxSqlServerScanConnector', () => {
|
|||
|
||||
await expect(connector.getTableRowCount('orders')).resolves.toBe(2);
|
||||
await expect(connector.listSchemas()).resolves.toEqual(['dbo', 'sales']);
|
||||
await expect(connector.listTables(['dbo'])).resolves.toEqual([
|
||||
{ catalog: 'analytics', schema: 'dbo', name: 'customers', kind: 'table' },
|
||||
{ catalog: 'analytics', schema: 'dbo', name: 'order_summary', kind: 'view' },
|
||||
{ catalog: 'analytics', schema: 'dbo', name: 'orders', kind: 'table' },
|
||||
]);
|
||||
await expect(
|
||||
connector.columnStats(
|
||||
{ connectionId: 'warehouse', table: { catalog: 'analytics', db: 'dbo', name: 'orders' }, column: 'status' },
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { KtxSqlServerDialect } from './dialect.js';
|
||||
import { KtxSqlServerDialect } from '../../../src/connectors/sqlserver/dialect.js';
|
||||
|
||||
describe('KtxSqlServerDialect', () => {
|
||||
const dialect = new KtxSqlServerDialect();
|
||||
|
|
@ -7,7 +7,9 @@ describe('KtxSqlServerDialect', () => {
|
|||
it('quotes identifiers and formats schema-qualified table names', () => {
|
||||
expect(dialect.quoteIdentifier('events')).toBe('[events]');
|
||||
expect(dialect.quoteIdentifier('odd]name')).toBe('[odd]]name]');
|
||||
expect(dialect.formatTableName({ catalog: 'warehouse', db: 'dbo', name: 'events' })).toBe('[dbo].[events]');
|
||||
expect(dialect.formatTableName({ catalog: 'warehouse', db: 'dbo', name: 'events' })).toBe(
|
||||
'[warehouse].[dbo].[events]',
|
||||
);
|
||||
expect(dialect.formatTableName({ catalog: null, db: null, name: 'events' })).toBe('[events]');
|
||||
});
|
||||
|
||||
|
|
@ -20,7 +22,7 @@ describe('KtxSqlServerDialect', () => {
|
|||
expect(dialect.mapToDimensionType('')).toBe('string');
|
||||
});
|
||||
|
||||
it('builds sampling, distinct-value, pagination, and time SQL', () => {
|
||||
it('builds sampling, distinct-value, and pagination SQL', () => {
|
||||
expect(dialect.generateSampleQuery('[dbo].[events]', 25, ['id', 'event_name'])).toBe(
|
||||
'SELECT TOP 25 [id], [event_name] FROM [dbo].[events]',
|
||||
);
|
||||
|
|
@ -28,22 +30,8 @@ describe('KtxSqlServerDialect', () => {
|
|||
"SELECT TOP 10 [event_name] FROM [dbo].[events] WHERE [event_name] IS NOT NULL AND LTRIM(RTRIM(CAST([event_name] AS NVARCHAR(MAX)))) != ''",
|
||||
);
|
||||
expect(dialect.generateDistinctValuesQuery('[dbo].[events]', '[event_name]', 5)).toContain('SELECT TOP 5 val');
|
||||
expect(dialect.getTopClause(10)).toBe('TOP 10');
|
||||
expect(dialect.getLimitOffsetClause(10, 20)).toBe('OFFSET 20 ROWS FETCH NEXT 10 ROWS ONLY');
|
||||
expect(dialect.getTimeTruncExpression('created_at', 'month')).toBe(
|
||||
'DATEFROMPARTS(YEAR(created_at), MONTH(created_at), 1)',
|
||||
);
|
||||
expect(dialect.getTopClause(10)).toBe('TOP (10)');
|
||||
expect(dialect.getLimitOffsetClause(10, 20)).toBe('');
|
||||
});
|
||||
|
||||
it('prepares named parameters using SQL Server @ parameters', () => {
|
||||
expect(
|
||||
dialect.prepareQuery('select * from events where id = :id and name = :name', {
|
||||
id: 10,
|
||||
name: 'signup',
|
||||
}),
|
||||
).toEqual({
|
||||
sql: 'select * from events where id = @id and name = @name',
|
||||
params: { id: 10, name: 'signup' },
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
import { buildDefaultKtxProjectConfig, type KtxProjectConfig } from './context/project/config.js';
|
||||
import { buildDefaultKtxProjectConfig, type KtxProjectConfig } from '../src/context/project/config.js';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import type { KtxPublicIngestProject, KtxPublicIngestTargetResult } from './public-ingest.js';
|
||||
import type { KtxPublicIngestProject, KtxPublicIngestTargetResult } from '../src/public-ingest.js';
|
||||
import {
|
||||
type ContextBuildTargetState,
|
||||
extractProgressMessage,
|
||||
|
|
@ -11,7 +11,7 @@ import {
|
|||
renderContextBuildView,
|
||||
runContextBuild,
|
||||
viewStateFromSourceProgress,
|
||||
} from './context-build-view.js';
|
||||
} from '../src/context-build-view.js';
|
||||
|
||||
function makeIo(options: { isTTY?: boolean; columns?: number } = {}) {
|
||||
let stdout = '';
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { normalizeBigQueryProjectId, normalizeBigQueryRegion } from './bigquery-identifiers.js';
|
||||
import { normalizeBigQueryProjectId, normalizeBigQueryRegion } from '../../../src/context/connections/bigquery-identifiers.js';
|
||||
|
||||
describe('BigQuery identifier normalization', () => {
|
||||
it('normalizes project ids and regions for information schema paths', () => {
|
||||
316
packages/cli/test/context/connections/dialects.test.ts
Normal file
316
packages/cli/test/context/connections/dialects.test.ts
Normal file
|
|
@ -0,0 +1,316 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
|
||||
import type { KtxConnectionDriver, KtxTableRef } from '../../../src/context/scan/types.js';
|
||||
|
||||
interface DialectFixture {
|
||||
driver: KtxConnectionDriver;
|
||||
table: KtxTableRef;
|
||||
quoteInput: string;
|
||||
quotedIdentifier: string;
|
||||
formattedTable: string;
|
||||
display: string;
|
||||
invalidDisplay: string;
|
||||
columnDisplayTablePartCount: 1 | 2 | 3;
|
||||
limitClause: string;
|
||||
topClause: string;
|
||||
randomFilter: string;
|
||||
tableSampleClause: string;
|
||||
sampleQuery: string;
|
||||
columnSampleContains: string;
|
||||
nullCountExpression: string;
|
||||
distinctCountExpression: string;
|
||||
textLengthExpression: string;
|
||||
castToText: string;
|
||||
sampleValueAggregation: string;
|
||||
cardinalityContains: string;
|
||||
randomizedCardinalityContains: string;
|
||||
distinctValuesContains: string;
|
||||
statisticsContains: string | null;
|
||||
dimensionInput: string;
|
||||
dimensionType: 'time' | 'string' | 'number' | 'boolean';
|
||||
nativeTypeInput: string;
|
||||
normalizedType: string;
|
||||
}
|
||||
|
||||
const innerSampleSql = 'SELECT status AS value FROM orders';
|
||||
|
||||
const fixtures: DialectFixture[] = [
|
||||
{
|
||||
driver: 'postgres',
|
||||
table: { catalog: null, db: 'public', name: 'orders' },
|
||||
quoteInput: 'order"items',
|
||||
quotedIdentifier: '"order""items"',
|
||||
formattedTable: '"public"."orders"',
|
||||
display: 'public.orders',
|
||||
invalidDisplay: 'orders',
|
||||
columnDisplayTablePartCount: 2,
|
||||
limitClause: 'LIMIT 25 OFFSET 5',
|
||||
topClause: '',
|
||||
randomFilter: 'RANDOM() < 0.25',
|
||||
tableSampleClause: 'TABLESAMPLE SYSTEM (25)',
|
||||
sampleQuery: 'SELECT "id", "status" FROM "public"."orders" LIMIT 5',
|
||||
columnSampleContains: 'TRIM(CAST("status" AS TEXT)) != \'\'',
|
||||
nullCountExpression: 'COUNT(*) FILTER (WHERE "status" IS NULL)',
|
||||
distinctCountExpression: 'COUNT(DISTINCT "status")',
|
||||
textLengthExpression: 'LENGTH(CAST("status" AS TEXT))',
|
||||
castToText: 'CAST("status" AS TEXT)',
|
||||
sampleValueAggregation:
|
||||
'(SELECT STRING_AGG(CAST(value AS TEXT), CHR(31)) FROM (SELECT status AS value FROM orders) AS relationship_profile_values)',
|
||||
cardinalityContains: 'SELECT COUNT(DISTINCT val) AS cardinality',
|
||||
randomizedCardinalityContains: 'ORDER BY RANDOM()',
|
||||
distinctValuesContains: 'SELECT DISTINCT "status"::text AS val',
|
||||
statisticsContains: 'FROM pg_stats s',
|
||||
dimensionInput: 'timestamp with time zone',
|
||||
dimensionType: 'time',
|
||||
nativeTypeInput: 'numeric(12,2)',
|
||||
normalizedType: 'numeric(12,2)',
|
||||
},
|
||||
{
|
||||
driver: 'mysql',
|
||||
table: { catalog: null, db: 'analytics', name: 'orders' },
|
||||
quoteInput: 'order`items',
|
||||
quotedIdentifier: '`order``items`',
|
||||
formattedTable: '`analytics`.`orders`',
|
||||
display: 'analytics.orders',
|
||||
invalidDisplay: 'orders',
|
||||
columnDisplayTablePartCount: 2,
|
||||
limitClause: 'LIMIT 25 OFFSET 5',
|
||||
topClause: '',
|
||||
randomFilter: 'RAND() < 0.25',
|
||||
tableSampleClause: '',
|
||||
sampleQuery: 'SELECT `id`, `status` FROM `analytics`.`orders` LIMIT 5',
|
||||
columnSampleContains: 'TRIM(CAST(`status` AS CHAR)) != \'\'',
|
||||
nullCountExpression: 'SUM(CASE WHEN `status` IS NULL THEN 1 ELSE 0 END)',
|
||||
distinctCountExpression: 'COUNT(DISTINCT `status`)',
|
||||
textLengthExpression: 'CHAR_LENGTH(CAST(`status` AS CHAR))',
|
||||
castToText: 'CAST(`status` AS CHAR)',
|
||||
sampleValueAggregation:
|
||||
'(SELECT GROUP_CONCAT(CAST(value AS CHAR) SEPARATOR CHAR(31)) FROM (SELECT status AS value FROM orders) AS relationship_profile_values)',
|
||||
cardinalityContains: 'SELECT COUNT(DISTINCT val) AS cardinality',
|
||||
randomizedCardinalityContains: 'ORDER BY RAND()',
|
||||
distinctValuesContains: 'SELECT DISTINCT CAST(`status` AS CHAR) AS val',
|
||||
statisticsContains: null,
|
||||
dimensionInput: 'tinyint(1)',
|
||||
dimensionType: 'boolean',
|
||||
nativeTypeInput: 'varchar(255)',
|
||||
normalizedType: 'varchar(255)',
|
||||
},
|
||||
{
|
||||
driver: 'clickhouse',
|
||||
table: { catalog: null, db: 'analytics', name: 'events' },
|
||||
quoteInput: 'order`items',
|
||||
quotedIdentifier: '`order``items`',
|
||||
formattedTable: '`analytics`.`events`',
|
||||
display: 'analytics.events',
|
||||
invalidDisplay: 'events',
|
||||
columnDisplayTablePartCount: 2,
|
||||
limitClause: 'LIMIT 25 OFFSET 5',
|
||||
topClause: '',
|
||||
randomFilter: 'rand() / 4294967295.0 < 0.25',
|
||||
tableSampleClause: '',
|
||||
sampleQuery: 'SELECT `id`, `status` FROM `analytics`.`events` LIMIT 5',
|
||||
columnSampleContains: 'trim(toString(`status`)) != \'\'',
|
||||
nullCountExpression: 'countIf(`status` IS NULL)',
|
||||
distinctCountExpression: 'COUNT(DISTINCT `status`)',
|
||||
textLengthExpression: 'length(toString(`status`))',
|
||||
castToText: 'toString(`status`)',
|
||||
sampleValueAggregation:
|
||||
'(SELECT arrayStringConcat(groupArray(toString(value)), \'\\x1F\') FROM (SELECT status AS value FROM orders) AS relationship_profile_values)',
|
||||
cardinalityContains: 'SELECT COUNT(DISTINCT val) AS cardinality',
|
||||
randomizedCardinalityContains: 'ORDER BY rand()',
|
||||
distinctValuesContains: 'SELECT DISTINCT toString(`status`) AS val',
|
||||
statisticsContains: null,
|
||||
dimensionInput: 'Nullable(DateTime64(3))',
|
||||
dimensionType: 'time',
|
||||
nativeTypeInput: 'LowCardinality(String)',
|
||||
normalizedType: 'LowCardinality(String)',
|
||||
},
|
||||
{
|
||||
driver: 'sqlite',
|
||||
table: { catalog: null, db: null, name: 'orders' },
|
||||
quoteInput: 'order"items',
|
||||
quotedIdentifier: '"order""items"',
|
||||
formattedTable: '"orders"',
|
||||
display: 'orders',
|
||||
invalidDisplay: 'public.orders',
|
||||
columnDisplayTablePartCount: 1,
|
||||
limitClause: 'LIMIT 25 OFFSET 5',
|
||||
topClause: '',
|
||||
randomFilter: '(RANDOM() % 100) < 25',
|
||||
tableSampleClause: '',
|
||||
sampleQuery: 'SELECT "id", "status" FROM "orders" LIMIT 5',
|
||||
columnSampleContains: 'TRIM(CAST("status" AS TEXT)) != \'\'',
|
||||
nullCountExpression: 'SUM(CASE WHEN "status" IS NULL THEN 1 ELSE 0 END)',
|
||||
distinctCountExpression: 'COUNT(DISTINCT "status")',
|
||||
textLengthExpression: 'LENGTH(CAST("status" AS TEXT))',
|
||||
castToText: 'CAST("status" AS TEXT)',
|
||||
sampleValueAggregation:
|
||||
'(SELECT GROUP_CONCAT(CAST(value AS TEXT), char(31)) FROM (SELECT status AS value FROM orders) AS relationship_profile_values)',
|
||||
cardinalityContains: 'SELECT COUNT(DISTINCT val) AS cardinality',
|
||||
randomizedCardinalityContains: 'ORDER BY RANDOM()',
|
||||
distinctValuesContains: 'SELECT DISTINCT CAST("status" AS TEXT) AS val',
|
||||
statisticsContains: null,
|
||||
dimensionInput: 'INTEGER',
|
||||
dimensionType: 'number',
|
||||
nativeTypeInput: 'VARCHAR(255)',
|
||||
normalizedType: 'VARCHAR(255)',
|
||||
},
|
||||
{
|
||||
driver: 'snowflake',
|
||||
table: { catalog: 'ANALYTICS', db: 'PUBLIC', name: 'ORDERS' },
|
||||
quoteInput: 'order"items',
|
||||
quotedIdentifier: '"order""items"',
|
||||
formattedTable: '"ANALYTICS"."PUBLIC"."ORDERS"',
|
||||
display: 'ANALYTICS.PUBLIC.ORDERS',
|
||||
invalidDisplay: 'PUBLIC.ORDERS',
|
||||
columnDisplayTablePartCount: 3,
|
||||
limitClause: 'LIMIT 25 OFFSET 5',
|
||||
topClause: '',
|
||||
randomFilter: 'UNIFORM(0::FLOAT, 1::FLOAT, RANDOM()) < 0.25',
|
||||
tableSampleClause: 'SAMPLE (25)',
|
||||
sampleQuery: 'SELECT "id", "status" FROM "ANALYTICS"."PUBLIC"."ORDERS" SAMPLE ROW (5 ROWS)',
|
||||
columnSampleContains: 'TRIM(CAST("status" AS STRING)) != \'\'',
|
||||
nullCountExpression: 'COUNT_IF("status" IS NULL)',
|
||||
distinctCountExpression: 'APPROX_COUNT_DISTINCT("status")',
|
||||
textLengthExpression: 'LENGTH(CAST("status" AS TEXT))',
|
||||
castToText: 'CAST("status" AS VARCHAR)',
|
||||
sampleValueAggregation:
|
||||
'(SELECT LISTAGG(CAST(value AS VARCHAR), \'\\x1f\') FROM (SELECT status AS value FROM orders) AS relationship_profile_values)',
|
||||
cardinalityContains: 'SELECT COUNT(DISTINCT val) AS cardinality',
|
||||
randomizedCardinalityContains: 'SAMPLE ROW (100 ROWS)',
|
||||
distinctValuesContains: 'SELECT DISTINCT "status"::VARCHAR AS val',
|
||||
statisticsContains: null,
|
||||
dimensionInput: 'TIMESTAMP_NTZ',
|
||||
dimensionType: 'time',
|
||||
nativeTypeInput: 'NUMBER(38,0)',
|
||||
normalizedType: 'NUMBER(38,0)',
|
||||
},
|
||||
{
|
||||
driver: 'bigquery',
|
||||
table: { catalog: 'analytics-project', db: 'warehouse', name: 'orders' },
|
||||
quoteInput: 'order`items',
|
||||
quotedIdentifier: '`order\\`items`',
|
||||
formattedTable: '`analytics-project`.`warehouse`.`orders`',
|
||||
display: 'analytics-project.warehouse.orders',
|
||||
invalidDisplay: 'warehouse.orders',
|
||||
columnDisplayTablePartCount: 3,
|
||||
limitClause: 'LIMIT 25 OFFSET 5',
|
||||
topClause: '',
|
||||
randomFilter: 'RAND() < 0.25',
|
||||
tableSampleClause: 'TABLESAMPLE SYSTEM (25 PERCENT)',
|
||||
sampleQuery: 'SELECT `id`, `status` FROM `analytics-project`.`warehouse`.`orders` ORDER BY RAND() LIMIT 5',
|
||||
columnSampleContains: 'TRIM(CAST(`status` AS STRING)) != \'\'',
|
||||
nullCountExpression: 'COUNTIF(`status` IS NULL)',
|
||||
distinctCountExpression: 'APPROX_COUNT_DISTINCT(`status`)',
|
||||
textLengthExpression: 'LENGTH(CAST(`status` AS STRING))',
|
||||
castToText: 'CAST(`status` AS STRING)',
|
||||
sampleValueAggregation:
|
||||
'(SELECT STRING_AGG(CAST(value AS STRING), \'\\u001F\') FROM (SELECT status AS value FROM orders) AS relationship_profile_values)',
|
||||
cardinalityContains: 'SELECT APPROX_COUNT_DISTINCT(val) AS cardinality',
|
||||
randomizedCardinalityContains: 'ORDER BY RAND()',
|
||||
distinctValuesContains: 'SELECT DISTINCT CAST(`status` AS STRING) AS val',
|
||||
statisticsContains: null,
|
||||
dimensionInput: 'INT64',
|
||||
dimensionType: 'number',
|
||||
nativeTypeInput: 'INT64',
|
||||
normalizedType: 'BIGINT',
|
||||
},
|
||||
{
|
||||
driver: 'sqlserver',
|
||||
table: { catalog: 'warehouse', db: 'dbo', name: 'events' },
|
||||
quoteInput: 'odd]name',
|
||||
quotedIdentifier: '[odd]]name]',
|
||||
formattedTable: '[warehouse].[dbo].[events]',
|
||||
display: 'warehouse.dbo.events',
|
||||
invalidDisplay: 'dbo.events',
|
||||
columnDisplayTablePartCount: 3,
|
||||
limitClause: '',
|
||||
topClause: 'TOP (25)',
|
||||
randomFilter: 'ABS(CHECKSUM(NEWID())) % 100 < 25',
|
||||
tableSampleClause: 'TABLESAMPLE (25 PERCENT)',
|
||||
sampleQuery: 'SELECT TOP 5 [id], [status] FROM [warehouse].[dbo].[events]',
|
||||
columnSampleContains: 'LTRIM(RTRIM(CAST([status] AS NVARCHAR(MAX)))) != \'\'',
|
||||
nullCountExpression: 'SUM(CASE WHEN [status] IS NULL THEN 1 ELSE 0 END)',
|
||||
distinctCountExpression: 'COUNT(DISTINCT [status])',
|
||||
textLengthExpression: 'LEN(CAST([status] AS NVARCHAR(MAX)))',
|
||||
castToText: 'CAST([status] AS NVARCHAR(MAX))',
|
||||
sampleValueAggregation:
|
||||
'(SELECT STRING_AGG(CAST(value AS NVARCHAR(MAX)), CHAR(31)) FROM (SELECT status AS value FROM orders) AS relationship_profile_values)',
|
||||
cardinalityContains: 'SELECT COUNT(DISTINCT val) AS cardinality',
|
||||
randomizedCardinalityContains: 'ORDER BY NEWID()',
|
||||
distinctValuesContains: 'SELECT TOP 20 val',
|
||||
statisticsContains: null,
|
||||
dimensionInput: 'datetime2',
|
||||
dimensionType: 'time',
|
||||
nativeTypeInput: 'uniqueidentifier',
|
||||
normalizedType: 'uniqueidentifier',
|
||||
},
|
||||
];
|
||||
|
||||
describe('getDialectForDriver', () => {
|
||||
it.each(fixtures)('returns a full KtxDialect for $driver', (fixture) => {
|
||||
const dialect = getDialectForDriver(fixture.driver);
|
||||
const column = dialect.quoteIdentifier('status');
|
||||
|
||||
expect(dialect.type).toBe(fixture.driver);
|
||||
expect(dialect.quoteIdentifier(fixture.quoteInput)).toBe(fixture.quotedIdentifier);
|
||||
expect(dialect.formatTableName(fixture.table)).toBe(fixture.formattedTable);
|
||||
expect(dialect.formatDisplayRef(fixture.table)).toBe(fixture.display);
|
||||
expect(dialect.parseDisplayRef(fixture.display)).toEqual(fixture.table);
|
||||
expect(dialect.parseDisplayRef(fixture.invalidDisplay)).toBeNull();
|
||||
expect(dialect.columnDisplayTablePartCount()).toBe(fixture.columnDisplayTablePartCount);
|
||||
expect(dialect.getLimitOffsetClause(25, 5)).toBe(fixture.limitClause);
|
||||
expect(dialect.getTopClause(25)).toBe(fixture.topClause);
|
||||
expect(dialect.getRandomSampleFilter(0.25)).toBe(fixture.randomFilter);
|
||||
expect(dialect.getTableSampleClause(0.25)).toBe(fixture.tableSampleClause);
|
||||
expect(dialect.generateSampleQuery(fixture.formattedTable, 5, ['id', 'status'])).toBe(fixture.sampleQuery);
|
||||
expect(dialect.generateColumnSampleQuery(fixture.formattedTable, 'status', 10)).toContain(
|
||||
fixture.columnSampleContains,
|
||||
);
|
||||
expect(dialect.getNullCountExpression(column)).toBe(fixture.nullCountExpression);
|
||||
expect(dialect.getDistinctCountExpression(column)).toBe(fixture.distinctCountExpression);
|
||||
expect(dialect.textLengthExpression(column)).toBe(fixture.textLengthExpression);
|
||||
expect(dialect.castToText(column)).toBe(fixture.castToText);
|
||||
expect(dialect.getSampleValueAggregation(innerSampleSql)).toBe(fixture.sampleValueAggregation);
|
||||
expect(dialect.generateCardinalitySampleQuery(fixture.formattedTable, column, 100)).toContain(
|
||||
fixture.cardinalityContains,
|
||||
);
|
||||
expect(dialect.generateRandomizedCardinalitySampleQuery(fixture.formattedTable, column, 100)).toContain(
|
||||
fixture.randomizedCardinalityContains,
|
||||
);
|
||||
expect(dialect.generateDistinctValuesQuery(fixture.formattedTable, column, 20)).toContain(
|
||||
fixture.distinctValuesContains,
|
||||
);
|
||||
const statistics = dialect.generateColumnStatisticsQuery(fixture.table.db ?? '', fixture.table.name);
|
||||
if (fixture.statisticsContains) {
|
||||
expect(statistics).toContain(fixture.statisticsContains);
|
||||
} else {
|
||||
expect(statistics).toBeNull();
|
||||
}
|
||||
expect(dialect.mapToDimensionType(fixture.dimensionInput)).toBe(fixture.dimensionType);
|
||||
expect(dialect.mapDataType(fixture.nativeTypeInput)).toBe(fixture.normalizedType);
|
||||
});
|
||||
|
||||
it('accepts three-part ANSI display refs while keeping one-part names caller-owned', () => {
|
||||
for (const driver of ['postgres', 'mysql', 'clickhouse'] as const) {
|
||||
const dialect = getDialectForDriver(driver);
|
||||
expect(dialect.parseDisplayRef('warehouse.public.orders')).toEqual({
|
||||
catalog: 'warehouse',
|
||||
db: 'public',
|
||||
name: 'orders',
|
||||
});
|
||||
expect(dialect.parseDisplayRef('orders')).toBeNull();
|
||||
}
|
||||
});
|
||||
|
||||
it('throws with a supported-driver list for unknown drivers', () => {
|
||||
expect(() => getDialectForDriver('oracle')).toThrow(
|
||||
'Unsupported warehouse driver "oracle". Supported drivers: bigquery, clickhouse, mysql, postgres, sqlite, snowflake, sqlserver',
|
||||
);
|
||||
});
|
||||
|
||||
it('rejects legacy driver aliases', () => {
|
||||
expect(() => getDialectForDriver('postgresql')).toThrow('Unsupported warehouse driver "postgresql"');
|
||||
expect(() => getDialectForDriver('sqlite3')).toThrow('Unsupported warehouse driver "sqlite3"');
|
||||
});
|
||||
});
|
||||
145
packages/cli/test/context/connections/drivers.test.ts
Normal file
145
packages/cli/test/context/connections/drivers.test.ts
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
import { mkdtemp, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import {
|
||||
driverRegistrations,
|
||||
getDriverRegistration,
|
||||
listSupportedDrivers,
|
||||
} from '../../../src/context/connections/drivers.js';
|
||||
import type {
|
||||
KtxDriverConnectorModule,
|
||||
KtxScopeConfigKey,
|
||||
} from '../../../src/context/connections/drivers.js';
|
||||
import type { KtxConnectionDriver } from '../../../src/context/scan/types.js';
|
||||
|
||||
type FixtureFactory = (projectDir: string) => Record<string, unknown>;
|
||||
|
||||
const connectionFixtures: Record<KtxConnectionDriver, FixtureFactory> = {
|
||||
postgres: () => ({
|
||||
driver: 'postgres',
|
||||
url: 'postgresql://reader:secret@localhost:5432/analytics', // pragma: allowlist secret
|
||||
schemas: ['public'],
|
||||
}),
|
||||
sqlite: () => ({ driver: 'sqlite', path: 'warehouse.db' }),
|
||||
mysql: () => ({
|
||||
driver: 'mysql',
|
||||
host: 'localhost',
|
||||
database: 'analytics',
|
||||
username: 'reader',
|
||||
password: 'secret', // pragma: allowlist secret
|
||||
schemas: ['analytics'],
|
||||
}),
|
||||
clickhouse: () => ({
|
||||
driver: 'clickhouse',
|
||||
url: 'http://localhost:8123',
|
||||
database: 'analytics',
|
||||
username: 'reader',
|
||||
password: 'secret', // pragma: allowlist secret
|
||||
}),
|
||||
sqlserver: () => ({
|
||||
driver: 'sqlserver',
|
||||
host: 'localhost',
|
||||
database: 'analytics',
|
||||
username: 'reader',
|
||||
password: 'secret', // pragma: allowlist secret
|
||||
schemas: ['dbo'],
|
||||
}),
|
||||
bigquery: () => ({
|
||||
driver: 'bigquery',
|
||||
dataset_id: 'analytics',
|
||||
credentials_json: JSON.stringify({
|
||||
project_id: 'project-1',
|
||||
client_email: 'reader@example.test',
|
||||
private_key: '-----BEGIN PRIVATE KEY-----\nsecret\n-----END PRIVATE KEY-----\n', // pragma: allowlist secret
|
||||
}),
|
||||
location: 'US',
|
||||
}),
|
||||
snowflake: () => ({
|
||||
driver: 'snowflake',
|
||||
account: 'example-account',
|
||||
username: 'reader',
|
||||
password: 'secret', // pragma: allowlist secret
|
||||
warehouse: 'COMPUTE_WH',
|
||||
database: 'ANALYTICS',
|
||||
schema: 'PUBLIC',
|
||||
}),
|
||||
};
|
||||
|
||||
const allowedScopeKeys = new Set(['dataset_ids', 'databases', 'schemas', 'schema_names']);
|
||||
const historicSqlReaderDrivers = new Set<KtxConnectionDriver>(['postgres', 'bigquery', 'snowflake']);
|
||||
const localExecutorDrivers = new Set<KtxConnectionDriver>(['postgres', 'sqlite']);
|
||||
|
||||
function assertExportedRegistryBoundaryTypes(input: {
|
||||
scopeConfigKey: KtxScopeConfigKey;
|
||||
connectorModule: KtxDriverConnectorModule;
|
||||
}): {
|
||||
scopeConfigKey: KtxScopeConfigKey;
|
||||
connectorModule: KtxDriverConnectorModule;
|
||||
} {
|
||||
return input;
|
||||
}
|
||||
|
||||
describe('driverRegistrations', () => {
|
||||
let projectDir: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
projectDir = await mkdtemp(join(tmpdir(), 'ktx-driver-registry-'));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await rm(projectDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('lists every supported warehouse driver', () => {
|
||||
const registryDrivers = Object.keys(driverRegistrations).sort();
|
||||
expect(listSupportedDrivers()).toEqual(registryDrivers);
|
||||
expect(listSupportedDrivers()).toEqual([
|
||||
'bigquery',
|
||||
'clickhouse',
|
||||
'mysql',
|
||||
'postgres',
|
||||
'snowflake',
|
||||
'sqlite',
|
||||
'sqlserver',
|
||||
]);
|
||||
});
|
||||
|
||||
it('resolves registered drivers case-insensitively', () => {
|
||||
expect(getDriverRegistration(' Postgres ')?.driver).toBe('postgres');
|
||||
expect(getDriverRegistration('unknown')).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each(Object.values(driverRegistrations))('adapts $driver connector exports', async (registration) => {
|
||||
const connectorModule = await registration.load();
|
||||
const connection = connectionFixtures[registration.driver](projectDir);
|
||||
const exportedBoundary = assertExportedRegistryBoundaryTypes({
|
||||
scopeConfigKey: registration.scopeConfigKey ?? 'schemas',
|
||||
connectorModule,
|
||||
});
|
||||
expect(exportedBoundary.connectorModule.createScanConnector).toEqual(expect.any(Function));
|
||||
|
||||
expect(connectorModule.isConnectionConfig(connection)).toBe(true);
|
||||
expect(connectorModule.isConnectionConfig({})).toBe(false);
|
||||
|
||||
const connector = connectorModule.createScanConnector({
|
||||
connectionId: 'warehouse',
|
||||
connection,
|
||||
projectDir,
|
||||
});
|
||||
|
||||
expect(connector.driver).toBe(registration.driver);
|
||||
expect(connector.listSchemas).toEqual(expect.any(Function));
|
||||
expect(connector.listTables).toEqual(expect.any(Function));
|
||||
await connector.cleanup?.();
|
||||
|
||||
if (registration.driver === 'sqlite') {
|
||||
expect(registration.scopeConfigKey).toBeNull();
|
||||
} else {
|
||||
expect(registration.scopeConfigKey).not.toBeNull();
|
||||
expect(allowedScopeKeys.has(registration.scopeConfigKey ?? '')).toBe(true);
|
||||
}
|
||||
expect(registration.hasHistoricSqlReader).toBe(historicSqlReaderDrivers.has(registration.driver));
|
||||
expect(registration.hasLocalQueryExecutor).toBe(localExecutorDrivers.has(registration.driver));
|
||||
});
|
||||
});
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { createDefaultLocalQueryExecutor } from './local-query-executor.js';
|
||||
import { createDefaultLocalQueryExecutor } from '../../../src/context/connections/local-query-executor.js';
|
||||
|
||||
describe('createDefaultLocalQueryExecutor', () => {
|
||||
it('dispatches postgres and sqlite drivers to their executors', async () => {
|
||||
|
|
@ -3,7 +3,7 @@ import {
|
|||
localConnectionInfoFromConfig,
|
||||
localConnectionToWarehouseDescriptor,
|
||||
localConnectionTypeForConfig,
|
||||
} from './local-warehouse-descriptor.js';
|
||||
} from '../../../src/context/connections/local-warehouse-descriptor.js';
|
||||
|
||||
describe('localConnectionToWarehouseDescriptor', () => {
|
||||
it('maps local Postgres URLs to canonical warehouse descriptors', () => {
|
||||
|
|
@ -7,7 +7,7 @@ import {
|
|||
parseNotionConnectionConfig,
|
||||
redactNotionConnectionConfig,
|
||||
resolveNotionAuthToken,
|
||||
} from './notion-config.js';
|
||||
} from '../../../src/context/connections/notion-config.js';
|
||||
|
||||
describe('standalone Notion connection config', () => {
|
||||
let tempDir: string;
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { createPostgresQueryExecutor } from './postgres-query-executor.js';
|
||||
import { createPostgresQueryExecutor } from '../../../src/context/connections/postgres-query-executor.js';
|
||||
|
||||
function makeClient() {
|
||||
const calls: unknown[] = [];
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from './read-only-sql.js';
|
||||
import { assertReadOnlySql, limitSqlForExecution } from '../../../src/context/connections/read-only-sql.js';
|
||||
|
||||
describe('assertReadOnlySql', () => {
|
||||
it('allows select and with queries', () => {
|
||||
|
|
@ -4,7 +4,7 @@ import { tmpdir } from 'node:os';
|
|||
import { join } from 'node:path';
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import { createSqliteQueryExecutor, sqliteDatabasePathFromConnection } from './sqlite-query-executor.js';
|
||||
import { createSqliteQueryExecutor, sqliteDatabasePathFromConnection } from '../../../src/context/connections/sqlite-query-executor.js';
|
||||
|
||||
describe('createSqliteQueryExecutor', () => {
|
||||
let tempDir: string;
|
||||
|
|
@ -2,7 +2,7 @@ import { mkdir, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { resolveKtxConfigReference, resolveKtxHomePath } from './config-reference.js';
|
||||
import { resolveKtxConfigReference, resolveKtxHomePath } from '../../../src/context/core/config-reference.js';
|
||||
|
||||
describe('KTX config references', () => {
|
||||
it('resolves env references without returning empty values', () => {
|
||||
|
|
@ -3,9 +3,9 @@ import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import type { SimpleGit } from 'simple-git';
|
||||
import type { KtxCoreConfig } from './config.js';
|
||||
import { createSimpleGit } from './git-env.js';
|
||||
import { GitService } from './git.service.js';
|
||||
import type { KtxCoreConfig } from '../../../src/context/core/config.js';
|
||||
import { createSimpleGit } from '../../../src/context/core/git-env.js';
|
||||
import { GitService } from '../../../src/context/core/git.service.js';
|
||||
|
||||
describe('GitService.assertWorktreeClean', () => {
|
||||
let workdir: string;
|
||||
|
|
@ -3,9 +3,9 @@ import { mkdir, mkdtemp, readdir, rm, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import type { SimpleGit } from 'simple-git';
|
||||
import type { KtxCoreConfig } from './config.js';
|
||||
import { createSimpleGit } from './git-env.js';
|
||||
import { GitService } from './git.service.js';
|
||||
import type { KtxCoreConfig } from '../../../src/context/core/config.js';
|
||||
import { createSimpleGit } from '../../../src/context/core/git-env.js';
|
||||
import { GitService } from '../../../src/context/core/git.service.js';
|
||||
|
||||
describe('GitService.deleteDirectories', () => {
|
||||
let workdir: string;
|
||||
|
|
@ -2,7 +2,7 @@ import { mkdir, mkdtemp, readFile, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { GitService } from './git.service.js';
|
||||
import { GitService } from '../../../src/context/core/git.service.js';
|
||||
|
||||
async function makeGit() {
|
||||
const homeDir = await mkdtemp(join(tmpdir(), 'ktx-git-patch-'));
|
||||
|
|
@ -3,9 +3,9 @@ import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import type { SimpleGit } from 'simple-git';
|
||||
import type { KtxCoreConfig } from './config.js';
|
||||
import { createSimpleGit } from './git-env.js';
|
||||
import { GitService } from './git.service.js';
|
||||
import type { KtxCoreConfig } from '../../../src/context/core/config.js';
|
||||
import { createSimpleGit } from '../../../src/context/core/git-env.js';
|
||||
import { GitService } from '../../../src/context/core/git.service.js';
|
||||
|
||||
describe('GitService.resetHardTo', () => {
|
||||
let workdir: string;
|
||||
|
|
@ -2,8 +2,8 @@ import { mkdtemp, readFile, realpath, rm, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import type { KtxCoreConfig } from './config.js';
|
||||
import { GitService } from './git.service.js';
|
||||
import type { KtxCoreConfig } from '../../../src/context/core/config.js';
|
||||
import { GitService } from '../../../src/context/core/git.service.js';
|
||||
|
||||
// These tests drive a real git repo inside a temp directory — simple-git shells out to the
|
||||
// system `git` binary. They are fast enough to run as unit tests and catch real issues that
|
||||
|
|
@ -2,9 +2,9 @@ import { mkdtemp, realpath, rm, stat } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { KtxCoreConfig } from './config.js';
|
||||
import { GitService } from './git.service.js';
|
||||
import { SessionWorktreeService, type WorktreeConfigPort } from './session-worktree.service.js';
|
||||
import type { KtxCoreConfig } from '../../../src/context/core/config.js';
|
||||
import { GitService } from '../../../src/context/core/git.service.js';
|
||||
import { SessionWorktreeService, type WorktreeConfigPort } from '../../../src/context/core/session-worktree.service.js';
|
||||
|
||||
interface TestWorktreeConfig extends WorktreeConfigPort<TestWorktreeConfig> {
|
||||
workdir?: string;
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
import { once } from 'node:events';
|
||||
import { createServer } from 'node:http';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { createHttpSemanticLayerComputePort, createPythonSemanticLayerComputePort } from './semantic-layer-compute.js';
|
||||
import { createHttpSemanticLayerComputePort, createPythonSemanticLayerComputePort } from '../../../src/context/daemon/semantic-layer-compute.js';
|
||||
|
||||
const source = {
|
||||
name: 'orders',
|
||||
|
|
@ -2,10 +2,10 @@ import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import type { KtxEmbeddingPort } from '../../context/core/embedding.js';
|
||||
import { initKtxProject, loadKtxProject, type KtxLocalProject } from '../../context/project/project.js';
|
||||
import { SqliteKnowledgeIndex } from '../wiki/sqlite-knowledge-index.js';
|
||||
import { reindexLocalIndexes } from './reindex.js';
|
||||
import type { KtxEmbeddingPort } from '../../../src/context/core/embedding.js';
|
||||
import { initKtxProject, loadKtxProject, type KtxLocalProject } from '../../../src/context/project/project.js';
|
||||
import { SqliteKnowledgeIndex } from '../../../src/context/wiki/sqlite-knowledge-index.js';
|
||||
import { reindexLocalIndexes } from '../../../src/context/index-sync/reindex.js';
|
||||
|
||||
class FakeEmbeddingPort implements KtxEmbeddingPort {
|
||||
readonly maxBatchSize = 8;
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { actionTargetConnectionId, memoryActionIdentity } from './action-identity.js';
|
||||
import { actionTargetConnectionId, memoryActionIdentity } from '../../../src/context/ingest/action-identity.js';
|
||||
|
||||
describe('memory action target identity', () => {
|
||||
it('keys SL actions by target connection and wiki actions by run connection', () => {
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { parseDbtSchemaFile, parseDbtSchemaFiles } from './parse-schema.js';
|
||||
import { parseDbtSchemaFile, parseDbtSchemaFiles } from '../../../../../src/context/ingest/adapters/dbt-descriptions/parse-schema.js';
|
||||
|
||||
describe('dbt descriptions schema parser', () => {
|
||||
it('resolves shared dbt vars and defaults before parsing schema YAML', () => {
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { chunkDbtProject } from './chunk.js';
|
||||
import { chunkDbtProject } from '../../../../../src/context/ingest/adapters/dbt/chunk.js';
|
||||
|
||||
describe('chunkDbtProject', () => {
|
||||
const diffSet = (modified: string[]) => ({ added: [], modified, deleted: [], unchanged: [] });
|
||||
|
|
@ -2,8 +2,8 @@ import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import type { SourceAdapter } from '../../types.js';
|
||||
import { DbtSourceAdapter } from './dbt.adapter.js';
|
||||
import type { SourceAdapter } from '../../../../../src/context/ingest/types.js';
|
||||
import { DbtSourceAdapter } from '../../../../../src/context/ingest/adapters/dbt/dbt.adapter.js';
|
||||
|
||||
describe('DbtSourceAdapter', () => {
|
||||
let stagedDir: string;
|
||||
|
|
@ -2,7 +2,7 @@ import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { fetchDbtRepo } from './fetch.js';
|
||||
import { fetchDbtRepo } from '../../../../../src/context/ingest/adapters/dbt/fetch.js';
|
||||
|
||||
describe('fetchDbtRepo', () => {
|
||||
let tempDir: string;
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import { normalizeDbtPath } from './parse.js';
|
||||
import { normalizeDbtPath } from '../../../../../src/context/ingest/adapters/dbt/parse.js';
|
||||
|
||||
describe('normalizeDbtPath', () => {
|
||||
it('normalizes Windows separators to POSIX separators', () => {
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue